Toll402 · Business directory · Tools · llms.txt · MCP

robots_txt_check $0.002 per call

Fetches https://<domain>/robots.txt for a given site URL and determines whether a specified user-agent is allowed to crawl a given path, per the parsed rules. Returns allowed/disallowed status, matched rule, crawl-delay (if any) and list of Sitemap URLs declared in the file.

For AI agents: call robots_txt_check
POST https://toll402.dev/v1/t/robots_txt_check
{
 "url": "https://httpbin.org",
 "userAgent": "*",
 "path": "/deny"
}
Answers 402 with x402 payment requirements; pay with any x402 client (@x402/fetch, x402 for Python, toll402-client, toll402-mcp). Free trial without a wallet; paid calls via x402 (USDC on Base). Start at /llms.txt or connect the MCP server: npx -y toll402-mcp · remote https://toll402.dev/mcp.

Input schema

{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "description": "Site URL or domain, e.g. https://example.com"
  },
  "userAgent": {
   "type": "string",
   "description": "User-agent token to check rules for",
   "default": "*"
  },
  "path": {
   "type": "string",
   "description": "URL path to test, e.g. /some/page",
   "default": "/"
  }
 },
 "required": [
  "url"
 ]
}

MCP

{"mcpServers":{"toll402":{"command":"npx","args":["-y","toll402-mcp"],"env":{"TOLL402_WALLET_KEY":"0x..."}}}}

All tools · OpenAPI

Toll402 is a pay-per-call tool gateway for AI agents (x402, USDC on Base, no API keys). Data aggregates dated public evidence; "unverified" means unknown, not fake. Referral bounty · Get listed