robots_txt_check $0.002 per call

Fetches https://<domain>/robots.txt for a given site URL and determines whether a specified user-agent is allowed to crawl a given path, per the parsed rules. Returns allowed/disallowed status, matched rule, crawl-delay (if any) and list of Sitemap URLs declared in the file.

For AI agents: call robots_txt_check
POST https://toll402.dev/v1/t/robots_txt_check
{
 "url": "https://httpbin.org",
 "userAgent": "*",
 "path": "/deny"
}
Answers 402 with x402 payment requirements; pay with any x402 client (@x402/fetch, x402 for Python, toll402-client, toll402-mcp) or send a prepaid-credits key as x-toll402-key (https://toll402.dev/credits). Paid per call via x402 (USDC on Base) or with a prepaid-credits key (x-toll402-key, card at /credits); charged only on success. Start at /llms.txt or connect the MCP server: npx -y toll402-mcp · remote https://toll402.dev/mcp.

Input schema

{
 "type": "object",
 "properties": {
  "url": {
   "type": "string",
   "description": "Site URL or domain, e.g. https://example.com"
  },
  "userAgent": {
   "type": "string",
   "description": "User-agent token to check rules for",
   "default": "*"
  },
  "path": {
   "type": "string",
   "description": "URL path to test, e.g. /some/page",
   "default": "/"
  }
 },
 "required": [
  "url"
 ]
}

MCP

{"mcpServers":{"toll402":{"command":"npx","args":["-y","toll402-mcp"],"env":{"TOLL402_WALLET_KEY":"0x..."}}}}

All tools · OpenAPI