{"id":"t/robots_txt_check","kind":"forged","name":"robots_txt_check","description":"Fetches https://<domain>/robots.txt for a given site URL and determines whether a specified user-agent is allowed to crawl a given path, per the parsed rules. Returns allowed/disallowed status, matched rule, crawl-delay (if any) and list of Sitemap URLs declared in the file.","tags":["robots","seo","crawling","compliance"],"method":"POST","path":"/v1/t/robots_txt_check","url":"https://toll402.dev/v1/t/robots_txt_check","specUrl":"https://toll402.dev/v1/spec/t/robots_txt_check","price":"$0.002","priceUsd":0.002,"available":true,"inputSchema":{"type":"object","properties":{"url":{"type":"string","description":"Site URL or domain, e.g. https://example.com"},"userAgent":{"type":"string","description":"User-agent token to check rules for","default":"*"},"path":{"type":"string","description":"URL path to test, e.g. /some/page","default":"/"}},"required":["url"]},"example":{"url":"https://httpbin.org","userAgent":"*","path":"/deny"},"meta":{"creator":null,"createdAt":"2026-09-09T07:09:34.538Z","calls":19,"failures":17,"network":true,"notes":"Fetches <origin>/robots.txt via GET, parses User-agent groups (handling multi-agent grouping and blank-line-free records), converts Allow/Disallow patterns (supporting * wildcards and $ end-anchors) to regex, selects the best-matching group for the requested user-agent (exact match, else '*'), applies longest-match-wins with allow-over-disallow tie-break to determine allowed/matchedRule, extracts crawl-delay from the matched group, and collects all Sitemap: URLs found anywhere in the file; if the fetch fails or returns non-OK it reports robotsFound:false and defaults to allowed:true."}}