⚙️ Active Directives
User-agent: *
Allow: /
Disallow: /api/v2/internal
Disallow: /admin/
Crawl-delay: 2
# Auto-synced: 2024-05-12T14:32:00Z
Allow: /
Disallow: /api/v2/internal
Disallow: /admin/
Crawl-delay: 2
# Auto-synced: 2024-05-12T14:32:00Z
Edge Cached
Syntax Valid
14 Crawlers Active
📡 Live Crawler Log
| Agent | Path | Status |
|---|---|---|
| Googlebot/2.1 | /products | ALLOWED |
| Bingbot/2.0 | /blog | ALLOWED |
| Slurp/3.0 | /private | BLOCKED |
| DuckDuckBot | / | ALLOWED |
| Unknown/0.9 | /api | THROTTLED |
🔌 API & CLI Endpoint
# Fetch current configuration
curl -X GET https://api.robots.txt/v1/config \\ -H "Authorization: Bearer" \\
-H "Content-Type: application/json"
# Response payload:
{
"status": "active",
"directives_count": 4,
"last_modified": "2024-05-12T14:32:00Z",
"hash": "sha256:a1b2c3d4e5...",
"path": "/%F0%9F%A4%96%0A------------------------robots.txt"
}
curl -X GET https://api.robots.txt/v1/config \\ -H "Authorization: Bearer
# Response payload:
{
"status": "active",
"directives_count": 4,
"last_modified": "2024-05-12T14:32:00Z",
"hash": "sha256:a1b2c3d4e5...",
"path": "/%F0%9F%A4%96%0A------------------------robots.txt"
}