📄 /%F0%9F%A4%96%0A------------------------robots.txt
PRODUCTION

⚙️ Active Directives

User-agent: *
Allow: /
Disallow: /api/v2/internal
Disallow: /admin/
Crawl-delay: 2
# Auto-synced: 2024-05-12T14:32:00Z
Edge Cached
Syntax Valid
14 Crawlers Active

📡 Live Crawler Log

AgentPathStatus
Googlebot/2.1/productsALLOWED
Bingbot/2.0/blogALLOWED
Slurp/3.0/privateBLOCKED
DuckDuckBot/ALLOWED
Unknown/0.9/apiTHROTTLED

🔌 API & CLI Endpoint

# Fetch current configuration
curl -X GET https://api.robots.txt/v1/config \\ -H "Authorization: Bearer " \\ -H "Content-Type: application/json"

# Response payload:
{
"status": "active",
"directives_count": 4,
"last_modified": "2024-05-12T14:32:00Z",
"hash": "sha256:a1b2c3d4e5...",
"path": "/%F0%9F%A4%96%0A------------------------robots.txt"
}
"}{