Crawl Dashboard
Monitor indexing health, bot activity, and rule compliance across your properties.
Total Crawls (24h)
14,208
↑ 12.4% vs last week
Indexed Pages
8,942
↑ 3.2% growth
Crawl Errors
24
↓ 5 new since 08:00
SEO Health
94/100
↑ Optimized today
Crawl Activity
MonTueWedThuFriSatSun
Bot Distribution
Googlebot
68%
Bingbot
18%
Yandex
8%
Others
6%
Recent Crawl Events
| Timestamp | URL Path | Bot | Rule | Response | Status |
|---|---|---|---|---|---|
| 14:22:05 UTC | /api/v2/products/search | Googlebot | Allow | 200 OK | Crawled |
| 14:18:33 UTC | /admin/dashboard/settings | Bingbot | Disallow | 403 Forbidden | Blocked |
| 14:05:11 UTC | /blog/post-123-draft | Googlebot-Image | Allow | 200 OK | Crawled |
| 13:58:44 UTC | /checkout/payment-gateway | Unknown | Rate Limit | 429 Too Many | Throttled |
| 13:42:19 UTC | /sitemap-index.xml | Googlebot | Allow | 200 OK | Crawled |
Quick Actions
AI Rule Generator
Analyze your site structure and auto-generate optimized crawl directives.
Push to Google
Submit updated robots.txt and request immediate re-indexing via Search Console.
Scan for Violations
Identify bots ignoring your rules and automatically deploy blocking measures.
Export Compliance Report
Generate PDF/JSON reports for audits, compliance, and team documentation.