Overview

Code 1

Issues 12

Pull requests 3

Actions

Projects

Wiki

Security

📁 packages Updated 3 days ago

📁 cli/ feat: add dry-run flag for policy validation 2d ago

📁 sdk-node/ fix: handle malformed allow directives 4d ago

📁 sdk-python/ chore: bump version to 2.4.1 1w ago

📄 README.md docs: update installation instructions 5d ago

📄 package.json build: add workspace configuration 3d ago

📄 LICENSE initial commit 2y ago

robots-txt-org/developer-hub

Build Passing TypeScript Python CLI

About

A monorepo containing open-source tools for parsing, validating, and generating robots.txt policies across modern web stacks. Designed for scale, accuracy, and zero-config integration.

Quick Start

# Install CLI
npm i -g @robots.txt/cli

# Validate local robots.txt
robots validate ./robots.txt

# Generate optimized policy from sitemap
robots generate --sitemap ./sitemap.xml --output ./robots.txt

# Dry-run against live endpoint
robots check https://example.com/robots.txt --verbose

Supported SDKs

Node.js - @robots.txt/sdk-node (v2.4.1)
Python - robots_txt_sdk (v2.4.0)
Go - github.com/robots-txt/sdk-go (v1.8.2)
Rust - robots_txt (v0.9.4)

Architecture

All packages share a core spec engine that implements RFC 9309 and Google/Bing extensions. Policies are compiled to a deterministic AST before rendering, ensuring consistent behavior across crawlers.

Contributing

See CONTRIBUTING.md for guidelines on running the monorepo, writing tests, and submitting PRs. All contributions require passing lint, type-check, and integration suites.