Files
/mcp.json, /llms.txt, /ai/context.md and /mcp/setup.md are served with Cache-Control: public, max-age=3600, so a cached copy can be up to an hour old.
For AI crawlers
robots.txt opens the site to search, AI input and AI training. Its catch-all group carries this line:
/ and disallows a short list of private paths, including sign-in, account, dashboard, admin and /api/. Named AI crawlers each get their own Allow: / group. They include OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Meta-WebIndexer, Amazonbot and Applebot-Extended.
Which file to use
- Connect an AI app: point it at
/mcp/setup.md, or follow Install Private Clubs. - Find the MCP server from code: read
/mcp.json. - Build on the REST API: read
/openapi.json. - Give an AI tool a site overview: use
/llms.txt, or/ai/context.mdfor a shorter summary. - Check crawler rules: read
/robots.txt. - Check the terms of use: read
/data/license.