# Biffen Site is the public, indexable surface — search engines are wanted here and the SEO work
# depends on them (see the tag/cinema/city pages and the sitemap below). Only two things are
# closed off: the JSON endpoints, and crawlers that take the content for AI training rather than
# to send readers back.

User-agent: *
Disallow: /api/
Allow: /

Sitemap: https://biffen.info/sitemap.xml

# --- AI crawlers -------------------------------------------------------------------------
# The showtimes here are a scraped, normalised and enriched collection that costs real, ongoing
# work to maintain (see FINGERPRINTS.md in the bio-app repo, and /vilkaar). These crawlers consume
# it to answer questions elsewhere, which is exactly the substitution the site exists to avoid —
# unlike a search engine, they send no reader back.
#
# Deliberately robots.txt AND Cloudflare's zone-level "Block AI Scrapers and Crawlers" toggle:
# this file only binds crawlers that choose to honour it, while the Cloudflare rule blocks at the
# edge but lags on newly-appearing agents. Neither alone is sufficient.
#
# NOTE: Google-Extended governs Gemini/Vertex training ONLY. It has no effect on Googlebot or on
# ranking in Google Search — normal indexing is untouched. Same split for Applebot-Extended,
# which is separate from the Applebot that powers Siri/Spotlight results.

User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: anthropic-ai
User-agent: CCBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Bytespider
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: Google-Extended
User-agent: meta-externalagent
User-agent: FacebookBot
User-agent: Diffbot
User-agent: Omgilibot
User-agent: ImagesiftBot
User-agent: cohere-ai
User-agent: AI2Bot
User-agent: Timpibot
Disallow: /