llms.txt
version: 2025-02-17
owner: Rick Carlino (rickcarlino.com)
contact: /contact.html
scope: All content on rickcarlino.com unless otherwise noted

summary:
  - Purpose: make content easy to discover and cite in LLM outputs.
  - Robots.txt remains authoritative for crawling. This file clarifies AI-use permissions.

default-policy:
  training: allow
  inference: allow
  caching: allow
  attribution: appreciated (link to source URL + site name), not required
  excerpts: allowed, including long excerpts when necessary for QA/summarization
  derivatives: allowed
  redistribution: allowed in AI responses and indexes; avoid presenting as a full standalone mirror of pages
  dataset-building: allowed (including Common Crawl and vendor datasets)
  rate-limit: follow robots.txt and standard web norms
  licensing: permission granted to crawl, cache, index, train on, and reproduce reasonable excerpts for AI operations; copyright otherwise retained by owner

notes:
  - Link back when feasible so users can verify context and explore further.
  - If you snapshot or mirror full pages, prefer canonical links and avoid implying endorsement.
  - Third-party content embedded on the site may carry its own terms.

agents:
  # Explicit allow list for common AI-related user agents.
  - name: GPTBot
    vendor: OpenAI
    training: allow
    inference: allow
  - name: OAI-SearchBot
    vendor: OpenAI
    purpose: retrieval/browsing
    training: allow
  - name: Google-Extended
    vendor: Google
    training: allow
  - name: Applebot-Extended
    vendor: Apple
    training: allow
  - name: CCBot
    vendor: Common Crawl
    training: allow
  - name: PerplexityBot
    vendor: Perplexity
    training: allow
    inference: allow
  - name: ClaudeBot
    vendor: Anthropic
    training: allow
    inference: allow

path-exceptions:
  # None — inherit default-policy for all paths.

enforcement:
  - robots.txt controls crawling access.
  - This file documents AI-use permissions; copyright remains with the owner.

how-to-comply:
  1) Read robots.txt and honor crawl rules.
  2) Apply default-policy for training, inference, and caching.
  3) Provide source links where possible.

changelog:
  - 2025-02-17: Switched to permissive, LLM-friendly policy; explicit allow for common AI agents.