Begin with the access decision
A robots file communicates which URLs a named crawler may request. It is a policy tool, not a universal switch for all AI products. Providers use different user agents for search discovery, user-initiated fetching, and model training. An organization may allow one purpose and disallow another. Before editing rules, agree with legal, security, and content owners on the policy you intend to express.
Inspect the live robots file at the site's root, then test important URLs against each relevant user agent. Check the page response too: redirects, authentication, JavaScript rendering, and web application firewall rules can block content even when robots allows it. Conversely, an allowed URL may still be omitted from an index or answer for other reasons. Do not equate “crawlable” with “will be cited.”
| Layer | Check | Typical failure |
|---|---|---|
| Robots policy | Rules for the actual user agent | Broad disallow overrides an intended allow |
| HTTP delivery | Status and final URL | Redirect loop, login page, or 403 response |
| Page content | Main text in rendered HTML | Critical answer exists only in a script or download |
| Structured data | Markup matches visible facts | Old price or unsupported feature in JSON-LD |
Keep schema accurate and modest
Structured data helps describe a page and can support eligible rich results in Google Search. It is not a guaranteed AI citation mechanism. Use a type that fits the page, supply required properties, and keep values aligned with visible content. A product feature listed in markup but absent from the page is a quality problem. Test templates after releases, especially when pricing or availability changes.
For Google AI search features, Google's current guidance emphasizes core SEO and says no special schema is required. OpenAI and Perplexity document their own crawlers and controls; read the provider's current documentation before setting a rule. Keep a dated access inventory because user-agent definitions and IP guidance can change.
Turn findings into a safe change set
Group affected URLs by template and business importance. Propose the smallest change that expresses the intended policy. Test on staging when possible, then verify production responses and logs after release. Capture before-and-after examples. If a WAF exception is needed, coordinate with security rather than broadly disabling protection. Continue to monitor access and search outcomes separately: technical access is only the first step in being useful to a reader.
Test access in a controlled sequence
Start with a URL inventory rather than a single homepage check. Select the templates that carry important answers: product details, pricing, documentation, and editorial pages. For each URL, note its canonical destination and whether it is public by design. Then inspect the robots file for the exact user agent and path. A broad rule for all bots may interact with a more specific rule, so test the resolved behavior instead of reading one line in isolation.
Next, request the URL and inspect the final status, redirect chain, response headers, and rendered content. A 200 response that contains an empty shell or a consent wall may not give a crawler the answer the business expects. Check logs for actual requests, but remember that a lack of requests during a short window is not proof of a block. If a WAF rule appears responsible, work with security on a narrow exception and verify the provider's bot identity using current documentation.
Separate the three decisions
The organization has at least three distinct decisions: whether a provider may fetch the page, whether the page should be indexed or surfaced, and whether the content is good enough to support an answer. Robots and access controls address the first. Indexing settings and provider behavior affect the second. Clear, useful, current content affects the third. Treating them as one switch leads to overconfident conclusions.
Keep a dated policy record with the reason for each crawler rule and the person who approved it. Revisit the record when providers change their user agents or when new content types are published. After any technical fix, check the page as a human visitor too. A technically accessible page with contradictory pricing or hidden documentation still creates a poor discovery experience.
Questions to settle before changing access
Which bot are we discussing?
Do not treat “AI crawler” as one identity. Providers may distinguish a search-discovery bot, a user-triggered fetcher, and a training crawler. Their published controls and user agents can change. Name the provider, purpose, and exact agent in the access record. Then ask whether the organization wants that purpose to reach the selected page class. A security exception should be scoped to verified traffic, not to any request that happens to claim a familiar user-agent string.
Why is a page still absent after access is fixed?
The page may be low quality, duplicate another URL, change frequently, be excluded by a different setting, or simply not be selected by the provider. A successful fetch proves that one access barrier has been removed. It does not prove indexing or citation. Continue the audit with the page's visible answer, canonical URL, internal links, and the provider's available diagnostics.
Copy this URL test record
Record URL and template, intended public status, provider and agent, robots result, HTTP status and redirect chain, WAF behavior, rendered main content, structured-data parity, test date, and owner. Attach a log excerpt or test output where possible. Classify the issue as policy, delivery, content, or unknown. After a change, rerun the same request and separately monitor whether discovery behavior changes over time.