Search traffic to content sites now splits three ways: classic blue links, AI Overviews and other in-search summaries, and assistants like ChatGPT, Claude, and Perplexity that fetch pages directly. A Gatsby site is unusually well placed for all three — it ships real HTML, no client-side hydration is needed to read it, and every page is a file you control. But most Gatsby sites still leak the details these systems rely on: JSON-LD stuck inside a React component that never renders to the static HTML, a robots.txt that blocks nothing and allows nothing on purpose, and no machine-readable summary of what the site is.
This tutorial covers the four concrete things we implement on client Gatsby sites for AI-era discoverability, with code that works on Gatsby 5.
1. Make sure your metadata is actually in the HTML
The first check is not a code change. Build the site and read the output:
npx gatsby build
grep -c "application/ld+json" public/blog/some-post/index.html
node -e "console.log(require('fs').readFileSync('public/blog/some-post/index.html','utf8').length)"
If the JSON-LD count is zero, or the file is tiny and the body is a single empty <div id="___gatsby">, everything below is pointless — crawlers that do not execute JavaScript, which includes several assistant fetchers, are seeing nothing. Common causes: metadata injected from a useEffect, a component gated behind typeof window !== 'undefined', or content loaded client-side from an API instead of at build time.
On Gatsby 5, use the built-in Head API rather than react-helmet, which is unmaintained and had a habit of dropping tags during hydration:
// src/templates/post.js
export const Head = ({ data, location }) => {
const post = data.markdownRemark;
const siteUrl = 'https://www.example.com';
const canonical = `${siteUrl}${location.pathname}`;
return (
<>
<title>{post.frontmatter.title}</title>
<meta name="description" content={post.frontmatter.description} />
<link rel="canonical" href={canonical} />
<meta property="og:type" content="article" />
<meta property="og:title" content={post.frontmatter.title} />
<meta property="og:description" content={post.frontmatter.description} />
<meta property="og:url" content={canonical} />
</>
);
};
Anything exported as Head is rendered into the static HTML at build time. That is the property that matters here.
2. Emit JSON-LD that matches what is on the page
Assistants and search engines both use structured data as a shortcut to "what is this page and who published it". Two schema types cover most of a consulting or content site: Article for posts and Organization for the site itself. Keep the JSON-LD generated from the same data the page renders — hand-written JSON-LD drifts within a month and mismatched markup is worse than none.
// src/components/structured-data.js
export const ArticleJsonLd = ({ post, url, siteUrl }) => {
const data = {
'@context': 'https://schema.org',
'@type': 'Article',
headline: post.title,
description: post.description,
datePublished: post.datePublished,
dateModified: post.dateModified || post.datePublished,
mainEntityOfPage: { '@type': 'WebPage', '@id': url },
author: { '@type': 'Person', name: post.author },
publisher: {
'@type': 'Organization',
name: 'Example Co',
url: siteUrl,
logo: { '@type': 'ImageObject', url: `${siteUrl}/logo.png` },
},
};
return (
<script
type="application/ld+json"
dangerouslySetInnerHTML={{ __html: JSON.stringify(data) }}
/>
);
};
Render it from Head:
export const Head = ({ data, location }) => (
<>
{/* ...title, description, canonical... */}
<ArticleJsonLd
post={{
title: data.markdownRemark.frontmatter.title,
description: data.markdownRemark.frontmatter.description,
datePublished: data.markdownRemark.frontmatter.date,
dateModified: data.markdownRemark.parent.modifiedTime,
author: data.markdownRemark.frontmatter.author,
}}
url={`https://www.example.com${location.pathname}`}
siteUrl="https://www.example.com"
/>
</>
);
dateModified from the file's modifiedTime (available via gatsby-source-filesystem's File node) is worth wiring up: freshness is one of the few signals you can influence honestly, and a fabricated one is easy to catch.
Put the Organization block, with sameAs links to your real profiles, in the site layout's Head on the home page only. Validate everything with Google's Rich Results Test and validator.schema.org before you ship — a JSON syntax error silently disables the whole block.
3. Decide, in writing, which bots you allow
robots.txt now has two audiences: search crawlers, and AI crawlers that either index for training or fetch a page live to answer a question. These are different decisions and the second one is a business decision, not a technical one. The named agents worth knowing about:
| User-agent | Operator | What it does |
|---|---|---|
GPTBot | OpenAI | Crawls for model training |
OAI-SearchBot | OpenAI | Indexes for ChatGPT search results |
ChatGPT-User | OpenAI | Fetches a page when a user's prompt needs it |
ClaudeBot | Anthropic | Crawls for training |
Claude-User / Claude-SearchBot | Anthropic | Live fetch and search indexing |
PerplexityBot | Perplexity | Indexes for answers with citations |
Google-Extended | Controls Gemini/AI training use, not Search ranking | |
Applebot-Extended | Apple | Controls Apple Intelligence training use |
A typical stance for a consulting site — allow everything that might cite you, block nothing, because citations are the point:
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap-index.xml
A typical stance for a site with proprietary content — allow the fetchers that cite, disallow the training crawlers:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap-index.xml
Generate it rather than hand-editing, so staging never gets indexed:
// gatsby-config.js
module.exports = {
siteMetadata: { siteUrl: 'https://www.example.com' },
plugins: [
'gatsby-plugin-sitemap',
{
resolve: 'gatsby-plugin-robots-txt',
options: {
resolveEnv: () => process.env.GATSBY_ENV || 'development',
env: {
development: { policy: [{ userAgent: '*', disallow: ['/'] }] },
production: {
policy: [
{ userAgent: 'GPTBot', disallow: ['/'] },
{ userAgent: 'ClaudeBot', disallow: ['/'] },
{ userAgent: 'Google-Extended', disallow: ['/'] },
{ userAgent: '*', allow: '/' },
],
},
},
},
},
],
};
Two honest caveats. robots.txt is voluntary; it is respected by the major named agents and ignored by scrapers, so it is a policy statement, not a control. And blocking training crawlers does not remove you from answers that are generated from live fetches or from third-party copies of your content.
4. Publish an llms.txt, and keep it generated
llms.txt is a proposed convention (see llmstxt.org) for a Markdown file at the site root that tells a model what the site is and where the important content lives. It is not a standard, no crawler is obliged to read it, and it will not rank you anywhere. It is cheap, though, and it forces a useful exercise: writing down, in a hundred words, what your site is for.
The mistake is committing it as a static file that goes stale. Generate it at build time from the same GraphQL data that builds your pages, using createPagesStatefully or a small onPostBuild hook:
// gatsby-node.js
const fs = require('node:fs/promises');
const path = require('node:path');
exports.onPostBuild = async ({ graphql, reporter }) => {
const { data, errors } = await graphql(`
{
site { siteMetadata { title description siteUrl } }
allMarkdownRemark(sort: { frontmatter: { date: DESC } }, limit: 50) {
nodes {
fields { slug }
frontmatter { title description }
}
}
}
`);
if (errors) { reporter.panicOnBuild('llms.txt query failed', errors); return; }
const { title, description, siteUrl } = data.site.siteMetadata;
const lines = [
`# ${title}`,
'',
`> ${description}`,
'',
'## Guides',
'',
...data.allMarkdownRemark.nodes.map(
(n) => `- [${n.frontmatter.title}](${siteUrl}${n.fields.slug}): ${n.frontmatter.description}`,
),
'',
];
await fs.writeFile(path.join('public', 'llms.txt'), lines.join('\n'), 'utf8');
reporter.info(`llms.txt written with ${data.allMarkdownRemark.nodes.length} entries`);
};
onPostBuild runs after the static files are emitted, so writing straight into public/ is safe and the file ships with the deploy. If you also want plain-Markdown versions of each page (/blog/some-post.md), the same hook can write those from the raw rawMarkdownBody field — that is more useful to a model than your rendered HTML, and it costs nothing to emit.
5. Verify after every deploy
Add these to your post-deploy smoke test so a refactor cannot silently break the whole thing:
BASE=https://www.example.com
curl -sf "$BASE/robots.txt" | head -20
curl -sf "$BASE/llms.txt" | head -5
curl -sf "$BASE/sitemap-index.xml" > /dev/null && echo "sitemap ok"
curl -sf -A "OAI-SearchBot" "$BASE/blog/some-post/" | grep -o 'application/ld+json' | wc -l
The last line is the important one: fetch a real page with a bot user-agent and confirm the structured data is present in the raw response, not just in the browser after hydration. Most "we're invisible to AI search" problems we get called about turn out to be exactly that gap — the content exists, but only after JavaScript runs, and only for a browser.
What this does and does not buy you
It makes your content readable, attributable, and citable by systems that increasingly stand between a reader and your site. It does not make thin content rank, and no amount of markup substitutes for having something worth quoting. Structured data, crawler policy, and llms.txt are the plumbing; the pages still have to be good.
If you would like a review of how a Gatsby or static site presents itself to search and AI crawlers — or help wiring the build-time generation above into an existing project — get in touch.