Crawled 16 out of 16: how our prerender succeeded at doing nothing

The problem

Our website is a React single-page application. The articles on it — including this one — live in our internal knowledge base and are fetched through an API when the page loads in the browser.

For a visitor that is fine. For anything that does not run JavaScript, it is not. Link previews on LinkedIn or Slack, and a good share of crawlers, saw every article as the same empty shell with the same generic title. Share a link to an article and the preview described the company, not the article.

The usual fix is prerendering: at build time, open each page in a headless browser, wait until it has rendered, and save the resulting HTML as a static file. We used react-snap, which does exactly that on top of Puppeteer. This is the story of getting it to work for content that comes from an API — and of the day it reported complete success while doing nothing at all.

Prerendering pages you don't know in advance

Static pages are easy: you list the routes. Articles are not known when you write the configuration, so our prerender script asks the API for the list first and builds the routes from it:

const articles = await fetchArticles();   // [] if the API does not answer

await run({
    include: [
        '/', '/products', '/contact', '/articles',
        ...articles.map((a) => `/post/${a.id}`),
    ],
    // ...
});

If the API is down during a build, we prerender only the static pages and carry on. A build should never fail because of content.

One setting needed rethinking. react-snap can block every request to another domain during rendering, and we had that switched on to keep analytics out of our builds. With the content living behind an API on another domain, it has to be off — otherwise every article is prerendered as "Loading…". To keep builds from registering as page views, we block analytics inside the headless browser at the DNS level instead:

'--host-resolver-rules=MAP www.googletagmanager.com 127.0.0.1,MAP *.google-analytics.com 127.0.0.1'

The API resolves normally; the analytics hosts resolve to nowhere.

Hydration: the page must not forget what it knows

Prerendering gives you HTML. Then the JavaScript loads and React hydrates — it renders the same component tree and expects the result to match the HTML already on the page.

A component that fetches its data on mount starts in a "loading" state. Its first render is a spinner, not the article list. That does not match the prerendered HTML, so React throws the markup away and re-renders: the visitor sees the content, then "Loading articles…", then the content again.

react-snap has an escape hatch for this. Before saving a page it calls window.snapSaveState() and writes every key it returns into an inline script placed before the application bundle. The component then starts from that data instead of from nothing:

const preloaded = window.__SNAP_ARTICLES__;

const [posts, setPosts]     = useState(preloaded || []);
const [loading, setLoading] = useState(!preloaded);

// after a successful fetch:
window.snapSaveState = () => ({ __SNAP_ARTICLES__: forSnapshot(list) });

Two things we learned here.

The saved state is payload. Our first version saved the whole API response, including the full body of every article. The article list page went to 207 kB of HTML — 43 kB compressed — almost all of it data the cards never display. The cards need a title, a date, tags and a short excerpt. Saving only that, with the excerpt already computed, brought the page down to 37 kB, 7 kB compressed. The full content still arrives, a moment later, from the normal fetch.

Hydration compares text nodes, not text. Our footer rendered the copyright line as {year} © Company - {translatedText}. In React that is three adjacent text nodes; in the saved HTML the browser has merged them into one. Same text on screen, different structure — and a hydration mismatch. Rendering it as a single template string fixed it. It is the kind of bug you only find by comparing the prerendered DOM with the hydrated one node by node, which is what we ended up doing for every prerendered page.

"Crawled 16 out of 16"

Locally, everything worked. On the build server, the log said:

✅  crawled 16 out of 16

The build contained none of the seven article pages. The final error the script caught was an empty string.

The cause was the browser. On the server, react-snap used the Chromium that Puppeteer downloads for itself — and our version of Puppeteer ships a build from 2019, roughly Chrome 76. Our JavaScript bundle contains optional chaining, a?.b. Babel leaves it untranspiled on purpose, because the production browser targets are all browsers that support it. Chrome 76 does not; it arrived in Chrome 80.

So the bundle failed to parse with a SyntaxError, the application never started, and the page rendered nothing. From the prerender tool's point of view, loading a page and saving what it rendered had worked perfectly. It just rendered nothing.

The prerender had been working with that same browser before. What changed was our code — new syntax the old browser could not read. Our development machines run a current Chrome, so we could not have seen it there.

A two-line probe makes the difference obvious:

data:text/html,<script>document.title = String(({a:1})?.a)</script>

A current browser sets the title to 1. The 2019 one never runs the script.

The fix: verify the browser, don't assume it

Installing a current Chromium from the operating system's packages solved the immediate problem. The more useful change was in the script: it no longer uses the first browser that exists on disk, but the first one that actually starts and understands the code we ship:

for (const exe of CANDIDATES) {
    const browser = await puppeteer.launch({ executablePath: exe, args: LAUNCH_ARGS });
    const page = await browser.newPage();
    await page.goto('data:text/html,<script>window.__probe=({a:1})?.a</script>');
    const modern = await page.evaluate(() => window.__probe === 1);
    await browser.close();
    if (modern) return exe;
}
return null;

The system browser is tried first; Puppeteer's own is the last resort. If nothing passes, the prerender is skipped with a warning and the site deploys as a plain single-page application. A site that works for visitors but shows generic link previews is a much better outcome than a failed deployment — prerendering is an optimisation, not a requirement.

Smaller traps along the way

• After prerendering, index.html is your homepage. Any route you did not prerender still needs a clean application shell to fall back to. Serve that instead, and unknown URLs start by hydrating the homepage's markup. We keep the untouched shell as 200.html and point the server's fallback at it.
• react-snap refuses to run if 200.html already exists, so the shell has to be written after the render, not before.
• It resolves its source directory relative to the working directory. Pass an absolute path and it quietly looks in the wrong place.

Keeping it fresh

An article published in the knowledge base is visible on the website immediately, because the page still fetches the latest list. It only reaches the static HTML — the part link previews and crawlers see — on the next build.

A nightly job closes that gap. It fetches the article list, compares its hash with the last one that was prerendered, and rebuilds only when something changed. It deliberately does not pull new code: content refreshes on its own, code deployments stay a conscious act.

What we took away

• "Success" from a prerender tool means it saved something, not that the something is right. Check the output: count the files, look for the content you expect in them.
• The headless browser is part of your toolchain. Its age matters as much as the Node version, and it is much easier to forget.
• Hydration state is payload. Ship what the first render needs, not what the API happens to return.
• Fail open. If an optimisation cannot run, deploy without it and say so loudly.