Guessing a visitor's language deleted our English pages from Google

Search Console listed every English marketing URL under one status: “Discovered - currently not indexed”. Not some of them. All of them, plus their Indonesian counterparts. Google knew the URLs existed, had them queued, and had decided they were not worth keeping.

The usual explanations for that status are thin content, duplicate pages, or a crawl budget spent on junk. None of them applied. The pages were fine. Google had the addresses and had never been served what was behind them.

What the code was doing

The marketing layout ran a locale decision on every request. If the path carried no locale prefix and the request looked Indonesian, it issued a 302 to the same path under /id. The signal it used was the country code header that our edge proxy attaches to every incoming request. Reasonable on the surface: a visitor in Jakarta lands on the site and gets Indonesian without hunting for a toggle.

Crawlers do not work that way. A crawler asks for /about from wherever its fetcher happens to sit that day, and some of those egress points sit in Indonesia. That request got a 302 to /id/about. The crawler followed it, indexed the Indonesian page, and moved on. The English page at /about was never served to it. Not served slowly, not served and judged weak. Never served at all.

So the English tree was invisible to the one visitor whose opinion decided whether it existed in search. Every internal link pointing at it resolved to a redirect, and the sitemap listed URLs that did the same. From Google’s position the English site was a set of addresses that forward somewhere else.

There is a second problem stacked on the first, and it is the one that turns a bug into a penalty. Serving different content based on the requester’s IP address is what Google calls cloaking. We were not trying to hide anything, but intent does not enter into it. The pattern is what gets classified.

The site was arguing with itself

Each page also emitted an hreflang block declaring the English URL as its own canonical and naming the Indonesian URL as the alternate. That block was correct. It was also being served from a URL that, for the requester in question, answered with a redirect away from itself.

Two claims in one response. The markup said this address is the English page. The status line said this address is not a page, go elsewhere. When a site contradicts itself about which URL is authoritative, the resolution is not in your favour. Google picks one, and the one it picks is the one it actually received.

Redirect on a choice, not on a guess

The fix was to change what the redirect keys on. An IP address is a weak signal about intent. It tells you where a packet entered the network, which is a fact about routing, not about the person. A click on the language toggle is a strong signal, because someone performed it on purpose.

So the toggle now writes a locale cookie, and only that cookie moves a visitor between the two trees:

const preferred = cookies.get('locale');
if (!params.lang && preferred === 'id') {
	redirect(302, `/id${url.pathname === '/' ? '' : url.pathname}${url.search}`);
}
if (params.lang === 'id' && preferred === 'en') {
	redirect(302, `${url.pathname.replace(/^/id/, '') || '/'}${url.search}`);
}

Two properties matter here and neither is decorative.

The first is that the rule is symmetric. The original code only pushed people into /id. It had no way to push them back out, so an English-preferring visitor who followed an Indonesian link was stuck in that tree for the rest of the session. Once the decision is a stored preference rather than a property of the network, both directions cost the same to implement, and leaving one out is just a missing branch.

The second is that a crawler carries no cookies. It has no preference to honour, so it falls through both conditions and gets a 200 with the page it asked for. That is the whole repair. The English pages became reachable for the client that decides whether they exist.

The cache header this forces

Once a cookie decides the response, the response is no longer a pure function of the URL. Any shared cache in front of the app, ours or an intermediary’s, will happily store the first response it sees for /about and replay it to everyone who asks for that URL afterwards. One visitor with an id preference primes the cache, and the next hundred requests get Indonesian HTML from an English address.

So the handler appends Vary: Cookie to every marketing response. It is one line, and it is not optional the moment a cookie enters the decision. Skipping it converts a correctness fix into a much stranger bug that only appears under traffic.

What we rejected

The alternative on the table was to keep the IP redirect and soften it with a dismissible banner: land the Indonesian visitor on the Indonesian page, show a bar offering English, remember the dismissal.

We threw it out for two reasons. The first is that it does not touch the actual defect. The crawler still gets a redirect, the English pages are still never served, and the hreflang block still contradicts the status line. The banner is a message shown to humans about a problem that only harms us with machines.

The second reason is more general. Adding UI to explain a wrong assumption is a way of keeping the assumption. The assumption here was that where a request originates tells you what language its sender wants. That is false often enough to matter: people travel, use VPNs, work in one country and read in another, and in our case the visitor was not a person at all. Once you write down that IP is a guess and a click is a fact, the banner stops looking like a fix and starts looking like an apology for the guess.

The pages are all live now, all answering 200 from any origin, and the pricing page is finally something a crawler can read without being sent somewhere else first. Whether Google forgives the earlier pattern quickly or slowly is not something we control.

The part I keep thinking about: this shipped months ago and looked correct in every browser we opened it in. The only client that could see the bug was the one we never tested with.