How Yahoo Handled Redirects

Yahoo's crawler, and why its redirect handling was its own

Yahoo handled redirects on its own terms. For a few years in the middle of the 2000s it ran its own crawler and its own index, and the way it decided which URL to keep after a redirect — the old address or the new one, and for how long — was not the way Google decided it, or MSN. This page has been on this site since that period, at this address.

The original was a short note of the kind people wrote before blogging conventions settled: a paragraph recording what one engine appeared to do with one kind of redirect, for readers maintaining sites against three or four crawlers at once. Its wording and its exact date are not recoverable, so neither is asserted here. The problem it described is what survives. Redirect handling was then, and remains now, a matter of what each crawler actually does — some documented, some not — rather than a settled standard every engine implements identically.

Optimising against four engines at once

Yahoo became a search engine in its own right by acquisition rather than by building one. It bought Inktomi, and it bought Overture, which had itself absorbed AltaVista and AlltheWeb, and around 2004 it stopped serving Google's results and started serving its own. Its crawler was Slurp. Microsoft was building its own index over the same period; Ask had Teoma behind it. That meant four sets of behaviour to accommodate, none small enough to ignore.

There was almost no documentation. An engine might publish a page of guidelines; it did not publish how its duplicate detection worked, how many hops its crawler followed, or whether it kept the source or the target URL after a temporary redirect. The reference material was a conference session, a forum thread, and your own server logs. Practitioners tested: moved a page, watched each index for weeks, reported what happened. This site's author was speaking at the conferences where much of that reporting happened. It was real work, and also the source of a great deal of confident error.

What a redirect was actually deciding

Strip away the era and a redirect answers three questions: which URL an engine treats as the real one, which it shows to a searcher, and whether the history attached to the old address follows to the new one. In the mid-2000s only the first two were observable, and engines disagreed on both:

• Temporary redirects. Given a 302, one engine kept indexing the source and another substituted the target. A site could look correct in one index and wrong in another with no change to the server.
• Meta refresh and script redirects. Both were common, because shared hosting often gave no access to server configuration, and engines varied in whether they followed them.
• Chains. No engine published a hop limit, so the rule was to keep chains short — advice that survived because it happened to be right.
• Default server behaviour. Plenty of platforms emitted a temporary redirect where the author meant a permanent one. That is still true of some frameworks today, and still the most common redirect defect found on a migration.

How the multi-engine world ended

It ended with a commercial deal, not a technical one. Yahoo and Microsoft entered a search alliance in the late 2000s under which Yahoo's organic results came to be supplied from Microsoft's index, Yahoo keeping its own front end and audience. The arrangement was renegotiated more than once, and the precise terms at any given date are not something this page will assert. The consequence is what matters: Slurp stopped being an independent target. Yahoo Search still exists as a destination, but its results have been supplied by partners rather than a Yahoo-operated web index for most of the time since.

That left, in English-language markets, one dominant crawler, one shared Microsoft index behind several front ends, and regional engines that matter enormously where they matter and not at all elsewhere. For roughly a decade afterwards, multi-engine testing stopped being a real constraint. That decade is over: a busy site's access log now shows Googlebot, Bingbot and a growing set of assistant and training crawlers, each with its own fetching behaviour and its own documentation of varying quality.

Where this site, and the rest of the industry, got it wrong

The error was not in testing. Testing was all anyone had. It was in how results were written up, and this site was part of it. A note saying an engine does X was a statement about one crawler, in one week, on whatever URLs somebody happened to move — published without the sample, the dates, or any commitment to re-test. Readers took it as a rule, and some rules outlived the behaviour they described: the refusal to use a temporary redirect at all, limits on redirect counts no engine had published, superstitions about header order.

Three mistakes are worth naming, because they are still being made:

• Treating observed behaviour as specification. A status code's meaning is defined in the standards; what a crawler does with it is a product decision, and product decisions change without notice.
• Treating one engine's behaviour as the web's. Multi-engine testing existed because that assumption was false, and it was made anyway.
• Publishing findings without their conditions. A finding with no date, sample or method cannot be corrected later, because there is nothing to correct it against.

The same sentences are being written now about AI crawlers.

What is documented about redirects now

Twenty years later there is documentation, and it is specific. The full treatment belongs on the permanent-redirects guide; the short version makes the comparison. Google classifies redirects by whether they signal permanence, because permanence drives canonicalisation.

Google ranks implementation methods by reliability: server-side redirects, then meta refresh, then JavaScript, then plain-text redirects it may not recognise at all. Its crawlers follow up to 10 redirect hops by default, specific products' crawlers may differ, and Google Inspection Tools does not follow redirects. Keep redirects for as long as possible, generally at least a year. And a rel=canonical annotation is a hint, not a rule.

One absence is worth stating plainly: Google's redirects documentation does not mention PageRank or link equity at all. The widely repeated claim that permanent redirects lose no PageRank traces to a 2016 statement by a Google engineer on Twitter, not to documentation.

What changed in twenty years, and what did not

What did not change is the thing the original note was really about. Redirect handling is still per-crawler behaviour rather than a guarantee. The hop limit is documented; what happens when a chain exceeds it is not. Google states that specific products' crawlers may use different limits, and that one of its own tools does not follow redirects at all — precisely the class of quirk that made the mid-2000s difficult, still present, now merely written down. Documentation reduced the guesswork. It did not turn crawler behaviour into a standard, and a migration plan that assumes it did will fail in the same way plans failed then.

Testing a migration against the crawlers that matter

The inheritance from that era is a testing discipline. Before a migration goes live:

1. Build the URL map from a crawl and from logs, not the sitemap. The sitemap has the URLs someone remembered; the logs have the URLs actually being requested and linked.
2. Verify status codes across the full chain. A checker that reports only the endpoint will report success on a three-hop chain that starts with a temporary redirect.
3. Test for the crawlers you can name. Googlebot and Bingbot, plus any regional engine with real volume in your logs. Confirm each can fetch the new host, and that bot-mitigation rules there are not blocking them.
4. Watch both properties after the cut, for weeks rather than days. Fluctuation is normal; a flat line is not.
5. Keep the map in version control and the redirects at least a year. The redirect file is infrastructure, and someone who was not there will have to read it.

Then write down what you tested, when, and on how many URLs — the habit the mid-2000s should have taught the industry and did not.

Frequently Asked Questions