Berners-Lee's Model URL Died in Four Months

A Link That Died in Four Months, an Essay Still Alive After 28 Years

In 1998, Tim Berners-Lee wrote a short piece called “Cool URIs don’t change.” Translated loosely, it means something like “a good address doesn’t change.” It reads almost like a scolding from the web’s inventor to web administrators everywhere, and its core line is this:

“URIs¹ don’t change: people change them.”

In the essay, he picked out a few examples of what he considered good addresses. One was a URL for an article in the American magazine Money, which Berners-Lee even singled out with the comment that it “looks OK.” But the site that URL lived on, pathfinder.com, was shut down entirely by Time Warner in April 1999 — just about four months after he’d held it up as a model.

Reader, meanwhile, Berners-Lee’s own essay still opens today at the exact same address it was published at in 1998. For 28 years running.

The address he cited as best practice disappeared in four months. The essay pointing that out never moved. What kills a link isn’t technology — it’s who decides to keep maintaining that address. And this old problem has suddenly become urgent again in 2026, because as the primary reader of the web shifts from humans to machines, the role of a link is changing from “a path to follow” to “evidence to cite.”

The Web Disappears Much Faster Than You’d Think

Let’s start with how much actually disappears. The most citable empirical study in this area is the one Pew Research Center published in 2024. They pulled roughly 900,000 pages per year from Common Crawl² data, tracking them from 2013 through 2023.

Here’s what they found. 38% of web pages that existed in 2013 were no longer accessible as of October 2023. In other words, more than one in three pages vanishes over a decade. Even pages from 2023 itself — less than a year old — were already 8% inaccessible.

What’s even more striking is Wikipedia. Researchers examined 50,000 Wikipedia articles and found that 54% had at least one dead reference link. That means more than half of the pages we open first when we want to verify something have already lost part of their own supporting evidence.

Move to the courtroom, and it gets worse. A 2014 study by a team at Harvard Law School led by Jonathan Zittrain and Lawrence Lessig found that about 50% of the URLs cited in U.S. Supreme Court opinions between 1996 and 2011 no longer contained their original content. Links cited in Harvard Law Review articles fared even worse — only 26.8% still had their original content intact.

There’s an important distinction the study drew here. Link rot is when clicking the address produces an error. Reference rot³ is when the address opens fine, but the content has changed underneath it. What the researchers found was that the latter is far more common. In other words, there are more links that open but show altered content than links that simply don’t open at all. I’ll come back to why this is dangerous a bit later.

One caveat worth flagging: all these figures come from U.S.-centric samples. I haven’t yet seen any public data measuring the disappearance rate of the Korean-language web using the same methodology. But when I think about Naver’s blog and cafe service shutdowns, platforms that vanished wholesale like Daum View or Me2Day, and the repeated overhauls of news outlets’ article databases, there’s no particular reason to believe Korea is doing any better.

Looking at this fragment against the Korean source, everything checks out well: no Hangul remains, numbers (1998, 2, 20, 200) are preserved, headings/images match, and the meaning is faithfully carried over.

Berners-Lee’s Diagnosis: This Isn’t a Tech Problem, It’s an Organizational One

Let’s go back to that 1998 piece again. What makes Berners-Lee interesting is that he didn’t treat this as a technical flaw.

He wrote down every excuse managers gave him, one by one, and rebutted each: “We redesigned the site,” “We had to move the files,” “That file’s managed by Jane now, not John,” “We used to run CGI scripts, now we use a binary program.” And to each, he asked the same question back: why was John’s name in the address to begin with?

His prescription boils down to one sentence: designing an address is mostly a matter of deciding what not to put in it.

Here’s what he said to leave out: author names (people leave companies), subject categories (classification systems keep changing), document status (things like draft, latest, old), access-level distinctions (team-only, members-only, public), file extensions (even .html will change someday), traces of software (cgi-bin, .pl), even the disk name. Practically the only thing safe to keep is the publication date — because a date never changes.

Reading this list again, one pattern jumps out: everything he says to remove is internal organizational business. Who’s in charge, which department, what stage something’s at, what tech stack is being used. Addresses break not because servers fail, but because organizations carve their own internal structure into names exposed to the outside world. When the organization changes, the name collapses with it.

I’d put it this way: an address is a fossil of an org chart. Scan through any company’s URL structure and you can roughly read off which year it reorganized which team — paths like /2019/newbiz/, /ai-innovation/, /dx-center/. And the moment that team disappears, so does the path.

Berners-Lee pushes this all the way to a moral obligation. “It is the duty of a webmaster to allocate URIs which you will be able to stand behind in 2 years, in 20 years, in 200 years,” he wrote. Even in 1998 that sounds like a bit much — but looking at it now, if anything, it wasn’t strong enough.

As AI starts reading the web, the role of the link has changed

For the past 28 years, a broken link was mostly just an inconvenience. You’d click it, get a 404, feel annoyed, and search again. That’s not the case anymore. Instead of searching directly, people now ask AI, and AI reads the web to construct an answer, attaching sources underneath. In this structure, a link no longer functions as a “path to travel”—it functions as “proof that the claim is true.”

The problem is, that proof doesn’t work very well.

Columbia Journalism Review’s Tow Center published a study in March 2025 on exactly this. Researchers pulled excerpts from articles by 20 news outlets and asked 8 chatbots to identify the sources. Across 1,600 queries total, here’s what they found: over 60% of the answers were wrong. Grok 3 had an error rate of 94%, and of the 200 links it cited as sources, 154 (77%) led to error pages. Gemini and Grok 3 both included nonexistent or broken URLs in more than half of their responses.

What struck me most in this study wasn’t any of that, though. The paid, premium models actually had higher error rates. Rather than admitting uncertainty, they confidently delivered wrong answers. And because a source link was attached, the answer looked more trustworthy—except when you actually clicked it, there was nothing there.

This is where the reference rot I mentioned earlier comes back into play. If a link dies with a 404, at least we can notice. The real problem is when the address stays alive but the content underneath has changed. If an AI cites a URL as evidence for something it learned three years ago, there’s no way for a person to know unless they actually click through and check. And most people don’t click through.

And the web is closing from the other direction, too

There’s one more problem. If the story above is about “machines misreading the web,” this one is about “the web shutting its door on machines.”

Cloudflare changed its policy on July 1, 2025. Newly registered domains now block AI crawlers by default, and site owners have to explicitly grant access. This was the first time an infrastructure provider flipped that default. At the same time, Cloudflare rolled out “Pay Per Crawl,” a system for charging crawlers a fee.⁴

Behind this decision is a metric Cloudflare published: the crawl-to-refer ratio⁵ — a number showing how many times an AI company scrapes a given site before sending back a single visitor. As of the fourth week of June 2025, Anthropic’s ratio was about 70,900 to 1. That means 70,000 scrapes for every one visitor sent back. Mistral, by contrast, sat at 0.1 to 1 — sending back ten times more visitors than it scraped. The variance across companies is enormous.

That said, this number needs careful reading. As Cloudflare itself noted, traffic generated from apps doesn’t carry a referrer header, so the ratios for web-based tools may look worse than they actually are. Still, the broad trend is clear: on this metric, most AI companies crawl sites far more often than they send visitors back to them.

So what happens next? Sites close their doors. They block with robots.txt⁶, put up paywalls, hide content behind logins. It’s a rational choice. But it puts anyone trying to verify something in a bind.

• If a person tries to check a source directly, they have to click the link — and a large share of links are already dead
• If they ask an AI instead, they get an answer, but 60% of its citations are wrong
• If they go to check the original source, they run into crawler blocks and paywalls

In all three directions, the path to verification is disappearing. This is the real reason link rot matters again in 2026.

Oswarld’s Lens

I find it a bit frustrating when people treat this as purely an “archiving problem” — as if it’s fine so long as the Internet Archive dutifully saves everything. Archives let you look up a page after it’s already gone. They don’t stop the address from breaking in the first place.

Through consulting work, I’ve watched several website overhaul projects unfold, and the conversation about URL structure almost always comes last. The brand gets decided, the design gets locked in, the CMS gets chosen — and only then does a developer ask, “So how do we handle redirects?” By that point, the information architecture has already been rebuilt around the new org chart. Then, months after launch, when traffic drops, everyone blames the content. In reality, it’s usually the accumulated links being severed by the redesign itself.

There’s a line I use often when building go-to-market strategy: “The name you create today is one your future self, three years from now, will have to live with.” Product names, plan names, feature names, and URLs all follow the same logic. The more of the present moment you cram into a name, the easier it is to explain right now — and the more certain it is to break later. /2024_final_actually_final.pptx and /cgi-bin/browse.pl are exactly the same failure.

But one thing has changed decisively in the AI era. A stable URL used to be an SEO asset. Now it’s a question of whether AI can cite that address as the basis for an answer. I think the gap between organizations that keep their URLs fixed and those that reshuffle them with every redesign is going to widen dramatically. When a link persists, citations accumulate over time — and once citations pile up, the domain itself starts to be treated as a source of authority.

Of course, this is my own interpretation, and there isn’t yet enough public data to fully back it up. Long-term tracking of how often a given domain gets cited in AI answers is still in its early days as a field of research. But I think it’s worth preparing for this direction now, since the cost of maintaining a single redirect rule is close to zero.

Closing

Here’s the summary.

1. The web disappears fast. 38% of 2013 pages are already gone, and 54% of Wikipedia articles have dead references.
2. The cause is organizational, not technical. When a URL encodes internal details like who’s responsible, which department, or which tech stack, the name collapses the moment the organization changes.
3. As AI reads the web, the cost of broken links has grown larger. A link is now both a path a human clicks through and a source an AI answer cites as evidence. Yet the Tow Center’s investigation found that more than 60% of chatbot citations were wrong, all while the web is moving toward blocking crawler access.

What you can actually do right now turns out to be small. The next time you’re restructuring a site or a document system, before you decide on the new addresses, start by listing what you can strip out of the current addresses. Department names, the person in charge, status, file extensions. Removing just that much sharply raises the odds your links survive the next reorganization. Berners-Lee’s original document is still sitting in the same place it always has been (see References below), and you can read the whole thing in 30 minutes.

---

💬 Have you ever lost a whole set of old links to a redesign or migration? Tell me in the comments what you regretted losing most. I’ll turn the responses into a link-preservation checklist for a future issue.

📨 If someone near you is about to redesign their site, pass this along.

The draft looks accurate and complete—no Hangul remains, all numbers match, structure and footnote counts are preserved. No edits needed.

References & Further Reading

Primary sources

• Tim Berners-Lee, “Cool URIs don’t change”, W3C Style Guide, 1998. Read the original ··· This is where today’s newsletter starts. The fact that it’s lived at the same address for 28 years is the whole argument of this piece.
• Athena Chapekis, Samuel Bestvater, Emma Remy, Gonzalo Rivero, “When Online Content Disappears”, Pew Research Center, May 17, 2024. Read the report ··· This is the most solid empirical measurement to cite when talking about link rot. I’d recommend reading through the methodology section too.
• Jonathan Zittrain, Kendra Albert, Lawrence Lessig, “Perma: Scoping and Addressing the Problem of Link and Reference Rot in Legal Citations”, Harvard Law Review Forum, Vol. 127, 2014. Read the paper ··· This is the research that drew the distinction between “link rot” and “reference rot.” The single most important concept in today’s piece comes from here.
• Klaudia Jaźwińska, Aisvarya Chandrasekar, “AI Search Has a Citation Problem”, Columbia Journalism Review / Tow Center, March 6, 2025. Read the investigation ··· This compares 8 chatbots across 1,600 cases. The table of error rates by tool is especially worth a look.
• Cloudflare, “The crawl before the fall… of referrals”, Cloudflare Blog, July 1, 2025. Read the analysis ··· This lays out the crawl-to-refer ratio concept along with company-by-company figures. It’s also honest about the measurement’s limitations right in the text.

Background

• Cloudflare, “Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large”, July 1, 2025. Read the press release ··· This is the original announcement of the default-block switch for AI crawlers. I think this will be remembered as the document marking the turning point in the web’s access rules.
• “Pathfinder (website)”, Wikipedia. Read the entry ··· This is the fate of the very site Berners-Lee cited as a best-practice example. It shut down in April 1999.
• Perma.cc, Harvard Law School Library Innovation Lab. See the service ··· This is a case where the solution proposed in the paper above became an actual running service. It’s a rare instance of a problem statement turning into infrastructure — worth a look.

---

The author is Oswarld (Kwangseob Ahn). Current roles: Adjunct Professor at Sejong University, Strategy Consultant at INLEVEL9. Career, research, books, and recent work are kept current on the About page. Latest · July 2026: HEMA-2: A Consolidation-Aware Tri-Memory Architecture with Multi-Channel Scheduling for Lifelong Conversational AI.

📝 Glossary

Footnotes

1. URI (Uniform Resource Identifier): An addressing scheme for pointing to resources on the web. What we usually call a URL is one type of URI. Berners-Lee used the term URI to emphasize that “what it refers to” matters more than “where it’s located.” ↩

2. Common Crawl: A nonprofit archive that periodically scrapes the entire web and releases it for free. It’s also widely used as a starting point for large language model training data. ↩

3. Reference rot: A state where the link opens fine, but the content inside has changed from what was originally cited. Since no error appears, it’s hard to notice — which makes it trickier than link rot. ↩

4. Pay Per Crawl: A Cloudflare feature that lets site owners charge a fee for AI crawler access. Instead of the binary choice of blocking or allowing crawling, it attaches a price to it. ↩

5. Crawl-to-refer ratio: A ratio showing how many times an AI service has to crawl a given site in order to send back one visitor to that site. The larger the number, the more it means “taking a lot, giving back little.” ↩

6. robots.txt: A text file placed at the top level of a site that spells out which crawlers may go how far. It has no legal force and is closer to a gentleman’s agreement, relying on crawlers’ voluntary compliance. ↩