kernel.org says just 2% of its git traffic is real developers

Only about 2% of the traffic hitting git.kernel.org looks like a real developer. Konstantin Ryabitsev published that estimate on August 29, 2026, in a post called "Creepy crawlies." The rest is AI scrapers, and they now cost the Linux kernel's own infrastructure more computing power than every legitimate use combined.

Ryabitsev is the kernel infrastructure maintainer, as LWN.net describes him. He runs git.kernel.org, the public website that lets anyone browse the Linux kernel's source history in a web browser.

The numbers behind the claim

The site takes about 6 million requests a day. Most of them ask to see a random old commit. A commit is a single recorded change to the code, and the site has to build a web page for each one on demand.

That work adds up. Ryabitsev writes that 14 to 16 of the site's 90 processor cores are busy at any moment doing nothing but rendering commits for scrapers. That is roughly a fifth of the whole system's capacity, spread across five servers in different parts of the world.

His summary is blunt. "We spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones," he wrote.

The challenge page is not stopping them

kernel.org already runs Anubis, which it deployed about a year ago. Anubis is a proof-of-work gate. Before serving a page, it makes the visitor's browser solve a small math puzzle. The puzzle is trivial for one person and expensive for a program fetching millions of pages.

It works, partly. Ryabitsev reports that Anubis turns away 66% of scrapers outright. But 33% now solve the puzzle and walk through the gate anyway. Paying the compute cost is apparently worth it to whoever is collecting the data.

Why banning IP addresses does not work

The obvious fix is to block the offending addresses. According to LWN's report, that fails here. The scrapers run through residential proxies, which route requests through ordinary home internet connections rented from providers such as Bright Data. Each request arrives from a different household address, so a blocklist never catches up.

Telling the two groups apart is the harder problem. "It's impossible to tell with certainty which of these are bots and which are real humans," Ryabitsev wrote. His heuristic is a good one: a request for an old commit in a random old fork is probably not a developer doing their job.

Commenters on Lobsters noted that ClaudeBot, Anthropic's crawler, shows up prominently in the proxy logs of exposed git servers. One reader pointed out the waste on both sides. Scrapers are choosing, as they put it, "the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it."

What this means for developers

If you host a public git server, assume this is already happening to you. The kernel's numbers are extreme because the kernel is famous, but the crawlers are not selective. Check what share of your traffic asks for deep commit URLs rather than clones. That ratio is the tell.

Serve the cheap path instead of the expensive one. A git clone is one efficient transfer; a scraper walking every commit page forces your server to render thousands of documents. Rate-limiting the web view while leaving clone and fetch untouched protects the budget without blocking real work.

Expect challenge pages to spread, and expect them to annoy you. Anubis and tools like it now sit in front of many open-source project sites. When a build script or a CI job suddenly fails to fetch a patch, a proof-of-work gate is a likely cause. Point automation at the git protocol or a mirror rather than at the web interface.

The deeper point is a funding one. Volunteer-run infrastructure is absorbing the cost of training data collection it never agreed to supply. Nobody is billing the crawlers, so the bill lands on the projects.