Here's a weird thought: when an AI agent reads a news article, 90% of what it's looking at is garbage.
Not the writing. Not the facts. The actual webpage itself — all the code, the ads, the menus, the cookie banners, the stuff built for a human eye and a browser window. None of that means anything to a machine. It just gets in the way.
That's the problem Jon Roberts has been thinking about. He's the chief innovation officer at People Inc., the company behind sites like People, Better Homes & Gardens, and a stack of other media brands you've probably clicked on without noticing who owns them. Speaking at Digiday's Media Strategies event in London, Roberts laid out something I think a lot of us haven't considered yet: AI agents don't read the internet the way we do, and right now, we're making them work way too hard to get at the good stuff.
The Problem With Feeding AI a Whole Webpage
Think about what's actually on a typical article page. Navigation bars. Sidebars. Autoplay videos. Newsletter pop-ups. Footer links to forty other stories. A human brain filters all that out in about half a second without thinking.
An AI agent can't do that instinctively. It has to process every bit of it — every line of HTML, every script tag — just to find the paragraph that actually matters.
"If you are an agent and you're reading an HTML page, 90% of the content is junk meant for a browser to read, not an agent," Roberts said. "It will really distract the agent, and it will be incredibly expensive to process all these tokens that it doesn't need."
That word "expensive" isn't just a figure of speech. Processing tokens costs real computing power and real money. So People Inc. decided to stop handing AI systems a messy house and start handing them a packed suitcase instead.
Building a Cleaner Way In
The company is now developing custom APIs — direct channels that deliver just the article text, formatted in simple Markdown, stripped of all the browser clutter. Along with the text, it includes what Roberts calls "trust signals that matter": who wrote the piece, when it was published, when it was last updated.
It's the same information a reader would want to know anyway. Who's behind this? Is it current? Can I trust it? Turns out AI agents need that context too.
This isn't People Inc.'s first attempt at solving the access problem. They started by letting AI companies crawl their sites the regular way. Then they moved to sending content through Amazon S3 storage buckets, which let them push updates and pull back content when needed. Now they're building something more precise — a direct line that skips the crawling step altogether.
As Roberts put it, bluntly: "If you throw it the entire HTML header, that's abuse. Right, that's not reasonable."
Why This Matters Beyond One Company
Here's the part that caught my attention. This shift is happening at the same time People Inc. cut about 6% of its workforce. Those two facts sitting next to each other tell a bigger story — publishers are scrambling to figure out how to survive as AI changes how people find and consume news. Fewer people are clicking through to articles the old way. More are asking an AI assistant to just summarize it for them.
I've spent years being curious about how things actually work underneath the surface, whether that's an engine, a visa system, or now, apparently, the plumbing of the internet. What's happening here isn't really about APIs. It's about control. By building a direct, efficient pipeline, People Inc. decides exactly what version of its journalism reaches an AI, and on what terms.
That's not a small thing. As more of us ask AI tools questions instead of typing them into a search bar, who controls that pipeline starts to matter a lot.
So here's a question worth sitting with: if most publishers eventually build something like this, does journalism get easier for AI to use, or easier for AI to control? I don't think anyone has that answer yet. But it's worth paying attention to, because this is one of those quiet shifts that ends up reshaping how we all get our information — long before most of us notice it happened.
Source: Digiday
Read our previous article: AI Doesn't Browse. It Answers. So Who Wins When There's No List to Climb?
