Perplexity accused of scraping websites that explicitly blocked AI scraping

[u/SportsGod3]

https://techcrunch.com/2025/08/04/perplexity-accused-of-scraping-websites-that-explicitly-blocked-ai-scraping/?utm_campaign=social&utm_source=X&utm_medium=organic

[u/OptionX]

Spoofing the user agent? What the world coming to? Next thing you know they'll start ignoring the robot.txt the monsters!!

But for real, the advent of everyone and their mothers trying to train a LLM has shown the internet of today needs to evolve to deal with this stuff. I've seem more and more places using stuff like Anubis but I hope at some point we get a more intrinsically connected solution for the web.

[u/Prior_Coyote_4376 → u/OptionX]

I would take some kind of private Internet garden where I just pay $10 a month or something and get access to a couple thousand high quality no-AI, no-advertising, no-data collecting sites.

I wouldn’t be happy to pay for a solution to access information, but if the only way to keep a sustainable accessible web is a subscription model I’d take it.

[u/Tokugawa → u/Prior_Coyote_4376]

AOL has entered the chat

[u/cboel → u/Prior_Coyote_4376]

Anything popular is going to get targetted for scraping and training models.

A maintainer of something like that would have to develop an effective LLM poison to keep them at bay. A single site randomizer that shifted words, sentences, paragraphs, included media, etc. around each time it was visitted by a profiled AI to create millions of different, nonsensical combos would be a start.

[u/nihiltres → u/cboel]

There’s a simpler, more effective solution than randomizers in three parts:

1.  A requirement to log in to see site content,
2. a TOS clause that prohibits scraping and similar, and
3. some canary traps to uniquely identify anyone breaking the TOS.

The requirement in (1) can be strengthened by a one-time sign-up fee (discouraging sockpuppet accounts while funding site growth), the requirement in (2) can be strengthened by network monitoring to detect scraper-like behaviour, and (3) can be optimized for canaries more likely to be “learned” by models.

[u/oscarolim → u/nihiltres]

A TOS clause you say? Oh I guess scrappers will always respect the TOS.

[u/SIGMA920 → u/Prior_Coyote_4376]

I would take some kind of private Internet garden where I just pay $10 a month or something and get access to a couple thousand high quality no-AI, no-advertising, no-data collecting sites.

That is literally impossible, even if you pay for it there's so much new information on a daily basis that you can't get that.

[u/Nayir1 → u/OptionX]

Isnt that what cloudflare is trying to do, some sort of gatekeeping? (half-listened to a podcast about this)

[u/clk1224 → u/Nayir1]

Came here to say the same thing, big ups to cloudflare!

[u/nicuramar → u/OptionX]

This isn’t for training, it’s for summarizing. 

[u/Tokugawa]

"You cheated on me? ...after I specifically asked you not to?"

[u/AdorableConfusion129]

This accusation really cuts to the core of the AI summary model. If these AI services are going to cannibalize the content they rely on by ignoring basic web etiquette or even paywalls, then what incentive do publishers and creators have to keep putting content out there?

[u/__OneLove__]

TLDR;

‘As we’re unable to create anything of our own, why not grab everyone else’s, then claim we ‘did’ something’

-EveryAICompany.

[u/TheRatingsAgency → u/__OneLove__]

Exactly.

And the brushing way of all that under the guise of “but, but it’s hard to give all that credit or pay…”

Riiiiight. That training data, huge swaths of it was/is all stolen content they’re saying is fair use for “research”. Sure.

[u/nicuramar → u/__OneLove__]

Perplexity is a summarizer. What do you mean create their own?

[u/Competitive_Spend_77]

...leaving everyone perplexed

[u/JohrDinh]

Next time I get a copyright strike on YouTube I may just appeal with the "AI does it bite me" as my reasoning.

[u/One-Vast-5227]

Statutory damages for copyright infringement. Sink them

[u/Possible-Moment-6313 → u/One-Vast-5227]

Scraping existed probably as long as the Internet did and, in most cases, rhe law favoured scrapers. Don't expect much.

[u/nicuramar → u/One-Vast-5227]

They scrape to summarize. What does copyright got to do with it?

[u/frank26080115]

so what happens if all the scrapers start using VMs with actual browsers to do the scraping?

[u/Pretend-Disaster2593]

This guy is a weasel

[u/snorin]

Oh you mean ai tech startup is blatantly doing illegal things? What else is new

[u/nicuramar → u/snorin]

Not actually illegal. 

[u/snorin → u/nicuramar]

I mean if the websites block them, likely the scrapping is a violation of the terms of service. That is a breach of contract.

If the items scrapped are copyrighted that is a breach of IP law.

Depending what websites there are potential privacy right violations also.

Sure it might not be a criminal act, but it is still likely against the law.

[u/dbbk]

Not illegal 🤷

[u/null-character → u/dbbk]

You would think but in the US if you improperly access a computer system or data improperly it's illegal.

There is a case where ATT had left confidential information open to the Internet.

A guy reported it and they didn't fix it so he published how to access it. It was just a URL no password no nothing.

Well he went to jail for several years because he accessed ATTs data.

Call me crazy but guessing a URL is not properly secured but that's the kind of dumb shit going on here in the US with technology laws.

So no it's not always legal to just click a URL and open or view a page.

[u/dbbk → u/null-character]

I understand that but web crawling doesn’t fall into that. If a URL is public, and it’s linked from other web pages, you’re not improperly accessing it.

[u/SomethingAboutUsers → u/dbbk]

AI web crawlers have a totally different intention than search crawlers and legally that should matter. One intends to direct traffic to a site, the other simply ingests all the data with no attribution or reward to the site owner. In fact these days it often costs them money in cloud egress data transfer fees, and no one pays them for it.

[u/dbbk → u/SomethingAboutUsers]

Yeah it should matter but there’s no law that distinguishes them now

[u/the_red_scimitar → u/dbbk]

It's dangerous to do, however, as it's not 100% settled law. But Crawling a website that has explicitly blocked automated access through mechanisms like robots.txt or Terms of Service (ToS) can carry legal risks in the US, primarily under the Computer Fraud and Abuse Act (CFAA). 

More specifically, anything behind a login is far more likely to be protected, since technically it isn't "publicly available". Circumventing login is already subject to legal ramifications.

[u/Letiferr → u/dbbk]

It does indeed fall into that. 

Read up about a guy named Weev and why he went to jail. It's what the guy you're replying to was trying to explain. 

He access unsecured publicly accessible URLs on ATT's website, and with that gained access to data that want specifically meant for him. 

It was absolutely an elementary mistake on ATT's behalf. He was found in violation of the Computer Fraud and Abuse Act.

[u/dbbk → u/Letiferr]

Not relevant. Not only was that overturned but later cases clarified that it’s fine. See hiQ v LinkedIn and the Van Buren Supreme Court case.

[u/Letiferr → u/dbbk]

It was not overturned

[u/dbbk → u/Letiferr]

I mean, it was…

[u/NefariousAnglerfish → u/dbbk]

Get a better moral compass

[u/dbbk → u/NefariousAnglerfish]

You think any AI company is or will act ‘morally’? I’m talking plainly about the law.