AI Crawlers Just Got Blocked by Default — What It Means for Your Malaysian Business
If you run a website, blog, or even just have your business listed online, you’ve probably been feeding AI for free. But that just changed. Cloudflare, the massive internet infrastructure company, just flipped a switch that changes how your content gets used.
What Happened
On July 1, 2025, Cloudflare announced it will now block known AI web crawlers by default for new domain owners (source). This means any AI scraper caught trying to access your site without permission is automatically turned away—no extra setup required.
The company is also testing a “Pay Per Crawl” program with major publishers like The Associated Press, The Atlantic, Fortune, Stack Overflow, and Quora (source). These publishers can now set a price for AI companies to access their content. AI companies can choose to pay or get blocked.
Cloudflare has been working on this for a while. In 2023, they offered the option to block AI crawlers through robots.txt, but that only worked if the crawlers respected it. By 2024, they enabled blocking of all AI bots regardless of robots.txt. Now it’s the default for new customers (source).
“Original content is what makes the Internet one of the greatest inventions in the last century, and we have to come together to protect it.” — Cloudflare CEO Matthew Prince
Why This Matters for Your Business
If you’re a Malaysian SME owner, you might not think you’re a target for AI scrapers. But think again. Every product description, blog post, FAQ section, or customer review on your website is valuable training data for AI models. They’ve been using it for free.
This change puts control back in your hands. If your site is on Cloudflare (and many Malaysian SMEs use it for performance), you’ll now have this protection by default. If not, it’s worth considering—not just for speed, but for content security.
For businesses with specialty content—like detailed guides, industry reports, or unique photos—there could even be opportunities to charge for access. The Pay Per Crawl model is still early, but it signals that AI companies will need to start paying for quality data.
It feels likely that more platforms will follow Cloudflare’s lead. This could become the standard way to handle AI crawlers.
The Bigger Picture
Cloudflare’s move reflects a broader shift: the value of original content is rising again. As AI chatbots become the go-to for people seeking answers, the source of that information becomes critically important.
Publishers like The Associated Press and Fortune have been fighting AI scrapers for months. They’ve argued that their content fuels AI answers, without compensation or traffic back to their sites. This deal with Cloudflare is a step toward fairer treatment.
For Malaysian SMEs, this trend means you should start seeing your content as an asset. The days of freely offering information while others profit from it might be ending. It’s probably a good time to audit what you have online and consider how it’s being used.
Ultimately, this isn’t just about blocking bots. It’s about rethinking the relationship between content creators and AI companies. And for small businesses, that’s a conversation worth paying attention to.
Book a free 15-min call to see how protecting your online content from AI crawlers applies to your business → https://autorunbiz.com