Google has cracked down on web scrapers that pull data from search results. This has caused a lot of headaches for popular rank tracking tools like Semrush that rely on fresh data from search pages.
What if Google’s search results pages were completely off-limits? Tracking services have used different data sources to gather info for a long time. One way to get around this block might be to pull data from other sources.
SEO Tools Hit Hard
Google’s actions are really changing how fresh the data is for SEO tools. Many of them are struggling to collect data that usually comes from scraping Google’s search results pages.
A popular SEO tool, SE Ranking, is also seeing less new data coming in.
@LauraChiocciora shared A screenshot showing a message from SE Ranking saying that position tracking is back, but some SERP features are still offline because of “technical issues.”
The full message in the screenshot is:
“Position tracking has resumed, but SERP Features tracking remains unavailable due to technical difficulties. Our team is working hard to fix the issue and will get the data to you as soon as we can.”
No Scraping Allowed By Google
Google has always banned automated rank checking in its search results. But it seems they’ve allowed a few companies to scrape these results and charge for access to ranking data for tracking keywords.
According to Google’s guidelines :
“Machine-generated traffic, or automated traffic, means submitting automated queries to Google. This includes scraping search results to check rankings or using automated access to Google Search without clear permission. This traffic uses up resources and makes it harder for us to serve our users effectively. These actions go against our spam policies and Google Terms of Service.”
Today it’s not about ‘get the traffic, it’s about ‘get the targeted and relevant traffic.’
Tough to Block Scrapers
Blocking scrapers takes a lot of resources since they often change their IP addresses and user agents to avoid detection. One way to do it is to watch specific patterns, like how many pages a user requests. If someone makes too many requests, they might get blocked. But this also uses a lot of resources to manage the ever-growing list of blocked IPs, which can quickly reach millions.
Updates on Social Media
A post in the private SEO Signals Lab Facebook Group mentioned that Google is really going after web scrapers. One member said that the Scrape Owl tool wasn’t working right for them, and others noted that Semrush’s data hasn’t been updated.
Another LinkedIn post also pointed out that several tools weren’t updating their info. But it was mentioned that not all data providers were affected, as Sistrix and MonitorRank are still working fine. A rep from a company called HaloScan said they made changes to start scraping from Google again and have come back strong. Meanwhile, someone else noted that MyRankingMetrics is still reporting data.
Right now, Google’s moves don’t seem to be hitting all scrapers. They might be focusing on specific scraping behaviors and checking how it’s working, possibly improving their blocking methods. In the coming weeks, we may find out if Google is amping up their scraper-blocking skills or if they’re just targeting the biggest players.
A different LinkedIn post shared thoughts on the effects, including how this blockage could lead to higher costs and fees for users of SaaS SEO tools.
Natalia Witczyk ( LinkedIn profile ) shared her thoughts on LinkedIn :
“Recently, Google has been stepping up its game against data scraping with tougher measures like IP blocking and CAPTCHAs. This is affecting major SEO tools such as Semrush and SE Ranking.”
Google’s choices are making data gathering trickier and costlier, which might lead to higher subscription fees for users.
Is anyone else having data issues with their SEO tools?
EDIT: Ahrefs reported no outages, so I took their name off the first paragraph, but some users said they noticed data delays. Sistrix seems to be unaffected.
No Word from Google
So far, Google hasn’t said anything publicly. But with all the chatter online, they might want to consider making a statement soon.
