Published on LinkedIn — July 2023
Nowadays, ARC Publishing is probably one of the most popular CMS among big publishers. Property of Jeff Bezos and the Washington Post, websites like Infobae or Reuters use its technology.
Based in AWS to serve infrastructure, it helps many publishers to just “forget” about all that stuff and focus in the core business: creating good content for all the acquisition platforms.
React JS, the front-end gem
If we analyze ARC websites, they are all built in React JS, a modern and more efficient framework of JavaScript.
However, using all this stuff sometimes could derivate in some crawl issues that could impact in Search or Discover traffic.
Websites in ARC rely in a big bunch of JSON and inline JS code to paint the pre-render (the static HTML that the platform serves to Google to efficiently crawl the content).
A curious case in Google Discover
We have in the agency a website that out of the blue they lost almost 100% of their Google Discover traffic.
We talked with our client and they told us that they didn’t deploy anything so we started to explore in content stuff and technical SEO.
In terms of content, almost all entities and topics were close to zero so it was almost impossible to see if this was a Google’s decision or an unannounced update.
Analyzing Crawl Stats report
First, we crossed Discover traffic with Googlebot’s smartphone hits and we saw a strong correlation between the Discover critical moments and the bot’s trend.
Another interesting point: during the last two months the Image Googlebot’s crawl rate moved in a very similar way to Discover’s traffic (logical considering that image is crucial for CTR in the platform).
And after we did the technical fix we can see a peak in smartphone crawling, we recovered image crawling and traffic recovered from the almost 100% loss.
JSONs used to be 30-40% of the crawl stats hits. The zero hits in JSON was a set-up that we did with tech team to block a resources sub-folder that had lots of JSONs with image metadata brought by ARC.
After that we saw that an uptrend in these files started when Image googlebot hits were reduced and Discover traffic was sent to almost zero.
What we did to fix the JSON issue
We mapped that all the JSONs were allocated in /pf/api/v3 subfolder (be careful about blocking /pf/ because it contains all ARC assets to paint the DOM).
We are not confirming that this is 100% a technical stuff, but for websites that use ARC we recommend monitoring these issues if you see downtrends in your traffic.
Summary
- Image Googlebot is important for Google Discover
- Googlebot Smartphone crawl seems to move similar with Discover traffic
- Re-allocating crawl resources from JSONs or other non-important assets could improve Googlebot Smartphone crawling
