robotted resources (old, probably outdated)
I see a bunch of posts about the robotted resources message that we’re sending out. I haven’t had time to go through & review them all (so include URLs if you can :)), but I’ll spend some time double-checking the reports tomorrow.
Looking back a lot of years, blocking CSS & JS is something that used to make sense when search engines weren’t that smart, and ended up indexing & ranking those files in search. For a long time now, that’s no longer the caseĀ (happy to get reports if you do see it though), and nowadays it’s extremely helpful to be able to access CSS & JS files so that we understand what a page really looks like. Is it mobile-friendly? Is there content that we’d miss without JavaScript? Blocked CSS & JS files are some of the bigger problems that we run across in the meantime.
Search Console gives you tools to find blocked content across your site, to test individual pages to see if all embedded content is accessible, and to test your robots.txt files to find why a specific URL is blocked (and to submit an updated robots.txt so that we can crawl it ASAP). We’ve also documented this problem in various ways over time, for example in https://developers.google.com/webmasters/mobile-sites/mobile-seo/common-mistakes/blocked-resources (archive.org)
That said, we hear you that these messages aren’t as clear as they could be, and we’ll double-check your reports (here on G+, on Twiter, in the forums, etc) to see what we can do to improve that.
Original atĀ https://plus.google.com/+JohnMueller/posts/8iCsUmstaHy (archive.org)
Comments / questions
There's currently no commenting functionality here. If you'd like to comment, please use Bluesky and mention me there: @johnmu.com. Thanks!
Related pages
- robots.txt (2015)
- 429 or 503 (2015)
- Staging Site indexing (2019)
- crawl budget & 404s (2016)
- 307s (2016)