.Google's John Mueller answered an inquiry about why Google indexes web pages that are disallowed from crawling through robots.txt and also why the it's risk-free to dismiss the relevant Browse Console reports concerning those creeps.Bot Traffic To Inquiry Guideline URLs.The person talking to the question documented that robots were actually producing web links to non-existent inquiry specification Links (? q= xyz) to web pages with noindex meta tags that are actually additionally obstructed in robots.txt. What urged the concern is that Google.com is actually crawling the links to those webpages, getting shut out through robots.txt (without watching a noindex robots meta tag) then receiving shown up in Google Explore Console as "Indexed, though blocked out by robots.txt.".The person talked to the following question:." Yet here's the significant inquiry: why will Google index pages when they can't also observe the web content? What's the conveniences because?".Google's John Mueller affirmed that if they can not creep the webpage they can not find the noindex meta tag. He additionally creates an intriguing mention of the web site: search operator, suggesting to neglect the end results due to the fact that the "normal" individuals will not view those end results.He created:." Yes, you're proper: if we can not crawl the webpage, our company can not find the noindex. That said, if our company can't creep the pages, then there's certainly not a lot for us to mark. Therefore while you might find several of those pages with a targeted website:- concern, the typical user will not observe all of them, so I would not fuss over it. Noindex is actually also great (without robots.txt disallow), it simply means the URLs will certainly find yourself being crept (as well as end up in the Browse Console report for crawled/not recorded-- neither of these statuses trigger concerns to the remainder of the site). The integral part is actually that you don't create all of them crawlable + indexable.".Takeaways:.1. Mueller's answer affirms the limitations being used the Site: hunt evolved search operator for diagnostic main reasons. One of those explanations is actually due to the fact that it is actually not linked to the routine hunt mark, it is actually a separate factor entirely.Google's John Mueller discussed the site hunt driver in 2021:." The quick response is actually that a website: question is certainly not suggested to become comprehensive, nor utilized for diagnostics objectives.A web site inquiry is a particular sort of hunt that confines the results to a certain web site. It's primarily just the word site, a bowel, and afterwards the website's domain name.This query restricts the end results to a certain web site. It is actually certainly not meant to become a comprehensive assortment of all the pages from that web site.".2. Noindex tag without making use of a robots.txt is fine for these kinds of circumstances where a robot is actually connecting to non-existent pages that are receiving found by Googlebot.3. URLs along with the noindex tag will definitely create a "crawled/not recorded" entry in Search Console and that those will not possess an unfavorable effect on the rest of the web site.Review the inquiry and also answer on LinkedIn:.Why will Google.com mark webpages when they can not also view the material?Featured Photo by Shutterstock/Krakenimages. com.