.Google.com's John Mueller answered an inquiry about why Google indexes web pages that are actually disallowed coming from creeping through robots.txt and why the it's safe to disregard the associated Look Console files regarding those creeps.Robot Traffic To Concern Criterion URLs.The person talking to the question chronicled that crawlers were making web links to non-existent query specification URLs (? q= xyz) to pages with noindex meta tags that are also blocked in robots.txt. What motivated the question is that Google.com is actually creeping the web links to those webpages, acquiring obstructed through robots.txt (without noticing a noindex robotics meta tag) at that point getting shown up in Google.com Browse Console as "Indexed, though blocked out by robots.txt.".The person talked to the observing question:." Yet right here is actually the significant question: why will Google.com mark pages when they can not also see the web content? What's the benefit in that?".Google.com's John Mueller confirmed that if they can not crawl the web page they can't view the noindex meta tag. He also makes an exciting reference of the internet site: search driver, urging to disregard the end results given that the "ordinary" individuals will not observe those outcomes.He wrote:." Yes, you're appropriate: if our experts can't creep the webpage, our company can't observe the noindex. That claimed, if our team can not creep the web pages, after that there's certainly not a lot for us to mark. So while you could view a few of those webpages with a targeted web site:- question, the typical consumer won't find them, so I would not bother it. Noindex is actually likewise alright (without robots.txt disallow), it simply means the Links will wind up being actually crawled (and also wind up in the Search Console report for crawled/not indexed-- neither of these conditions cause issues to the remainder of the internet site). The vital part is actually that you do not create all of them crawlable + indexable.".Takeaways:.1. Mueller's response confirms the restrictions being used the Web site: search advanced hunt operator for diagnostic factors. Among those main reasons is actually considering that it is actually certainly not attached to the frequent hunt mark, it's a separate trait altogether.Google.com's John Mueller discussed the website search operator in 2021:." The quick response is that a web site: inquiry is actually not suggested to be full, nor used for diagnostics objectives.An internet site question is a specific sort of hunt that restricts the results to a particular web site. It's basically only words site, a bowel, and afterwards the website's domain name.This inquiry confines the results to a certain site. It's not suggested to be a complete selection of all the pages coming from that website.".2. Noindex tag without utilizing a robots.txt is alright for these type of scenarios where a robot is connecting to non-existent webpages that are actually obtaining found through Googlebot.3. URLs along with the noindex tag are going to produce a "crawled/not recorded" entry in Look Console which those won't have a bad result on the rest of the site.Review the inquiry and also respond to on LinkedIn:.Why would certainly Google mark webpages when they can't even see the information?Featured Graphic by Shutterstock/Krakenimages. com.