Introduction
Some time before the site migration, Barbour Magento began generating an infinite number of faceted URLs, which Google started crawling.
Lacking the development resources to fix the issue or at least add a noindex tag to prevent indexation and reduce Google’s crawl rate, we opted to disallow faceted URLs in the robots.txt file to prevent millions of unnecessary crawls.
The disallow rules were added on the 8th of January.
Results
Google’s response was unexpected—immediately after we disallowed the URLs in the robots.txt file, it began rapidly indexing them.
We then removed the disallowed parameters from the robots.txt file and immediately saw Google gradually removing the pages from its index.
Conclusion
It appears that the canonical tags were effectively preventing these URLs from being indexed, as expected. However, by disallowing them in robots.txt, we inadvertently blocked Google from seeing the canonicals, causing it to index the URLs instead.
