The site returns 200 OK, robots.txt doesn’t block crawling, and the meta tags are clean—but the pages still aren’t in the index. This complaint usually comes up on projects where a contractor has already worked on the site, so the first thing to check is not what’s in plain sight, but what is usually overlooked.
Indexing can be blocked in places other than robots.txt
The first things checked when diagnosing indexing are robots.txt and the robots meta tag on the page. But crawling and appearing in search can also be blocked at the server level, bypassing both of those places:
- with rules in .htaccess—for example, the X-Robots-Tag: noindex header, which is sent to bots and isn’t visible in the page source;
- with conditional redirects or response substitution specifically for search bots based on their user agent;
- with a script that runs immediately after the page is generated and changes its content or sends it to another address—while a regular visitor, and even a quick check using “View Page Source,” will show nothing suspicious.
If the project’s history includes an outstanding debt to a previous contractor, such hidden blocks are not uncommon: removing the directive takes a minute, but its effect continues until the dispute is resolved. That’s why, when intentional indexing restrictions are suspected, .htaccess and the server responses sent to bots should be checked first—before analyzing content and links.
The tool people usually don’t check, but should
There is a much simpler and much more common explanation: the page may not be in the index because it was sent there for removal by the site owners themselves—through the “Removals” tool in Google Search Console.
Temporarily removing a URL through this tool blocks the page from appearing in search for approximately six months. The page continues to return 200 OK and remains accessible via a direct link—the service merely hides it from search results and clears the cached copy. Six months is the period Google gives the site owner to either fix the page or permanently block indexing (404, a canonical tag, or server-side noindex).
If a previous contractor submitted the pages for hiding through this tool and then simply stopped working on the project, the symptoms will be exactly these: 200 OK, clean robots.txt, clean meta tags—and the page is missing from search for no apparent reason. Checking Removals in Search Console should be part of the basic technical audit checklist, not a fallback option for when nothing else has worked.
This same tool also has a practical use beyond diagnosis: it speeds up the removal of duplicates and already-configured redirects from the index when an old version of a URL needs to be removed quickly without waiting for it to be recrawled.
Internal linking doesn’t enable indexing, but it does change the bots’ schedule
Another common myth is that on a large site, it’s enough to “play around with internal linking” for pages to start being indexed. Changing the link structure alone doesn’t initiate indexing: if a page is blocked by a directive or doesn’t meet the search engine’s quality requirements, internal linking won’t change that.
What internal linking really changes is the frequency and route of bot crawling, and this differs between search systems. For Google, server behavior also plays a role: correct Last-Modified headers in the response help prioritize recrawling of changed pages. Yandex’s crawling rotation works differently and may behave less predictably. Internal linking shouldn’t be treated as a tool for “turning on” indexing—it adjusts crawling speed rather than acting as a visibility switch.
Diagnostic sequence
- Check .htaccess and the server response headers sent to bots—look for X-Robots-Tag and conditional redirects based on the user agent.
- Open Google Search Console → Removals and check whether the URL is temporarily hidden—this is the most common and most easily overlooked cause.
- Check robots.txt and the meta tags—if they’re clean and the first two items are also clear, look for a content-quality issue or a penalty.
- If you address the issue by changing internal linking, treat it as a way to speed up the recrawling of pages that are already open to indexing, not as a way to remove a block.
Pay particular attention to the places that aren’t visible at a glance: server responses for bots specifically and tools that can hide a page without changing either the code or the content.
How to distinguish causes by their symptoms
| Symptom | Likely cause | What to check |
|---|---|---|
| The page is visible to users but cannot be opened by a bot or returns different content to it | Conditional blocking by user agent or a directive in .htaccess | A request with the User-Agent substituted for Googlebot/YandexBot; compare the server response |
| The page returns 200, and the webmaster panel shows the status “Page is canonical, but not indexed” or directly indicates removal | Submitted for temporary hiding through Search Console | The “Removals” section in Search Console |
| Indexing is happening overall, but slowly; changes are picked up with a delay | Low crawl priority rather than a block | Last-Modified headers, how often the content changes, and internal linking |
This distinction saves time: there’s no point rewriting content or building more internal links if the problem is physically located in the “Removals” section of Search Console—first remove the block, and only then assess whether the page really lacks link equity or freshness.
The “Removals” section should be part of a regular technical audit, not a one-off measure for when problems arise: the tool is actively used both to clean up old duplicates and to remove redirects that have already been configured but remain from the site’s previous structure—and specialists who check it systematically on every project are less likely to miss cases like these.