Help Centre
Everything you need to use Robotsread: how it works, what the verdict means, what the tool can and cannot tell you, and how to get help.
Getting started
Using Robotsread takes about ten seconds. There's no account to create and nothing to install.
1. Go to the tool
Open the Robotsread tool page. You'll see a single text input and a "Check" button.
2. Paste a URL
Enter the full URL of the page you want to check — for example,
https://example.com/blog/my-article. The URL must be a public
page that the tool can fetch.
3. Click "Check"
Robotsread fetches the page and its origin's /robots.txt. It
reads all four indexability signals from the response, reconciles them, and
delivers a single verdict.
4. Read the verdict
The verdict is one of three results: Indexable — no blocking rule found; Blocked — a rule prevents indexing, with the exact rule and location named; or Unknown — the tool could not reach the page or parse its response.
5. Inspect the signal breakdown
Below the verdict, each signal is shown in its own card with the raw content fetched from the server. Use this to verify the tool's reading or investigate further.
Frequently asked questions
What does Robotsread do exactly?
It answers one question: can a search crawler index this page, and if
not, which specific rule stops it? It reconciles four separate
indexability signals — robots.txt Disallow rules, the
<meta name="robots"> tag, the X-Robots-Tag
HTTP response header, and the <link rel="canonical">
directive — into a single verdict. If the page is blocked, it names the
exact rule (robots.txt line number, tag content, or header value) that
produced the verdict.
How is the verdict determined?
The tool applies the most-restrictive-wins logic across all four signals. If any one signal says "do not index," the verdict is Blocked and the blocking signal is named. If all signals permit indexing, the verdict is Indexable. If the page could not be fetched at all, or returned a non-2xx status, the verdict is Unknown — the tool does not attempt to grade an error page as if it were a real reading of the site's indexability.
Why does it say "Blocked" when I think the page should be indexable?
One of the four signals is preventing indexing. Common causes:
• A Disallow: / rule in robots.txt — check the
"robots.txt" signal card for the exact rule and line number.
• A noindex directive in a <meta name="robots">
tag — check the "meta robots" signal card.
• An X-Robots-Tag: noindex HTTP header set by the server —
check the "X-Robots-Tag" signal card.
• A <link rel="canonical"> pointing to a different URL —
check the "Canonical" signal card.
If you believe the page is actually indexable despite the tool's verdict,
double-check the raw signals in the breakdown. A typo in a URL parameter,
a CMS setting you forgot about, or a security plugin that injects a
noindex header are all common real-world causes.
What does the Cloudflare warning mean?
A host behind Cloudflare (or a similar reverse-proxy service) cannot be reached from this diagnostic tool because Cloudflare's challenge/verification page blocks automated fetch requests. When this happens, the tool shows a yellow warning next to the URL input. Every verdict signal that depends on a successful fetch will show Unknown — this is a platform limitation, not a finding about the target site's indexability. If you own the site, you can verify its indexability by checking the page directly in your browser or using Google Search Console's URL Inspection tool.
Does Robotsread save or log the URLs I check?
No. Robotsread is stateless. The tool fetches the URL you submit and its
robots.txt, parses the response, displays the result, and
discards everything. There are no accounts, no databases, no server-side
logs of checked URLs, and no analytics tracking. Each check is independent
— refreshing the page resets everything.
Can I use Robotsread as an API or check multiple URLs?
No. Robotsread is a single-URL diagnostic tool accessed through a web interface. There is no public API, no batch submission feature, and no programmatic access. Repeated, automated, or high-frequency requests may be rate-limited to protect both the tool and the servers it queries. If you need to check many pages, consider using a search engine's own URL inspection tool or a dedicated crawler.
Known limitations
Robotsread is a focused diagnostic tool, not a full SEO platform. The following are honest limitations of the current product.
- Not a ranking predictor. The tool reports what the indexability rules say. It does not predict whether Google (or any other search engine) will actually index, rank, or display the page. Google's algorithms consider many factors beyond the four signals we check.
- Cloudflare-hosted sites return "Unknown." The tool cannot bypass Cloudflare's challenge page. The verdict will be Unknown for any host behind Cloudflare, with a warning displayed next to the input.
- No scheduled re-checks, alerts, or monitoring. There is no system that re-checks pages periodically or notifies you of changes. Each check is a one-off manual request. You must revisit the tool to check again.
- No email, SMS, or push notifications. The platform does not have a sender for any kind of notification. If a feature mentions alerts or reminders, it is not yet built.
- No accounts or saved history. There is no way to save results, bookmark a history of checks, or share a permalink to a verdict. Take a screenshot or copy the breakdown if you need a record.
- No bulk or batch mode. The tool processes one URL at a time. There is no API, no CSV upload, and no scriptable interface.
- Rate-limited by design. The tool may reject rapid or automated requests to protect itself and the target servers. This is not a paid-tier upgrade — it is a hard constraint of the current architecture.
Contact us
Robotsread does not have a built-in contact form or email system. The best way to report a bug, ask a question, or share feedback is through the project issue tracker. If you need to reach the owner directly, please use the contact details provided in the repository.