Do I need a robots.txt file if I am not blocking anything?

A missing robots.txt does not break your site, but it removes the one file where you tell crawlers what to skip and where your sitemap lives. Even a site that blocks nothing benefits from a file that declares an explicit policy and points to the sitemap. The spec is strict about where the file must live and how it must be encoded.

Why this matters

RFC 9309 defines the Robots Exclusion Protocol and states that the rules must be accessible in a file named /robots.txt, in all lowercase, at the top-level path of the service, encoded as UTF-8 with the Internet media type text/plain. When no such file exists, a crawler that receives a 404 treats the resource as unavailable and may crawl anything, so you lose the ability to exclude staging paths, search results, or admin routes without editing application code. There is a sharper risk on the other side: if the file exists but the server returns a 5xx error, RFC 9309 requires the crawler to assume a complete disallow, which can silently drop your site from indexing. Google's own guidance stresses that robots.txt manages crawler traffic and is not a mechanism for keeping a page out of search results, since a blocked URL can still appear with no description. A deliberate, well-formed file avoids all of this guesswork and gives you a reliable hook for your sitemap. It is also worth remembering that robots.txt is public by design, so never list secrets or private paths you would not want a competitor to discover.

How to fix it

Create a plain-text file at the site root. Use UTF-8 encoding, keep every directive lowercase, and include a Sitemap line. The Sitemap directive is independent of any user-agent group.

# https://example.com/robots.txt  (UTF-8, text/plain)
User-agent: *
Disallow: /admin/
Disallow: /checkout/
Allow: /

Sitemap: https://example.com/sitemap.xml

Group rules under a user-agent line, and remember that the longest matching path wins. Use a trailing dollar sign to anchor an exact match and an asterisk for wildcards, both of which RFC 9309 requires crawlers to support. If you intend to allow everything, an empty group with just a user-agent line is valid and is interpreted as allowing access.

How to verify the fix

Confirm the file is served at the correct path with the correct content type. A healthy response returns 200 and advertises text/plain.

curl -sI https://example.com/robots.txt
curl -s https://example.com/robots.txt

The first command shows the status line and Content-Type header; the second prints the rules exactly as a crawler would read them. Check in an incognito window too, because a valid robots.txt should never require authentication. If you recently edited the file, wait for the next crawl cycle before drawing conclusions from search results, since crawlers cache the previous version for up to a day.

Related issues