What Is the X-Robots-Tag?
The X-Robots-Tag is an HTTP header that provides the same indexing and crawling directives as a meta robots tag, but is applied at the server level rather than within a page’s HTML.
Its Use for Non-HTML Files
Because the meta robots tag only works within HTML `<head>` sections, it can’t be used to control indexing for non-HTML files — PDFs, images, or other downloadable documents. The X-Robots-Tag solves this by applying the same directives (like `noindex`) directly through the server’s HTTP response headers, regardless of file type.
Syntax and Common Use Cases
A typical implementation looks like: `X-Robots-Tag: noindex, nofollow`, set within the server configuration for a specific file type or path. It’s commonly used to keep PDFs, internal documents, or specific file directories out of search results without needing to modify each individual file.
WRITTEN BY
Annabelle Cochrane
MANAGING DIRECTOR & SEO SPECIALIST
Annabelle holds a Bachelor of Business (Marketing) from QUT and works directly with every ALC Consulting client. She’s based on the Sunshine Coast and works with businesses across Australia.
Not sure where your site stands?
Get a free audit from the person who’d do the work.