How to create a robots.txt file

What robots.txt is for

It gives crawlers hints about which paths they may fetch. It is not a security boundary.

Minimal useful example

Allow public pages, block private admin areas, and point to your sitemap.

Common mistakes

• Blocking CSS/JS needed to render the page
• Using robots.txt to “hide” confidential URLs (use authentication instead)
• Forgetting HTTPS sitemap URLs

Generate yours

Use the Robots.txt Generator, then host the file at your domain root.