How to create a robots.txt file

## What robots.txt is for It gives crawlers hints about which paths they may fetch. It is not a security boundary. ## Minimal useful example Allow public pages, block private admin areas, and point to your sitemap. ## Common mistakes - Blocking CSS/JS needed to render the page - Using robots.txt to “hide” confidential URLs (use authentication instead) - Forgetting HTTPS sitemap URLs ## Generate yours Use the Robots.txt Generator, then host the file at your domain root.