Guide · 2026-10-16

robots.txt and sitemap.xml for a new site

robots.txt

Put this at https://yoursite.com/robots.txt:

User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml

Watch out for Disallow: / under User-agent: *. That blocks all search engines and often gets left over from staging. Also check for <meta name="robots" content="noindex"> in your page head.

sitemap.xml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url><loc>https://yoursite.com/</loc></url>
  <url><loc>https://yoursite.com/pricing</loc></url>
</urlset>

List every public page. Next.js can generate this for you with an app/sitemap.js file.

Submit it

Add your site to Google Search Console, verify it, and submit the sitemap URL. Bing Webmaster Tools can import from Search Console.