What is a robots.txt file and do I need one?

A robots.txt file tells search engine crawlers which pages they can and cannot access on your site. Every website should have one. Without it, crawlers will attempt to index everything, including admin panels, API routes, and staging pages you'd prefer to keep private.

What should I include in my sitemap.xml?

Include all canonical, indexable URLs you want search engines to find. Each URL entry should have a lastmod date, change frequency hint, and priority value. Exclude URLs blocked by robots.txt, noindex pages, paginated variants, and duplicate content. Keep your sitemap under 50,000 URLs or 50MB.

What are common robots.txt mistakes?

The most common mistakes are: accidentally blocking CSS/JS files that Google needs to render pages, using incorrect wildcard patterns, forgetting to include a sitemap reference, blocking the entire site with 'Disallow: /' during development and forgetting to remove it, and not testing changes before deploying.

How often should I update my sitemap?

Update your sitemap whenever you add, remove, or significantly change pages. For dynamic sites like SaaS platforms, use a programmatic sitemap that generates automatically from your database. Next.js App Router supports this natively via a sitemap.ts file.

Free robots.txt & Sitemap Generator — robots.txt and sitemap generator

The Foundation of Technical SEO: robots.txt and Sitemap Files

Every search engine crawler that visits your website looks for two files first: robots.txt and sitemap.xml. Together, these files form the foundation of your technical SEO — they tell search engines what to crawl, what to skip, and where to find your most important pages. Getting them right is straightforward; getting them wrong can make your entire site invisible to search engines.

Understanding robots.txt

Your robots.txt file lives at the root of your domain (e.g., yoursite.com/robots.txt) and contains rules that instruct crawlers which parts of your site they can and cannot access. Each rule specifies a User-agent (which crawler the rule applies to) and Disallow or Allow directives (which paths to block or permit).

For startup websites, a well-configured robots.txt typically blocks: admin dashboards, API routes, authentication pages, search result pages (to prevent crawler traps), staging environments, and any dynamically generated pages that could waste crawl budget. It should always include a reference to your sitemap.xml file so crawlers can find it automatically.

Understanding Sitemap.xml

Your sitemap.xml is a structured XML file that lists every page you want search engines to index, along with metadata about each URL: when it was last modified, how frequently it changes, and its relative priority. While search engines can discover pages through crawling links, a sitemap ensures they find every important page — especially those buried deep in your site's architecture.

For dynamic websites built with Next.js, sitemaps should be generated programmatically from your database. The App Router supports this natively through a sitemap.ts file that queries your data and returns the sitemap structure at build time or on request.

Common Configuration Mistakes

robots.txt pitfalls: Accidentally blocking CSS and JavaScript files prevents Google from rendering your pages correctly. Using overly broad wildcards like "Disallow: /app" when you only meant to block "Disallow: /app/admin". Forgetting to remove "Disallow: /" after migrating from a staging environment — this single line makes your entire site invisible.

Sitemap issues: Including URLs that return non-200 status codes. Listing pages blocked by robots.txt (creates conflicting signals). Exceeding the 50,000 URL or 50MB file size limits without splitting into a sitemap index. Missing lastmod dates, which are the most useful signal for crawlers to know when to re-visit a page.

Using This Tool

Switch between Generator and Validator modes. The Generator walks you through creating standards-compliant files step by step — add user-agent rules, specify allowed and disallowed paths, and list your sitemap URLs. The Validator parses your existing files and flags common errors, syntax issues, and configuration problems. Download the generated files or copy them directly to your project.