Skip to content
M2TOOLKIT

Popular tools

Menu

Robots.txt Generator

Create a correct robots.txt file to guide search engine crawlers.

  • Free
  • No sign-up
  • Runs in your browser
Default for all crawlers

Use paths, not full URLs. * matches anything; $ marks the end.

Google ignores this; some other crawlers respect it.

Upload it to the root of your domain (example.com/robots.txt). Robots.txt controls crawling, not indexing — to keep a page out of search results, use a noindex tag instead and let it be crawled.

Everything you type is processed in your browser. Nothing you enter is sent to our servers or stored by us.

How to use the Robots.txt Generator

  1. Choose whether crawlers may crawl the site.
  2. List any paths to block, like /admin/ or /cart.
  3. Add your sitemap URL.
  4. Download robots.txt and upload it to your site's root.

What does this tool do?

robots.txt tells crawlers which parts of your site they may visit. It's a request, not security — reputable search engines follow it, but it doesn't hide pages. Never use it to protect private content.

Blocking a page in robots.txt can actually keep it in search results (without a description), because Google can't see a noindex tag on a page it isn't allowed to crawl.

Why use it?

  • Keep crawlers out of admin, cart and search pages.
  • Point search engines to your sitemap.
  • Opt out of AI training crawlers.

A typical robots.txt

User-agent: *
Disallow: /admin/
Disallow: /cart

Sitemap: https://example.com/sitemap.xml

Privacy

Everything you type is processed in your browser. Nothing you enter is sent to our servers or stored by us. There's no account to create and nothing to install.

Frequently asked questions

Should I block CSS and JavaScript?

No. Google needs them to render your pages and judge mobile-friendliness. Blocking them can hurt rankings.

Last reviewed by the M2Toolkit team.

Other tools people use alongside the robots.txt generator.