EpiFive
All free tools

robots.txt generator

Start from a preset or build your own rules. The robots.txt file updates as you type, and you can copy or download it.

Runs in your browser. Nothing you enter is sent to or stored by EpiFive.

Start from a preset
Rule group 1

Your robots.txt

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

Upload it to the root of your domain so it opens at https://yourdomain.com/robots.txt.

How to use it

  1. Tap a preset, or edit the first rule group. * means every crawler.
  2. List the paths to block under Disallow, and any exceptions under Allow. Paths start with /.
  3. Add more groups for specific bots, and your sitemap address at the bottom.
  4. Copy or download the file and upload it to the root of your site, then open yourdomain.com/robots.txt to check it.

Useful patterns

  • Disallow: /admin/ blocks everything under /admin/.
  • Disallow: /*? blocks URLs with query strings (Google and Bing support *).
  • Disallow: /*.pdf$ blocks PDF files; $ marks the end of the URL.
  • An empty Disallow: allows everything for that group.
  • Meta tag generator: Title, description, canonical, robots, Open Graph and Twitter tags with a live Google preview.
  • Keyword density checker: Top one, two and three-word phrases with counts and percentages, plus focus-keyword density.
  • DNS lookup: Look up A, AAAA, CNAME, MX, TXT, NS, CAA and SOA records for any domain, with TTLs and a share link.
  • JSON formatter: Format, validate and minify JSON, with the line and column of any error.

Related guides

Questions people ask

Where does robots.txt go?

At the root of your domain, so it opens at https://yourdomain.com/robots.txt. Each subdomain (such as blog.yourdomain.com) needs its own file. On WordPress, an SEO plugin usually lets you edit it from the dashboard.

Does Disallow remove a page from Google?

No. Disallow stops crawling, but a blocked page can still appear in results if other sites link to it. To keep a page out of Google, let it be crawled and add a noindex meta tag instead.

How do I block AI crawlers like ChatGPT and Claude?

Tap the “Block AI crawlers” preset. It disallows GPTBot, ChatGPT-User and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini training), PerplexityBot, CCBot (Common Crawl), Bytespider, Applebot-Extended, meta-externalagent and Amazonbot, while normal search engines can still crawl. Reputable bots obey robots.txt, but it is a request, not a lock.

Will blocking Google-Extended hurt my Google rankings?

No. Google-Extended only controls whether your content is used to train and ground Gemini models. Googlebot, which crawls for search, is separate and is not affected.

Does Google follow Crawl-delay?

No. Google ignores Crawl-delay and adjusts its crawl rate automatically. Bing, Yandex and some other crawlers do respect it.