# AI Chatbots User-agent: GPTBot User-agent: ChatGPT-User User-agent: Claude-Web User-agent: ClaudeBot User-agent: anthropic-ai User-agent: cohere-ai User-agent: Bytespider User-agent: Google-Extended User-agent: PerplexityBot User-agent: Applebot-Extended User-agent: Diffbot Disallow: / # Scrapers User-agent: Scrapy User-agent: magpie-crawler User-agent: CCBot User-agent: omgili User-agent: omgilibot User-agent: Node/simplecrawler Disallow: / # General Restrictions Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /wp-includes/ Disallow: /wp-content/ Disallow: /trackback Disallow: /feed Disallow: */comments Disallow: ?replytocom* Disallow: */comments-page-* Disallow: */trackback Disallow: */feed Disallow: *s=* Disallow: /*.pdf$ Disallow: /assets/js/pixels.js # Allowed Content Allow: /wp-content/cache/ Allow: /wp-content/uploads/ Allow: /wp-content/themes/ Allow: /wp-content/plugins/ Allow: /wp-includes/js/ Allow: /wp-includes/css/ # Events Calendar Restrictions Disallow: *post_type=tribe_events* Disallow: *hide_subsequent_recurrences=* Disallow: *tribe-bar-date=* Disallow: *tribe-venue=* Disallow: *eventDisplay=* Disallow: *eventDate=* Disallow: *paged=* Disallow: *pagename=* Disallow: *shortcode=* Disallow: *ical=* Disallow: *outlook-ical=* Disallow: *related_series=* Disallow: *tribe_geofence=* # Allowed Events Calendar Content Allow: /events/ Allow: /event/ # Crawl Rate Limitation Crawl-delay: 3