# Core: Disallow User-agent: 008 Disallow: / User-agent: AhrefsBot Disallow: / User-agent: Baiduspider Disallow: / User-agent: BLEXBot Disallow: / User-agent: CCBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: Yandex Disallow: / # Core: Allow, Delay User-Agent: Bingbot Crawl-Delay: 5 User-Agent: msnbot Crawl-Delay: 5 # Custom User-agent: Twitterbot Allow: /