Custom Robots.txt Code to Block New AI Web Scrapers and Crawlers in Blogger

Here at PADAChecker, we noticed an influx of aggressive bots scraping site data, so we built this quick 2026 configuration framework to help you stop them.

Web content theft has fundamentally changed. In the past, website owners worried about cheap scraping plugins copying their text layout. Today, data training bots and generative engines are aggressively crawling private servers to feed foundational large language models, draining valuable hosting bandwidth without giving back any click-through referral traffic.

If your blog is hosted on Blogger, you do not have server-level firewalls or access to a localized .htaccess structure. Your ultimate line of data defense is an optimized Custom Robots.txt configuration.

---

⚠️ The Golden Rule of 2026 AI Scraping: Do Not Use Wildcards

Many site owners think writing a broad restriction block like User-agent: * followed by a generic block will protect their site.

This is a critical mistake. A global catch-all restriction can accidentally de-index your entire site layout from organic platforms like Google Search and Bing Search. In the modern web ecosystem, you must separate AI Data Scraping Bots (which steal text layout content for model training) from AI Indexing/Search Engines (which actively crawl to give your site high-intent citation traffic).

💡 Before Modifying Your Site Code: Ensure your security changes haven't caused a spike in backend structural vulnerabilities. Use the Free Site Checker Tool on PADAChecker.com to analyze your live domain authority, backlinks, and current platform spam scores instantly.

---

📋 The Ultimate Master Code Block for Blogger (2026 Setup)

This code blocks the major scrapers that harvest your data for AI training while explicitly keeping standard search engines open so your traffic continues to grow. Copy the exact code below:

# --------------------------------------------------
# PHASE 1: BLOCK AGGRESSIVE AI TRAINING SCRAPERS
# --------------------------------------------------

# OpenAI Training Bot (Feeds ChatGPT models)
User-agent: GPTBot
Disallow: /

# Anthropic Core Scraper (Feeds Claude models)
User-agent: ClaudeBot
Disallow: /

# Google AI Model Training (Does not impact standard Googlebot search)
User-agent: Google-Extended
Disallow: /

# ByteDance Scraper (Feeds TikTok & alternative global AI tools)
User-agent: Bytespider
Disallow: /

# Meta Platform Training Bots
User-agent: Meta-ExternalAgent
Disallow: /

# Common Crawl (Massive open-source dataset scraped by various AI systems)
User-agent: CCBot
Disallow: /

# Omgili / Webzio AI Data Harvesting Engine
User-agent: webzio-extended
User-agent: Omgilibot
Disallow: /

# --------------------------------------------------
# PHASE 2: ALLOW HIGH-INTENT AI CITATION ENGINES
# --------------------------------------------------
# These bots bring direct click traffic via dynamic source attribution.

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# --------------------------------------------------
# PHASE 3: PROTECT CORE BLOGGER LAYOUT STRUCTURES
# --------------------------------------------------

User-agent: *
Disallow: /search
Allow: /

# Replace with your exact custom domain path layout
Sitemap: https://padachecker.comsitemap.xml

---

🛠️ Step-by-Step Layout Guide to Install on Blogger

To update your crawl parameters safely without accidentally breaking your existing Google indexing states, follow this simple process:

[ Blogger Dashboard Settings ] → [ Crawlers and Indexing Section ] → [ Enable Custom Robots.txt & Paste Code ] → [ Save Modifications ]

1. Log into your account at the Blogger Admin Dashboard.
2. From the sidebar layout, select Settings.
3. Scroll down until you locate the subheader section labeled Crawlers and indexing.
4. Click the toggle switch next to Enable custom robots.txt to turn it on.
5. Click directly on Custom robots.txt to open the modal text window block.
6. Paste the master code block provided above into the open field container.
7. Important: Change the example sitemap URL at the bottom to match your exact domain layout (e.g., https://padachecker.comsitemap.xml).
8. Click Save.

---

🔍 How to Test and Verify Your New Rules

Once saved, do not assume the settings have propagated correctly. You need to verify that Google’s crawling infrastructure registers the instructions accurately.

• Access the live text file: Open a new browser tab and navigate straight to your root text route: https://yourdomain.com. Ensure the new layout appears cleanly without any missing or malformed line breaks.
• Audit using Google Search Console: Access your properties panel in Google Search Console. Use the URL Inspection tool on your homepage to verify that standard search crawlers are allowed while checking that your explicit AI blocks function correctly.