Overview
Gregius Optimizer provides two layers of crawl control:
- Robots.txt — Access policies that govern which crawlers (search engines, AI answer engines, AI trainers) can reach which parts of your site.
- Robots meta tag — Per-page indexing directives (
noindex,noarchive) dispatched by page type and configurable via a sidebar toggle.
The default robots.txt grants access to major search engines and AI answer services (ChatGPT, Perplexity, Claude) and blocks AI model trainers and administrative paths (wp-admin, except admin-ajax.php). These defaults establish a crawl posture that balances discoverability with a secure administrative surface.
Technical Prerequisites & Blocks
- Gregius Optimizer plugin installed and activated
- Settings → Reading → “Discourage search engines from indexing this site” must remain unchecked. When enabled, WordPress ignores custom robots.txt directives.
Edit your robots.txt
- Open any post or page in the Block Editor.
- Locate the Robots panel in the right sidebar.
- Click Settings to open the editor modal.
- Review your current directives in the text area. The site’s robots.txt URL appears as a clickable link at the top of the modal.
- Edit the directives. The text area auto-expands as you add lines.
- Click Update to save.
Example — Restrict AI crawlers
Block AI model trainers while preserving access for search engines and AI answer services:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Cohere-ai
Disallow: /Reset to defaults
- Open the Robots modal.
- Click Reset to defaults.
- The button is disabled when no custom override exists — a grayed-out control confirms default directives are active.
Resetting restores the original composition — the built-in directives that allow all major crawlers.
How robots meta works
Every page receives a <meta name="robots"> tag automatically. The directive is dispatched by page type:
| Page type | Directive |
|---|---|
| Standard pages and posts | index, follow |
| Search results | noindex, follow |
| 404 pages | noindex, follow |
| Posts hidden from search (via the Sitemap panel) | noindex, follow |
How to noindex a specific post
- Open the post in the Block Editor.
- In the Sitemap panel, toggle Hide page from search engines to on.
- Update or publish the post.
The post receives a noindex, follow directive and is stripped from the XML sitemap.
Permissions
| Action | Required capability |
|---|---|
| View robots.txt settings | manage_options |
| Update robots.txt content | manage_options |
| Hide a post from search engines | edit_post (on that post) |
Global robots.txt configuration requires manage_options — administrators and super admins. Content editors with edit_post can noindex their own posts.
Next Steps
Gregius Optimizer is fully open-source, transparent, and built for modern development workflows.
Available on WordPress.org: Install the plugin directly from the official in the WordPress Plugin Directory.
View on GitHub: You can review, fork, and inspect the entire codebase and core logic over at the repository on GitHub.
Documentation: Read the setup guides and technical documentation below to get started.
- Sitemap – Control which content appears in your XML sitemap
- Robots.txt – Configure crawl directives for search engines and AI bots
- Schema – Add structured data markup to help search engines understand your content
- Social Card – Control how your content appears when shared on social media and in search results.
- LLMs – Optimize your site for AI agents and large language models with an
/llms.txtfile