Composing Scores: SEO, SMO, AEO, LLMO, AIO

robots.txt

Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.

https://www.robotstxt.org/

Action filter

if ( ! function_exists( 'starter_output_robots_txt' ) ) {
	/**
	 * Build robots.txt directives for production/public sites.
	 *
	 * @filter starter_robots_txt_enabled  Disable extra robots.txt directives. Default true.
	 *
	 * @param string $output    Existing robots output from WordPress.
	 * @param string $is_public Whether search indexing is enabled.
	 * @return string
	 */
	function starter_output_robots_txt( $output, $is_public ) {
		if ( '0' === (string) $is_public ) {
			return $output;
		}

		// phpcs:ignore -- Example usage documentation.
		// Usage: add_filter( 'starter_robots_txt_enabled', '__return_false' );
		if ( ! apply_filters( 'starter_robots_txt_enabled', true ) ) {
			return $output;
		}

		$sitemap = home_url( '/wp-sitemap.xml' );

		$rules = <<<ROBOTS
			User-agent: *
			Disallow: /wp-admin/
			Allow: /wp-admin/admin-ajax.php

			Sitemap: $sitemap

			# Traditional Search & Live RAG Retrieval
			User-agent: Googlebot
			Allow: /

			User-agent: Bingbot
			Allow: /

			# Conversational & Generative Answer Engines
			User-agent: OAI-SearchBot
			Allow: /

			User-agent: ChatGPT-User
			Allow: /

			User-agent: PerplexityBot
			Allow: /

			User-agent: ClaudeBot
			Allow: /

			User-agent: Claude-Web
			Allow: /

			# AI Data Model Trainers
			User-agent: Google-Extended
			Allow: /

			User-agent: GPTBot
			Allow: /

			User-agent: Applebot-Extended
			Allow: /

			User-agent: Cohere-ai
			Allow: /
			ROBOTS;

		return $rules;
	}

	add_filter( 'robots_txt', 'starter_output_robots_txt', 10, 2 );
}

Robots.txt Output

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://gregius.com/wp-sitemap.xml

# Traditional Search & Live RAG Retrieval
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# Conversational & Generative Answer Engines
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

# AI Data Model Trainers
User-agent: Google-Extended
Allow: /

User-agent: GPTBot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Cohere-ai
Allow: /

Schema.org

Schema.org is a collaborative, community activity with a mission to create, maintain, and promote schemas for structured data on the Internet, on web pages, in email messages, and beyond.

https://schema.org/

To simplify the process, we developed custom inspector control components that make injecting structured data directly from the editor effortless.

Schema.org output

{
  "@context": "https://schema.org",
  "@graph": [
    {
      "@type": "Organization",
      "@id": "https://gregius.com#organization",
      "name": "Gregius",
      "url": "https://gregius.com/",
      "description": "The Orchestration Layer for AI workflows in WordPress",
      "logo": "https://gregius.com/wp-content/uploads/2026/05/gregius-logo-600x600-gradient-vibrant.png",
      "sameAs": [
        "https://github.com/gregius/",
        "https://www.linkedin.com/company/gregius/"
      ]
    },
    {
      "@type": "WebSite",
      "url": "https://gregius.com/",
      "name": "Gregius",
      "description": "The Orchestration Layer for AI workflows in WordPress",
      "potentialAction": {
        "@type": "SearchAction",
        "target": "https://gregius.com/?s={search_term_string}"
      },
      "@id": "https://gregius.com#website",
      "publisher": {
        "@id": "https://gregius.com#organization"
      }
    },
    {
      "@type": "WebPage",
      "url": "https://gregius.com/",
      "headline": "Gregius",
      "description": "The open-source AI orchestration layer for WordPress. Gregius gives you full pipeline control, sovereign AI, and complete telemetry from top to bottom.",
      "mainEntityOfPage": {
        "@type": "WebPage",
        "@id": "https://gregius.com/"
      },
      "publisher": {
        "@id": "https://gregius.com#organization"
      },
      "image": "https://gregius.com/wp-content/uploads/2026/05/Screenshot-2026-05-14-203956.webp",
      "dateModified": "2026-05-21T13:55:47+00:00",
      "datePublished": "2026-05-15T20:14:37+00:00",
      "@id": "https://gregius.com#webpage",
      "isPartOf": {
        "@id": "https://gregius.com#website"
      }
    },
    {
      "@type": "FAQPage",
      "@id": "https://gregius.com#faq",
      "isPartOf": {
        "@id": "https://gregius.com#webpage"
      },
      "mainEntity": [
        {
          "@type": "Question",
          "name": "Why does Gregius exist?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "We are experiencing a structural shift toward agentic web applications, where digital interfaces are shifting from static click-and-scroll pages to conversational, context-aware environments. Gregius exists to provide WordPress operators with an on-premise alternative to control this intelligence layer completely."
          }
        },
        {
          "@type": "Question",
          "name": "What are the core design principles of Gregius?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "Gregius is built on three core principles: data custody, architectural decoupling, and open-source sovereignty. We believe that your website's data is your greatest asset and that its semantic memory, telemetry, and prompt pipelines should remain under your infrastructure control to avoid vendor lock-in."
          }
        },
        {
          "@type": "Question",
          "name": "Why is a self-hosted AI architecture important right now?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "As digital environments evolve, businesses require greater choice in how they manage digital assets and user interactions. By containerizing your intelligence stack within WordPress and your own database, you gain complete visibility into operational costs, clear boundary controls for data privacy, and a stable infrastructure insulated from external vendor changes."
          }
        },
        {
          "@type": "Question",
          "name": "How does Gregius bridge WordPress with modern AI engineering?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "Gregius is crafted at the intersection of two distinct architectural paradigms. It embraces native WordPress systems\u2014such as the Abilities API, WP-CLI, and action/filter hooks, while introducing advanced AI engineering patterns like vectors, telemetry streams, and recursive context compression."
          }
        },
        {
          "@type": "Question",
          "name": "Why use WordPress as the core orchestration layer?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "Powering over 40% of the web, WordPress is a proven, familiar content management hub. Gregius leverages this existing data environment to turn standard site content and metadata into structured, machine-discoverable capabilities. This enables teams to deploy advanced AI workflows natively, using the content backend they already know and manage every day."
          }
        },
        {
          "@type": "Question",
          "name": "What is the underlying data architecture?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "Gregius uses a decoupled dual-layer storage design. WordPress handles your standard content infrastructure and configuration logic, while a secondary PostgreSQL instance runs alongside it as an isolated execution layer for high-fidelity vector operations, semantic search, and embeddings. This split ensures heavy mathematical AI workloads never compromise standard web performance."
          }
        },
        {
          "@type": "Question",
          "name": "How does Gregius protect against rapid AI model obsolescence?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "The platform uses a unified provider registry that decouples application logic from underlying model providers. Connection layers and prompt templates can be dynamically changed without altering code base."
          }
        },
        {
          "@type": "Question",
          "name": "How should an enterprise think about the Gregius ecosystem?",
          "acceptedAnswer": {
            "@type": "Answer",
            "text": "Think of Gregius as an open, modular stack. Gregius Data provides the foundational infrastructure layer to handle database synchronization, pipeline routing, and basic orchestration. This foundation is completely extensible; your internal development team can use our native action and filter hooks to build custom solutions tailored exactly to your business logic. In fact, this is the exact same methodology we used to extend the platform and build Gregius Intelligence, a premium upgrade layer you can deploy to immediately introduce advanced features like session continuity and editorial curation."
          }
        }
      ]
    }
  ]
}

The Path Forward

The ultimate goal of this exercise was not to build another feature-heavy plugin. Instead, the intent was to create a lightweight, pragmatic approach that treats content, search, social, and artificial intelligence as a single, unified responsibility. By building this directly into the site and block editor, creators can seamlessly align the document’s data boundaries to match their intent while editing content in real time.

Our goal is to make this a portable component for anyone needing a lightweight, drop-in implementation. Before packaging it as a standalone plugin, we need to decouple critical pieces by moving the core configurations out of PHP action filters and directly into the WordPress editor UI. This will allow creators to natively tune these three foundational discovery files right from the dashboard:

• wp-sitemap.xml: Dynamically handling search and indexation.
• robots.txt: Managing traditional crawling permissions.
• llms.txt: Real-time context provisioning for downstream inference.

Taking back control of how our brand identity is parsed, synthesized, and cited is not optional anymore. Navigating the shift from UX to AIX just means building cleaner, more intentional bridges.

The score is set. Comments are open and we would love to hear how you are handling this shift in your own sites.

Frequently Asked Questions

Photo credit: Arco. Speaking of scores, we were lucky enough to catch this film during a TIFF screening in Toronto and it was incredible. If you need a backdrop while composing your own technical optimization scores, give the soundtrack a listen.