Gregius Data – RAG Benchmark

Overview

Gregius Data’s RAG benchmark validates response behavior through repeatable, evidence-backed runs against the live pipeline. It executes a configurable prompt matrix, captures raw output and structured JSON artifacts, and generates a Markdown scorecard for human review.

Prerequisites

• WordPress 6.9+ with Gregius Data plugin installed and activated
• WP-CLI available in the execution environment
• RAG subsystem configured with active connection and models
• A benchmark configuration JSON file (a template is shipped with the plugin)

---

How the Benchmark Works

The benchmark executes each prompt from a configured prompt matrix against the live RAG pipeline and captures durable evidence for manual QA review.

Prompt matrix

Each prompt in the matrix has a stable ID, a slug, the prompt text sent to RAG, and the expected behavior description. The canonical baseline covers three prompts (P1–P3) for single-entity, compare, and abstain scenarios.

Artifact outputs

For each executed prompt, the benchmark produces:

• A raw output file containing the complete command output.
• A JSON artifact with the extracted RAG response payload (query, answer, sources, metadata).
• A Markdown scorecard linking to all artifacts with columns for ID, prompt, expected outcome, actual behavior, and blank Verdict and Notes fields.

Human review process

No automated scoring is implemented. A human reviewer inspects the raw output, JSON artifact, and scorecard, then assigns PASS, PARTIAL, or FAIL verdicts for each prompt. This decouples data capture from assessment, keeping artifacts valid regardless of scoring rubric changes.

---

Running a Benchmark

Benchmark commands are executed through wp gg-data benchmark from the command line.

Full benchmark run

wp gg-data benchmark run

Executes all prompts in the configured matrix. Scorecard includes all configured prompts by default.

Single-prompt targeted run

wp gg-data benchmark run --only=P1 --scorecard-scope=selected

Executes a single prompt and limits scorecard rows to the executed set. Useful for targeted remediation validation after tuning.

Scorecard regeneration

wp gg-data benchmark scorecard --from-dir=<path>

Regenerates the scorecard from a previous run’s artifacts without re-executing prompts. Populates the Actual Behavior column from stored JSON artifacts.

---

Configuration

The benchmark reads runtime configuration from a JSON file. A template is shipped in the plugin at includes/cli/resources/benchmark/rag-benchmark-quality-config.example.json. Copy it to wp-content/uploads/gregius-data/config/ — the writable location survives plugin updates.

Required fields include the connection name, model identifiers (embedding, agentic, rerank, answer), and the prompt matrix entries. Missing required fields cause a non-zero exit with diagnostic output.

---