Gregius Data – RAG Evaluations

Overview

Gregius Data’s evaluation runner produces framework-ready evaluation datasets from pre-collected RAG interaction prompts. It executes prompts through the live RAG pipeline, generates a canonical JSONL dataset, and dispatches framework-specific adapter output for downstream quality measurement.

Prerequisites

• WordPress 6.9+ with Gregius Data plugin installed and activated
• WP-CLI available in the execution environment
• RAG subsystem configured with active connection and models
• An evaluation configuration JSON file (a template is shipped with the plugin)

---

How the Evaluation Runner Works

The evaluation runner processes a prompt corpus against the live RAG pipeline and produces three layers of output: canonical data, framework-specific adapter files, and a provenance manifest.

Canonical-first output

Each prompt is executed through the RAG pipeline, and the results are written to a framework-agnostic prompts.jsonl file. This decouples the evaluation dataset from any single framework’s schema, keeping artifacts stable and inspectable regardless of which adapter is used.

Framework adapter dispatch

After canonical output is written, a framework adapter transforms the data into a framework-specific format. Currently the Ragas adapter is supported, producing frameworks/ragas/dataset.jsonl compatible with Ragas SingleTurnSample field expectations.

Run manifest

Every run produces a manifest.json recording run identity, schema version, framework, prompt counts, execution scope, config path, and artifact paths with SHA-256 integrity hashes. This enables provenance tracking and cross-run comparison without directory scanning.

---

Running an Evaluation

Evaluation commands are executed through wp gg-data evaluation from the command line.

---

Configuration

The evaluation runner reads configuration from a JSON file. A template is shipped in the plugin at includes/cli/resources/evaluation/rag-evaluation-config.example.json. Copy it to wp-content/uploads/gregius-data/config/ — the writable location survives plugin updates.

Required fields include connection, framework, and a non-empty prompts array. Missing required fields cause a non-zero exit with diagnostic output.

Prompt corpus

Each prompt in the corpus must have a stable unique id and a user_input field. Optional fields include reference (ground-truth answer for reference-based metrics), reference_contexts (ideal context strings), and rubric (scoring guidance). The response and retrieved_contexts fields are populated at runtime from live pipeline execution.

---