Home
Studio

Offline evaluations

The Evaluation SDK (mistralai-evaluations) lets you run offline evaluations on your LLM pipelines in a few lines of Python. Define a dataset, a task, and scorers to get results in your terminal and in Studio.

i
Information

Offline evaluations are available to Enterprise-tier organizations only. Contact your Mistral representative to enable them for your organization.

The SDK follows a simple mental model:

Dataset  →  Task  →  Evaluators  →  Results
  • Dataset: a list of input records (dicts) with your test cases.
  • Task: an async or sync function that receives a TaskContext and produces an output.
  • Evaluators: functions that receive a ScorerContext and score each output.
  • Results: statistics and per-record scores, displayed in the terminal and uploaded to Studio.
Installation

Installation

Prerequisites: Python 3.12+ and a Mistral API key.

Install the package from PyPI:

pip install mistralai-evaluations

Set your API key:

export MISTRAL_API_KEY=<your-api-key>
Quickstart

Quickstart

Create an eval.py file:

import asyncio
import os

from mistralai.evaluations import (
    Evaluation, Evaluator, Goal, Mistral, Project, ScorerContext, System, TaskContext,
)

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])

dataset = [
    {"sentence": "Hello, how are you?", "groundtruth": "English"},
    {"sentence": "Bonjour, comment ça va?", "groundtruth": "French"},
    {"sentence": "Hola, ¿cómo estás?", "groundtruth": "Spanish"},
]

async def task(ctx: TaskContext) -> str:
    response = await client.chat.complete_async(
        model=str(ctx.system.params["model"]),
        messages=[
            {"role": "system", "content": str(ctx.system.params["system_prompt"])},
            {"role": "user", "content": ctx.input_record["sentence"]},
        ],
    )
    return str(response.choices[0].message.content)

def scorer(ctx: ScorerContext) -> int:
    return 1 if ctx.input_record["groundtruth"].lower() == str(ctx.output).lower() else 0

async def main():
    run = await client.evaluation.run(
        project=Project(name="Language Detection"),
        evaluation=Evaluation(name="Accuracy Eval"),
        system=System(name="mistral-small", params={
            "model": "mistral-small-latest",
            "system_prompt": "What language is this sentence in? Reply with ONLY the language name.",
        }),
        dataset=dataset,
        task=task,
        evaluators=[
            Evaluator(
                name="accuracy",
                description="1 if the detected language matches the groundtruth, 0 otherwise.",
                scorer=scorer,
                goal=Goal.gte(0.8),
            ),
        ],
    )
    run.show(level="records")

asyncio.run(main())

Run it:

uv run eval.py

Results are printed in your terminal and uploaded to Studio under Observability > Evaluate > Evaluations.

Core concepts

Core concepts

A task is an async (or sync) function that receives a TaskContext and returns an output. This is the code you want to evaluate:

from mistralai.evaluations import TaskContext

async def task(ctx: TaskContext) -> str:
    response = await client.chat.complete_async(
        model=str(ctx.system.params["model"]),
        messages=[
            {"role": "system", "content": str(ctx.system.params["system_prompt"])},
            {"role": "user", "content": ctx.input_record["prompt"]},
        ],
    )
    return str(response.choices[0].message.content)

The task can return any type: strings, Pydantic models, or dicts. The SDK serializes the output for upload.

TaskContext gives you typed access to ctx.input_record (the current dataset record), ctx.system (the System config from evaluation.run()), and ctx.metadata (run metadata).

Organizing runs in Studio

Organizing runs in Studio

Use Projects and Evaluations to organize runs:

  • Project: groups related evaluations for a system (for example, "Language Detection").
  • Evaluation: a specific evaluation within a project (for example, "Accuracy Eval").
  • Run: each call to evaluation.run() creates a new run under the evaluation.
run = await client.evaluation.run(
    project=Project(name="My Project"),       # created if it doesn't exist
    evaluation=Evaluation(name="My Eval"),    # created if it doesn't exist
    dataset=dataset,
    task=task,
    evaluators=[...],
)

You can also reference existing entities by slug:

project=Project(slug="my-project")
evaluation=Evaluation(slug="my-eval")

Tags and metadata help you filter and compare runs in Studio:

run = await client.evaluation.run(
    ...
    tags=["model:mistral-small", "prompt:v2"],
    metadata={"commit": "abc123"},
)
Local mode

Local mode

Use local=True to run evaluations without uploading results to Studio. This is useful for fast iteration during development:

run = await client.evaluation.run(
    dataset=dataset,
    task=task,
    evaluators=[Evaluator(name="accuracy", scorer=scorer)],
    local=True,  # no upload, no project/evaluation required
)
run.show(level="records")

A typical development workflow:

  1. Iterate locally: set local=True and adjust your task and scorers until results look right.
  2. Push to Studio: remove local=True and add project and evaluation to track results over time.
Displaying results

Displaying results

The run.show() method prints results at different levels of detail:

run.show(level="run")          # Summary statistics only
run.show(level="records")      # Per-record results
run.show(level="generations")  # Per-generation details
run.show(level="scores")       # Full score breakdown

Export as JSON for further processing:

run.show(mode="json", level="records")
Results in Studio

Results in Studio

Once a run completes, results are available at Observability > Evaluate > Evaluations in Studio. Each run uploads its scores, statistics, and metadata automatically.

The Studio UI gives you four ways to explore results:

  • Trend charts: score averages over time with min/max bands, so you can spot regressions across runs.
  • Distribution charts: pie and stacked area charts showing score distribution across records and runs.
  • Composable filters: filter by tag, run ID, or metric threshold to isolate specific subsets.
  • Run detail view: open any run to inspect per-record inputs, outputs, and scores — with expandable generation details when num_generations > 1.

Cell values render automatically based on type: plain text, JSON objects, and LLM conversation arrays each get their own display format.

Explore the docs

Explore the docs

Work through the building blocks in the order you use them to build an evaluation:

  • Datasets: structure your test cases as typed Python records.
  • Configure system params: externalize task configuration for side-by-side comparison in Studio.
  • Evaluators: define scorers for each metric — rule-based functions or LLM-as-judge.
  • Set goals: define pass/fail thresholds and set direction and normalization on evaluators.
  • Configure statistics: choose which run-level values (averages, totals, percentiles) an evaluator exposes.
  • Optimize prompts and parameters: automatically search for better prompts and system params.

Advanced guides:

Reference:

FAQ

FAQ