olmo-eval

Overview

This project provides a unified workbench for evaluating language models throughout the model development loop.

Features:

Registry of benchmark tasks and composable suites, with named variants for few-shot settings, formatting, and scoring (e.g. humaneval:3shot:bpb).
Support for inference via vLLM, SGLang, LiteLLM for commercial APIs, and a mock provider for dry runs and debugging.
Harness abstraction that separates execution policy from task definition, so any task can be run baseline or tool-augmented without modification.
Multi-turn agentic evaluation with tool calling, scaffolds, and sandboxed environments via Docker, Podman, or Modal.
LLM-as-judge scoring with auxiliary providers, including locally served judge models.
Aggregate and instance-level prediction storage.
Inspection tooling for viewing instances, formatted prompts, token arrays, and model responses.

To contribute, start with CONTRIBUTING.md. The reasoning behind the design is in docs/design.md, and AGENTS.md holds the instructions for coding agents.

Quick Start

This project uses uv with a checked-in uv.lock for reproducible builds. To get started, sync the repo with uv, browse the available tasks and suites, and preview a run with the built-in mock provider.

Run Your First Eval

# Install uv if not already installed
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install Python 3.12 if your machine does not already have it
uv python install 3.12

# Install dependencies + the package (editable) from the lockfile.
# The default groups (`dev` + `vllm`) are installed automatically, which
# pulls in storage, beaker, hf, and the vLLM inference provider. vLLM
# deps are marked Linux-only via PEP 508 markers, so this works on macOS
# too — no extra flags needed.
uv sync --frozen

# Install pre-commit hooks
make setup

# To update the lockfile after changing pyproject.toml
uv lock

# Add an optional extra on top of the defaults (e.g. agents, litellm)
uv sync --frozen --extra agents

# `openhands` conflicts with vllm — opt out of the vllm group when using it
uv sync --frozen --no-group vllm --extra openhands

# Browse a few suites
uv run olmo-eval suite inspect mmlu
uv run olmo-eval suite inspect gpqa
uv run olmo-eval suite inspect olmobase:code

# Preview a run without loading a model
uv run olmo-eval run -m mock -t gsm8k --dry-run

# Preview another run with a different task spec
uv run olmo-eval run -m mock -t humaneval:3shot:bpb --dry-run

Key Concepts

The evaluation framework is built around these core abstractions:

Abstraction	Description
Task	Benchmark specification defining dataset slice, request construction, and scoring logic
Suite	Benchmark collection that composes tasks and/or nested suites and defines result aggregation
Harness	Execution runtime around the inference provider, tools, scaffolds, and runtime behavior
Formatter	Prompt renderer from an instance and few-shot context to an LM request
Scorer	Per-example evaluator from model output to raw score or judgment
Metric	Dataset-level aggregator over per-example scores

Tasks

Tasks define how to load data, format prompts, and score outputs. Register with @register:

from olmo_eval.evals.tasks.common import Task, register
from olmo_eval.data import DataSource

@register("my_task")
class MyTask(Task):
    # DataSource specifies path, subset (optional), and split
    data_source = DataSource(path="cais/mmlu", subset="abstract_algebra", split="test")
    ...

Variants can also act as named evaluation presets (for example, few-shot settings):

from olmo_eval.evals.tasks.common import register_variant

register_variant("my_task", "3shot", num_fewshot=3, fewshot_seed=42)
# Built-in example: uv run olmo-eval run -m llama3.1-8b -t humaneval:3shot:bpb

Runtime Dependencies allow tasks to specify packages installed at job startup:

@register("code_eval")
class CodeEvalTask(Task):
    data_source = DataSource(path="my-org/code-dataset", split="test")
    dependencies = ["code-sandbox==1.0", "git+https://github.com/user/repo@v2.0"]
    ...

Suites

Suites group multiple tasks for batch evaluation:

from olmo_eval.evals.suites import Suite, register

register(Suite(
    name="my_suite",
    tasks=("task_a:3shot", "task_b:3shot", "task_c:3shot"),
))

Aggregation

Suites support different strategies for combining task results:

Strategy	Description
`AVERAGE`	Simple average of all task scores (default)
`WEIGHTED_AVERAGE`	Average of all task scores, each weighted by the task's instance count
`AVERAGE_OF_AVERAGES`	Average over child suite averages (equal weight per child)
`DISPLAY_ONLY`	Display child results without computing suite average
`GAP`	Two tasks, reference then companion: both primary scores and the companion minus the reference
`NONE`	No suite score: tasks report individually, and each nested suite reports its own aggregate

Average of Averages Example:

from olmo_eval.evals.suites import Suite, AggregationStrategy, register

# Nested suite with 3 tasks
multilingual_code = Suite(
    name="multilingual_code",
    tasks=("mbpp_python", "mbpp_java", "mbpp_rust"),
    aggregation=AggregationStrategy.AVERAGE,
)

# Parent suite using average of averages
register(Suite(
    name="code_eval",
    tasks=(
        "humaneval",        # Single task (score: 0.80)
        multilingual_code,  # Nested suite with 3 tasks (scores: 0.40, 0.50, 0.60)
    ),
    aggregation=AggregationStrategy.AVERAGE_OF_AVERAGES,
))

# Results:
# - humaneval: 0.80
# - mu
10BC8
ltilingual_code average: (0.40 + 0.50 + 0.60) / 3 = 0.50
#
# AVERAGE_OF_AVERAGES: (0.80 + 0.50) / 2 = 0.65
# vs AVERAGE:          (0.80 + 0.40 + 0.50 + 0.60) / 4 = 0.575

Note: AVERAGE_OF_AVERAGES gives each child equal weight regardless of how many tasks it contains.

Weighted Average Example:

register(Suite(
    name="mmlu_pro",
    tasks=("mmlu_pro_math", "mmlu_pro_history"),  # 1351 and 381 questions
    aggregation=AggregationStrategy.WEIGHTED_AVERAGE,
))

# With scores of 0.30 (math) and 0.45 (history):
# WEIGHTED_AVERAGE: (0.30 * 1351 + 0.45 * 381) / 1732 = 0.333
# vs AVERAGE:       (0.30 + 0.45) / 2 = 0.375

WEIGHTED_AVERAGE matches the instance-weighted "micro" average that oe-eval reports for some suites. It weights each task by the number of instances that task scored.

A weighted suite reports its weighted mean or no score at all. If a contributing task has no instance count, the suite aggregate is omitted and the runner logs which tasks were missing, rather than publishing an unweighted mean under the same suite name. Instance counts are required when a result is stored, so this only affects results written before that check existed.

Gap Example:

register(Suite(
    name="omega:dev",
    tasks=("omega_500:hillclimb", "omega_500_out"),  # in-distribution, then held-out
    aggregation=AggregationStrategy.GAP,
))

# With scores of 0.60 (in) and 0.45 (out):
# primary_score: reference 0.60, companion 0.45 (the suite's primary), gap -0.15

A gap suite retains its definition in suite_metrics, but is omitted from summary unless both tasks scored on the same primary metric. It can be nested only in a NONE suite, which reports it under its own name: a gap is not a score to average.

The held-out companion score is the value to rank and optimize. The gap is a diagnostic guard, shown in percentage points in the log summary; reducing the reference score never improves the suite's primary score. In metrics.json, suite_metrics["omega:dev"]["primary_score"] retains reference, companion, and gap; summary selects the companion.

Formatters

Formatters convert instances into LM requests. See olmo_eval.common.formatters for available options.

from olmo_eval.common.formatters import MultipleChoiceFormatter, ChatFormatter

# Multiple choice with logprob scoring
formatter = MultipleChoiceFormatter(template="Q: {question}\n\nA:")

# Chat-based formatting
formatter = ChatFormatter(system_prompt="You are a helpful assistant.")

Scorers

Scorers compute a score for each instance/output pair. See olmo_eval.common.scorers for available options.

from olmo_eval.common.scorers import ExactMatchScorer, MultipleChoiceScorer

# Exact string match
scorer = ExactMatchScorer()

# Multiple choice comparison
scorer = MultipleChoiceScorer()

Metrics

Metrics aggregate scores across responses. See olmo_eval.common.metrics for available options.

from olmo_eval.common.metrics import AccuracyMetric, F1Metric
from olmo_eval.common.scorers import ExactMatchScorer, F1Scorer

# Mean accuracy
metric = AccuracyMetric(scorer=ExactMatchScorer)

# Mean F1 score
metric = F1Metric(scorer=F1Scorer)

Model Presets

Pre-configured model settings in olmo_eval/common/constants/models.py:

from olmo_eval.common.constants import get_model_presets

# Returns dict of preset name -> ModelConfig
presets = get_model_presets()
# {
#     "llama3.1-8b": ModelConfig(model="meta-llama/Meta-Llama-3.1-8B"),
#     "olmo-2-7b": ModelConfig(model="allenai/OLMo-2-1124-7B"),
#     ...
# }

Harness

A Harness is the runtime orchestration layer for an evaluation run. It combines the primary inference provider with execution policy such as system prompts, tools, auxiliary providers, sandboxing, metrics collection, and an optional scaffold for multi-turn control. This lets the same task run in plain, tool-using, or scaffolded modes without changing the task definition.

Key concept: Any task can be run with or without tools—that's determined by the Harness configuration, not the task definition. This allows comparing baseline vs tool-augmented performance on the same task.

Using Harness via CLI

# Run task without tools or a scaffold (baseline)
uv run olmo-eval run -m llama3.1-8b -t simpleqa:judge

# Run task with search tools via harness preset
uv run olmo-eval run -m llama3.1-8b -t simpleqa:judge --harness dr_tulu

# Use a custom harness config file
uv run olmo-eval run -m llama3.1-8b -t simpleqa:judge --harness-config ./my_harness.yaml

HarnessConfig

Configuration for a harness:

from olmo_eval.harness import HarnessConfig, ProviderConfig, get_harness_preset
from olmo_eval.harness.tools.search import (
    semantic_scholar_search,
    serper_web_search,
    serper_fetch_page,
)

# Get a preset
config = get_harness_preset("dr_tulu")

# Or create custom config with tools
config = HarnessConfig(
    name="my_harness",
    provider=ProviderConfig(model="gpt-4o", kind="litellm"),
    tools=(semantic_scholar_search, serper_web_search, serper_fetch_page),
    system_prompt="You are a helpful assistant with search tools.",
    max_turns=10,
    max_concurrency=8,
    scaffold="openai_agents",
    required_secrets=("S2_API_KEY", "SERPER_API_KEY"),
)

Field	Type	Default	Description
`name`	`str`	Required	Harness identifier
`provider`	`ProviderConfig`	`ProviderConfig()`	Model provider configuration
`tools`	`tuple[Tool \| str, ...]`	`()`	Tool instances or registered tool names
`system_prompt`	`str \| None`	`None`	System prompt to inject
`tool_choice`	`str`	`"auto"`	Tool selection mode (`auto`, `none`, `required`)
`scaffold`	`str \| None`	`None`	Execution scaffold (e.g., `openai_agents`)
`max_turns`	`int \| None`	`None`	Max turns for multi-turn execution
`max_concurrency`	`int \| None`	`None`	Concurrent executions
`scoring_concurrency`	`int \| None`	`None`	Max concurrent scoring operations
`sandboxes`	`tuple[SandboxConfig, ...]`	`()`	Sandbox configurations for isolated tool execution
`scaffold_kwargs`	`dict[str, Any]`	`{}`	Scaffold-specific options (e.g., `enable_compaction`)
`metrics`	`MetricsConfig \| None`	`None`	Inference metrics collection config
`batching`	`BatchConfig \| None`	`None`	Batching strategy configuration
`max_hard_failure_rate`	`float \| None`	`None`	The failure budget: fraction of a task's instances allowed to produce no output or have no scored output before the task is marked failed, its metrics are withheld and the run exits non-zero (default `0.05`). `1.0` never fails a run on partial failures; a task with nothing scored still fails
`required_secrets`	`tuple[str, ...]`	`()`	Required environment variables

Scaffolds

Scaffolds define how the Harness executes multi-turn requests with tool calling. A scaffold handles the agentic loop: calling the model, executing tools, and feeding results back.

# List available scaffolds
uv run olmo-eval scaffolds

When to use a scaffold:

For multi-turn execution with harness.run(), you must specify a scaffold
For single-turn generation with harness.generate(), no scaffold is needed

# Multi-turn execution requires a scaffold
config = HarnessConfig(
    name="my_agent",
    provider=ProviderConfig(model="gpt-4o", kind="litellm"),
    tools=(semantic_scholar_search, serper_web_search),
    scaffold="openai_agents",  # Required for run()
)
harness = Harness(config)
result = await harness.run(request)  # Uses the scaffold

# Single-turn generation works without a scaffold
config = HarnessConfig(
    name="simple",
    provider=ProviderConfig(model="gpt-4o", kind="litellm"),
)
harness = Harness(config)
outputs = harness.generate(requests)  # No scaffold needed

Inference Metrics

Harness configurations can include MetricsConfig to collect inference performance metrics during evaluation. A run without --harness or --harness-config collects them with the default MetricsConfig(), writing metrics/<provider>_<model>-inference.jsonl in the output directory, as the built-in presets do:

from olmo_eval.harness import HarnessConfig, ProviderConfig
from olmo_eval.inference.metrics import MetricsConfig

config = HarnessConfig(
    name="with_metrics",
    provider=ProviderConfig(model="llama3.1-8b", kind="vllm_server"),
    metrics=MetricsConfig(
        enabled=True,
        reporters=("file", "db"),  # Save to file and database
        collect_vllm_server=True,  # Poll vLLM server /metrics endpoint
    ),
)

Visualizing Metrics:

# Plot metrics from database (requires at least one filter)
uv run olmo-eval metrics plot -G my-benchmark-group
uv run olmo-eval metrics plot -m OLMo-3 --metric throughput

# Show statistics table without interactive plots
uv run olmo-eval metrics plot -e experiment_123 --stats-only

When using the db reporter, metrics are stored in a PostgreSQL database (default name: olmo_eval_metrics). You must configure your own database connection using the OLMO_EVAL_DB_* environment variables (see Database Configuration).

Auxiliary Providers and Local Judge Models

Some tasks or custom scorers use LLM-as-judge scoring, where a separate model evaluates responses. The auxiliary_providers configuration lets you specify additional inference providers for scoring or judging. Harness overrides must come immediately after --harness, while task overrides like limit=... must come after -t.

Local example with uv run olmo-eval run:

uv run olmo-eval run \
    --harness default \
    -o provider.max_model_len=16384 \
    -o provider.num_instances=1 \
    -o 'metrics.reporters=[file]' \
    -o 'metrics.collect_gpu=true' \
    -o 'provider.kwargs.timeout=300' \
    -o auxiliary_providers.judge.kind=vllm_server \
    -o auxiliary_providers.judge.model=Qwen/Qwen3-8B \
    -o auxiliary_providers.judge.num_instances=1 \
    -o scoring_concurrency=4 \
    -m Qwen/Qwen3-8B \
    -t simpleqa:judge \
    -o limit=10

Key configuration options:

Option	Description
`auxiliary_providers.judge.kind`	Provider type: `vllm_server`, `litellm`, etc.
`auxiliary_providers.judge.model`	Model to use for judging
`auxiliary_providers.judge.num_instances`	Number of parallel vLLM instances
`auxiliary_providers.judge.base_url`	URL for external servers (when not spawning locally)
`scoring_concurrency`	Number of concurrent scoring requests

Defining Tools

Tools combine schema (for the LLM) and implementation (for execution) in a single definition:

from olmo_eval.harness import tool, registered_tool

# Option 1: @tool decorator (local use)
@tool(description="Search the web for information")
async def web_search(query: str) -> str:
    """Search implementation."""
    return await search_api(query)

# Option 2: @registered_tool decorator (global registry, for cross-process use)
@registered_tool(description="Fetch a webpage")
async def fetch_page(url: str) -> str:
    """Fetch implementation."""
    return await fetch_url(url)

Tools are automatically registered when using @registered_tool, making them available by name in HarnessConfig.

Custom Harness Config File

Create a YAML file for custom harness configurations:

# my_harness.yaml
name: custom_search
tool_names:
  - semantic_scholar_snippet_search
  - serper_google_webpage_search
system_prompt: |
  You are a research assistant with web search capabilities.
  Use search tools to find accurate information before answering.
max_turns: 15
max_concurrency: 4
required_secrets:
  - S2_API_KEY
  - SERPER_API_KEY

uv run olmo-eval run -m llama3.1-8b -t simpleqa:judge --harness-config my_harness.yaml

Programmatic Usage

from olmo_eval.harness import Harness, HarnessConfig, ProviderConfig, get_harness_preset
from olmo_eval.harness.tools.search import (
    semantic_scholar_search,
    serper_web_search,
)

# Create harness with preset and provider override
config = get_harness_preset("dr_tulu").with_provider(
    ProviderConfig(model="meta-llama/Llama-3.1-8B-Instruct", kind="vllm")
)
harness = Harness(config)

# Or create from scratch
config = HarnessConfig(
    name="my_harness",
    provider=ProviderConfig(model="gpt-4o", kind="litellm"),
    tools=(semantic_scholar_search, serper_web_search),
    system_prompt="You are a helpful assistant.",
    scaffold="openai_agents",
)
harness = Harness(config)

# Multi-turn execution with tool calling
result = await harness.run(request, sampling_params)
print(result.trajectory)  # Shows all turns including tool calls
print(result.final_output)  # Final model response

Adding New Tasks

This section explains how to create new evaluation tasks.

Quick Start: Minimal Task Example

A multiple-choice task scored by the log-likelihood of each answer choice:

"""Example: a minimal multiple-choice task scored by log-likelihood."""

from collections.abc import Iterator
from typing import Any

from olmo_eval.common.formatters import MultipleChoiceFormatter
from olmo_eval.common.metrics import LogprobMCAccuracyMetric
from olmo_eval.common.types import Instance, LMRequest
from olmo_eval.data import DataSource
from olmo_eval.evals.tasks.common import Task, register


@register("my_task")
class MyTask(Task):
    """MMLU abstract algebra, scored by the highest-likelihood answer choice."""

    # Pin the revision so the task's data cannot change underneath stored results.
    data_source = DataSource(
        path="cais/mmlu",
        subset="abstract_algebra",
        split="test",
        revision="c30699e8356da336a370243923dbaf21066bb9fe",
    )
    formatter = MultipleChoiceFormatter(
        template="Question: {question}",
        choice_template=" {choice}",
        prompt_suffix="\nAnswer:",
    )
    metrics = (LogprobMCAccuracyMetric(),)

    @property
    def instances(self) -> Iterator[Instance]:
        # Loads the configured data source, calls process_doc, and caches the result.
        yield from self._load_instances_cached()

    def process_doc(self, doc: dict[str, Any], index: int = 0) -> Instance | None:
        return Instance(
            question=doc["question"],
            choices=tuple(doc["choices"]),
            gold_answer=doc["choices"][doc["answer"]],
            metadata={"id": f"my_task_{index}", "gold_idx": doc["answer"]},
        )

    def format_request(self, instance: Instance) -> LMRequest:
        assert self.config.formatter is not None
        return self.config.formatter.format(instance, self.get_fewshot())

Class attributes that match TaskConfig fields (data_source, formatter, metrics, and so on) become the task's default configuration. Put the module in src/olmo_eval/evals/tasks/ and it is registered on import. Check it with uv run olmo-eval task inspect my_task --request -n 1 and uv run olmo-eval run -m mock -t my_task -o limit=5. Generation tasks use a chat or completion formatter instead, extract an answer in extract_answer, and score it with a metric such as AccuracyMetric(scorer=ExactMatchScorer).

Task Class Overview

Method	Required	Purpose
`instances`	Yes	Property that yields `Instance` objects; usually `yield from self._load_instances_cached()`
`format_request(instance)`	Yes	Converts an `Instance` into an `LMRequest` for the model
`process_doc(doc, index)`	When using `_load_instances*`	Converts a raw document into an `Instance`, or `None` to skip it
`extract_answer(output)`	No	Extracts the answer from `LMOutput`; defaults to `config.answer_extractor` or the raw text
`_build_fewshot()`	No	Override to customize few-shot example loading
`score_responses(...)`	No	Override to customize scoring logic
`compute_metrics(...)`	No	Override to customize metric computation

TaskConfig Reference

Commonly used fields. TaskConfig in src/olmo_eval/evals/tasks/common/base.py documents the rest.

Field	Type	Default	Description
`name`	`str`	Required	Task identifier used in CLI
`data_source`	`DataSource \| str \| None`	`None`	Dataset source (HuggingFace, S3, GCS, local, or URI string)
`formatter`	`Formatter \| None`	`None`	Request formatter
`metrics`	`tuple[Metric, ...]`	`()`	Evaluation metrics (scorers are inferred from metrics)
`num_fewshot`	`int`	`0`	Number of few-shot examples
`limit`	`int \| None`	`None`	Max instances to evaluate
`split`	`Split`	`Split.TEST`	Dataset split to use
`primary_metric`	`MetricName \| Metric \| None`	`None`	Primary metric for ranking (defaults to single metric if only one)
`sampling_params`	`SamplingParams \| None`	`None`	Default sampling parameters for this task
`answer_extractor`	`Callable[[str], str] \| None`	`None`	Function applied to output text by the default `extract_answer`
`response_parser`	`str`	unset	Set `harmony` to score only structured Harmony final-channel text
`strip_thinking`	`bool`	`False`	Drop `<think>...</think>` traces before scoring
`dependencies`	`list[str] \| None`	`None`	Runtime packages to install (e.g., `["pkg==1.0"]`)
`judge_model`	`str \| None`	`None`	Judge model for LLM-as-judge scoring

For inline vLLM, or a provider that returns raw Harmony delimiters, use -o response_parser=harmony. Inline vLLM preserves the generated Harmony token boundaries before its default decoder removes special tokens, saving them in text_with_special_tokens for auditing or re-scoring. Saved predictions also record response_parser_found: false means no delimited transcript was found (including an already parsed plain answer). If a known structured decode cannot be aligned with the stopped completion, explicit parsing raises a scoring error; the diagnostic and raw text remain in saved predictions. This opt-in changes the task hash, keeps the original decoded generation in original_text, and scores an empty answer when a Harmony analysis trace has no final channel. It accepts the documented <|start|>/<|channel|>/<|message|> delimiters of the gpt-oss tokenizer. Already parsed plain text passes through. Decoding with special tokens removed loses message boundaries: analysis...assistantfinal... cannot be parsed reliably; retain special tokens or use a provider's native Harmony reasoning parser before evaluating.

Generation diagnostics

Generation diagnostics count outputs, including each sample of an instance. cap_hit counts known length finish reasons and is a lower bound when finish_reason_unknown is nonzero. unclosed_think detects emitted <think> tags without a closing tag in the original generation. empty counts empty scoring text and excludes tool calls. Tasks that strip or transform text inside extraction, without changing LMOutput.text, set output.metadata[SCORING_TEXT_EMPTY_KEY] from that scoring text before parsing; a nonempty answer that fails parsing is still nonempty.

Data Sources

Tasks can load data from multiple sources using DataSource:

< 10BC8 div class="highlight highlight-source-python notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="from olmo_eval.data import DataSource # HuggingFace datasets - specify path, subset, and split DataSource(path="cais/mmlu", subset="abstract_algebra", split="test") # Using URI string (alternative syntax) DataSource.from_uri("hf://cais/mmlu?subset=abstract_algebra&split=test") # Without subset (for datasets that don't have subsets) DataSource(path="openai_humaneval", split="test") # With specific data files and revision DataSource(path="my-org/dataset", data_files="data/test.jsonl", revision="v1.0") # Local JSONL files DataSource(path="/path/to/dataset.jsonl") # S3 DataSource(path="s3://my-bucket/datasets/data.jsonl") # GCS DataSource(path="gs://my-bucket/datasets/data.parquet")">

from olmo_eval.data import DataSource

# HuggingFace datasets - specify path, subset, and split
DataSource(path="cais/mmlu", subset="abstract_algebra", split="test")

# Using URI string (alternative syntax)
DataSource.from_uri("hf://cais/mmlu?subset=abstract_algebra&split=test")

# Without subset (for datasets that don't have subsets)
DataSource(path="openai_humaneval", split="test")

# With specific data files and revision
DataSource(path="my-org/dataset", data_files="data/test.jsonl", revision="v1.0")

# Local JSONL files
DataSource(path="/path/to/dataset.jsonl")

# S3
DataSource(path="s3://my-bucket/datasets/data.jsonl")

# GCS
DataSource(path="gs://my-bucket/datasets/data.parquet")

Name		Name	Last commit message	Last commit date
.githooks		.githooks
.github		.github
dashboard		dashboard
docs		docs
examples/beaker/configs		examples/beaker/configs
infra/terraform		infra/terraform
scripts		scripts
src/olmo_eval		src/olmo_eval
tests		tests
.dockerignore		.dockerignore
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CONTRIBUTING.md		CONTRIBUTING.md
DEVELOPMENT.md		DEVELOPMENT.md
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
pyproject.toml		pyproject.toml
skiff2.json		skiff2.json
uv.lock		uv.lock

Field	Type	Default	Description
`path`	`str`	Required	Dataset path (HuggingFace repo, S3/GCS URI, or local path)
`subset`	`str \| None`	`None`	Dataset subset/config name
`split`	`str`	`"test"`	Dataset split
`source_type`	`SourceType \| None`	`None`	Backend type; inferred from `path` when unset
`data_files`	`str \| tuple[str, ...] \| None`	`None`	Specific data files to load
`revision`	`str \| None`	`None`	Dataset revision/version

Family	Suite	Tasks
Image-QA	`molmo2_imageqa`	`chart_qa`, `vqa2`, `doc_qa`, `info_qa`, `text_vqa`, `real_world_qa`, `mmmu`, `mmmu_pro`, `math_vista`, `countbench_qa`, `pixmo_count`, `ai2d`, `charxiv_descriptive`, `charxiv_reasoning`
Dense caption	`molmo2_imageqa_caption`	the image-QA tasks plus `dense_caption`
Pointing	`molmo2_pointing`	`pixmo_points_eval`, `sa_co_gold_subset`
Pointing (model prompts)	`molmo2_pointing_mp`	`pixmo_points_eval_mp`, `sa_co_gold_subset_mp`, `sa_co_gold_point_4k_mp`
Multi-image	`molmo2_multiimage`	`muir_bench`, `mmiu`, `blink`
Document OCR, English (default for Molmo2)	`ocr_en`	`olmocr_bench`, `cc_ocr_multi_scene_en`, `omnidocbench_en`
Document OCR, every language	`ocr`	`olmocr_bench`, `cc_ocr_multi_scene`, `omnidocbench`

	v1.6 (`omnidocbench`)	v1.5 (`omnidocbench_v15`)
LaTeX	TeX Live 2025, `pdflatex` + `CJK`	TeX Live 2025, `xelatex` + `xeCJK`
Font	`gkai` (TeX Live `arphic`)	Source Han Sans SC
ImageMagick	7.1.1-47	7.1.1-47
Ghostscript	9.55.0	9.55.0
Other		Node.js 16.13.1 (KaTeX tokenizer)

Variable	Default	Purpose
`OMNIDOCBENCH_CDM_DIR`	`~/.cache/olmo_eval/omnidocbench/cdm`	Where the toolchain is installed; point several runs at one shared install to reuse it
`OMNIDOCBENCH_EVAL_DIR`	`~/.cache/olmo_eval/omnidocbench`	Where each version's evaluator checkout and virtualenv are provisioned
`OMNIDOCBENCH_EVAL_WORKERS`	`min(16, cpus / 4)`	Evaluator worker threads (matching, TEDS, CDM); 48 suits a Beaker GPU node
`OMNIDOCBENCH_EVAL_TIMEOUT_S`	`43200` (12 h)	Wall-clock cap on evaluation; past it, as on any evaluator failure, the task reports no metrics and an error, and keeps the predictions for rescoring
`OMNIDOCBENCH_DIR` / `OMNIDOCBENCH_V15_DIR`	Hub download	Local copy of the dataset (`OmniDocBench.json` + `images/`)

Format	Flag	Description
Table	`--format table`	Rich terminal tables (default)
JSON	`--format json`	Structured JSON with pagination metadata
CSV	`--format csv`	CSV output to stdout

Variable	Default	Description
`OLMO_EVAL_DB_HOST`	`localhost`	Database host
`OLMO_EVAL_DB_PORT`	`5432`	Database port
`OLMO_EVAL_DB_NAME`	`olmo_eval`	Database name
`OLMO_EVAL_DB_USER`	`postgres`	Database user
`OLMO_EVAL_DB_PASSWORD`	-	Database password (use this OR `OLMO_EVAL_DB_SECRET_ARN`)
`OLMO_EVAL_DB_SECRET_ARN`	-	AWS Secrets Manager ARN for password (fetched on auth failure)

Flag	Description
`--resume`	Record progress under `<output_dir>/progress/` and continue from any already there
`--fresh`	Delete the recorded progress of this run's tasks and model before the run; cannot be combined with `--resume`

Option	Description
`-n, --count`	Number of instances to display
`-s, --skip`	Number of instances to skip
`--instance`	Show instance details (default if no other flags)
`--request`	Show the LM request
`-T, --tokenizer`	Tokenizer for formatting/tokenization
`--formatted`	Show prompt after template applied (requires `-T`)
`--tokens`	Show token array (requires `-T`)
`--max-tokens`	Max tokens to display (0 for no limit)
`--max-chars`	Max chars for formatted prompt (0 for no limit)
`--max-string-length`	Max chars for instance field values (0 for no limit)
`--json`	Output as JSON

Flag	Description
`--inspect`	Enable all inspection flags below
`--inspect-instance`	Print the first instance of each task before running
`--inspect-request`	Print the first LM request before model generation
`--inspect-formatted`	Show formatted prompt (after chat template applied)
`--inspect-tokens`	Show token array before evaluation
`--inspect-response`	Print the first response after model generation

Field	Type	Description
`name`	`str`	Eval identifier
`metrics`	`dict[str, float]`	Evaluation metrics
`metadata`	`dict`	Additional metadata
`success`	`bool`	Whether the eval completed successfully
`error`	`str \| None`	Error message if failed
`duration_seconds`	`float \| None`	Execution time
`raw_output`	`str \| None`	Raw stdout/stderr from the evaluation
`predictions`	`list[dict] \| None`	Instance-level predictions

Field	Type	Default	Description
`image`	`str`	Required	Container image
`mode`	`SandboxMode`	Required	`LOCAL`, `DOCKER`, or `MODAL`
`capabilities`	`frozenset[str]`	`Capability.DEFAULT`	Capabilities this sandbox provides, such as `Capability.BASH` or `Capability.PYTHON`
`instances`	`int \| None`	`None`	Executors to create; `None` lets the caller allocate capacity, otherwise one executor
`container_runtime`	`str`	`"podman"`	`"docker"` or `"podman"`
`command_timeout`	`float`	`30.0`	Default timeout per command (seconds)
`working_dir`	`str`	`"/workspace"`	Working directory in the container
`environment`	`tuple[tuple[str, str], ...]`	`()`	Environment variables as `(name, value)` pairs
`volumes`	`tuple[tuple[str, str], ...]`	`()`	Volume mounts as `(host_path, container_path)` pairs

Mode	Description
`LOCAL`	Run commands locally (development only)
`DOCKER`	Run in Docker/Podman containers
`MODAL`	Run on Modal cloud platform

Option	Short	Default	Description
`--config`	`-f`	none	YAML config file (CLI args override config values)
`--name`	`-n`	auto	Experiment name (auto-generated from model/tasks if not provided)
`--model`	`-m`	required	Model name or HuggingFace path (can specify multiple)
`--task`	`-t`	required	Task name with optional `@priority` suffix (can specify multiple)
`--harness`	`-H`	none	Harness preset name
`--override`	`-o`	none	Override for preceding `-t` or `-H` (can specify multiple)
`--cluster`	`-c`	required	Cluster alias (`h100`, `a100`, `aus`) or full name
`--gpus`	`-G`	auto	Number of GPUs (defaults to 1 for GPU providers, 0 otherwise)
`--max-gpus-per-node`		`8`	Maximum GPUs per node (tasks split if exceeded)
`--priority`	`-p`	`normal`	Job priority (`low`, `normal`, `high`, `urgent`)
`--preemptible`		`true`	Allow preemption (deprecated by Beaker; prefer `--min-runtime`); `--no-preemptible` means `--min-runtime 8h`
`--min-runtime`		none	Minimum runtime before Beaker may preempt the job (e.g., `2h`)
`--timeout`	`-T`	`24h`	Job timeout (e.g., `24h`, `30m`)
`--retries`	`-r`	none	Number of retries on failure
`--workspace`	`-w`	required	Beaker workspace
`--budget`	`-B`	required	Beaker budget
`--image`	`-I`	default	Custom Beaker image
`--group`	`-g`	auto	Add experiments to Beaker group(s) (auto-generated if not specified)
`--external-eval`	`-E`	none	External evaluation name(s) to run instead of tasks
`--eval-arg`	`-A`	none	Arguments for external evals (`key=value`)
`--provider-kwarg`	`-K`	none	Provider kwargs for external evals (`key=value`)
`--force-download-model`		`false`	Refresh Hugging Face model/tokenizer cache before loading
`--uv-cache-dir`		default	UV cache directory for package downloads
`--weka-bucket`		none	Extra Weka bucket to mount at `/weka/<bucket>`, on top of `oe-training-default`, `oe-eval-default` and `oe-adapt-default` (can specify multiple)
`--resume / --no-resume`		enabled	Recover interrupted task progress; opt out to run without a resume-volume claim
`--resume-volume`		`ai2/olmo-eval`	Native Beaker volume for resumable task state; requires a read-write workspace claim
`--resume-retention-days`		7	Positive whole number of days to retain inactive recovery files
`--dry-run`	`-d`	`false`	Print spec without launching
`--yes`	`-y`	`false`	Skip confirmation prompt
`--follow/--no-follow`		`true`	Follow logs after launch
`--secret-env`		none	Map Beaker secret to env var (`SECRET:VAR`)
`--aws-credentials`		auto	Inject AWS credentials (auto-detected from s3:// paths)
`--gcp-credentials`		auto	Inject GCP credentials (auto-detected from gs:// model paths)
`--store`		`false`	Persist results to configured database

Kind	Example
Number	`-o limit=100`, `-o num_fewshot=5`
Sampling shortcut	`-o temperature=0.6`, `-o max_tokens=512`
Boolean	`-o strip_thinking=true`
List (JSON)	`-o 'dependencies=["pkg1==1.0", "pkg2"]'`
Harness, nested	`--harness default -o provider.max_model_len=16384`

Field	Type	Required	Description
`name`	string	no	Experiment name (generated from models and tasks if omitted)
`models`	list of strings	yes	Model names or presets
`tasks`	list	yes	Task specs (with optional `@priority`)
`cluster`	string	yes	Cluster alias or full name
`workspace`	string	yes	Beaker workspace
`budget`	string	when the workspace has no bound budget	Beaker budget
`gpus`	int	no	GPUs per model instance (auto-detected from the provider if omitted)
`max_gpus_per_node`	int	no	Max GPUs per node, splits tasks if exceeded (default: `8`)
`priority`	string	no	Default priority (default: `normal`)
`min_runtime`	string	no	Minimum runtime before preemption (e.g., `2h`); see Preemption
`preemptible`	bool	no	Deprecated by Beaker; `false` means `min_runtime: 8h`; cannot be combined with `min_runtime`
`timeout`	string	no	Job timeout (default: `24h`)
`retries`	int	no	Retry count on failure
`resume`	bool	no	Recover interrupted task progress (default: `true`); `false` needs no resume-volume claim
`resume_volume`	string	no	Native Beaker volume for resumable task state (default: `ai2/olmo-eval`)
`resume_retention_days`	int	no	Days to retain inactive recovery files (default: 7; must be positive)
`beaker_image`	string	no	Container image to use (config-only)
`groups`	list	no	Beaker groups to add experiments to

Latest commit

History

Folders and files

Repository files navigation

olmo-eval

Overview

Quick Start

Run Your First Eval

Key Concepts

Tasks

Suites

Aggregation

Formatters

Scorers

Metrics

Model Presets

Harness

Using Harness via CLI

HarnessConfig

Scaffolds

Inference Metrics

Auxiliary Providers and Local Judge Models

Defining Tools

Custom Harness Config File

Programmatic Usage

Adding New Tasks

Quick Start: Minimal Task Example

Task Class Overview

TaskConfig Reference

Generation diagnostics

Data Sources

Common Patterns

Adding Variants

Tool-Augmented Evaluation

Function Calling (BFCL)

Multimodal Evaluation

Setup

Running

OmniDocBench formula scoring (CDM)

Providers

Adding a multimodal task

Hill-Climb Dev Suite

Post-Training Dev and Test Suites

Querying Results

Basic Queries

Instance-Level Predictions

Output Formats

Database Configuration

AI2 Users (Recommended)

All Environment Variables

Advanced Usage

Multi-GPU and Tool-Augmented Evaluation

Resuming Interrupted Runs

Debugging and Inspection

Task Inspection (uv run olmo-eval task inspect)

Runtime Inspection Flags

Mock Provider for Testing

External Evals

Defining an External Eval

Running External Evals

ExternalEvalResult

Sandboxes

Configuration

SandboxConfig Fields

Using SandboxManager

Sandbox Modes

Launching on Beaker

Installation

CLI Usage

Advanced Usage

Local Judge Models

Per-Task Priorities

Experiment Groups

Inference Provider Configuration

CLI Options

Preemption

Per-Task Overrides

OpenAI models through OpenRouter

Secret Environment Overrides

YAML Configuration

Task Inspection (`uv run olmo-eval task inspect`)