Environment Recipes

AgentV combines Promptfoo-compatible eval authoring with AgentV-owned environment recipes for coding-agent testbeds. Use prompts, tests, vars, default_test, assert, targets, top-level env, and extensions for the Promptfoo-shaped eval matrix. Use environment for the host or Docker state a coding agent should inspect or modify.

environment is an AgentV extension because Promptfoo does not define a typed primitive for repo materialization, Docker images, setup commands, fixture generation, services, and working directories. The recipe can be inline or loaded from a field-local file reference:

environment: file://.agentv/environments/local-python.yaml

.agentv/environments/local-python.yaml

type: host
workdir: ./workspaces/example
setup:
  command: ["bash", "-lc", "bun install && bun run build"]
  cwd: "."
  timeout_ms: 120000

For containerized testbeds, use a Docker environment recipe:

environment:
  type: docker
  context: ./environment
  dockerfile: Dockerfile
  workdir: /app

Vocabulary

Field	Meaning
environment	The AgentV testbed recipe: host or Docker placement, cwd, setup command, fixtures, services, and provenance.
environment.type	Current recipe discriminator. Supported values are host and docker.
environment.workdir	The cwd inside the prepared testbed. Host paths are local paths; Docker paths are container paths.
environment.setup.command	A non-empty argv array. Use ["bash", "-lc", "..."] when shell behavior is required.
Top-level env	Promptfoo-compatible provider/eval variables and template inputs, such as OPENAI_API_KEY: "{{ env.OPENAI_API_KEY }}".
environment.env	Recipe-scoped process environment for the host or container testbed when a recipe needs it. It is distinct from top-level env.
extensions	Promptfoo-style lifecycle callbacks and instrumentation. They can customize flow, but they are not the canonical testbed materialization contract.

Targets Stay Separate

Targets select the system under test. The environment prepares the world those targets run against. This separation lets Codex, Claude, Copilot, and Pi see the same repo state, fixtures, cwd, and setup output.

environment: file://.agentv/environments/local-python.yaml

targets:
  - id: codex-host
    provider: codex-cli
    runtime: host
  - id: claude-host
    provider: claude-cli
    runtime: host
  - id: copilot-host
    provider: copilot-cli
    runtime: host
  - id: pi-host
    provider: pi-cli
    runtime: host

prompts:
  - "{{ task }}"

tests:
  - id: fix-parser
    vars:
      task: Fix the parser regression in ./src/parser.ts.
    assert:
      - type: script
        command: ["bash", "-lc", "bun test"]

Target runtime describes how the provider is invoked. It does not replace the authored environment recipe.

Eval Setup Lifecycle

Each run follows the same high-level order:

eval start
  |
  v
+-----------------------------+
| 1. Resolve environment      |  Inline object or file:// recipe
+-----------------------------+
  |
  v
+-----------------------------+
| 2. Prepare testbed          |  Host directory or Docker image/container context
+-----------------------------+
  |
  v
+-----------------------------+
| 3. Run environment setup    |  setup.command argv in setup.cwd, with timeout_ms
+-----------------------------+
  |
  v
+-----------------------------+
| 4. beforeAll lifecycle      |  extensions, then target hook
+-----------------------------+
  |
  v
+-----------------------------+
| 5. Test loop                |  beforeEach -> target -> grading -> afterEach
+-----------------------------+
  |
  v
+-----------------------------+
| 6. afterAll / cleanup       |  target hook, extensions, cleanup
+-----------------------------+

Use environment.setup for installs, builds, repo checkout scripts, fixture generation, and other authored testbed setup that should be visible in config and provenance. Use extensions for lifecycle callbacks after the testbed is prepared, such as staging agent rules or applying per-case patches from metadata.

Repo Provenance Vs Acquisition

When a benchmark needs a real repository checkout, make that operational in the environment recipe. Keep source pins in setup argv or in the setup script’s own inputs, and duplicate them in tests[].metadata only for audit and slicing.

environment:
  type: host
  workdir: ./workspaces/widget
  setup:
    command:
      - bash
      - ./scripts/materialize-repo.sh
      - ./workspaces/widget
      - https://github.com/example/widget.git
      - 4f3e2d19b6e4e8f1c2b7d9a0e5a6b7c8d9e0f123
    cwd: "."
    timeout_ms: 120000

tests:
  - id: widget-parser
    vars:
      task: Fix the widget parser regression in the prepared checkout.
    metadata:
      repo_url: https://github.com/example/widget.git
      source_commit: 4f3e2d19b6e4e8f1c2b7d9a0e5a6b7c8d9e0f123

metadata.source_commit is audit data. A SHA in metadata or prompt prose does not give the agent an actual checkout; the environment setup must materialize the checkout.

Existing Local Directories

--workspace-path points AgentV at an existing local mutable directory for a one-off run. It is useful for local debugging, but it is machine-local state, not a portable authored testbed recipe.

agentv eval evals/my-eval.yaml --workspace-path /path/to/workdir

Do not commit local absolute paths in eval YAML. Use an environment file for portable setup and keep machine-local overrides in local config.