RealReplicaBench
Nested detail page

Harness 装配链 深度解析

TaskSpec、公开输入、runtime mock 如何从 task.toml 落成容器运行契约。

§ 00 · IMPORT CHAIN

导入链与边界

▸为什么这个模块是入口

Harness 装配链 的关键不是单个函数,而是几组契约一起生效:src/real_replica_bench/cli.py:328、src/real_replica_bench/cli.py:358、src/real_replica_bench/cli.py:415、src/real_replica_bench/cli.py:481。这些行分别描述任务定位、能力声明、评分或 harness 行为,因此改动其中任何一处都会改变 benchmark 的可比较性。

src/real_replica_bench/cli.py · L328-434Python
328def load_task(task_id: str) -> TaskSpec:
329    case_dir = DATASETS_DIR / task_id
330    task_toml = case_dir / "task.toml"
331    if not task_toml.is_file():
332        for candidate in DATASETS_DIR.rglob("task.toml"):
333            cdata = tomllib.loads(candidate.read_text(encoding="utf-8"))
334            if cdata.get("task", {}).get("id") == task_id:
335                case_dir = candidate.parent
336                task_toml = candidate
337                break
338        else:
339            raise SystemExit(f"task not found: {task_id} (searched {DATASETS_DIR})")
340    data = tomllib.loads(task_toml.read_text(encoding="utf-8"))
341    task = data.get("task", {})
342    agent = data.get("agent", {})
343    verifier = data.get("verifier", {})
344    environment = data.get("environment", {})
345    host_port_raw = environment.get("host_published_port")
346    host_published_port: int | None = None
347    if host_port_raw is not None:
348        try:
349            host_published_port = int(host_port_raw)
350        except (TypeError, ValueError):
351            raise SystemExit(
352                f"task {task_id}: [environment].host_published_port must be an integer, got {host_port_raw!r}"
353            )
354        if host_published_port <= 0 or host_published_port > 65535:
355            raise SystemExit(
356                f"task {task_id}: [environment].host_published_port out of range: {host_published_port}"
357            )
358    runtime_mocks = _parse_runtime_mocks(task_id, environment)
359    implicit_host_port = None
360    runtime_mock_ports = [mock.port for mock in runtime_mocks if mock.port is not None]
361    if host_published_port is None and len(runtime_mock_ports) == 1:
362        implicit_host_port = runtime_mock_ports[0]
363    early_terminate: EarlyTerminate | None = None
364    et_raw = environment.get("early_terminate")
365    if et_raw is not None:
366        if not isinstance(et_raw, dict):
367            raise SystemExit(
368                f"task {task_id}: [environment.early_terminate] must be a table"
369            )
370        required = ("poll_path", "match_kind", "match_field", "match_value")
371        for key in required:
372            if key not in et_raw:
373                raise SystemExit(
374                    f"task {task_id}: [environment.early_terminate].{key} is required"
375                )
376        match_kind = str(et_raw["match_kind"])
377        if match_kind != "list_any_field_equals":
378            raise SystemExit(
379                f"task {task_id}: unsupported early_terminate.match_kind {match_kind!r}"
380            )
381        if host_published_port is None and implicit_host_port is None:
382            raise SystemExit(
383                f"task {task_id}: early_terminate requires host_published_port or exactly one runtime mock port"
384            )
385        early_terminate = EarlyTerminate(
386            poll_path=str(et_raw["poll_path"]),
387            poll_interval_sec=float(et_raw.get("poll_interval_sec", 5)),
388            match_kind=match_kind,
389            match_field=str(et_raw["match_field"]),
390            match_value=str(et_raw["match_value"]),
391            kill_pattern=str(et_raw.get("kill_pattern", "bun run runner.ts")),
392        )
393    output_file_early_terminate: OutputFileEarlyTerminate | None = None
394    output_et_raw = environment.get("output_file_early_terminate")
395    if output_et_raw is not None:
396        if not isinstance(output_et_raw, dict):
397            raise SystemExit(
398                f"task {task_id}: [environment.output_file_early_terminate] must be a table"
399            )
400        output_path = str(output_et_raw.get("output_path") or "")
401        if not output_path.startswith("/"):
402            raise SystemExit(
403                f"task {task_id}: [environment.output_file_early_terminate].output_path must be an absolute path"
404            )
405        output_file_early_terminate = OutputFileEarlyTerminate(
406            output_path=output_path,
407            poll_interval_sec=float(output_et_raw.get("poll_interval_sec", 2)),
408            kill_pattern=str(output_et_raw.get("kill_pattern", "bun run runner.ts")),
409        )
410    # Layout-aware default for the verifier entrypoint: the new layout uses
411    # grader/run.sh, the old (frozen v2) layout uses test.sh. The task prompt file
412    # is `task.md` in BOTH layouts. An explicit [task].entrypoint / [task].test in
413    # task.toml always wins over the detected default.
414    test_default = "grader/run.sh" if (case_dir / "grader" / "run.sh").is_file() else "test.sh"
415    return TaskSpec(
416        task_id=str(task.get("id") or task_id),
417        case_dir=case_dir,
418        name=str(task.get("name") or task_id),
419        language=str(task.get("language") or "zh-CN"),
420        timeout_sec=int(agent.get("timeout_sec", 900)),
421        max_actions=int(agent.get("max_actions", 6)),
422        verifier_timeout_sec=int(verifier.get("timeout_sec", 120)),
423        entrypoint=str(task.get("entrypoint") or "task.md"),
424        test_script=str(task.get("test") or test_default),
425        rubric=str(task.get("rubric") or "rubric.json"),
426        private_dir=(case_dir / "private") if (case_dir / "private").is_dir() else None,
427        mock_services=bool(environment.get("mock_services", False) or runtime_mocks),
428        host_published_port=host_published_port,
429        requires_browser=bool(environment.get("requires_browser", False)),
430        requires_vision=bool(environment.get("requires_vision", False)),
431        runtime_mocks=runtime_mocks,
432        early_terminate=early_terminate,
433        output_file_early_terminate=output_file_early_terminate,
434    )
§ 01 · CORE IMPL

核心实现,逐段拆

▸状态与输入如何被收束

第二段源码展示了实现层的防漏设计:src/real_replica_bench/cli.py:500、src/real_replica_bench/cli.py:507、src/real_replica_bench/cli.py:620、src/real_replica_bench/cli.py:632、src/real_replica_bench/cli.py:636。它不是为了让页面好看,而是把任务输入、运行时状态、模型协议和输出证据都保持在可审计范围内,方便后续报告复原。

src/real_replica_bench/cli.py · L481-539Python
481def copy_public_task_dir(src: Path, dst: Path) -> None:
482    """Copy the agent-visible task content, never exposing verifier-only data.
483
484    New layout (allowlist): the agent sees ONLY ``workspace/`` (its inputs) plus
485    the task statement ``task.md``. Everything verifier-only — ``grader/``,
486    ``services/`` (mock source), ``private/``, ``rubric.json``, ``task.toml`` —
487    lives outside ``workspace/`` and is structurally excluded by not copying it.
488    This replaces the old blocklist, where a mock/verifier filename missing from
489    the ignore tuple would leak answers to the agent.
490
491    Old layout (frozen v2, no ``grader/``/``workspace/``): fall back to the
492    historical blocklist over the whole task dir. Removed once v2 migrates (Phase 2).
493    """
494    workspace = src / "workspace"
495    # Detect the new layout by grader/ or workspace/ so browse-only tasks (no
496    # agent inputs → no workspace/) still take the allowlist path instead of
497    # falling through to the old blocklist (which would leak grader/ + services/).
498    is_new_layout = (src / "grader").is_dir() or workspace.is_dir()
499    if is_new_layout:
500        dst.mkdir(parents=True, exist_ok=True)
501        if workspace.is_dir():
502            shutil.copytree(
503                workspace,
504                dst / "workspace",
505                ignore=shutil.ignore_patterns(".DS_Store", "__pycache__", "*.pyc"),
506            )
507        task_md = src / "task.md"
508        if task_md.is_file():
509            shutil.copy2(task_md, dst / "task.md")
510        return
511    ignore = shutil.ignore_patterns(
512        ".DS_Store",
513        "fixtures",
514        "attachments",
515        "node_modules",
516        "private",
517        "__pycache__",
518        "*.pyc",
519        "rubric.json",
520        "test.sh",
521        "task.toml",
522        "source_metadata.json",
523        "REFERENCE_SOLUTION.py",
524        "_grader.py",
525        "host_deps.txt",
526        "package.json",
527        "package-lock.json",
528        "start_mock_services.sh",
529        "mock*.py",
530        "mock*.js",
531        "*server.py",
532        "*server.js",
533        "*.mjs",
534        "verify*.py",
535        "INPUTS_TO_BUILD.md",
536        "ASSET_INVENTORY.md",
537        "EXPECTED_BEHAVIOR.md",
538    )
539    shutil.copytree(src, dst, ignore=ignore)

▸运行时细节如何暴露给调试

第三段代码补上执行末端:src/real_replica_bench/cli.py:328 与 src/real_replica_bench/cli.py:636 附近的实现让失败不只表现为 pass/fail,还能回到日志、配置、工具调用和标准输出定位原因。

src/real_replica_bench/cli.py · L620-636Python
620def copy_runtime_mock_sources(runtime_mocks: tuple[RuntimeMockConfig, ...], dst: Path) -> None:
621    if not runtime_mocks:
622        return
623    copyable = [mock for mock in runtime_mocks if not _runtime_mock_uses_baked_source(mock)]
624    if not copyable:
625        return
626    dst.mkdir(parents=True, exist_ok=True)
627    for mock in copyable:
628        target = dst / mock.name
629        if target.exists():
630            shutil.rmtree(target)
631        shutil.copytree(mock.source_dir, target, ignore=_runtime_mock_ignore(mock))
632    if any(mock.kind == "cli" for mock in copyable):
633        bridge_src = PROJECT_ROOT / "real_replica_bench" / "mock_services" / "bench_bridge.py"
634        if not bridge_src.is_file():
635            raise SystemExit(f"CLI runtime mocks require missing bench bridge: {bridge_src}")
636        shutil.copy2(bridge_src, dst / "bench_bridge.py")
§ 02 · KEY FLOW

关键流程

task config → contract object → runtime materialization → agent execution → verifier/report

从这些源码行可以看到,RealReplicaBench 的流程不是把 Agent 输出丢给一个脚本评分,而是先构造可复现任务世界,再让 harness 运行,最后把 verifier、trajectory 和 report 合并成证据链。

读这条链路时还要把入口和物化点连起来看:src/real_replica_bench/cli.py:342 负责把磁盘任务转成对象,src/real_replica_bench/cli.py:371 读取 harness 声明,src/real_replica_bench/cli.py:428 固定 provider 参数,src/real_replica_bench/cli.py:508 过滤私有资源,src/real_replica_bench/cli.py:522 保留公开 fixture,src/real_replica_bench/cli.py:631 再把 runtime mock 写进副本目录。

§ 03 · EDGE CASES

边界 / 失败模式

▸失败在哪里会被放大

  • 如果任务声明不完整,src/real_replica_bench/cli.py:358 附近的解析会提前中断,而不是让 Agent 在缺环境时跑出伪结果。
  • 如果 mock 或 provider 配置漂移,src/real_replica_bench/cli.py:415 附近的检查会让问题在发布前暴露。
  • 如果 Agent 只完成部分业务动作,最终分数会被 verifier 端硬门禁压成失败,需要回看 trajectory。
§ 04 · RELATED

关联模块

相关子页
Generated · 2026-08-07Source · Accio-org/RealReplicaBench