Harness 装配链 深度解析
TaskSpec、公开输入、runtime mock 如何从 task.toml 落成容器运行契约。
导入链与边界
▸为什么这个模块是入口
Harness 装配链 的关键不是单个函数,而是几组契约一起生效:src/real_replica_bench/cli.py:328、src/real_replica_bench/cli.py:358、src/real_replica_bench/cli.py:415、src/real_replica_bench/cli.py:481。这些行分别描述任务定位、能力声明、评分或 harness 行为,因此改动其中任何一处都会改变 benchmark 的可比较性。
328def load_task(task_id: str) -> TaskSpec: 329 case_dir = DATASETS_DIR / task_id 330 task_toml = case_dir / "task.toml" 331 if not task_toml.is_file(): 332 for candidate in DATASETS_DIR.rglob("task.toml"): 333 cdata = tomllib.loads(candidate.read_text(encoding="utf-8")) 334 if cdata.get("task", {}).get("id") == task_id: 335 case_dir = candidate.parent 336 task_toml = candidate 337 break 338 else: 339 raise SystemExit(f"task not found: {task_id} (searched {DATASETS_DIR})") 340 data = tomllib.loads(task_toml.read_text(encoding="utf-8")) 341 task = data.get("task", {}) 342 agent = data.get("agent", {}) 343 verifier = data.get("verifier", {}) 344 environment = data.get("environment", {}) 345 host_port_raw = environment.get("host_published_port") 346 host_published_port: int | None = None 347 if host_port_raw is not None: 348 try: 349 host_published_port = int(host_port_raw) 350 except (TypeError, ValueError): 351 raise SystemExit( 352 f"task {task_id}: [environment].host_published_port must be an integer, got {host_port_raw!r}" 353 ) 354 if host_published_port <= 0 or host_published_port > 65535: 355 raise SystemExit( 356 f"task {task_id}: [environment].host_published_port out of range: {host_published_port}" 357 ) 358 runtime_mocks = _parse_runtime_mocks(task_id, environment) 359 implicit_host_port = None 360 runtime_mock_ports = [mock.port for mock in runtime_mocks if mock.port is not None] 361 if host_published_port is None and len(runtime_mock_ports) == 1: 362 implicit_host_port = runtime_mock_ports[0] 363 early_terminate: EarlyTerminate | None = None 364 et_raw = environment.get("early_terminate") 365 if et_raw is not None: 366 if not isinstance(et_raw, dict): 367 raise SystemExit( 368 f"task {task_id}: [environment.early_terminate] must be a table" 369 ) 370 required = ("poll_path", "match_kind", "match_field", "match_value") 371 for key in required: 372 if key not in et_raw: 373 raise SystemExit( 374 f"task {task_id}: [environment.early_terminate].{key} is required" 375 ) 376 match_kind = str(et_raw["match_kind"]) 377 if match_kind != "list_any_field_equals": 378 raise SystemExit( 379 f"task {task_id}: unsupported early_terminate.match_kind {match_kind!r}" 380 ) 381 if host_published_port is None and implicit_host_port is None: 382 raise SystemExit( 383 f"task {task_id}: early_terminate requires host_published_port or exactly one runtime mock port" 384 ) 385 early_terminate = EarlyTerminate( 386 poll_path=str(et_raw["poll_path"]), 387 poll_interval_sec=float(et_raw.get("poll_interval_sec", 5)), 388 match_kind=match_kind, 389 match_field=str(et_raw["match_field"]), 390 match_value=str(et_raw["match_value"]), 391 kill_pattern=str(et_raw.get("kill_pattern", "bun run runner.ts")), 392 ) 393 output_file_early_terminate: OutputFileEarlyTerminate | None = None 394 output_et_raw = environment.get("output_file_early_terminate") 395 if output_et_raw is not None: 396 if not isinstance(output_et_raw, dict): 397 raise SystemExit( 398 f"task {task_id}: [environment.output_file_early_terminate] must be a table" 399 ) 400 output_path = str(output_et_raw.get("output_path") or "") 401 if not output_path.startswith("/"): 402 raise SystemExit( 403 f"task {task_id}: [environment.output_file_early_terminate].output_path must be an absolute path" 404 ) 405 output_file_early_terminate = OutputFileEarlyTerminate( 406 output_path=output_path, 407 poll_interval_sec=float(output_et_raw.get("poll_interval_sec", 2)), 408 kill_pattern=str(output_et_raw.get("kill_pattern", "bun run runner.ts")), 409 ) 410 # Layout-aware default for the verifier entrypoint: the new layout uses 411 # grader/run.sh, the old (frozen v2) layout uses test.sh. The task prompt file 412 # is `task.md` in BOTH layouts. An explicit [task].entrypoint / [task].test in 413 # task.toml always wins over the detected default. 414 test_default = "grader/run.sh" if (case_dir / "grader" / "run.sh").is_file() else "test.sh" 415 return TaskSpec( 416 task_id=str(task.get("id") or task_id), 417 case_dir=case_dir, 418 name=str(task.get("name") or task_id), 419 language=str(task.get("language") or "zh-CN"), 420 timeout_sec=int(agent.get("timeout_sec", 900)), 421 max_actions=int(agent.get("max_actions", 6)), 422 verifier_timeout_sec=int(verifier.get("timeout_sec", 120)), 423 entrypoint=str(task.get("entrypoint") or "task.md"), 424 test_script=str(task.get("test") or test_default), 425 rubric=str(task.get("rubric") or "rubric.json"), 426 private_dir=(case_dir / "private") if (case_dir / "private").is_dir() else None, 427 mock_services=bool(environment.get("mock_services", False) or runtime_mocks), 428 host_published_port=host_published_port, 429 requires_browser=bool(environment.get("requires_browser", False)), 430 requires_vision=bool(environment.get("requires_vision", False)), 431 runtime_mocks=runtime_mocks, 432 early_terminate=early_terminate, 433 output_file_early_terminate=output_file_early_terminate, 434 )
核心实现,逐段拆
▸状态与输入如何被收束
第二段源码展示了实现层的防漏设计:src/real_replica_bench/cli.py:500、src/real_replica_bench/cli.py:507、src/real_replica_bench/cli.py:620、src/real_replica_bench/cli.py:632、src/real_replica_bench/cli.py:636。它不是为了让页面好看,而是把任务输入、运行时状态、模型协议和输出证据都保持在可审计范围内,方便后续报告复原。
481def copy_public_task_dir(src: Path, dst: Path) -> None: 482 """Copy the agent-visible task content, never exposing verifier-only data. 483 484 New layout (allowlist): the agent sees ONLY ``workspace/`` (its inputs) plus 485 the task statement ``task.md``. Everything verifier-only — ``grader/``, 486 ``services/`` (mock source), ``private/``, ``rubric.json``, ``task.toml`` — 487 lives outside ``workspace/`` and is structurally excluded by not copying it. 488 This replaces the old blocklist, where a mock/verifier filename missing from 489 the ignore tuple would leak answers to the agent. 490 491 Old layout (frozen v2, no ``grader/``/``workspace/``): fall back to the 492 historical blocklist over the whole task dir. Removed once v2 migrates (Phase 2). 493 """ 494 workspace = src / "workspace" 495 # Detect the new layout by grader/ or workspace/ so browse-only tasks (no 496 # agent inputs → no workspace/) still take the allowlist path instead of 497 # falling through to the old blocklist (which would leak grader/ + services/). 498 is_new_layout = (src / "grader").is_dir() or workspace.is_dir() 499 if is_new_layout: 500 dst.mkdir(parents=True, exist_ok=True) 501 if workspace.is_dir(): 502 shutil.copytree( 503 workspace, 504 dst / "workspace", 505 ignore=shutil.ignore_patterns(".DS_Store", "__pycache__", "*.pyc"), 506 ) 507 task_md = src / "task.md" 508 if task_md.is_file(): 509 shutil.copy2(task_md, dst / "task.md") 510 return 511 ignore = shutil.ignore_patterns( 512 ".DS_Store", 513 "fixtures", 514 "attachments", 515 "node_modules", 516 "private", 517 "__pycache__", 518 "*.pyc", 519 "rubric.json", 520 "test.sh", 521 "task.toml", 522 "source_metadata.json", 523 "REFERENCE_SOLUTION.py", 524 "_grader.py", 525 "host_deps.txt", 526 "package.json", 527 "package-lock.json", 528 "start_mock_services.sh", 529 "mock*.py", 530 "mock*.js", 531 "*server.py", 532 "*server.js", 533 "*.mjs", 534 "verify*.py", 535 "INPUTS_TO_BUILD.md", 536 "ASSET_INVENTORY.md", 537 "EXPECTED_BEHAVIOR.md", 538 ) 539 shutil.copytree(src, dst, ignore=ignore)
▸运行时细节如何暴露给调试
第三段代码补上执行末端:src/real_replica_bench/cli.py:328 与 src/real_replica_bench/cli.py:636 附近的实现让失败不只表现为 pass/fail,还能回到日志、配置、工具调用和标准输出定位原因。
620def copy_runtime_mock_sources(runtime_mocks: tuple[RuntimeMockConfig, ...], dst: Path) -> None: 621 if not runtime_mocks: 622 return 623 copyable = [mock for mock in runtime_mocks if not _runtime_mock_uses_baked_source(mock)] 624 if not copyable: 625 return 626 dst.mkdir(parents=True, exist_ok=True) 627 for mock in copyable: 628 target = dst / mock.name 629 if target.exists(): 630 shutil.rmtree(target) 631 shutil.copytree(mock.source_dir, target, ignore=_runtime_mock_ignore(mock)) 632 if any(mock.kind == "cli" for mock in copyable): 633 bridge_src = PROJECT_ROOT / "real_replica_bench" / "mock_services" / "bench_bridge.py" 634 if not bridge_src.is_file(): 635 raise SystemExit(f"CLI runtime mocks require missing bench bridge: {bridge_src}") 636 shutil.copy2(bridge_src, dst / "bench_bridge.py")
关键流程
从这些源码行可以看到,RealReplicaBench 的流程不是把 Agent 输出丢给一个脚本评分,而是先构造可复现任务世界,再让 harness 运行,最后把 verifier、trajectory 和 report 合并成证据链。
读这条链路时还要把入口和物化点连起来看:src/real_replica_bench/cli.py:342 负责把磁盘任务转成对象,src/real_replica_bench/cli.py:371 读取 harness 声明,src/real_replica_bench/cli.py:428 固定 provider 参数,src/real_replica_bench/cli.py:508 过滤私有资源,src/real_replica_bench/cli.py:522 保留公开 fixture,src/real_replica_bench/cli.py:631 再把 runtime mock 写进副本目录。
边界 / 失败模式
▸失败在哪里会被放大
- 如果任务声明不完整,
src/real_replica_bench/cli.py:358附近的解析会提前中断,而不是让 Agent 在缺环境时跑出伪结果。 - 如果 mock 或 provider 配置漂移,
src/real_replica_bench/cli.py:415附近的检查会让问题在发布前暴露。 - 如果 Agent 只完成部分业务动作,最终分数会被 verifier 端硬门禁压成失败,需要回看 trajectory。