DAG 状态机与调度预算 深度解析
从 PENDING/READY/RUNNING 到 handoff、条件路由、纠正和派发重试,逐段还原 HomeRail 如何用确定性代码约束非确定性 Agent。
导入链与契约
DAGRun 将图结构与运行态分离:sources/manager/src/orchestration/dag-engine.ts:3、sources/manager/src/orchestration/dag-engine.ts:12、sources/manager/src/orchestration/dag-engine.ts:15、sources/manager/src/orchestration/dag-engine.ts:16、sources/manager/src/orchestration/dag-engine.ts:17、sources/manager/src/orchestration/dag-engine.ts:18 保存状态、循环源、已交接节点和依赖集合,sources/manager/src/orchestration/dag-engine.ts:19、sources/manager/src/orchestration/dag-engine.ts:20、sources/manager/src/orchestration/dag-engine.ts:23、sources/manager/src/orchestration/dag-engine.ts:31、sources/manager/src/orchestration/dag-engine.ts:35、sources/manager/src/orchestration/dag-engine.ts:40 则负责输入满足、mailbox、失败端口和条件匹配。Agent 只选择已声明端口,真正的状态迁移仍由引擎执行,因此模型不能靠自然语言跳过依赖。
65function _initialState(graph: DAGGraphData, nodeId: string): NodeState { 66 const deps = _incomingAfterDeps(graph.edges, nodeId); 67 return deps.length === 0 ? "READY" : "PENDING"; 68} 69 70export function createDAGRun(parsedDAG: ParsedDAG, runId: string): DAGRun { 71 const nodeStates = new Map<string, NodeState>(); 72 const afterSatisfied = new Map<string, Set<string>>(); 73 const inputSatisfied = new Map<string, Set<string>>(); 74 const mailboxes = new Map<string, Map<string, unknown[]>>(); 75 76 for (const node of parsedDAG.graph.nodes) { 77 nodeStates.set(node.node_id, _initialState(parsedDAG.graph, node.node_id)); 78 afterSatisfied.set(node.node_id, new Set<string>()); 79 inputSatisfied.set(node.node_id, new Set<string>()); 80 mailboxes.set(node.node_id, new Map<string, unknown[]>()); 81 } 82 83 return { 84 runId, 85 graph: parsedDAG.graph, 86 loopSources: new Set(parsedDAG.loop_sources), 87 nodeStates, 88 handoffedNodes: new Set<string>(), 89 afterSatisfied, 90 inputSatisfied, 91 mailboxes, 92 }; 93}
核心实现,逐段拆
▸handoff 同时完成状态收口、数据路由与下游唤醒
handoff 先在 sources/manager/src/orchestration/dag-engine.ts:234、sources/manager/src/orchestration/dag-engine.ts:240、sources/manager/src/orchestration/dag-engine.ts:244、sources/manager/src/orchestration/dag-engine.ts:249、sources/manager/src/orchestration/dag-engine.ts:251、sources/manager/src/orchestration/dag-engine.ts:252 校验源节点、筛选命中边并判断无下游的失败端口;随后 sources/manager/src/orchestration/dag-engine.ts:253、sources/manager/src/orchestration/dag-engine.ts:262、sources/manager/src/orchestration/dag-engine.ts:264、sources/manager/src/orchestration/dag-engine.ts:265、sources/manager/src/orchestration/dag-engine.ts:266、sources/manager/src/orchestration/dag-engine.ts:271 更新源节点状态、满足 after_dep、向目标端口邮箱推送内容,并逐个尝试晋升。循环源和 loop gateway 还有单独唤醒路径,保证迭代图可以再次进入 READY。
234export function handoff( 235 run: DAGRun, 236 fromNode: string, 237 port: string, 238 content: unknown, 239): DAGTransitionResult { 240 if (!run.nodeStates.has(fromNode)) { 241 throw new Error(`Unknown node: ${fromNode}`); 242 } 243 244 const matchingDownstream = run.graph.edges.filter( 245 (edge) => 246 edge.from_node === fromNode && 247 edge.label !== "after_dep" && 248 edge.to_node !== "" && 249 edgeMatchesHandoff(edge, port), 250 ); 251 const terminalFailure = isFailurePort(port) && matchingDownstream.length === 0; 252 run.handoffedNodes.add(fromNode); 253 run.nodeStates.set( 254 fromNode, 255 terminalFailure 256 ? "FAILED" 257 : run.loopSources.has(fromNode) 258 ? "RUNNING" 259 : "COMPLETED", 260 ); 261 262 const affected = terminalFailure ? new Set<string>() : _satisfyAfterDeps(run, fromNode); 263 const mailboxReceivers = new Set<string>(); 264 for (const edge of matchingDownstream) { 265 _ensureMailbox(run, edge.to_node, edge.to_port).push(content); 266 run.inputSatisfied.get(edge.to_node)?.add(fromNode); 267 affected.add(edge.to_node); 268 mailboxReceivers.add(edge.to_node); 269 } 270 271 for (const nodeId of affected) { 272 _tryPromote(run, nodeId); 273 _skipUntakenSatisfiedBranches(run, nodeId); 274 } 275 for (const nodeId of mailboxReceivers) { 276 _wakeLoopSource(run, nodeId); 277 _wakeLoopGatewayReceiver(run, fromNode, nodeId); 278 } 279 if (terminalFailure) { 280 _skipDependentNodes(run, fromNode); 281 } 282 return { 283 affectedNodes: Array.from(affected).sort(), 284 routedNodes: Array.from(mailboxReceivers).sort(), 285 terminalFailure, 286 };
▸dispatchReadyNodes 把预算、能力选择和瞬时重试放在同一门禁
调度循环从 sources/manager/src/runtime/active-runs.ts:1389、sources/manager/src/runtime/active-runs.ts:1393、sources/manager/src/runtime/active-runs.ts:1398、sources/manager/src/runtime/active-runs.ts:1400、sources/manager/src/runtime/active-runs.ts:1402、sources/manager/src/runtime/active-runs.ts:1407 获取 READY 节点并先处理 gateway;普通节点在 sources/manager/src/runtime/active-runs.ts:1412、sources/manager/src/runtime/active-runs.ts:1416、sources/manager/src/runtime/active-runs.ts:1420、sources/manager/src/runtime/active-runs.ts:1421、sources/manager/src/runtime/active-runs.ts:1423、sources/manager/src/runtime/active-runs.ts:1428 检查总派发预算、调用 dispatcher,并只对 retryable 故障记录一次重试。最终只有 sources/manager/src/runtime/active-runs.ts:1444、sources/manager/src/runtime/active-runs.ts:1445、sources/manager/src/runtime/active-runs.ts:1446、sources/manager/src/runtime/active-runs.ts:1447 收到 dispatched 的节点才进入 RUNNING,避免“已尝试发送”被当成“已开始执行”。
1389export function dispatchReadyNodes( 1390 runId: string, 1391 dispatcher: DAGDispatcher, 1392): number { 1393 const run = store.get(runId); 1394 if (!run) return 0; 1395 1396 let count = 0; 1397 const before = _snapshotNodeStates(run); 1398 const ready = getReadyNodes(run.dagRun); 1399 for (const nodeId of ready) { 1400 const node = run.dagRun.graph.nodes.find((n) => n.node_id === nodeId); 1401 if (!node) continue; 1402 if (_isGatewayNode(node)) { 1403 if (_executeGatewayNode(runId, run, node)) count++; 1404 continue; 1405 } 1406 1407 const built = _buildDispatchEnvelope(run, nodeId); 1408 if (!built.ok) { 1409 failActiveRun(runId, nodeId, built.reason); 1410 continue; 1411 } 1412 if (run.counters.dispatches >= run.limits.max_dispatches) { 1413 abortActiveRun(runId, `max_dispatches (${run.limits.max_dispatches}) exceeded`, nodeId); 1414 break; 1415 } 1416 run.counters.dispatches++; 1417 1418 const envelope = built.envelope; 1419 1420 let result = dispatcher.dispatch(envelope); 1421 if (result.status === "failed") { 1422 const retryable = result.retryable !== false; 1423 if (retryable && recordNodeDispatchRetry(runId, nodeId, result.reason)) { 1424 if (run.counters.dispatches >= run.limits.max_dispatches) { 1425 abortActiveRun(runId, `max_dispatches (${run.limits.max_dispatches}) exceeded`, nodeId); 1426 break; 1427 } 1428 run.counters.dispatches++; 1429 result = dispatcher.dispatch({ 1430 ...envelope, 1431 inputs: { 1432 ...envelope.inputs, 1433 dispatch_retry: [ 1434 `Retrying DAG node ${nodeId} after transient dispatch failure: ${result.reason}`, 1435 ], 1436 }, 1437 }); 1438 } 1439 if (result.status === "failed") { 1440 failActiveRun(runId, nodeId, result.reason); 1441 continue; 1442 } 1443 } 1444 if (result.status !== "dispatched") continue; 1445 startNode(run.dagRun, nodeId); 1446 _markNodeSessionStatus(run, nodeId, "running"); 1447 emit("dag:node_dispatched", { runId, nodeId, agentId: envelope.agentId, sessionId: envelope.sessionId }); 1448 count++; 1449 } 1450 if (count > 0) { 1451 writeRunMetadata(runId, serializeRunMetadata(run)); 1452 _emitNodeStateChanges(run, before); 1453 } 1454 return count; 1455}
关键流程
运行预算不仅是 max_dispatches。sources/manager/src/runtime/active-runs.ts:110、sources/manager/src/runtime/active-runs.ts:111、sources/manager/src/runtime/active-runs.ts:112、sources/manager/src/runtime/active-runs.ts:113、sources/manager/src/runtime/active-runs.ts:114、sources/manager/src/runtime/active-runs.ts:115 同时限制派发、handoff、每节点纠正、边遍历和工具调用;默认值在 sources/manager/src/runtime/active-runs.ts:147、sources/manager/src/runtime/active-runs.ts:148、sources/manager/src/runtime/active-runs.ts:149、sources/manager/src/runtime/active-runs.ts:150、sources/manager/src/runtime/active-runs.ts:151、sources/manager/src/runtime/active-runs.ts:152 集中声明。这样循环网关、错误纠正与短暂网络失败不会无限放大成本。
▸纠正不是盲重跑,而是向 mailbox 注入结构化原因
当 Agent 未调用 handoff 时,sources/manager/src/runtime/active-runs.ts:765、sources/manager/src/runtime/active-runs.ts:775、sources/manager/src/runtime/active-runs.ts:776、sources/manager/src/runtime/active-runs.ts:781、sources/manager/src/runtime/active-runs.ts:783、sources/manager/src/runtime/active-runs.ts:784 检查纠正预算并记录次数,sources/manager/src/runtime/active-runs.ts:785、sources/manager/src/runtime/active-runs.ts:787、sources/manager/src/runtime/active-runs.ts:790、sources/manager/src/runtime/active-runs.ts:791、sources/manager/src/runtime/active-runs.ts:794、sources/manager/src/runtime/active-runs.ts:796 把带原始失败原因的提示写入 correction 端口、恢复 READY 并发出事件。次数耗尽后才走默认成功端口或由上层判定失败,容错过程因此可观察、可设上限。
边界 / 失败模式
- after_dep 已满足,但对应显式条件边未命中:节点转为 SKIPPED。
- 失败端口没有下游:源节点成为 FAILED,并递归跳过只依赖其成功路径的节点。
- 派发返回 retryable:false:不浪费重试预算,立即失败。
- 循环源邮箱只保留最新值,防止多轮积压把旧输入重复消费。