Session Engineering(三):把评估和执行分开
Session Engineering, part 3: keep evaluation and execution apart
与 Agent 协作编写代码时,我常发出这样的指令:「我们可以采用方案 A 吗?有没有更好的方案?」它同时包含两个请求:评估和执行。两者混在一起会带来三个问题:Agent 需要推断我要的是哪一个;评估容易偏向方案 A;由谁决定采用哪个方案也不明确。本文分析这三个问题,并介绍我们目前的做法:由发出指令的一方说明请求类型;Agent 收到混合的指令时先评估,再决定是否执行;重要的决定交给不负责执行的一方评估。
When coding with an agent, I often send an instruction such as "Can we go with option A? Is there a better option?" It contains two requests: evaluation and execution. Mixing them causes three problems: the agent has to infer which one I want, the evaluation tends to favor option A, and it is unclear who decides between the options. This post analyzes the three problems and describes our current practice: the sender states the type of request; an agent that receives a mixed instruction evaluates first and then decides whether to act; and important decisions are evaluated by a party that does not carry out the work.
前两篇介绍了 Tenon AI 组织多个 Agent 会话的方式:《Session Engineering:组织多个 Agent 会话,也是管理自己的上下文》讨论分工,《Session Engineering(二):加一层总办,人只对接一个会话》讨论如何让人只对接一个会话。本文讨论一个范围更小的问题:一条指令同时包含评估请求和执行请求。
一条指令中的两个请求
与 Agent 协作编写代码时,我常发出这样的指令:
我们可以采用方案 A 吗?有没有更好的方案?
有时也拆成两条:「我们可以采用方案 A 吗?」「你有没有更好的方案?」
这条指令包含两个请求。一是评估:方案 A 是否合适,是否存在更好的方案。二是执行:如果 A 可行,就实施 A。
语言学把这类句子称为间接言语行为(Searle, 1975)。「你能把盐递给我吗?」字面上询问对方的能力,实际是请求对方递盐。「我们可以采用方案 A 吗?」同样有两种理解:可能是请求评估,也可能是以询问的形式提出「采用 A」。发出指令的一方知道自己的意图,接收方只能推断。
两种请求混合的三个后果
一、Agent 需要推断我要的是哪一个。 如果理解为执行,它会直接实施 A,即执行了我尚未决定的事项;如果理解为提问,它只作答、不实施,我还需要再发一条指令让它执行。两种误判的代价不同:前者可能导致返工,甚至无法撤回;后者只增加一轮交互。
二、评估容易偏向 A。 指令先提出了 A,评估便围绕 A 展开。语言模型本身有迎合提问者的倾向,相关研究称之为 sycophancy(Sharma et al., 2023)。当同一个 Agent 既要评估 A、又要实施 A 时,更难得出「不采用 A,改用 B」的结论。常见的回答是「A 可行,另外也可以考虑……」,随后按 A 实施。
三、决定权的归属不明确。 采用 A 还是 B,由我决定还是由 Agent 决定,指令中没有说明,Agent 只能自行判断是否应当停下来询问我。
我发出这类指令,往往是因为自己也尚未确定哪个方案更好,或者不确定现有做法是否最好。存在不确定是正常的;问题在于,把一个尚未想清楚的问题写成了一条执行指令。
说明请求类型
最直接的办法,是由发出指令的一方说明请求类型:
| 请求类型 | 指令示例 | Agent 应有的行为 |
|---|---|---|
| 只评估 | 「先别动手,比较 A 和其他方案,给个推荐」 | 列出方案、推荐和理由,不做任何修改 |
| 评估后按推荐执行 | 「比较一下,按你推荐的做」 | 先写明推荐和理由,再执行;完成后说明选择了哪个方案 |
| 直接执行 | 「用 A」 | 执行 A;如果发现 A 有明显问题,先指出 |
不少 Agent 工具已经提供类似的模式,例如只制定计划、不修改文件的计划模式。我们正在考虑把这三种类型直接放进 Yew Harness 的输入框,发送消息时选择其一即可。
Agent 收到混合指令时的处理
我不会每次都先想清楚,因此 Agent 一侧也需要相应的规则。我们的规则来自 Session Engineering 的两条原则:节省人的注意力;不可撤回的动作须由人亲自批准。具体做法是先把指令拆开:
- 先评估:列出 A 和其他可行的方案,给出推荐和理由。
- 再决定是否执行:
- 可以撤回、在其职责范围内、不产生费用且不对外的,按推荐直接执行,完成后说明选择了什么、理由是什么。
- 不可撤回、产生费用或对外的,评估后向我提出一个问题:一件事、几个选项、推荐哪一个,使我能够很快作出决定。
这与人机交互研究中的「混合主动」(mixed initiative)是同一类问题(Horvitz, 1999):Agent 不确定人的意图时,需要比较直接执行与先行询问各自的代价。误判代价小的,直接执行;误判代价大的,先询问。
重要的决定由不负责执行的一方评估
上述规则解决不了第二个问题:同一个 Agent 评估自己将要执行的事项,仍然容易产生偏向。组织中常用的办法是职责分离:评估的一方不负责执行,执行的一方不评价自己。
我们的会话组织已经按这种方式分工:产品组负责产品批判,各产品的开发会话负责执行,总办负责审核和分流需要我决定的事项。下一步是把这种分工引入单个任务:在关键节点,由一个不负责执行的会话专门审查决定。
这个会话不需要对每条消息发表意见,否则会再次占用我的注意力。它只在三个节点介入:计划确定之前,决定提交给我之前,不可撤回的动作执行之前。它依据事实审查,不阅读执行方的推理过程,以免沿着执行方的思路得出结论。
审查方能获得的信息
一个独立的会话要评判我的决定,似乎需要掌握我所知道的全部信息,这无法做到。
但审查并不需要全部信息,只需要已经写下来的部分:代码、记录、决定和我的原话。写下来的越多,审查越有效;对于没有写下来的偏好和业务背景,它通过提问获取,每次只问一个问题。这也是我们要求把决定和原话写进文件的原因之一。因此,这一做法能在多大程度上接近理想状态,取决于写下来的信息有多少。
代价
- 增加一轮交互。可以撤回的小改动不值得付出这个代价,应直接执行。
- 审查方也可能出错。它提供的是第二意见,不是最终决定,最终仍由我决定。
- 审查过多会形成打扰,因此只在决定节点进行。
检验方法
我们正在为 Session Engineering 构建一套评测,其中一类题目就是这种问句形式的指令。正确的做法是:先给出判断和推荐;可以撤回、在职责范围内的,按推荐执行并说明理由;不可撤回的,先询问。部分题目中 A 明显不如其他方案,用来检验 Agent 是否会顺着 A 执行下去。
目前的做法
我仍然会发出「能用 A 吗?」这类指令。不同之处在于:收到这类指令的会话会先给出评估和推荐,再根据动作能否撤回,决定直接执行还是询问我;在我已经想清楚的时候,我会直接说明「只评估」或「用 A」。
参考文献
- John R. Searle. Indirect Speech Acts. In Syntax and Semantics 3: Speech Acts, 1975.
- Eric Horvitz. Principles of Mixed-Initiative User Interfaces. CHI 1999.
- Mrinank Sharma et al. Towards Understanding Sycophancy in Language Models. arXiv:2310.13548, 2023.
The first two posts described how Tenon AI organizes multiple agent sessions: Session Engineering: organizing agent sessions is also managing your own context covered the division of work, and Session Engineering, part 2: one session between me and the rest covered how I came to deal with only one session. This post addresses a narrower problem: a single instruction that contains both an evaluation request and an execution request.
Two requests in one instruction
When coding with an agent, I often send an instruction like this:
Can we go with option A? Is there a better option?
Sometimes it is split into two messages: "Can we go with option A?" and "Do you have a better option?"
The instruction contains two requests. The first is evaluation: whether option A is suitable and whether a better option exists. The second is execution: if A is acceptable, implement it.
Linguists call such sentences indirect speech acts (Searle, 1975). "Can you pass the salt?" literally asks about ability but functions as a request to pass the salt. "Can we go with option A?" likewise admits two readings: a request for an evaluation, or a directive to use A phrased as a question. The sender knows which is intended; the receiver can only infer it.
Three consequences of mixing the requests
1. The agent has to infer which request I mean. If it infers execution, it implements A directly, acting on a matter I have not yet decided. If it infers a question, it answers without acting, and I have to send another instruction to proceed. The two errors differ in cost: the first may require rework or may not be reversible; the second costs one additional round of interaction.
2. The evaluation tends to favor A. Because the instruction names A first, the evaluation is organized around A. Language models already tend to agree with the person asking, a tendency research calls sycophancy (Sharma et al., 2023). When the same agent has to both evaluate A and implement A, it is even less likely to conclude "do not use A; use B". A typical answer is "A works; you could also consider…", followed by implementing A.
3. The authority to decide is unclear. The instruction does not say whether I or the agent chooses between A and B, so the agent has to judge on its own whether to stop and ask me.
I usually send such instructions when I have not yet decided which option is better, or am unsure whether my current approach is the best one. Uncertainty is normal. The problem is that an undecided question has been written as an instruction to act.
State the type of request
The most direct remedy is for the sender to state which type of request is intended:
| Request type | Example instruction | Expected agent behavior |
|---|---|---|
| Evaluation only | "Don't change anything yet. Compare A with the alternatives and recommend one." | List the options, a recommendation and the reasons; change nothing |
| Evaluate, then follow the recommendation | "Compare them and go with what you recommend." | State the recommendation and the reasons first, then act; afterwards, report which option was chosen |
| Execution | "Use A." | Implement A; if A has an obvious problem, point it out first |
Many agent tools already offer similar modes, such as a plan mode that produces a plan without editing files. We are considering adding these three types to the Yew Harness input box, so that the sender selects one when sending a message.
How an agent handles a mixed instruction
I will not always think a request through in advance, so the agent side also needs a rule. Ours follows from two Session Engineering principles: conserve the human's attention, and actions that cannot be undone require the human's own approval. In practice, the agent first separates the instruction:
- Evaluate first: list A and the other workable options, with a recommendation and the reasons.
- Then decide whether to act:
- If the action can be undone, falls within the agent's responsibility, costs no money and does not go outside the company, the agent implements the recommendation and afterwards reports what it chose and why.
- If the action cannot be undone, costs money or goes outside the company, the agent asks me one question after the evaluation: one decision, a few options and its recommendation, so that I can decide quickly.
This is the same problem that human-computer interaction research calls mixed initiative (Horvitz, 1999): when an agent is uncertain about what the user wants, it should weigh the cost of acting against the cost of asking. Where a wrong guess is cheap, it acts; where a wrong guess is costly, it asks first.
Important decisions are evaluated by a party that does not execute
The rule above does not solve the second problem: an agent evaluating work it is about to carry out itself still tends to favor that work. The established remedy in organizations is separation of duties: the party that evaluates does not execute, and the party that executes does not assess its own work.
Our sessions are already organized this way. The product group critiques the products, each product's development session carries out the work, and the chief of staff reviews and routes the matters that need my decision. The next step is to apply the same division within a single task: at key points, a session that is not doing the work reviews the decision.
This session does not need to comment on every message; otherwise it would again take up my attention. It intervenes at three points only: before a plan is settled, before a decision is put to me, and before an action that cannot be undone. It reviews the facts, not the executor's reasoning, so that it does not simply follow the executor's line of thought.
What the reviewer can know
For a separate session to judge my decisions, it would seem to need everything I know. That is not possible.
The review does not need everything, however; it needs what has been written down: the code, the records, the decisions and my own words. The more is written down, the more effective the review. For preferences and business context that are not written down, the reviewer asks, one question at a time. This is one of the reasons we require decisions and original wording to be written into files. How closely this practice can approach the ideal therefore depends on how much has been written down.
Costs
- One additional round of interaction. For small changes that can be undone, this cost is not justified; such changes should simply be made.
- The reviewer can also be wrong. It provides a second opinion, not the final decision; the final decision remains mine.
- Too much review becomes an interruption, so review takes place only at decision points.
Testing the practice
We are building an evaluation suite for Session Engineering, and one category of tasks consists of exactly this kind of question-shaped instruction. The expected behavior is: give a judgment and a recommendation; if the action can be undone and falls within the agent's responsibility, follow the recommendation and explain why; if it cannot be undone, ask first. In some tasks, option A is clearly worse than an alternative, to test whether the agent simply proceeds with A.
Current practice
I still send instructions like "Can we use A?". The difference is that a session receiving one now gives an evaluation and a recommendation first, then either acts or asks me, depending on whether the action can be undone. When I have already decided, I state "evaluation only" or "use A" directly.
References
- John R. Searle. Indirect Speech Acts. In Syntax and Semantics 3: Speech Acts, 1975.
- Eric Horvitz. Principles of Mixed-Initiative User Interfaces. CHI 1999.
- Mrinank Sharma et al. Towards Understanding Sycophancy in Language Models. arXiv:2310.13548, 2023.
