---
translation_of: session-engineering-evaluation-execution
lang: en
title: 'Session Engineering, part 3: keep evaluation and execution apart'
summary: 'When coding with an agent, I often send an instruction such as "Can we go with option A? Is there a better option?" It contains two requests: evaluation and execution. Mixing them causes three problems: the agent has to infer which one I want, the evaluation tends to favor option A, and it is unclear who decides between the options. This post analyzes the three problems and describes our current practice: the sender states the type of request; an agent that receives a mixed instruction evaluates first and then decides whether to act; and important decisions are evaluated by a party that does not carry out the work.'
---

The first two posts described how Tenon AI organizes multiple agent sessions: [Session Engineering: organizing agent sessions is also managing your own context](/blog/session-engineering/) covered the division of work, and [Session Engineering, part 2: one session between me and the rest](/blog/session-engineering-chief-of-staff/) covered how I came to deal with only one session. This post addresses a narrower problem: a single instruction that contains both an evaluation request and an execution request.

## Two requests in one instruction

When coding with an agent, I often send an instruction like this:

> Can we go with option A? Is there a better option?

Sometimes it is split into two messages: "Can we go with option A?" and "Do you have a better option?"

The instruction contains two requests. The first is evaluation: whether option A is suitable and whether a better option exists. The second is execution: if A is acceptable, implement it.

Linguists call such sentences indirect speech acts (Searle, 1975). "Can you pass the salt?" literally asks about ability but functions as a request to pass the salt. "Can we go with option A?" likewise admits two readings: a request for an evaluation, or a directive to use A phrased as a question. The sender knows which is intended; the receiver can only infer it.

## Three consequences of mixing the requests

**1. The agent has to infer which request I mean.** If it infers execution, it implements A directly, acting on a matter I have not yet decided. If it infers a question, it answers without acting, and I have to send another instruction to proceed. The two errors differ in cost: the first may require rework or may not be reversible; the second costs one additional round of interaction.

**2. The evaluation tends to favor A.** Because the instruction names A first, the evaluation is organized around A. Language models already tend to agree with the person asking, a tendency research calls sycophancy (Sharma et al., 2023). When the same agent has to both evaluate A and implement A, it is even less likely to conclude "do not use A; use B". A typical answer is "A works; you could also consider…", followed by implementing A.

**3. The authority to decide is unclear.** The instruction does not say whether I or the agent chooses between A and B, so the agent has to judge on its own whether to stop and ask me.

I usually send such instructions when I have not yet decided which option is better, or am unsure whether my current approach is the best one. Uncertainty is normal. The problem is that an undecided question has been written as an instruction to act.

## State the type of request

The most direct remedy is for the sender to state which type of request is intended:

| Request type | Example instruction | Expected agent behavior |
| --- | --- | --- |
| Evaluation only | "Don't change anything yet. Compare A with the alternatives and recommend one." | List the options, a recommendation and the reasons; change nothing |
| Evaluate, then follow the recommendation | "Compare them and go with what you recommend." | State the recommendation and the reasons first, then act; afterwards, report which option was chosen |
| Execution | "Use A." | Implement A; if A has an obvious problem, point it out first |

![Three request types and how far the agent goes](/blog/assets/session-engineering-evaluation-execution/three-ways.en.svg)

Many agent tools already offer similar modes, such as a plan mode that produces a plan without editing files. We are considering adding these three types to the Yew Harness input box, so that the sender selects one when sending a message.

## How an agent handles a mixed instruction

I will not always think a request through in advance, so the agent side also needs a rule. Ours follows from two Session Engineering principles: conserve the human's attention, and actions that cannot be undone require the human's own approval. In practice, the agent first separates the instruction:

1. Evaluate first: list A and the other workable options, with a recommendation and the reasons.
2. Then decide whether to act:
   - If the action can be undone, falls within the agent's responsibility, costs no money and does not go outside the company, the agent implements the recommendation and afterwards reports what it chose and why.
   - If the action cannot be undone, costs money or goes outside the company, the agent asks me one question after the evaluation: one decision, a few options and its recommendation, so that I can decide quickly.

This is the same problem that human-computer interaction research calls mixed initiative (Horvitz, 1999): when an agent is uncertain about what the user wants, it should weigh the cost of acting against the cost of asking. Where a wrong guess is cheap, it acts; where a wrong guess is costly, it asks first.

![How an agent handles a mixed instruction](/blog/assets/session-engineering-evaluation-execution/mixed-message.en.svg)

## Important decisions are evaluated by a party that does not execute

The rule above does not solve the second problem: an agent evaluating work it is about to carry out itself still tends to favor that work. The established remedy in organizations is separation of duties: the party that evaluates does not execute, and the party that executes does not assess its own work.

Our sessions are already organized this way. The product group critiques the products, each product's development session carries out the work, and the chief of staff reviews and routes the matters that need my decision. The next step is to apply the same division within a single task: at key points, a session that is not doing the work reviews the decision.

This session does not need to comment on every message; otherwise it would again take up my attention. It intervenes at three points only: before a plan is settled, before a decision is put to me, and before an action that cannot be undone. It reviews the facts, not the executor's reasoning, so that it does not simply follow the executor's line of thought.

## What the reviewer can know

For a separate session to judge my decisions, it would seem to need everything I know. That is not possible.

The review does not need everything, however; it needs what has been written down: the code, the records, the decisions and my own words. The more is written down, the more effective the review. For preferences and business context that are not written down, the reviewer asks, one question at a time. This is one of the reasons we require decisions and original wording to be written into files. How closely this practice can approach the ideal therefore depends on how much has been written down.

## Costs

- One additional round of interaction. For small changes that can be undone, this cost is not justified; such changes should simply be made.
- The reviewer can also be wrong. It provides a second opinion, not the final decision; the final decision remains mine.
- Too much review becomes an interruption, so review takes place only at decision points.

## Testing the practice

We are building an evaluation suite for Session Engineering, and one category of tasks consists of exactly this kind of question-shaped instruction. The expected behavior is: give a judgment and a recommendation; if the action can be undone and falls within the agent's responsibility, follow the recommendation and explain why; if it cannot be undone, ask first. In some tasks, option A is clearly worse than an alternative, to test whether the agent simply proceeds with A.

## Current practice

I still send instructions like "Can we use A?". The difference is that a session receiving one now gives an evaluation and a recommendation first, then either acts or asks me, depending on whether the action can be undone. When I have already decided, I state "evaluation only" or "use A" directly.

## References

- John R. Searle. Indirect Speech Acts. In *Syntax and Semantics 3: Speech Acts*, 1975.
- Eric Horvitz. Principles of Mixed-Initiative User Interfaces. *CHI 1999*.
- Mrinank Sharma et al. Towards Understanding Sycophancy in Language Models. [arXiv:2310.13548](https://arxiv.org/abs/2310.13548), 2023.
