latte/bench

Abstract

LATTE-Bench evaluates whether a language model can construct and revise an executable plan for a continuous control task. A model specifies 33 bounded parameters for a simulated latte pour. A fixed executor runs the complete plan, after which an evaluator returns geometric scores, serving constraints and a spatial foam map. The next attempt can receive a replacement note written by the model itself. This separates initial planning ability from the use of accumulated feedback while keeping the action representation and environment schedule comparable across controllers. The benchmark addresses a practical evaluation problem: a model can explain a task convincingly yet fail to select effective actions, diagnose their consequences or retain useful evidence. LATTE-Bench makes these failures observable through numerical plans, deterministic execution and independently computed outcomes. Its principal value is as a compact experimental instrument for studying planning under incomplete information and adaptation through bounded textual memory. The current heart study covers 12 completed model routes, three matched milk schedules and five attempts per memory condition. It contains 330 valid model pours and 30 rejected plan slots. Notes change performance in both directions. One route produces all seven numerical passes, but its mean best score falls with notes because performance deteriorates on one seed. These findings motivate a multidimensional evaluation of quality, reliability, interface compliance and cost. They do not establish held-out transfer, model-weight learning or real-world pouring skill. [1], [2]

Scope and principal contribution

This paper defines the task and scorer, places the design in the agent-evaluation literature, analyses the released study and proposes a protocol for stronger causal and transfer claims. Existing behaviour and results are identified separately from proposed extensions. The central contribution is the combination of a shared continuous plan space, a restricted feedback channel and replayable evidence in a task whose visible outcome can also be inspected by a person. Repository and study identities are recorded in Section 12. Results are route-specific observations under the recorded inference settings. [1], [2], [3], [4]

1. The evaluation problem

Agent quality depends on the relationship between a proposed action and its consequences. A benchmark based only on final text can miss errors in units, action ordering, resource use and recovery from failure. Interactive benchmarks address parts of this problem, but a score can still combine several effects: the base model, the surrounding controller, repeated sampling, memory, tool access and the evaluation budget. A useful experiment must expose these components clearly enough to attribute a difference in performance.

LATTE-Bench concentrates on a small, inspectable action space. The model must translate a spatial target into continuous controls, commit to a complete trajectory, and use delayed feedback to choose another trajectory. The objective has competing requirements. More foam can increase target coverage while also filling the notch, exceeding the cup volume or spilling. A visually plausible trajectory is insufficient unless its simulated consequences meet the serving constraints. [3], [4]

Relationship to prior work

Research line	Established focus	latte/bench evaluation question
AgentBench [7]	Decisions across eight interactive environments.	Behaviour in a tightly specified continuous plan space.
ReAct [8]	Interleaving reasoning, actions and observations.	Planning a whole pour without observations during execution.
Reflexion [9]	Linguistic feedback and episodic text.	Retaining a replacement note versus discarding it.
Self-Refine [10]	Iterative generation, feedback and revision.	Revision using externally measured outcomes.
Voyager [11]	An expanding executable skill library in Minecraft.	Retention in a small text state.
PHYRE [12]	Physical reasoning and generalization.	Use of structured spatial feedback.

These connections locate LATTE-Bench within existing research; they do not establish that the task or memory mechanism is the first of its kind. Its useful distinction is experimental control. The same 33 coordinates support language models and numerical search, while the saved-note condition changes which past evidence the language model can access. The resulting comparison is narrower and easier to inspect than an unrestricted agent stack.

2. Task definition and control representation

Let T denote a target mask and its serving requirements, xₜ the revealed milk volume and starting temperature, and zₜ the hidden environmental coefficients on attempt t. The controller produces a plan pₜ in [0,1]³³ from T, xₜ and its permitted memory mₜ. A fixed compiler C converts that vector to actions, and a simulator F produces the final state and trace. The evaluator E then returns a quality score and a pass decision. [3], [4]

This is a partially observed episodic planning problem. Within one episode, the model makes a single open-loop commitment. Feedback closes the loop only between episodes. The model does not issue commands at 60 Hz, react to a live image or correct a pour in progress. Simulation speed and model response latency therefore affect runtime differently: latency changes evaluation throughput, but not the physical trajectory.

Coordinates	Decoded control	Range
0	Nominal cup volume after initial fill	100–200 ml
1	Initial fill height	100–220 mm
2	Initial fill tilt	45–85°
3–32	Six segments: x and z endpoints	Each axis −0.8 to 0.8 cup radii
Per segment	Height and tilt at endpoint	10–180 mm; 0–85°
Per segment	Movement duration	0.1–5 seconds

Each submitted value is decoded linearly between its physical bounds. The compiler supplies preparation, initial filling, settling and serving. The volume coordinate is a nominal pre-drawing cup volume, including 36 ml of espresso; it is not the final volume. The model controls six subsequent segments, each with five values. Position, height and tilt interpolate continuously at a fixed 1/60-second timestep. Segment durations are rounded to ticks. Tilt at or below 25 degrees stops flow. [3]

What the representation contributes

A common plan representation removes the advantage of one controller making more mid-pour decisions than another. It also makes every submitted action inspectable in physical units. However, six linear segments impose a ceiling on expressiveness. A controller may understand a desired shape yet be unable to express a suitable trajectory with this basis. Changing the segment count or timestep changes the benchmark configuration and requires renewed calibration. Task performance therefore measures the model together with this representation and compiler.

3. Environment and memory conditions

The simulator couples ballistic droplets to a stylized two-dimensional foam surface. Its canonical state and scoring grid contain 32 × 32 cells. Incoming foam is bounded by available surface capacity, with overflow spreading to nearby available cells. This provides an executable mapping from motion to a spatial pattern, but it is not a validated model of milk rheology, microfoam drainage or surface tension. Domain randomization explores the assumptions of this simulator. [1], [3]

Observed variables and hidden variation

The planning payload reveals starting milk volume and temperature, sampled from 205–245 ml and 14–22 °C. Hidden variation includes flow, spread, foam retention, mixing, damping, texture, initial pitcher offset and surface motion. For example, flow multiplier ranges from 0.85 to 1.15 and spread multiplier from 0.8 to 1.2. Each seed and attempt index generates a deterministic batch, so matched controllers face the same physical conditions at the same attempt. The target seed is separately fixed at 42 in the current study. [3], [5]

Hidden conditions change between attempts. A previous failure can inform a robust strategy or a hypothesis about the simulator, but it does not reveal the next batch’s latent coefficients. This matters for interpretation: an improvement after a parameter edit can arise from the edit, the changed milk or their interaction. The task supports studying adaptation under variation; it does not by itself identify successful online estimation of a persistent physical parameter.

Condition	Retained information	Current study
No notes	No earlier plans or feedback; reflections discarded.	Evaluated
Notes	Latest replacement note; no raw history.	Evaluated
Raw history	Earlier plans, traces and feedback.	Implemented; not evaluated here
CMA-ES	Optimizer state; one evaluation per candidate.	Cached five-evaluation comparison

Table 1. Best-of-five shape quality on a 0–100 scale; mean ± sample SD over three seeds, not confidence intervals. Each model-arm pass count has denominator 15. CMA-ES has 15 evaluations total and no notes arm. Rejected slots count as zero in aggregate curves but have no physical pour score. [2], [6]

Both language-model arms make a reflection call after each attempt. That call receives the plan, score breakdown, compact trace information and final foam map encoded as density digits from 0 to 9. The general schema allows a note up to 4,000 characters; the frozen study prompt requests at most 1,600 characters. Invalid notes leave the previous valid note in place. This distinction between a schema limit and a prompt instruction should be preserved in reproductions. [3], [5]

The client uses fresh requests and checks context isolation. It does not independently observe gateway-side memory, routing or model changes. The intervention is therefore the retained client-side note under the recorded route, rather than a verified reset of every possible upstream state. [2], [5]

4. Scoring and calibration

LATTE-Bench separates graded shape quality from a conjunction of success requirements. Let A be the set of cells with final foam density at least 0.25 and T the target foreground. Mask intersection over union is |A ∩ T| / |A ∪ T|; foam precision is |A ∩ T| / |A|. A four-connected outline defines the boundary. Boundary precision and recall count matches within one Euclidean grid cell, and their harmonic mean gives boundary F1, denoted B. [4]

Let H(T) be the rasterized convex hull of target cell centres and R = H(T) ∖ T its concave region. Let c be the mean raw foam density in R, with zero assigned when R is empty. The evaluated scorer computes quality as follows:

The minimum makes overlap and contour jointly limiting. The concavity penalty adds sensitivity to foam in negative space that overlap alone can underweight. It is continuous in raw density, even though the foreground mask and boundary remain thresholded. Quality is rounded to one decimal for display; pass checks use the underlying unrounded metrics. A displayed score above the quality floor is therefore insufficient to infer a pass.

Requirement	Heart threshold
Penalized shape quality	≥78.26 / 100
Mask IoU	≥78.26%
Boundary F1	≥81.98%
Foam precision	≥85.34%
Cup volume; temperature	190–240 ml; 55–65 °C
Other checks	Served; spill ≤5 ml; unscorched; time ≤120 s

Reachability and perceptual validity

Calibration supplies a reference plan demonstrating that the enabled task is reachable under nominal conditions. Existing heart thresholds were derived from accepted reference metrics, subject to predefined floors. The evaluated scorer retained the overlap, precision and contour thresholds while replacing the earlier binary concavity veto with a graded penalty; its quality floor equals the existing IoU floor. The reference scores approximately 85.8 and passes. This establishes an achievable instance, not a global optimum or a guarantee under every randomized batch. [1], [4]

The current target has only one concavity cell. A continuous penalty reduces the abruptness of the earlier veto, but does not create a rich representation of notch quality. The repository’s visual review identifies two weak visual matches among seven numerical passes. Scorer calibration and perceptual validation are consequently separate requirements. This is analogous to the broader proxy-objective problem studied by Gao et al.; their experiments do not directly validate this scorer. [2], [15]

5. Technical value of the design

Planning can be inspected independently of prose

A completed attempt contains both the normalized vector and the decoded schedule. This exposes mistakes that would be hidden in a verbal answer: a low pour held too long, a trajectory that deposits excess foam in the notch, or a high fill that leaves insufficient volume for drawing. The model’s explanation can be checked against the submitted controls and the actual trace. Effective pitcher motion remains the authority when requested settings and executed motion differ. [3], [6]

Memory is a bounded state with measurable consequences

The notes condition forces the controller to choose what survives. A replacement note may preserve a useful trajectory, summarize a failed action or encode a misleading explanation. It is a lossy compression of experience, and the next plan is an observable consequence of that compression. A shorter note can reduce prompt growth while discarding evidence needed for later decisions. Comparing notes with raw history and structured summaries would make this trade-off measurable; the present study only compares retained notes with discarded notes.

The memory update can be written as mₜ₊₁ = R(mₜ, pₜ, feedbackₜ), where R is the model’s reflection operation. The no-notes arm sets the next memory to empty. Model weights are not updated by the benchmark. The object under evaluation is the coupled planning-and-reflection controller, which may improve or degrade through changes in context. This is closely related to Reflexion, with a particularly constrained action and feedback interface. [9]

Geometric feedback permits localized diagnosis

The evaluator supplies more than a scalar reward. Overlap, boundary fit, precision, serving failures and a spatial density map can distinguish insufficient coverage from excess deposition or a malformed contour. These signals make it possible to test whether a model changes controls consistent with the observed failure. Such a test requires explicit counterfactual execution; reading a plausible note or observing a large parameter change is not enough.

Replay separates action quality from inference variability

Given the same plan, task and conditions, the simulator can replay the resulting trajectory. This supports score recomputation, inspection of suspicious passes and comparisons of alternative plans under identical conditions. It also permits future scorer revisions to be evaluated on archived outcomes without purchasing new model responses, provided the original version and scores remain available. Historical results must retain their original scorer identity. [1], [6]

The action space supports algorithmic controls

Numerical search and language models can submit the same bounded vector. This creates an opportunity to compare useful prior knowledge with adaptation from observations. The comparison is meaningful only when its budgets and information are explicit: a language model receives rich structured feedback, while the optimizer uses a scalar objective. Identical coordinates do not imply identical information or computation. CMA-ES is a suitable continuous-search baseline, but its population update schedule must be respected. [3], [13]

6. The current multi-model study

The frozen heart study uses seeds 42–44, five attempts per seed and two memory conditions. Each seed begins with empty memory. Twelve routes complete the protocol, giving 360 scheduled slots: 330 valid model pours and 30 rejected plans. A cached CMA-ES comparison adds 15 valid pours. Two provider-error episodes are recorded separately and excluded from model rankings. The recorded verification covers 345 physical replays, 799 provider calls and 107 format errors. [2], [5], [6]

Table 1 reports the mean and sample standard deviation of each seed’s best score after five attempts. These are descriptive statistics over three seeds, not confidence intervals. Display names identify the repository’s model routes; they are not independent attestations of model internals. All scores below were cross-checked against the released attempt CSV.

Route	No notes: best ± SD	Notes: best ± SD	Passes: no / notes
GPT-6 Astra	78.7 ± 2.4	74.7 ± 13.4	2 / 5
GPT-6.1 Sol	71.8 ± 4.6	76.9 ± 0.4	0 / 0
Claude Opus 5.5	38.8 ± 10.7	67.0 ± 7.8	0 / 0
Chat Latest	55.7 ± 5.1	64.9 ± 11.3	0 / 0
GPT-5.5	53.5 ± 8.2	55.9 ± 8.2	0 / 0
GPT-6 Sol	58.0 ± 10.2	49.0 ± 18.5	0 / 0
GPT-5.6 Terra	36.2 ± 9.7	44.3 ± 20.9	0 / 0
GPT-6 Luna	38.6 ± 14.9	42.3 ± 12.7	0 / 0
Claude Sonnet 5.5	36.6 ± 18.3	40.5 ± 7.3	0 / 0
GPT-5.6 Luna	51.5 ± 4.3	38.9 ± 6.0	0 / 0
GPT-5.6 Sol	41.0 ± 25.3	35.1 ± 5.9	0 / 0
GPT-5.5 Pro	No valid plan	No valid plan	0 / 0
CMA-ES	32.4 ± 8.2	Not applicable	0 / —

Astra is the only route with numerical passes: 2/15 without notes and 5/15 with notes. Both arms achieve at least one pass on 2/3 seeds. GPT-5.5 Pro exhausts the 4,096-token plan output allowance in all 90 submissions, including repairs, and returns no valid plan in 30 slots. Its outcome measures compatibility with this output budget, not unrestricted task capability. Equal output caps do not imply equal inference compute. [2]

7. What the notes comparison reveals

Explore the notes comparison in the interactive results.

Seven of the eleven routes that produce valid plans have a higher mean best score with notes; four have a lower mean. Opus has the largest observed gain, approximately 28.3 points, while GPT-5.6 Luna falls by approximately 12.7 points. These are absolute score-point differences. They should not be described as percentage gains in general intelligence or as statistically established memory benefits.

The directions differ enough to make a useful engineering point: adding a self-written note is a controller change whose effect must be tested. Retained feedback can preserve a successful plan, but it can also reinforce an inaccurate diagnosis, erase useful detail or induce unnecessary edits. The current experiment does not isolate which mechanism explains a particular result. It measures the combined effect of the note writer, note content and subsequent planner under the recorded budget.

CMA-ES reaches a mean best score of 32.4 in five evaluations. Its configured population is eight, so this budget ends before the first population update. The comparison shows performance of its initial sampling distribution at a very small budget. It cannot support a claim that language-model adaptation outperforms a developed optimizer search. A full comparison should show learning curves over budgets that include multiple updates, with local compute and simulator calls reported. [2], [13]

8. Quality and reliability can disagree

Explore Astra’s attempt scores and switch between seeds in the interactive results.

On seed 42, Astra with notes reaches its first pass on attempt 2 and passes every subsequent attempt. Without notes, the same seed produces a pass on attempt 3 and then declines. This sequence is consistent with useful retention, but it is one observed trajectory. Initial outputs differ despite equivalent first-attempt information, and milk changes between attempts.

Across seeds, notes improve the best result on seed 42 from 81.1 to 85.7, barely change seed 43 from 78.8 to 78.5, and reduce seed 44 from 76.3 to 59.8. The mean of these paired best-score differences is approximately −4.1 points. At the same time, total passes rise from two to five. Most of the extra passes are repeated successes within the already successful seed 42 run; the fraction of seeds ever solved remains 2/3.

Why rising curves do not demonstrate learning

For any sequence of scores, the best-so-far statistic Bₖ = max(Q₁, …, Qₖ) cannot decrease. Independent repeated sampling can therefore generate an upward curve without any retained experience. A notes curve needs a matched no-notes comparison, and even that comparison requires enough independent runs to distinguish a systematic effect from sampling variation. [2]

A mean best score describes what was found within the search budget. Per-attempt pass rate describes observed success across the sequence. Seed success describes whether each run ever found a passing solution. None alone establishes that the final controller can reliably reproduce a successful result on new conditions. Reporting them together prevents a single attractive statistic from hiding a different failure mode.

The repository labels two of Astra’s seven numerical passes as marginal visual matches. This reinforces the need to retain both the exact frozen pass decision and a separate perceptual assessment. Reclassifying results after inspection would change the endpoint; revising the scorer should instead create a new benchmark version. [2], [4]

9. Utility for evaluating models and agents

LATTE-Bench is useful when a team needs to understand how a model behaves inside a constrained action-and-feedback loop. It can complement task-specific evaluations by exposing failures that are easy to inspect and replay. Its current scope supports diagnostic comparisons; deployment decisions still require representative application tasks, operational constraints and broader evidence.

Evaluation goal	Measurement	Decision
Initial planning	First-attempt quality, pass and valid-plan rates	Can the model turn a specification into controls?
Use of feedback	Paired notes/no-notes outcomes	Does retaining feedback improve this controller?
Memory design	Notes, raw history, structured summaries	Which evidence is useful and affordable?
Constraint handling	Separate geometry and serving failures	Where is feasibility sacrificed?
Robustness	Frozen-memory evaluation on disjoint conditions	Does a strategy transfer? Proposed here.
Serving configuration	Output caps, repairs, tokens, time	Is the model usable under inference limits?

Comparing base models and fine-tuned models

A base model and an adapted checkpoint can use the same plan interface and matched seeds. To attribute a change to the checkpoint, hold prompts, compiler, memory mechanism and inference settings fixed, and document any unavoidable differences. Evaluate first attempts with empty memory separately from repeated attempts with notes. Otherwise a stronger reflection procedure or additional test-time computation can be mistaken for a benefit from training.

If benchmark trajectories are used to train a model, evaluation must use held-out targets, environment draws and task families that were not selected using the test results. The present release does not demonstrate a fine-tuning advantage. It supplies an environment in which such a comparison could be conducted under a new protocol.

Studying memory systems

For a memory product, the benchmark provides a concrete test of which evidence reaches the planner and how that evidence changes an executable action. A retrieval system could replace or augment the single-note state, but that would be a distinct condition with its own retrieval budget, provenance and isolation checks. The current saved-note study is not an evaluation of the full Mubit memory or model-adaptation stack.

Cost and accuracy should be evaluated together. The study makes both arms reflect to equalize call opportunities, which is useful for the ablation. A production no-memory controller might omit that call. Published experimental cost should therefore remain separate from projected deployment cost, consistent with the evaluation concerns raised by Kapoor et al. [14]

10. A protocol for stronger adaptation evidence

The following analyses are proposed extensions, not results of the current study. Their purpose is to distinguish the discovery of a good sample from a repeatable improvement in the controller, and to distinguish adaptation within practice from transfer beyond it.

Choose the unit of replication before analysis

An adaptive five-attempt run is one dependent sequence. Its attempts should not be treated as fifteen independent observations when three seeds are used. Increase the number of independent environment schedules and repeat model sampling within each schedule. Preserve pairing between memory conditions, predeclare the primary outcome and compute uncertainty over whole runs. For a larger task suite, report aggregate intervals and performance profiles alongside averages. The statistical concerns follow the evaluation analysis of Agarwal et al.; the exact design here is a recommendation for this benchmark. [16]

Measure the retained memory intervention

For seed s, define Bₛ,ₖ as the best score up to attempt k and define Dₛ = Bₛ,ₖ(notes) − Bₛ,ₖ(no notes). Report the mean paired difference, its distribution and an interval based on independent replicated runs. Also report final-attempt quality, fraction of runs with any pass and constraint-specific failures. First-attempt inputs are equivalent but model draws are independent, so early differences should not be attributed to memory.

Replay counterfactual plans under matched conditions

For a parameter revision from pₜ to pₜ₊₁, execute both on the same held-out condition set Z. Estimate the mean difference Q(pₜ₊₁, z) − Q(pₜ, z), with feasibility checked separately. This directly tests whether the revised plan is better under controlled conditions. Charge these additional simulator evaluations to the analysis budget and prevent their feedback from reaching the measured controller. Such replay controls environmental variation, although it does not isolate which phrase in the note caused the revision.

Separate practice and transfer

Finish practice, freeze the note or other learned state, and evaluate on disjoint milk conditions without updating memory or selecting a new state using test outcomes. Compare against an empty-memory controller under the same test protocol. A frozen note can still condition a newly generated plan on visible milk inputs; a frozen optimizer vector is a different inference policy and should be labelled accordingly. Additional target transforms and new shape families test different levels of generalization. Tulip and rosetta require renewed calibration before they can serve as enabled benchmark tasks.

Test whether information is being used correctly

Add budget-matched random search, a fixed generic plan, a reference feasibility control, raw history and structured summaries. Compare faithful notes with irrelevant or shuffled notes of similar length. A persistent-latent-condition variant would help separate system identification from robust planning across independently changing batches. All such variants require new protocol identities, predefined outcomes and checks that diagnostic information does not leak into planning.

11. Validity requirements and development priorities

Validate the outcome against independent judgments

Create an adversarial shape set that includes rounded ovals, filled notches, disconnected deposits, shifted hearts and overfilled cups. Obtain blinded judgments using a defined rubric and measure agreement, false passes and false failures. Tune any replacement scorer on a development set, then evaluate it on a separate set of shapes. Preserve the original score in archived records. A machine-readable target is valuable, but it does not eliminate the need to validate what the metric represents.

Establish simulator and representation sensitivity

Test timestep convergence and ranking stability across declared resolutions and trajectory bases. These experiments must separate numerical effects from changes in task difficulty: doubling the grid resolution changes the meaning of a one-cell contour tolerance, while adding segments changes what a controller can express. The current six-segment limit and the observed difficulty of separated tulip or rosetta structures are reasons to investigate representational capacity, not evidence that those tasks are impossible. [1], [6]

Expose inference budgets and operational failures

Report plan validity before and after repairs, rejection rate, provider failures and reflection failures separately. Keep rejected slots in fixed-budget controller outcomes while distinguishing them from valid low-scoring pours. Repeat comparisons at multiple output limits to reveal sensitivity to reasoning-token exhaustion. Record returned snapshot identifiers whenever available; a rolling alias can change even when the requested name stays constant. The current routes also differ in supported APIs and inference-effort settings. [2], [5]

Use more than one budget axis

Simulator evaluations, provider tokens, money and wall-clock time answer different questions. Equalizing one does not equalize the others. A useful report presents a quality-cost frontier with success and failure counts, and states which resource is constrained. A numerical optimizer has zero provider API charges, but still consumes local compute. Unknown provider charges should remain unknown or conservatively reserved rather than being reported as zero. [2], [14]

Control exposure and version changes

Public targets, reference plans and scorer code are useful for reproducibility and also create opportunities for task-specific tuning. A future external leaderboard should retain public development instances and reserve independently generated evaluation instances. Freeze the simulator, parameter guide, calibration, scoring and run protocol together. Report any post-start access or budget amendment explicitly. Scores from different scorer versions should not be pooled, even when the task name is unchanged.

The priority is to validate the metric and broaden independent replication before making general model-ranking claims. A larger model list alone will not resolve a narrow target set, an underdeveloped optimizer baseline or unmeasured perceptual disagreement. The benchmark becomes more informative when its failure modes are quantified as carefully as its highest scores.

12. Reproducibility and evidence boundaries

This v1 paper reports experiments collected with benchmark implementation 0.7, plan space pour-plan/2, and evaluator mask-iou-boundary-graded-concavity/6. These implementation identifiers are retained for reproducing the reported results.

This paper describes repository commit 3828bd8c201c272c8eb021608631f2217316ec33, inspected on 23 October 2026. The study records its original freeze at commit 6034172993aa25be1f989e2693b647102e66f437 under tag latte-bench-v0.7-multimodel-freeze. Later access, budget and queue amendments have separate identities in the summary. The inspected core physical source files match the original study’s recorded hashes. [1], [5], [6]

The repository stores per-attempt scores, plan evidence, model identities, request diagnostics and verification metadata. Its verification record reports 345 physical replays and checks of 31 source files. For this paper, the 250 best-score mean and sample-SD entries across 25 controller-condition groups were independently recomputed from the attempt CSV, and all group pass counts matched. This arithmetic check is distinct from the repository’s physical replay verification; no new paid model runs or independent full simulation replay were performed.

Evidence required for an external submission

Record	Evidence
Source and protocol	Commit, simulator, scorer, plan space, calibration
Model and inference	Requested and returned IDs, route, prompts, effort, output limits
Attempt	Plan, schedule, conditions, outcomes, constraints, errors
Memory	Exact context and notes; isolation checks
Cost	Usage, reservations, repairs, unknown charges
Replication	Seeds, repeated draws, holdouts, replays

The saved study reports a conservative cost bound of $116.2894563, comprising $115.7078263 for completed-call estimates and $0.58163 in retained unknown-charge reservations. These are historical token-based estimates from the study ledger, not current pricing or invoices. The initial budget was amended to $500. Provider-access exclusions and the Pro continuation are part of the experiment history and must remain visible. [2], [5], [6]

Conclusion

LATTE-Bench provides a concrete way to examine whether language-model controllers convert feedback into better executable plans. Its bounded controls, spatial outcomes and replayable records make planning errors and memory effects inspectable. The current results demonstrate that retaining notes can change performance substantially, with benefits and regressions depending on the route and seed. They also show why best score, success frequency and reliability across conditions must be reported separately.

Its strongest present use is controlled diagnostic evaluation. Independent metric validation, stronger baselines and frozen-state holdouts would extend that use toward credible adaptation and transfer measurement. Those additions would allow the benchmark to answer a more demanding question: whether experience produces a reproducible improvement that survives beyond the attempts used to acquire it.

Definitions

Condition set (milk seed)

A reproducible sequence of simulated milk conditions. A numeric seed, such as 42, generates the starting volume, temperature, and hidden properties for each attempt. Conditions change between attempts; the same seed gives each comparison the same sequence. The seed is an identifier, not a score or a physical property of milk.

With notes and without notes

The two comparison conditions, also called experimental arms. With notes, the model receives its latest self-written summary of earlier feedback. Without notes, that summary is discarded. Both conditions generate a reflection after each attempt, and neither updates model weights.

Attempt and run

An attempt is one submitted plan and its evaluation. A run contains five attempts under one condition set and one memory condition. Memory starts empty at the beginning of each run.

Model route

The model accessed through a particular provider or gateway with recorded inference settings. A route name identifies the tested configuration, rather than independently verifying the model’s internal implementation.

Best-of-five and best so far

Best-of-five is the highest score within a five-attempt run. The comparison chart averages that best score across three condition sets. Best so far is the highest score seen up to each attempt; it cannot decrease, even when attempts are independent.

Quality score and pass

Shape quality is a score from 0 to 100 based on overlap, boundary fit, and foam in the target’s concave region. A pass additionally requires all shape and serving thresholds, including volume, temperature, spills, and time. A high score alone is not a pass.

Success across condition sets

A condition set is counted as successful if at least one of its five attempts passes. Success in two of three sets does not mean that two of three individual pours passed, or that the final attempt was successful.

Sample standard deviation (SD)

The spread of the three best-of-five scores around their mean. A larger SD means more variation across condition sets. The displayed ± SD is not a confidence interval.

Intersection over union (IoU)

The area shared by the foam mask and target, divided by the area covered by either. It measures how closely the filled regions overlap.

Boundary F1 and foam precision

Boundary F1 combines precision and recall for matching outlines within one grid cell. Foam precision is the fraction of foam-covered cells that lie inside the target. These distinguish contour errors from excess foam outside the intended shape.

Concavity penalty

A deduction for foam filling the negative space in the target, such as the heart’s notch. The scorer uses the mean raw foam density in that region.

Held-out evaluation

Evaluation on new conditions or tasks that were not used to select or update the controller. The current study does not establish this kind of transfer.

References

Repository sources are pinned to the snapshot inspected for v1, 23 October 2026. Research references establish context; they do not independently validate this benchmark.

1. MubitLabs. LATTE-Bench. Repository snapshot and implementation scope. Back to text ↑
2. MubitLabs. Multi-model heart study: protocol, results and amendments. Back to text ↑
3. MubitLabs. Plan compiler, simulator and environment. Back to text ↑
4. MubitLabs. Scoring and calibration. Back to text ↑
5. MubitLabs. Frozen multi-model study protocol. Back to text ↑
6. MubitLabs. Released results and verification. Back to text ↑
7. Liu et al. AgentBench: Evaluating LLMs as Agents. ICLR, 2024. Back to text ↑
8. Yao et al. ReAct. ICLR, 2023. Back to text ↑
9. Shinn et al. Reflexion. NeurIPS, 2023. Back to text ↑
10. Madaan et al. Self-Refine. NeurIPS, 2023. Back to text ↑
11. Wang et al. Voyager. 2023. Back to text ↑
12. Bakhtin et al. PHYRE. NeurIPS, 2019. Back to text ↑
13. Hansen. The CMA Evolution Strategy: A Tutorial. 2016; revised 2023. Back to text ↑
14. Kapoor et al. AI Agents That Matter. 2024. Back to text ↑
15. Gao et al. Scaling Laws for Reward Model Overoptimization. ICML, 2023. Back to text ↑
16. Agarwal et al. Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS, 2021. Back to text ↑