Can Your Model Keep State? YANAMI Is a New Multiturn Multimodal Benchmark
YANAMI is a new multimodal dataset and benchmark for multiturn, stateful interaction. Here is what that tests and why single-prompt evals miss it.
Your model nails a single prompt. Then the user sends a follow-up about the screenshot from earlier, and it loses the thread. Who measures that?
A new Hugging Face daily paper does. It is called YANAMI: “A Large-Scale Multimodal Dataset and Benchmark for Multiturn Stateful Interaction.” The tempo of evaluation keeps shifting. First the test was whether a model could answer. Then whether it could see. Now it is whether it can stay correct while the conversation and the world change around it.
Three dated markers show the shift.
March 2023: OpenAI unveils GPT- with image input in the announcement. Multimodal becomes something products are expected to handle.
November 2024: Anthropic releases the Model Context Protocol. Models get a standard way to reach tools and data that can change between turns.
November 2026: YANAMI appears on Hugging Face’s daily papers. The paper’s ID places it in that month.
The through-line is simple. Inputs got richer, then the environment became mutable, and the yardstick stayed single-turn. A model can get every individual reply right and still fail the session. A “stateful” test is one where the correct answer on a given turn depends on what was shown or changed earlier. A wrong assumption about state does not stay local. It poisons every turn after it. A per-prompt score cannot see that failure. A session-level score can.
Here is what I can and cannot tell you. The source I was given is the paper’s title and listing. It names the dataset as large-scale, multimodal and multiturn, and it names stateful interaction as the target. I have not verified its size, tasks or results, so no numbers appear here. Read the paper before you cite it.
My position: if you ship a multimodal assistant or an agent, your regression suite of isolated prompts is measuring the easy part. Replay whole sessions from your own logs. Grade the end state, not each reply. Expect model rankings on single-prompt evals to reorder once state enters the picture. That is a judgement, and the benchmark’s own results can falsify it.
If you are building agents that carry state across steps, the full agentic SDLC breakdown shows where that state lives in a real pipeline. For the practical version of persistent context, see Claude Code as a daily driver.
Prediction: by the end of June 2027, a team other than YANAMI’s authors will publish results on it. If none does, it is a dataset, not a benchmark.
Want the next one in your inbox? Subscribe here. One AI signal a day. 90 seconds. No fluff.