I told the model to emit opaque identifiers and nothing else, it did exactly that, and then a few messages later it stopped, on its own, with nothing in the prompt changed. This was on Crystal aOS, a legal compliance platform where the answers carry citations a lawyer has to be able to audit, and I had built the citation rendering specifically to keep the model away from that job. The rule I took from the bug is that you engineer the boundary around a language model, and the model's own input is a surface people forget it includes. Confining the model to opaque output is necessary and not sufficient, because the output stops being opaque the moment your own code writes a nicer version of it back into the conversation.
Why citations could not be the model's job
In a legal product the citations have to be auditable and consistent, and they cannot depend on the model formatting or numbering them correctly. A model that renumbers its sources, invents a reference section, or drifts in how it writes a citation is a liability even when the underlying answer is fine. So the design kept the model out of it.
The model was instructed to emit only an opaque marker wherever it used a source, written inline as {ragieId}, keyed to the document it drew from in the retrieval layer, and it was told explicitly not to number or format anything. Deterministic backend code did the rest. It extracted the markers in order of appearance, assigned sequential numbers, looked up the canonical title and URL from a Document table in Postgres keyed by that id, replaced each marker with a real citation, and appended a references section. The model produced the meaning, and the code produced the citations.
The output came back as input
It worked on the first message. The problem was where the rendered message went next. The processed version, with numbered citations and a references section, was persisted, and the persisted transcript is what got fed back into the model's context on the following turn. So on message two the model read its own previous turn and found formatted citations like [1] Statute Title sitting where it had written raw {ragieId} markers. It did what a model does with a clear example in front of it and imitated the example. The system prompt still said to emit markers, but the most recent turn in the model's context showed formatted citations, and the model followed what it could see over what it had been told.
What I wrote about it at the time was that saving the post-processed output was shaping how the model answered later messages, that it had started copying the rendered format, and that the fix was to push the rendered version to the frontend without saving it to the database so it stayed out of the chat context. The determinism was all in place. The failure was letting the deterministic output rejoin the model's input.
Two representations of one message
The fix was architectural rather than a better prompt. One message now exists in two representations. The version that is persisted and sent to the model keeps the raw {ragieId} markers, so the model only ever sees the format I want it to keep producing. The version the user sees, with resolved titles, sequential numbers, and a references section, is produced at the boundary and streamed to the frontend, and it is never written back into the model's context. Historical conversations are post-processed when they are read and rendered, not stored in resolved form.
Once one message has a model-facing form and a user-facing form, the contamination cannot happen, because the surface the model reads back is guaranteed to hold the format the model is supposed to produce. Nothing about the prompt changed. The change was making sure my own output could not be read as an instruction on the next turn.
The surfaces where determinism leaks
Seen this way, the boundary around a language model is not one thing. It is several surfaces, and each one can leak.
The prompt is the surface everyone engineers. The persisted transcript the model reads back on each turn is the one that bit me, and it is easy to miss because it feels like "the conversation" rather than input you are responsible for. The tokens the model emits are a surface, which is the entire reason the opaque marker exists. And any stateful transformation you apply on the way in or out is a surface, because it has to behave the same way every time or it becomes noise the model has to read around. Confining the model to opaque output only addresses the tokens it emits. It does nothing for the transcript or the transformations, and each of those has to be engineered on its own terms.
The same discipline on a different surface
The transformation surface failed for me as well. Crystal stripped personally identifiable information before anything reached the model, using Microsoft Presidio's reversible anonymizer. It replaced names and other identifiers with fake stand-ins, kept the original content and the mapping stored on our side, and de-anonymized the model's response on the way back.
The anonymizer generated its replacements randomly on each call. So "John Smith" might become "Michael Douglas" in one message and some other invented name in the next. Across a multi-turn conversation the model then saw what looked like two different people where the user had meant one, and it lost the connection between them. The response could not be reliably de-anonymized either, because the mapping it depended on was not stable from one turn to the next.
The fix was to make the transformation deterministic and stable across the conversation. A fixed seed makes a given entity produce the same replacement, and, more importantly, seeding the anonymizer with the mapping already stored for that conversation means each new message extends the existing mapping instead of regenerating it. "John Smith" resolves to the same stand-in on every turn, the coreference survives, and the response de-anonymizes cleanly.
This failure is the reverse of the citation bug. There, my own code leaked a formatted version of the output back into the model's input. Here, a transformation on the input was not held steady from one turn to the next. Both are the same discipline, which is to stabilize what the model reads on whichever surface it reads it from.
The model's input is part of the contract
The thing I keep from both bugs is that the model's input is part of my contract with it, and my own code is a party to that contract whether I treat it as one or not. Anything my code writes back into the conversation is a de facto instruction, so the persisted transcript has to hold exactly the form I want the model to keep producing. Anything my code transforms on the way in has to produce the same result every time for the same input, or the variation reaches the model as signal it was never meant to carry.
The prompt is the surface everyone engineers. The transcript and the transformations are the ones that decide whether the prompt still holds after the first message.
About the author
Graham Morley , Software engineer and technical leader
I have built and delivered production software since 2011, including SOC 2 Type 1 and ISO 27001 certified platforms, DeFi systems that held more than 20 million dollars with zero exploits, and AI products that raised private equity funding. I build, review, and lead engineering across the stack, and I work with AI coding agents every day.
Get in touch