Why AI-written tickets get sent back in refinement
AI-written tickets are usually rejected for invented scope rather than for bad writing. A language model completes rather than reports: given thin notes and a request for a well-formed ticket, it produces the acceptance criteria a well-formed ticket normally has, whether or not the source supports them, and the invented lines carry the same confident register as the real ones. The fix is not a stronger model. It is an instruction that gives uncertainty somewhere to go, such as an Open questions section, plus a review that reads the ticket against the source instead of proofreading it on its own.
The item is well written. The title is in the right shape, the description is a proper user story, there are five acceptance criteria and a definition of done. Refinement takes about four minutes to send it back, and the sentence that does it is always some version of: where did this come from?
Nobody in the meeting said anything about rate limiting. The source notes are two lines long. The ticket has five criteria and three of them describe work the team has never discussed. The author did not make them up. A model did, and the author did not notice, because the invented criteria read exactly like the two that were real.
The rejection is almost never about writing quality
Models are good at ticket prose. Given a rough paragraph they will produce a clean user story, consistent tense, criteria in a testable shape, and formatting your team would accept without comment. If refinement were a copy review, AI-written tickets would sail through it.
Refinement is not a copy review. It is the meeting where a team agrees what is in scope, and its real question about every line is: who decided this. A ticket containing three lines nobody can trace fails that question however well those lines are written, and it fails slowly, because the room has to work out which lines are the untraceable ones.
The expensive version of this is not the rejected ticket. It is the plausible one that gets through a tired room, and a sprint later someone has built rate limiting nobody asked for. After two of those, a team starts reading every AI-drafted item with suspicion, including the good ones.
A model completes, it does not report
A language model continues text. Give it something that begins like a ticket and ask for a ticket, and it produces the rest of a ticket. A well-formed ticket has acceptance criteria, so acceptance criteria appear. Nothing in that process consults whether the source supports them, because the source is just earlier text in the same continuation. There is no path by which "the notes do not say" becomes a more likely ending than a plausible criterion, unless you deliberately make it one.
The second half of the mechanism is what makes it costly. Human writers leak uncertainty: they hedge, they shorten, they add a parenthesis saying they are not sure. A model writes the invented criterion in exactly the register it writes the sourced one, because register comes from the document type and not from the evidence. You cannot spot invention by reading for confidence. There is nothing to spot.
The same behavior shows up in a smaller place where it is easy to observe. When Khint drafts a Jira issue, the model is asked for field values by name and never by identifier, because a model asked for a priority id or a custom field key returns plausible-looking identifiers that do not exist in your project. The app resolves each name against the project's real options, and anything that resolves to nothing is left empty and reported under the card rather than sent. That is this article's thesis at the scale of one field: given a slot it cannot fill from evidence, a model fills it from likelihood.
The four shapes invented scope takes
- Criteria with no source line. The common one, and the easy one to catch once you look for it. Five criteria from a source that supports two.
- Numbers nobody specified. Response times, retention periods, page sizes, concurrent user counts. The most dangerous inventions, because a number reads as though it came from somewhere and reviewers assume the author had a reason for it.
- Actors nobody mentioned. An admin role, a second persona, a support flow. The notes described one user doing one thing, and the ticket quietly acquired a second stakeholder with permissions.
- Modality laundering.The source says "we should probably check what happens on a timeout" and the ticket says "the system must handle timeouts gracefully". Every word is traceable and the meaning is not, because the hedge was the information.
The fourth is the one worth training yourself on. The first three are absences you can find by looking. The fourth changes the certainty of content that really is there, so the source line exists, the check passes, and a discussion has become a commitment without anyone deciding to.
What to change in the prompt
The instruction people write is usually some version of: turn these notes into a Jira ticket with a user story and acceptance criteria. That asks for a shape and says nothing about evidence, so the model delivers the shape. The fix works by making the absence of evidence something the model is allowed to output.
Give uncertainty a destination. A model will not emit nothing, but it will happily emit a line under a heading called Open questions, because that is also a valid completion, and most of what would have become an invented criterion moves there instead. Then bind the criteria to the source, preserve the certainty of the original wording, and stop asking for a fixed count: a request for three to five criteria is an instruction to invent whenever the notes support one.
Every acceptance criterion must correspond to something actually
stated in the notes. If a criterion does not, do not write it as a
criterion: put it under "Open questions" instead. Do not add
non-functional requirements, numeric thresholds, roles or edge cases
that the notes do not mention. Preserve the certainty of the source:
if the notes say "probably" or "we should look at", do not write
"must". If the notes support only one criterion, return one.Be honest about what this buys. It does not eliminate the behavior. It relocates most of it under a heading that labels it as a guess, which matters because a labeled guess is a contribution and an unlabeled one is a defect. Expect leakage, particularly on long inputs where the instruction sits far from the material. This is also the kind of paragraph nobody types twice, which is the argument for keeping it as a saved agent you run on the selection rather than in a chat box, and for improving the one prompt instead of accumulating variants, for the reasons in why prompt libraries fail.
Give it context instead of a gap to fill
A good share of invented scope is the model compensating for something the author knew and never wrote down. The notes say "same as the export thing" because everyone in the room knew what that meant. The model does not, so it produces something that fits.
Two moves help. The first is unglamorous: give it the raw source rather than your tidied summary of it. Summarizing before drafting strips out the specifics that would have anchored the draft, and it is the step people add because it feels like helping.
The second is standing context. A Memory session keeps a compacted working state of what you have been doing, structured as an objective, open threads, artifacts, decisions and notes, and an agent that opts in sees it alongside the text you selected. A Knowledge index goes further and searches your own Jira, Confluence, Notion and Linear content, indexed on your machine rather than on our servers, attaching the closest excerpts to the run. Both replace guessing with something that was written down somewhere.
The honest cost runs against the sales pitch. More context does not stop a model from filling gaps. It changes what it fills them with, and it makes the output sound more like your project, which makes a wrong criterion harder to spot rather than easier. Still a good trade, and not a replacement for the review below.
Read the ticket against the source
Proofreading an AI-written ticket cannot catch invented scope, for the reason above: the invented line is frequently the best-written line on the page. The only review that works is a comparison, and it is quick once it is a habit.
Put the two side by side
Raw notes next to the draft. If you have to go looking for the source, you will skip this step by Thursday.
Point each criterion at a line
Take the criteria one at a time and find the words in the source that produced each one. Anything you cannot point at is the finding, and there are usually two or three.
Move the orphans, do not delete them
Send them to an Open questions section or a comment on the issue. They stop being defects the moment they stop claiming to be agreed, and several will turn out to be the right question.
Check the modal verbs
Search the draft for must and shall, and check what the source said. A hedge that became a requirement is a scope decision, and you want to be the one making it.
Check the numbers hardest
Any figure not present in the source is invented until proven otherwise, and numbers are the inventions that reach production.
Two minutes on a ticket, and under a minute once you know the shapes. It also survives a change of tool, because it depends on the source rather than on anything a vendor did.
A tool can help by keeping approval granular rather than wholesale. When Khint proposes a list of actions from what you have on screen, each line gets its own Run button, and the button that runs the whole list is deliberately the secondary one, because approving a list you have not read is what the per-line button exists to prevent. The pattern to look for in any tool is an approval step that is cheap to refuse.
When the invented line is worth keeping
A criterion the notes do not support is not automatically wrong. Often it is the question a careful analyst would have asked: what happens when the token expires, who sees this if the account is suspended. The defect is not that the model produced it. The defect is that the ticket presented it as agreed.
The same sentence is a contribution under Open questions and a liability under Acceptance criteria. That is the whole difference, and it is why the prompt change above is worth more than it looks: it is not suppressing the model's suggestions, it is filing them correctly.
There is a harder version, and it points at us as much as at any tool. If you are rejecting nearly every AI-drafted ticket, the model may be telling you something true about your notes. Two lines cannot become a ready ticket by any method: the missing content has to come from someone who was in the room. A tool that produces a confident, complete ticket from two lines is doing the wrong thing extremely well.
The realistic setup is a drafting agent explicitly allowed to say it does not know, run on the raw notes, and checked against the source before it goes near a sprint. The mechanics of running one on selected text are in turning meeting notes into Jira tickets, splitting extraction from drafting to get a checkpoint between them is what workflows are for, and a second agent whose only job is to judge a ticket against your own bar is the ticket quality check.