Priya runs a prompt against the ENGAGE timesheet story and gets eleven test cases. Marcus runs the same prompt, same model, same afternoon, and gets nine — with two she did not have and three of hers missing. Neither of them changed anything.
Description
Two QA engineers sit at one table with a single teal laptop between them. Priya, on the left, holds a printout so long that it spills across the table; Marcus, on the right, holds a single half-page. They look at each other with raised eyebrows and a half-smile — puzzled, not alarmed. The picture shows that one prompt, sent twice to one model, produced two different test suites. It does not show which suite is correct: neither length is evidence of quality, and both need review.
In a test tool, this is a defect. Here it is the specified behaviour, and a tester who does not know that will spend a week filing a bug against a model.
Why the output moves
A language model does not retrieve an answer. It predicts the next token, repeatedly, from a probability distribution — and then samples from that distribution rather than always taking the most likely token. Two runs walk two different paths through the same probabilities, and once they part at one token every later token is predicted from a different history.
Description
A left-to-right pipeline. The prompt is split into tokens, tokens become embedding vectors, the transformer relates every token to every other one, and the result is a probability distribution over the next token. The highlighted step is sampling, where one token is picked from that distribution. The picked token is appended to the output and the whole loop runs again for the next token. The sampling step is where two runs of the same prompt can part ways.
The word plausible is doing the work here. The model produces text that is statistically likely given its training data and your prompt. Likely is not the same as correct, and nothing in the mechanism checks the difference.
Turning the temperature down narrows the distribution but does not make the model a function. Even at temperature 0 providers do not guarantee identical output across calls — batching and floating-point arithmetic on shared hardware can tip a near-tie between two tokens. You will take this apart properly in u15; for now, plan for variation rather than hoping to switch it off.
This has a consequence you will use for the rest of the course: an LLM's output on a test task is evidence, not a result. It is reviewed the way you would review a colleague's first draft — not because the model is junior, but because you cannot tell from the output alone which parts it knew and which parts it constructed.
What the model actually reads
Your prompt does not reach the model as words.
Tokenization splits the text into tokens. For ordinary English prose a
workable rule of thumb is about four characters, or three-quarters of a word,
per token. Identifiers break that rule: login is likely one token,
ENGAGE_TIMESHEET_SUBMIT_BTN is likely six or more. Test artefacts are full of
identifiers, stack traces and GUIDs, so they cost more tokens than their word
count suggests.
Embeddings turn each token into a vector positioned so that tokens used in
similar ways sit near each other. This is why a model can connect "the submit
control" in your prompt with submitButton in your code without being told they
are the same thing — and why it will sometimes connect two things that merely
look similar.
The transformer is the architecture that lets the model weigh how every token relates to every other one across a long passage. It is the reason a model can hold a whole user story in mind rather than reading it a line at a time.
The window, and what happens at its edge
The context window is how much text, measured in tokens, the model can consider at once — your prompt, your input data, and its own answer, all together. In 2026 windows of 128,000 to 1,000,000 tokens are common. That sounds unlimited until you estimate:
- a 400-case regression suite at ~120 words per case is ~48,000 words — roughly 64,000 tokens before identifiers push it higher;
- a full nightly run log of run 2891 is tens of megabytes — millions of tokens. It fits nowhere.
What happens when you go over depends on which layer you are talking to:
- Directly through an API, an oversized request is rejected with a context-length error. Loud, and easy to handle.
- Through a chat tool, IDE assistant or agent, the tool manages the window for you. It may drop the earliest turns, truncate an attachment, summarise it, or retrieve only the chunks it judges relevant — and then answer with exactly the same confidence as if it had read everything.
The second case is the one testers meet, and the dangerous one: nothing in the answer tells you part of the input was never read. Check it the way you would check any unverified claim — ask about something you know sits at the end of the input, or compare the failure count the model reports with the count in the file.
A larger window is not free either. More tokens mean more computation, latency and cost per call — a thread you will pick up in u21 when you estimate what a task costs to run every night. And models attend less reliably to material in the middle of a very long context than to its beginning and end: a critical constraint buried at token 80,000 is weaker than the same constraint at the top.
Where GenAI sits
You will meet four terms, and they are not synonyms. Each is a subset of the one before it in how it learns.
Description
Symbolic AI stands in its own box on the left: it encodes explicit rules, such as routing defects by keyword. To its right, a frame labelled machine learning contains classical ML — learning from data with features a human chose, such as a severity predictor — and a nested frame labelled deep learning, where the model learns its own features. Inside deep learning sit a screenshot classifier and, highlighted, generative AI, which produces new content such as test cases drafted by an LLM. Dashed arrows name the line crossed at each step: from rules to data, from chosen to learned features, from classifying to generating.
- Symbolic AI encodes knowledge as explicit rules. Deterministic, inspectable, brittle.
- Classical machine learning learns from data you have prepared and features you have chosen. Defect triage models are usually this.
- Deep learning learns the features itself from large datasets.
- Generative AI uses deep learning to produce new content — text, code, images — by modelling patterns in what it was trained on.
The reason GenAI arrived in test teams and the others mostly did not: it is pre-trained. There is no dataset to assemble and no training phase before you get value from it on Monday. That convenience is exactly what makes the risk units worth a third of this course.
One more consequence of "generates rather than retrieves": unless a tool has explicitly given the model a document — through retrieval, an attachment or a connector — the model has not looked anything up. A sentence that sounds like it came from your Confluence page was produced from patterns. You will meet the tools that do fetch material in u18 and u19.
What you now know
Non-determinism is designed in, and temperature 0 narrows it without removing it. Tokens are what the model reads and what you pay for, and identifiers are expensive. At the edge of the window an API fails loudly while a tool fails quietly. And "plausible" is the model's actual objective — correctness is yours.