What actually happens when you ask AI a question

Most of the engineers and stakeholders I work with use AI every day and couldn't tell you, even roughly, how it produces an answer. I don't blame them. The vendor documentation is either marketing or a research paper, with nothing in between.

So this is the in between. The explanation I give junior devs when they ask, and the one I wish I'd had when I started building things on top of these APIs.

No math. No jargon. Just the mechanism.

Start here: it's autocomplete

An AI model has one job. Given some text, predict the next word. Write that word down. Then repeat the process with the new, slightly longer text. Keep going until the response is finished.

Not reasoning. Not retrieving. Not thinking. Predicting. Everything else you hear about these systems is an elaboration on that one task.

Once you accept this, the behaviors that used to feel mysterious start making sense. Why do hallucinations happen? Why does prompting wording matter so much? Why can't it do reliable math? Why can the same question give you different answers? All of it traces back to this one simple mechanism, run at enormous scale.

A concrete example

Take the question "What is the capital of Canada?" and follow it through the machine.

Your text is converted to numbers

The model doesn't read words. It works with numbers. So the first step is to chop your question into chunks called tokens and look each one up in a fixed dictionary of about hundreds of thousands of entries.

"What"   → 1867
"is"     → 318
"the"    → 262
"capital"→ 3139
"of"     → 286
"Canada" → 4881
"?"      → 30

Your question is now a list of IDs. No semantics attached, just numbers.

Each number becomes a position in a meaningful space

Every token gets mapped to a long vector of a few thousand numbers (the exact size depends on the model, often somewhere between 1,000 and 12,000). You can treat that vector as a coordinate in a very high dimensional space. The useful property of this space is that words with related meanings end up close to each other.

"Paris," "Toronto," and "Vancouver" sit near each other. "Ottawa" sits near "Poland" and "Italy." "Banana" is nowhere near any of them.

This is how the model represents meaning, not through definitions but through position. Two concepts are related if their vectors are close.

For those of us with a data modeling background, it's worth sitting with that for a moment. The model has no schema, no entities, no relationships, no foreign keys. The whole system is designed around geometry.

The words influence each other

Enter the step that made modern AI viable: 'attention.' It's one of those rare cases where the technical name actually fits.

Every word in the question gets to look at every other word and weigh how much each one matters to its own meaning. The word "capital" is ambiguous on its own. It could mean money, a city, or an uppercase letter. When it looks at "Canada" nearby, the link is strong, and the ambiguity resolves toward a city.

This happens in dozens of layers. Each layer refines the picture. By the end, the model has built an internal representation of the question that's much richer than the literal words.

Then every possible next word is scored

Here's the piece most people want to see, but rarely do.

After processing your question, the model produces a score for every word in its 100,000 word vocabulary. Almost all of those scores are effectively zero. A handful are real candidates.

For the first word of the answer, it looks roughly like this:

The      ████████████████████  42%  ← winner
Paris    ███████████████       31%
It       ████                   9%
Canada's ███                    6%
That     ██                     4%
Capital  █                      3%
Bonjour  █                      2%
Cheese   ▏                      1%
... (99,992 other words with near-zero scores)

Highest score wins. The model writes "The" and moves on.

And then the whole process repeats

This is the part that changed how I think about these systems. The model doesn't plan the answer. It picks one word, then runs the entire scoring process again, with that new word now part of the input.

After "The"                → "capital"  (68%)
After "The capital"        → "of"       (88%)
After "The capital of"     → "Canada"   (91%)
After "...of Canada"       → "is"       (82%)
After "...Canada is"       → "Ottawa"   (94%)
After "...is Ottawa"       → "."        (71%)

Final answer: "The capital of Canada is Ottawa."

Look at how "Ottawa" scored 94 percent after "Canada is." If you'd asked for its score immediately after "The," it would have been much lower. Every word shifts the probabilities for what comes next. The context gets rebuilt from scratch at every single step.

For a 100 word reply, this cycle runs 100 times. Each cycle involves billions of calculations. The reason it feels fast is that the entire thing runs on hardware built specifically for this kind of math.

What this means for the people building on top of it

Hallucinations are not a bug in the traditional sense. The model isn't trying to tell the truth and failing. It's picking the statistically most likely next word, and sometimes the most likely chain wanders away from reality. You cannot fix this with a better prompt alone. You need guardrails, retrieval, or validation layers around the model. Treat every generated answer as untrusted input until proven otherwise. That's a familiar pattern for anyone who's done integration work.

Prompt wording matters more than most people assume. Every word in your prompt moves the model through that meaning space. "Summarize this" and "as a senior reviewer, summarize this for an executive audience" don't just differ in instruction. They land the model in different regions of possible responses. This is why prompt engineering is a genuine skill. It's all about trial and error. You can't just guess the perfect prompt on paper; you have to dive in and iterate.

Context windows are the main thing we have to keep an eye on. Because the model rebuilds its understanding every step from the full current text, everything you put in the prompt is competing for space and influence. Long prompts with irrelevant content produce worse outputs, not better ones. When I design systems around these models now, I treat the prompt like a carefully curated cache, not a dumping ground.

Determinism is not guaranteed and shouldn't be designed for. The scoring process has some randomness built in. Even at temperature zero, minor variations can appear. Architect your systems to tolerate that. If you need the same answer every time, cache it.

My honest take

I've been through enough technology cycles to be skeptical of most of them. This one is different, but not for the reasons the hype cycle suggests.

It's different because the underlying mechanism is genuinely useful for a specific kind of work. Suddenly, tasks involving huge volumes of natural language, or translating tech-speak into plain English, are actually doable. For those of us working in complex enterprise ecosystems, where half the job is context switching between code, configuration, and stakeholder conversations, that's a real shift in what one person can accomplish.

This is a classic in AI circles because it perfectly captures the "useful but not autonomous" vibe. These models will happily produce a confident, plausible, wrong answer to a question about your domain. They don't know what your production data looks like, what your compliance constraints are, or what decisions your team made two years ago and regretted. Senior judgment is still the expensive and valuable part. The model gives your ideas more weight, but it's your expertise and intuition that give them direction.

The teams that will do well with this technology are the ones that understand the mechanism well enough to know when to trust the output and when not to. That's the whole reason I wrote this. Not to make anyone an expert. Just to move the conversation past the magic and into the engineering.