Open source · desktop app

I kept asking how AI works. So I built an instrument to look.

How does a language model choose its next word? When did it learn to understand? Where does it keep an idea? Does it know when it is wrong? Explanations online answer in prose. The Parameter Space Inspector (PSI) answers by opening a real model on your own computer and measuring it. Each section below is one of my questions and what PSI showed.

Get PSI Read the questions ↓
PSI's Generate view: a 4-billion-parameter model's answer to 'Why does a street market set prices that no single seller chooses?', each piece shaded by how torn the model was, with the fork map of alternatives and their odds below
Generate: a 4-billion-parameter model (Qwen3-4B) answering on an ordinary graphics card, with the lens off. Brighter pieces are where it was torn; the fork map lists what else it could have said, with its odds.

Nine questions, asked of a real model

Each answer is a view you can run

PSI runs open models on your machine: Qwen3-4B for answers, and EleutherAI’s Pythia family, which was saved at 154 points during its training. Every answer below is a measurement, with its controls, that you can repeat. Some of them went against what I expected, and those stayed in.

Question 01 · Generate

How does an LLM write an answer?

One piece at a time. At every step the model gives odds for every possible next piece, and one is drawn. PSI shows each step’s odds as a fork map: the road taken and the roads not taken, with how torn the model was. Click an alternative and it writes the other future.

Below the fork map, PSI samples many continuations from one fork and compares them. An early choice closes off whole families of answers, and at low creativity the text settles into ruts it cannot climb out of. The same choice, a few words in, decides where the answer can go.

What it showed: in an answer like this one, most steps are near-certain (90% or more), and a few forks, where the top choice has 60–70%, decide the direction. Those are the words worth looking at.

PSI's Generate view mid-answer, at step 37 of 70: the text so far, each piece shaded by how torn the model was, and the fork map with each step's alternatives and their odds
Mid-answer, step 37 of 70. Each column of the fork map is one step: the piece drawn on top, the roads not taken below with their odds.
Question 02 · Read and Tokens

What does an LLM actually read?

Not letters, and not words. It reads tokens, pieces of text from a fixed dictionary. The Tokens view shows the pieces: the same sentence costs more of them in Hindi than in English, so the model sees some languages through a coarser surface.

The Read view colours any text by how surprised the model is at each piece. Feed it an invented word and watch it learn in context: the first “zorblax” costs the model about 37 bits of surprise, the second about 4, as predictable as a real word after one exposure.

What it showed: a model absorbs a new word in one exposure, but never turns it into a piece of its own. That takes training.

PSI's Read view: a passage shaded by how surprised the model was at each piece, with a histogram of surprise in bits
Read: brighter is more surprising. Hover a piece to see what the model expected instead.
Question 03 · Neural net and Anatomy

Where inside the network does the answer form?

A second engine opens the model’s weights and records every layer while it answers. For each word it writes, PSI shows what each of the 36 layers would say if the model stopped there, which neurons fired hardest, and what the word looked at.

The Anatomy view draws the real network from its weights: 36 layers, 32 attention heads and 9,728 neurons in each. A patching tool finds where a fact is decided: give it “The Eiffel Tower is located in the city of” and the same sentence about the Colosseum, copy one piece of the first into the second, and see whether “Paris” comes back.

What it showed: the answer is present inside for about two thirds of the network’s depth before it becomes readable. I call that stretch the emergence gap.

36 × 9,728layers and neurons per layer in Qwen3-4B, drawn from the weights in Anatomy
0%of the time does the output head read the final answer through layer 23 of 36
⅔of the depth the answer is already there, not yet readable: the emergence gap
Question 04 · Emergence

When does an LLM start to understand?

Training only ever asks one thing: guess the next piece. Yet at some point a model starts using everything it was told earlier, not just the last few words. Pythia’s saved checkpoints let PSI ask the same questions at each stage of training and find that point.

It can be placed: between training steps 512 and 1,000 of 143,000, under 1% of the way in. Copying, keeping a new word and grammar across a sentence all switch on together, at the same step in three model sizes. Just before it, the model does worse than chance: it has learned which words are common before it has learned why.

What it showed: the training loss falls smoothly straight through that moment. The step is in what the model can do, not in the number training watches.

Chart of Pythia-410m across training: copying, context sensitivity, absorption and abstraction all jump between step 512 and 1,000, while understanding tests go from below chance to passing and the loss falls smoothly
Pythia-410m across its training run. Top: four ways of using the context, all switching on between step 512 and 1,000. Bottom: understanding tests dip below chance, then pass; the grey loss falls smoothly through it all.
Question 05 · Anatomy: light up a concept

Where does an LLM keep an idea?

Type a concept and PSI lights up the neurons that carry it. For Bluetooth, about 200 light up: a core that responds in every context, and slices that each carry one side of it: how it works, where the name came from, what it is used for.

Then delete everything that lit up and look again. Over several rounds I deleted 1,117 neurons. The model still answered correctly about Bluetooth, with about a third of the concept’s signal left, because new neurons took over each time.

What it showed: a concept is not stored in a place. The model rebuilds it from the word, at many layers, along many directions.

~200neurons light up for “Bluetooth”: one core, several slices
1,117neurons deleted over several rounds, and the model still answered correctly
≈ ⅓of the concept’s signal left at the end: new neurons took over
Question 06 · Storage ladder and Understanding

What does an LLM know about something it has never experienced?

A model has never seen red. It can still talk about it. PSI separates the two: what it says when asked, and where red sits inside it when nobody asks.

Asked, Qwen3-4B explains red with blood, fire, love and danger. Inside, unprompted, red sits among the colours: black, blue, white, pink. The Storage ladder measures how high an idea like red is stored, from the word itself up to a meaning shared across languages, and checks every level against an untrained twin of the same model.

What it showed: the colour map it built from text alone lines up with real colour space only weakly (a rank correlation of 0.21). It holds its own truth about red, even where that truth is thin.

blood, fire, love, dangerwhat it says when asked to explain red
black, blue, white, pinkwhere red sits inside it, unprompted
0.21how well its colour map, built from text alone, matches real colour space (rank correlation)
Question 07 · Tipping point

How much of an LLM can you remove before it fails?

Training builds structure. PSI runs it in reverse: remove a share of the neurons and heads, raw weights, chosen layers, or the neurons that carry one concept, and after each cut score what is left.

Random damage breaks things in a fixed order: coherent writing goes first, copying second, understanding last. At 40% removed, the model writes gibberish. Deleting one concept, cricket, dents it but does not remove it.

What it showed: abilities fail in the reverse of the order you might guess. The fluent surface is the most fragile part.

1stcoherent writing fails (46% retained at 20% removed)
2ndcopying what it was shown fails
3rdunderstanding tests hold out longest
Question 08 · Fight class and Self-portrait

Does an LLM know when it is about to be wrong?

Inside, yes. Every open model I tested, 15 of them from 6 families, carries an internal signal that predicts whether its answer will be right, and does so well. What the model says about its own confidence does much worse.

Below about 4 billion parameters, the gauge exists but is not wired to what the model says. In the site’s terms, that is the step from an Autopilot, which models the world, to a Strategist, which models itself modelling. The knowledge is there; the model does not use it. The Self-portrait view puts everything PSI has stored about one model on one page, written to be read by an AI agent as well as a person.

What it showed: knowing and being able to say are different things in a model, and they come apart at a size you can measure.

PSI's Self-portrait view: what PSI knows about one open model, each number beside its untrained twin
Self-portrait: one model’s measurements, each beside its untrained twin, with its noise.
Question 09 · Compare framings

Does an LLM think in the levels this site describes?

The Emergence Machine describes reality as fifteen levels, from fields to planetary coordination. I wanted to know whether a model holds anything like them. PSI asks the same question with and without a framing that names the levels, and reads where each level sits inside the network.

Mostly, no. The levels are not laid out layer by layer inside a model: about two thirds of a sentence’s level is already in its words. Only the relational levels, ecosystems and minds, needed depth. And models do not learn concepts in the order of the levels; they learn the concrete and frequent first.

What it showed: an instrument built to test an idea should be allowed to disagree with it. This one did, and the “no” made every later “yes” easier to believe.

SupportedAbilities arrive in steps, and the step can be placed. A concept is a slice held together by a word. Early choices narrow later ones.
PartlyThe fight classes describe something real, but knowing and saying come apart. Text alone grounds a concept like colour only modestly.
Not supportedLevels are not stacked by layer. Learning order does not follow the levels. No sign that models act differently when told they are watched.

These are one 4-billion-parameter model, a family of smaller ones, and larger models tested only from the outside. Probes find correlations, not causes, and several results match published research on how models work (Olsson, Nanda, Belrose and others), so they support this site’s framing more than they are discoveries unique to it. The full research log, with the scripts and raw numbers, ships with the code.

Ask your own questions

PSI is free and open source. It runs on your own computer, with open models you download yourself. Nothing you type leaves your machine unless you switch on the optional AI guide.

Get it on GitHub ↗︎

parameterspaceinspector.com: the full site, with every instrument on its own page, the manual, comparisons with other tools, and the use-case blog.

Developed and tested on Windows with a 16 GB NVIDIA graphics card. Linux code paths exist; macOS is untested. No graphics card? The browser inspector opens a small model with nothing to install.

Open Claude Code (or Codex, Gemini CLI, Cursor) in an empty folder and paste this. It looks at your computer, suggests a setup that fits, and does each step only after you say yes.

https://github.com/EmAssets/parameter-space-inspector
Help me use this

It never asks for passwords or keys, and installs only what the repository lists, from the official sites it names. Using a chat assistant that cannot open links? Paste the full prompt instead ↗︎

An AI to help you understand AI

Optional · your own key
For people Prometheus, the guide Words like “logit lens” and “activation patching” need explaining. Ask Prometheus and it explains the result on your screen, or runs the view for you: it opens the view, sets the controls and presses the button, so the answer appears where you are looking. ·Every action it takes is shown on screen ·Asks first before anything that costs money ·Uses your own OpenRouter key, with a spending cap
For agents Every instrument over MCP Everything the guide can do is also open to other AI systems through the Model Context Protocol. An agent can generate, read surprise, look layer by layer, knock out neurons or run the experiments, and study a model the way you do. ·Local only, behind a token made each launch ·Agents cannot change keys, caps or the engine ·Changing models is off unless you allow it

How I built it

October 2026 · made with Claude

It started with a gap in an explainer. In How AI Emerges, a tiny model trains in your browser on a street scene, but at the end you cannot open it and see what it kept. A viewer for that model came first. Then the real question: what happens when you open a real model, 3 GB or 15 GB? That meant leaving the browser.

01 · A questionEvery view began as a question from an essay.
The Arena, Abstraction, Language, The Sensing Surface and What Kind of Fight Is This? supplied the questions and the words for them.
02 · A measurementMeasure first, write it down, keep the “no”.
A script, a result and an entry in a research log, lettered A to V by the end. Results that went against the site stayed in.
03 · An instrumentThen turn it into a view anyone can run.
So a reader can repeat the measurement live instead of trusting a chart, and check it as a user would before it ships.
04 · A guideTalk to an AI while it drives the tool.
That turned out to be the best way to use it, so it became a feature: Prometheus in the app, and the same tools for any agent.

I set the direction, almost always as a question; Claude turned each one into a measurement, ran it, checked it in the running app and reported back; I chose what to keep. One instruction set the tone: treat the model as a stand-in for your own mind, and look for what you cannot see about yourself.

Ask Prometheus

Questions about PSI?

Prometheus, the site’s AI guide, can answer questions about the Parameter Space Inspector: what it measures, what it found, and how to run it. Start with one of these, or ask your own.

?What is the Parameter Space Inspector? ?When does a language model start to understand? ?Where does a model keep an idea? ?What do I need to run PSI?
Ask Prometheus →

Subscribe to the newsletter

Essays and new tools from The Emergence Machine, straight to your inbox.

Delivered via Substack · see our privacy policy