How a weight fits in 4 bits
Neural networks are mostly big tables of numbers, called weights. SmolLM2-135M (Allal et al., 2025), the small language model I put on a 2010 iPod touch, has 134.5 million of them. Stored as 32-bit floats they take 538 MB, twice the iPod’s 256 MB of RAM, and every word of an answer means reading all of them once. To fit in memory, and to be read quickly, each weight has to shrink to a few bits.
This article builds up one way to do that: codebook quantization, the scheme that the speech model Whistle ships in and that I used for SmolLM2. It is the technical core of a longer story about running Apple’s own Siri interface on that iPod, but it stands on its own.
We start with the basic idea, a set of levels that each weight rounds to, and how to place the levels. Then come four refinements, each fixing a problem the previous version runs into:
| Stage | Idea | Problem it fixes |
|---|---|---|
| Levels | Round each weight to the nearest of a few levels, placed where the weights are (a codebook) | Float formats and evenly spaced levels follow a fixed rule, not the weights |
| 1. Groups | Each group of 64 weights gets its own scale | One outlier stretches the levels for the whole matrix |
| 2. Rotation | Multiply each group by a Hadamard matrix | A spike still stretches the levels inside its own group |
| 3. One codebook | Divide each group by its length, so that one codebook made for a bell curve serves every group | A codebook for every group costs more to store than the weights |
| 4. Kernel | Rotate the activations once, never rebuild the weights | Decoding 135 million weights per word is too slow |
The figures are live: move the sliders, press the buttons, drag the 3D views. Collapsed “detours” hold side topics and the arithmetic, and margin notes point to published work (on a narrow screen, tap the number).
The “bell curve” of stage 3 is the normal (Gaussian) distribution: values pile up symmetrically around zero, most close to it and few far out.A normal distribution is fixed by its centre, here 0, and its spread . About 68% of the values lie within one of the centre and 99.7% within three, which is why a value beyond is rare (0.27%) in the histograms below.
Where do the levels go?
A weight is a number such as 0.137, and a float stores it in 32 bits. Quantization stores a rougher copy: a small set of allowed values, called levels, and for each weight only the number of its nearest level, its index. With levels the index takes bits, so 16 levels take bits, an eighth of the 32, and the model shrinks times. The price is the rounding error: the distance each weight moves to reach its nearest level.
The question is where to put the levels. The most intuitive answer is to keep the float format and cut bits off it. A float is scientific notation in binary: a sign, an exponent (the power of two) and a mantissa (the digits after the leading one). Here is how the bits of a 32-bit float spell a number:
Now chop the exponent and the mantissa independently, and watch the levels move:
Two things stand out:
- Each kind of bit does a different job. Fewer mantissa bits thin the levels out inside every power of two. Fewer exponent bits shrink the range the levels cover, until the largest weights are clipped.
- The levels follow a fixed rule, not the weights. Even with a good number of bits, many levels sit where almost no weight is: tiny values far below the typical weight, or large values beyond the largest. Evenly spaced levels, the other classic choice, make the opposite mistake: they spread out evenly while the weights crowd together in a few places.
So weights cluster, and a fixed rule wastes levels on the places where they are not. The next intuitive step is to put the levels where the weights are, so that each level serves about the same number of weights. That is what Lloyd’s algorithm does: send every weight to its nearest level, move each level to the average of the weights it got, and repeat. Try it on a lumpy cloud of values, and then on a bell curve:
The bits slider sets how many levels there are: 2 bits give 4 levels, 3 give 8, 4 give 16. The levels settle where the values crowd, and the error drops. A list of levels like this is called a codebook: to store a weight you keep the index of its entry, and to read it back you look the index up.Here a codebook is a list of levels: 16 numbers, and each weight rounds to one of them independently of the others. That is scalar quantization, and so is every step of this article. The rotation in step 2 only changes the coordinates the rounding happens in. Vector quantizers instead replace a whole group of weights with one entry from a codebook of points; QuIP# is an example.
Detour: other ways to quantize
Quantization methods differ on several independent axes, and this article covers one combination of them. In short:
| Axis | Options | What this article does |
|---|---|---|
| What is stored per number | A low-precision float (fp16, bf16, fp8, FP4), or a small integer index | An index, into a codebook |
| Level spacing | Evenly spaced, geometric (floats), or fitted to the weights (Lloyd–Max, NF4) | Evenly spaced at first, then fitted |
| Scalar or vector | One number at a time, or a whole group per codebook entry | Scalar |
| Scale | One per tensor, per row, or per group | One per group of 64 |
| Preprocessing | None, or a rotation (QuIP#, QuaRot) | A Hadamard rotation |
| What is quantized | Weights only, or activations and the attention cache too | Weights only |
| When | After training (post-training), or during it (quantization-aware training) | After training, with GPTQ rounding for SmolLM2 |
Where to read more:
- A Survey of Quantization Methods for Efficient Neural Network Inference (Gholami et al., 2021) is the standard overview of the axes above.
- A Visual Guide to Quantization (Maarten Grootendorst, 2024) is a picture-led tour from float formats through integer schemes to outliers.
- FP8 Formats for Deep Learning (Micikevicius et al., 2022) and Microscaling Data Formats for Deep Learning (Rouhani et al., 2023) describe the float-with-fewer-bits side, including formats that share one scale across a block of 32 numbers, which is group-wise scaling again.
- The Transformers quantization guide explains post-training against quantization-aware training and which libraries do what.
- The margin notes later in this article point to the papers behind rotation, fitted levels and GPTQ.
The trouble with outliers
Now that the levels fit the weights, real weight matrices bring a complication. Most weights are small, yet a few are much larger. These outliers are what much of the quantization literature is about. Try them: add an outlier to 40 weights, and compare levels spread evenly from the smallest weight to the largest with levels fitted by Lloyd’s algorithm:
- Evenly spaced levels suffer. They must reach the largest weight, so one outlier stretches them far apart, and the ordinary weights near zero round to just the few levels there. Their error jumps although almost no weight changed.
- Fitted levels cope much better, but not for free. The outlier takes a level or two of its own, out in the tail, which leaves fewer for the ordinary weights. With 2 bits there are only 4 levels to begin with, so this hurts most there.
- Either way, one set of levels for a whole matrix is the weak point. A few outliers anywhere among millions of weights steer all of the levels.
The next step limits the damage.
Step 1: cut each row into groups
From here to step 3, the figures use the simplest levels: evenly spaced from to , where is the largest weight in absolute value (the “scale”). With levels the gap between neighbours is , and level number , counting from 0, is . That one number is stored alongside, so the levels can be turned back into real values. Evenly spaced levels make the effect of groups and rotation easiest to see; step 3 brings the fitted levels back.
- A weight matrix has millions of numbers, and one outlier anywhere would stretch the levels for all of them.
- So each row is split, from left to right, into groups of 64 consecutive weights (128 in Whistle), and each group gets its own scale. A group never spans rows. An outlier now only spoils its own group.Group-wise scales for weights are common practice: ZeroQuant (Yao et al., 2022) uses them for transformers. The outliers behind them are well documented: LLM.int8() (Dettmers et al., 2022) found a few large-magnitude features in transformer hidden states, about 0.1% of them, that wreck quality when removed.
- The scale is a number stored once per group. In the plain scheme it is the group’s largest weight, which is what the figure below uses. Whistle stores the group’s length, , instead, for reasons that become clear in step 3.
- Inside the group, each weight only has to say “which of 16 levels am I closest to”, and 16 levels take 4 bits.
Detour: what exactly is a group, and why along a row?
A group is a run of consecutive columns inside one row. It never spans rows, and two rows never share a scale. For a weight matrix with rows and columns, and a group size :
- Row , which is one output of the layer, is cut into groups, left to right.
- Group of row holds the weights to .
- It has its own scale , so the whole matrix stores scales.
row i: W[i,0] … W[i,G−1] | W[i,G] … W[i,2G−1] | …
└──── group 0 ────┘ └───── group 1 ─────┘
scale s[i,0] scale s[i,1]
Why along a row. A layer turns the incoming numbers (the activations) into output with a dot product along row :
Each weight is stored as a scale and a level index, , where is the group that column falls in and is the list of levels. Put that into the dot product and group the terms by :
Inside the brackets every term shares the same scale, so the scale comes out of the sum. The inner sum uses only small level values and the activations, and the scale is multiplied in once per group instead of once per weight. Step 2 adds the rotation , which acts on the activations of one column block :
Here is the vector of the levels chosen in group of row . The rotated activations do not depend on , so they are computed once per column block and reused by every row. That works because every row cuts its columns at the same places.
Other layouts. Groups could be shaped differently, and the same formula shows what each shape does:
| Layout | Can the scale come out of the sum? | Notes |
|---|---|---|
| Run along a row (used here) | Yes, one multiply per group | Contiguous in memory, and the rotated activations are shared by all rows |
| Strip down a column | No: each term of the sum comes from a different strip with its own scale | One multiply per weight, and the reads are scattered |
| Square tile (some formats use 128×128) | Yes, once per row of the tile | Far fewer scales, but one outlier spoils a much larger block. This project did not measure it |
To get a feel for what grouping buys, try it on a toy matrix of 12 rows and 64 columns:
- Slide “Groups per row” up from one, which is a single scale for the whole row, and watch the rounding error fall.
- “Groups with outliers” and “Outlier size” decide how many groups contain an outlier and how big it is. Slide the first down to 0 to see how much of the effect the added outliers cause.
- The 16 levels are spread evenly up to the largest weight in each group: the plain scheme from the figure above.
Step 2: rotate the group
Grouping stops an outlier from spoiling its whole row, but inside its own group it still stretches the levels for the other 63 weights.
A spike costs the whole group
Here are the groups from the figure above, split into those that contain an outlier and the rest. Each dot is one group, and the tick marks the average:
- What matters is how spiky a group is: , its largest weight compared with the group’s length .
- We can’t remove a spike without changing the weights.
- But we can change the axes we write the weights against. That is what a rotation does.
Groups of two weights, as points
To visualize axes, let’s consider the smallest case: groups of just 2 weights. Below is a row of weights cut into groups of two, like one row of the toy matrix above, drawn larger. The groups run from a short one, where one weight is much bigger than the other, to a long one, where the two weights are equal:
The first group is short, and one of its two weights is much bigger than the other. The last is long, and its two weights are equal. The groups in between change gradually from one to the other, and the number of groups goes up in twos. In the first half one weight dominates: those groups are spiky (triangles). In the second half the weights are alike: those groups are flat (circles).
Weights, with each group's scale printed on it
Scale of each group, its largest weight (the more gold, the larger)
Rounding error, after rounding each weight to one of 4 levels (the redder, the larger)
Each group is just two numbers, w1 and w2, so it can also be read as a point on a plane: the first weight is its horizontal coordinate and the second weight is its vertical one. Here are the same groups drawn that way, with the same numbers and colors:
Quantizing the points
Now let’s see what quantization does to these points:
- The 2-bit levels, four per axis, form a grid of 16 points inside a square. A group is stored as the nearest grid point: each of its two weights is rounded to the nearest level, and the two level numbers (2 bits each) are what is saved.
- A trail of fading shadows shows how far that rounding moves a group, from its real weights (the faintest shadow) to the grid point it is stored as (the solid marker). The length of the trail is the group’s rounding error.
- A point close to an axis has one big weight and one tiny one, so its square must be as large as the point is long, and the grid is coarse.
- A point on the diagonal has two equal weights, which is as flat as it gets: the square is smaller and the grid finer.
Here is how much each group loses to rounding, the length of its trail as a share of the group's length, split into the first half (closer to an axis) and the second half (closer to a diagonal):
Rotating the points
Now rotate the groups and watch what happens to each half:
- The two curves show, at every angle, the largest weight as a share of the group’s length, averaged over the spiky first half (gold) and the flat second half (blue). Lower is flatter, and the rounding error follows it.
- The loss diagram under the plot shows how much each group loses at the current angle.
- The camera switch changes the viewpoint. Off, the camera stays with the axes and the points move. On, it stays with the points, and the axes and squares turn the other way.
And here is how much each group loses to rounding at the current angle, split into the first half (spiky) and the second half (flat). Turn the angle and watch the two rows move:
The rotation slider is only an animation: in practice a rotation is applied once, all at once.
Detour: how is this different from PCA?
If you have met principal component analysis (PCA), this may look familiar, because PCA is also a rotation. But what we are trying to do is exactly the opposite of PCA. PCA looks for the angle that makes data points more spiky, so that most of the information sits in a few coordinates. We look for the angle that flattens them instead:
| PCA | What we want | |
|---|---|---|
| Goal | Make groups spikier: line the axes up with the directions the data varies along most, so the energy piles into a few coordinates | Make groups flatter: turn spikes away from the axes, so the energy spreads over all coordinates |
| Good for | Compressing, by dropping the small coordinates | Rounding, where one dominant coordinate stretches the levels for all the others |
| In short | The best angle to concentrate the data | The best angle to flatten it |
Here are both, applied to the same 24 groups of 8 weights. PCA piles the energy into a few coordinates, and a flattening rotation spreads it out:
Here is the same contrast in three dimensions, where you can watch it happen:
- Each dot is a group of three weights, red when its 2-bit rounding error is large and blue when it is small.
- The flattening rotation is a 60° turn about the diagonal direction (1, 1, 1), the flattest rotation of three weights (the detour on three weights at the end of this step shows it).
- Choose a data set and a rotation, then drag the slider from 0 to 100% of that rotation. “Spin the view” turns the camera so you can see the cloud from all sides.
The slider is only an animation. A real PCA step applies one matrix, fitted to the data, all at once. The flattening rotation is the 60° turn about (1, 1, 1).
PCA and a flattening rotation do opposite jobs. When the data has no preferred direction (the first data set), PCA’s axes are close to arbitrary, so it can help or hurt by chance. When the weights move together (the second), PCA does exactly what compression wants and exactly what rounding does not: it makes every group spikier.
Spiky or flat? Telling them apart
So a rotation helps spiky groups and hurts flat ones. That raises a natural question: how do we tell which kind a group is?
- There is no sharp line between spiky and flat. In our figure the groups change gradually from sitting on an axis to sitting on a diagonal, and real groups do the same: how spiky a group is runs on a scale.
- A spiky group usually has one weight much larger than the rest. That weight is called an outlier, and it is what to look for in a real layer.
- In practice we use a heuristic. Compare the group’s largest weight with its overall size, and call the group spiky if that ratio is far bigger than ordinary weights would give.
- Whistle does not try to tell. The converter applies the same rotation to every group. Whether that is a good bet depends on how many groups are spiky and on how many weights each group has, which is where we are heading.
Detour: measuring how spiky a group is
Two numbers are commonly used. For a group of weights with length :
- Spikiness : the largest weight as a share of the group’s length. It is the quantity on the curves above. It runs from (all weights the same size) to 1 (a single weight).
- Crest factor , where is the typical size of a weight. It says how many typical weights the biggest one is worth, so it is easier to compare across group sizes.
To decide what counts as large, compare with groups of ordinary weights, which behave like Gaussian numbers. Here is what 20,000 random Gaussian groups give:
| Group size | Flat | Typical : mean | 99th percentile | Typical : mean | 99th percentile |
|---|---|---|---|---|---|
| 2 | 0.71 | 0.90 | 1.00 | 1.27 | 1.41 |
| 8 | 0.35 | 0.65 | 0.89 | 1.84 | 2.52 |
| 64 | 0.125 | 0.33 | 0.45 | 2.61 | 3.60 |
| 128 | 0.088 | 0.25 | 0.34 | 2.84 | 3.89 |
A reasonable rule of thumb is to say a group of 64 has an outlier if its crest factor is above about 4: a weight more than four times the typical size, rarer than 1 group in 100 among ordinary ones. The threshold is a choice. The useful readout is the share of groups above it, measured layer by layer.
- In two dimensions the crest factor can never exceed , so a rule like this has no bite. Our groups are spiky or flat only because of where we placed them.
- With more weights the gap between a flat group and a spike grows, which is why outliers only become a sharp problem for bigger groups.
Which rotation?
Many rotations would flatten a spike, as the figures above show. Whistle uses one that needs no multiplications and no stored matrix, because a simple rule generates it: the Walsh–Hadamard matrix.Rotating before quantizing is a published technique. QuIP# (Tseng et al., 2024) uses a randomized Hadamard transform on the weights, with random signs that my version does not add. QuaRot (Ashkboos et al., 2024) rotates a whole LLM with Hadamard matrices to remove outliers for 4-bit inference. Our groups so far have had two weights, but Whistle’s have 64 or 128, so we need one for bigger groups too.
It is a square matrix whose entries are all +1 or −1, divided by so that it keeps lengths. The 2×2 one is
and each bigger one is four copies of the previous one, with the bottom-right copy negated:
That gives a matrix for every size that doubles from 2: 4, 8, 16, and so on up to 64 and 128. Here are the entries, before the scaling:
Two properties make this the right tool:
- Every output mixes every input. Each output is a ± sum of all the group’s weights, so one large weight spreads evenly over all 64 outputs instead of dominating one.
- Sums of many ± terms tend toward a bell curve (the central limit theorem), so the rotated group looks much more Gaussian than the original, with no lone extremes.
- QuIP# and QuaRot use the same idea for large models.
- It is its own inverse. The rows are orthogonal and the matrix is symmetric, so applying it twice gives back the original. Rotating loses nothing.
To see the first property at work: orange is a group of weights as they come, with some outliers mixed in, and blue is the same group after rotation. The histograms below the bars show where the values sit:
Why and when the Hadamard matrix flattens best
The figures above showed that rotating a spiky group can help. Here is what makes the Hadamard rotation the one to use, and when it works.
Why: a lone spike of any size ends up, after a Hadamard rotation, equally spread over all outputs, so each output holds of its length. The more weights share the rotation, the flatter it gets:
| Weights in the group | Largest coordinate after rotating a lone spike |
|---|---|
| 1 (nothing to spread over) | 100% of the length |
| 3 (no Hadamard matrix exists for this size; see the detour on three weights) | at best 58% |
| 8 | 35% |
| 64 | 12.5% (1/8) |
| 128 | 8.8% |
In two dimensions the Hadamard matrix is exactly the 45° rotation with its two outputs swapped, so the two-weight picture shows when it helps. Here it is again, with that angle marked:
In two dimensions the match is exact: the Hadamard matrix is the 45° rotation with its two outputs swapped. Swapping outputs does not change the rounding error, because the grid is the same either way.
- Spiky groups, the triangles. They start near an axis, as far from 45° as possible, so turning by 45° brings them to the flat direction. The gold curve is lowest at 45°.
- Flat groups, the circles. They start near a diagonal, so they are already flat, and 45° turns them into spikes. The blue curve peaks at 45°.
For other group sizes the flattest a rotation can be is set by the length: each row of the matrix has length 1, so its largest entry is at least . The Hadamard matrix reaches that bound whenever it exists:
| Group size | Smallest largest entry | Which matrix gets there |
|---|---|---|
| 2 | 0.71 | The Hadamard matrix, exactly: the 45° rotation above |
| 3 | 0.67 | No Hadamard matrix of this size exists. The 60° rotation in the detour on three weights is the flattest we found (a numerical search agrees) |
| 4 | 0.5 | The 4×4 Hadamard matrix |
| 8 | 0.35 | The 8×8 Hadamard matrix |
| 64 | 0.125 | The 64×64 Hadamard matrix, which is why groups of 64 (or 128) are a natural size |
Detour: what about the 3D case (3 weights per group)?
Everything so far used groups of two weights, and Whistle’s groups have 64 or 128. Three weights is the case in between that you can still draw. It comes with a catch: a Hadamard matrix can’t be built for three weights. The doubling rule only makes sizes 2, 4, 8, 16 and so on, and in general a matrix of +1 and −1 entries with perpendicular rows exists only for sizes 1, 2 and multiples of 4. So for three weights we use the flattest rotation we know instead: a turn about the diagonal direction , which is best at 60°.
Here are some groups of three weights, laid out like the two-weight ones. They run from a short group where one weight dominates to a long group where all three weights are equal:
Weights, with each group's scale printed on it
Scale of each group, its largest weight (the more gold, the larger)
Rounding error, after rounding each weight to one of 4 levels (the redder, the larger)
Three weights make a point in 3D space, one weight per axis. Each group gets a cube of levels, 4 per axis, that just reaches its largest weight. Here is the same setup as for two weights, with the groups rotated about the diagonal:
And here is how much each group loses to rounding at the current angle, split into the first half (spiky) and the second half (flat):
The slider is only an animation: in practice a rotation is applied once, all at once.
Two things differ from the two-weight picture:
- The flat groups sit on the diagonal that the rotation turns about, so they hardly move, and the rotation does not hurt them here the way it did in 2D.
- At 60° the rotation matrix has entries of ±2/3 and ±1/3, which is as flat as any 3×3 rotation can be. That flattens a lone spike to two thirds of its length, a smaller gain than for bigger groups, where a Hadamard matrix takes it down to .
A whole matrix, rotated
Here is the toy matrix from step 1 again, with every group rotated by the same Hadamard matrix. The left half of each heatmap is the matrix as it was, and the right half is the same weights after rotation. Both halves use the plain scheme from step 1, with 16 evenly spaced levels up to the group’s largest weight:
- Weights: an outlier’s color is spread across its whole group on the right, as many mild cells instead of one strong one.
- Scales: the groups that held an outlier are no longer much more gold than their neighbors, because the largest weight in the group has shrunk.
- Errors: the rows of red cells in those groups fade. Rotation keeps lengths, so the error on the right is the same as the error after rotating back.
- The groups without an outlier barely change, or lose a little, as the table shows.
So the rotation is settled: one Hadamard matrix, applied to every group. Next come the levels.
Step 3: one codebook for every group
Per-group codebooks are expensive to store
The fitted levels from the start of this article come with a cost that we have not counted yet: the codebook has to be stored too. Evenly spaced levels are free in that respect, since the scale alone fixes all of them. Fitted levels are not, because every level is a number. And each group of weights has its own values, so each group would get its own codebook. Let us see what that costs.
Take a group of 64 weights, which is the group size I used for SmolLM2 (Whistle uses 128). Its 64 weights hardly spread evenly: most sit near zero and a few lie far out. So we want the fitted levels for this group, which are not evenly spaced.
To see such a set, pick two of the 64 weights at random, say the 21st and the 58th, and plot them as one point. The levels of the group, applied to both axes, make a grid around it. Then take the weights at the same two positions in another group: a different point, and a different set of levels, because the other group has its own weights. Add groups and levels below and watch the table:
The table keeps growing: one row for every group, one column for every level. Each number takes 32 bits, and a group’s 64 weights are stored as indices into its row. A full table is therefore a large part of what a group costs (at 16 levels, two thirds of it).
One shared codebook
That table is what the rest of this step gets rid of. The way out is to make every group look the same, so that one shared row serves them all. What “the same” has to mean is worked out in the next two parts. Then each group stores only its indices and one scale, and the codebook is stored once for the whole model. Here is what one group stores in each scheme, drawn in proportion. Move the sliders to see what the price depends on:
Detour: the arithmetic behind it
Take groups of weights, -bit indices (so levels), and -bit numbers ( by default in the figure). Anything stored once for the whole model is negligible, so what counts is what is stored for every group:
- The shared scheme pays for one scale per group. The fresh scheme pays for levels per group instead, which is times as much: with , sixteen times.
- The fresh codebook’s cost per weight equals the indices’ own bits when , which is 128 for and . Below that size, the codebooks outweigh the data they describe.
- To keep the fresh codebooks under 10% of the indices, the group would need , which is 1,280 weights for and . But groups have to stay small for step 1 to work: an outlier spoils its whole group, so a big group means more weights spoiled.
- The same logic rules out a different rotation for every group. A dense matrix costs bits per group, which is bits per weight, always far more than . A single Hadamard matrix, generated by a rule, costs 0.
So sharing is what makes small groups affordable: each group may only pay for a single scale.
What shape do the values have?
Trained weights are usually roughly bell-shaped: they start out as random Gaussian numbers, and training adds up many small nudges. Here are the real weights of SmolLM2. Each group is first divided by its length, so that every group is measured on the same scale. Under the plot, the readout gives the kurtosis of the values, which is 3 for a bell curve:Kurtosis measures how heavy the tails are. It is the average of when the values are measured in standard deviations: a bell curve scores 3, a sharper peak with long tails scores more, and values packed into a narrow range score less.
For groups of 64, the attention and feed-forward weights come close to a bell curve, with a kurtosis of 3.2 and 3.1 (a bell curve has 3).The embedding is the exception, with a kurtosis of 5.2: a sharper peak and longer tails.
Detour: does the rotation make them even more bell-shaped?
As one would expect, yes. Each rotated weight is a mix of all the weights in its group, so by the central limit theorem the values pile up in a bell curve even when the originals do not. Here are the same weights before and after rotating each group:
For groups of 64:
- Rotation trims the tails and evens out the layers. Every part lands at a kurtosis of 2.9. That is exactly what a random direction in 64 dimensions gives (), so it cannot get more bell-like than that. It sits a little below the 3 of a normal distribution because a group of fixed length cannot hold extreme values. The share of weights beyond 3σ falls from 0.50% to 0.20%, and the largest value from 7.9σ to 5.5σ.
- It helps most where the weights are least bell-shaped. The embedding goes from a kurtosis of 5.2 to 2.9. The same holds for groups with an outlier, as in the toy examples earlier. So the rotation is less a source of the bell shape than a guarantee of it.
- Without dividing by the length, the picture is much worse. Groups differ a lot in size, so the values are a mixture of many bell curves of different widths, which has heavy tails (kurtosis 5.4, largest value 51σ). Rotating helps (3.7, 25σ) but cannot make the groups alike: it mixes the weights inside a group and leaves the group’s size alone. That is why the length is divided out first.
- Each group, not just the pile. A histogram of all the weights together does not prove that each group is bell-shaped, and 64 values could not prove it anyway. What can be checked is whether the groups differ from bell-shaped ones by more than chance would. The kurtosis of a rotated group of 64 has a 10th, 50th and 90th percentile of 2.33, 2.81 and 3.59 in SmolLM2, and 2.33, 2.81 and 3.60 for groups drawn from a true normal distribution. The shared codebook also needs only the pooled shape to be right: if each group were given its own best 16 levels (fitted to those same 64 values, so this flatters it), the rounding error would fall from 9.6% to 7.2%, at the cost of 12 bits per weight instead of 4.5.
- Small groups show little. In a group of 8 no value can exceed , so there are no tails to trim. Slide the group size up to see the effect grow.
Placing the levels
Knowing the shape settles two things:
- The levels can be designed once, ahead of time. They need only be dense where the bell is tall and sparse where it is short, which evenly spaced levels are not.
- That only works if every group has the same spread as well as the same shape. The best levels for a bell curve of spread are the best levels for a bell curve of spread 1, multiplied by , so the spread is the one thing that must be taken out. Dividing each group by its own length does that: the length is times the group’s root-mean-square, so after the division every group has the same , namely per weight. One shared set of levels, made for that one , then serves every group, with nothing extra to store per group.A side benefit is speed. A 4-bit codebook has only 16 entries, which fit in one CPU register, so many weights can be looked up at once (step 4). Speed is not why the codebook exists: the aim of these steps is a smaller model.
Putting it together, the step does this:
- Divide each group by its length, which is stored as the group’s scale. What is left is a direction, so every group is on the same footing.
- The levels come from Lloyd’s algorithm run on a Gaussian: dense near zero, where most values are, and sparse in the tails. That set is the codebook.Levels built for normally distributed weights are not unique to Whistle’s scheme. The NF4 data type in QLoRA (Dettmers et al., 2023) is designed for zero-centred normal data too, but places its levels at quantiles of the normal distribution, so that each level serves the same expected number of values, rather than with Lloyd’s algorithm.
- Each weight stores the 4-bit index of its nearest level.
The levels come from Lloyd’s algorithm, exactly as in the figure near the start, only run on a bell curve instead of a lumpy cloud: switch that figure to “bell curve” and step through it to see where the levels settle.
Now the same idea on the two-weight groups from step 2. The length of each group is divided out, so the dots sit on a circle of length 1, and one grid is shared by every group:
Weights with the length divided out, and each group's length (its scale) printed on it
Scale of each group, its length (the more gold, the larger)
Rounding error, in units of the group's length (the redder, the larger)
Step 4: never rebuild the weights
Steps 1 to 3 made the weights small. This step keeps them small while the model runs, and makes using them fast.
- The problem: steps 2 and 3 store rotated, rounded weights. To use one you’d normally decode it: look up its level, multiply by the length, rotate back.
- Doing that for 135 million weights per word would be slow on a 2010 CPU, and it would rebuild the full-size weights we worked to avoid keeping.
- The way out: a layer only needs dot products, and rotating both sides of a dot product doesn’t change it. So rotate the other side, the activations, instead, and never touch the weights.
A layer computes dot products of the weights with the incoming numbers (the activations). Because the rotation is orthogonal, rotating both sides leaves the dot product unchanged:
Here is a group of weights, the matching activations, the group’s length, the codebook, and the index stored for weight . The first equality is exact. The is where the rounding enters: the stored level only approximates the -th entry of , divided by .
So in practice:
- The kernel rotates the activations once for each block of of them and reuses the result for every row of the matrix: thousands of rows share one rotation.QuaRot splits its rotations the same way: most are folded into the weights ahead of time, and the rest are applied while the model runs, using the fast Walsh–Hadamard transform.
- The weights stay as indices.
- On the iPod’s NEON unit, one table-lookup instruction (
vtbl) turns 8 indices into 8 small integer levels. - Multiply-accumulate instructions then sum 8 or 16 products at a time.
Here is the shortcut with actual numbers. Both routes give the same answer, but rotating the activations once is much cheaper when many rows share them:
The rotation itself is cheap thanks to the fast Walsh–Hadamard transform. Because the matrix is built by doubling, it can be applied in rounds of add-and-subtract on pairs, the same butterfly pattern as the FFT, with no multiplications:
Taking stock: what the steps buy, and what they cost
All four steps are in place. Here is the whole pipeline on one group, then its gains and its price:
What rotation and the codebook buy, measured on random groups of 64 and on SmolLM2’s real layers:
- Rotation is the big win when a group has outliers. For heavy-tailed groups it halves the error at 4 bits (18% → 9%) and cuts it from 86% to 36% at 2 bits.
- The Gaussian codebook adds a little on top, mostly at 2 bits (36% → 33%). SmolLM2’s real layers are closer to Gaussian already: at 2 bits the codebook takes their error from about 40% to 34%; at 4 bits every method lands near 10%.
- So at 4 bits most of the measured quality gain for SmolLM2 came from GPTQ (next), and the format’s other strength is speed, which is step 4.
What it costs to store:
- A group of 64 weights takes bits plus one 32-bit length, which is bits per weight on average, because the one length is shared by all 64.QLoRA (see above) has the same overhead: with 32-bit constants and blocks of 64 it pays 32/64 = 0.5 bits per parameter, and its “double quantization” shrinks that to about 0.127 by quantizing the constants as well.
- SmolLM2 went from 538 MB to 76 MB.
- Whistle is 16.9 MB.
The shared parts are almost free: the codebook is 16 numbers, 64 bytes for the whole model, and the Hadamard matrix costs nothing, because the doubling rule generates it. The alternatives that step 3 ruled out are not. Per weight, for groups of 64 and 32-bit numbers:
| What is stored | Bits per weight | SmolLM2 |
|---|---|---|
| 32-bit floats, for reference | 32 | 538 MB |
| What we do: 4-bit indices, one scale per group, one Hadamard matrix and one codebook for everything | 4.5 | 76 MB |
| A fresh 16-level codebook for every group instead of the shared one (it replaces the scale: 16 numbers instead of 1) | 12 | 202 MB |
| A different dense random rotation for every group, instead of the Hadamard matrix ( numbers each) | about 2,050 | about 34 GB |
Other choices cost other amounts. Try a few and compare with the iPod’s 256 MB of RAM:
Rounding smarter: GPTQ
One more refinement is for SmolLM2 only. It changes only which level each weight rounds to, so the stored format and the kernel stay the same.
- Rounding each weight to its nearest level is the simplest choice, but not the best one: the errors of neighbouring weights can add up or cancel out.
- GPTQ rounds the weights one at a time and nudges the not-yet-rounded ones to compensate, using statistics of real activations (here, from 40 chat questions).
- It ran on my Mac in a minute, in the rotated domain, with one GPTQ block per group.
Perplexity on held-out chats, where lower is better:
| SmolLM2-135M stored as | Size | Perplexity |
|---|---|---|
| 32-bit floats | 538 MB | 7.2 |
| Before: evenly spaced 4-bit layers, 8-bit embedding | 98 MB | 10.1 |
| 4-bit codebook, nearest level | 76 MB | 9.6 |
| 4-bit codebook, GPTQ rounding | 76 MB | 8.1–8.9 |
| 3-bit codebook | 59–66 MB | 29–81, unusable |
What it adds up to
A weight now costs 4.5 bits instead of 32: a 4-bit index, plus a 32-bit scale shared by 64 weights. SmolLM2 shrank from 538 MB to 76 MB, and the codebook costs 64 bytes for the whole model. Each step earned its place:
- Groups keep an outlier from spoiling a whole matrix. A group of 64 pays one scale for that.
- Rotation spreads what is left of an outlier across its group, so one big weight no longer stretches the levels for the rest.
- The codebook puts the levels where bell-shaped values actually are. All groups share it, which works only because each group is divided by its length first.
- The kernel rotates the activations instead of the weights, so the stored form never has to be rebuilt.
What the numbers do and do not show:
- Everything here was measured on SmolLM2-135M, one small model. At 4 bits, rotation and the codebook gave modest gains on its weights, which were already close to a normal distribution. Smarter rounding mattered more: GPTQ brought perplexity from 9.6 down to 8.1–8.9, against 7.2 for the 32-bit floats. Whistle’s weights were not analysed.
- Only the weights are quantized. Activations and the attention cache, which larger models also need to shrink, are a separate problem, and one where outliers are usually worse.
- The floor on bits depends on the model: at 3 bits, SmolLM2 was unusable.
To go further, the margin notes link the papers behind each step. The reference implementation (cq.py) and the scripts behind the histograms are in the project’s repository. The story of getting these models to answer questions on the iPod picks up where this article stops.