Blog

How a weight fits in 4 bits

Neural networks are mostly big tables of numbers, called weights. SmolLM2-135M (Allal et al., 2025), the small language model I put on a 2010 iPod touch, has 134.5 million of them. Stored as 32-bit floats they take 538 MB, twice the iPod’s 256 MB of RAM, and every word of an answer means reading all of them once. To fit in memory, and to be read quickly, each weight has to shrink to a few bits.

This article builds up one way to do that: codebook quantization, the scheme that the speech model Whistle ships in and that I used for SmolLM2. It is the technical core of a longer story about running Apple’s own Siri interface on that iPod, but it stands on its own.

We start with the basic idea, a set of levels that each weight rounds to, and how to place the levels. Then come four refinements, each fixing a problem the previous version runs into:

Stage Idea Problem it fixes
Levels Round each weight to the nearest of a few levels, placed where the weights are (a codebook) Float formats and evenly spaced levels follow a fixed rule, not the weights
1. Groups Each group of 64 weights gets its own scale One outlier stretches the levels for the whole matrix
2. Rotation Multiply each group by a Hadamard matrix A spike still stretches the levels inside its own group
3. One codebook Divide each group by its length, so that one codebook made for a bell curve serves every group A codebook for every group costs more to store than the weights
4. Kernel Rotate the activations once, never rebuild the weights Decoding 135 million weights per word is too slow

The figures are live: move the sliders, press the buttons, drag the 3D views. Collapsed “detours” hold side topics and the arithmetic, and margin notes point to published work (on a narrow screen, tap the number).

The “bell curve” of stage 3 is the normal (Gaussian) distribution: values pile up symmetrically around zero, most close to it and few far out.A normal distribution is fixed by its centre, here 0, and its spread σ\sigma. About 68% of the values lie within one σ\sigma of the centre and 99.7% within three, which is why a value beyond 3σ3\sigma is rare (0.27%) in the histograms below.

Where do the levels go?

A weight is a number such as 0.137, and a float stores it in 32 bits. Quantization stores a rougher copy: a small set of allowed values, called levels, and for each weight only the number of its nearest level, its index. With 2b2^b levels the index takes bb bits, so 16 levels take b=4b = 4 bits, an eighth of the 32, and the model shrinks 32/4=832/4 = 8 times. The price is the rounding error: the distance each weight moves to reach its nearest level.

The question is where to put the levels. The most intuitive answer is to keep the float format and cut bits off it. A float is scientific notation in binary: a sign, an exponent (the power of two) and a mantissa (the digits after the leading one). Here is how the bits of a 32-bit float spell a number:

How a float spells a number. A float is scientific notation in binary: a sign, an exponent and a mantissa. Its value is ±(1 + mantissa) × 2exponent, where the exponent is stored with a fixed number added to it. A 32-bit float gives the exponent 8 bits and the mantissa 23. Change how many bits each gets, pick a number to see its bits, or move the sliders for the stored values.
Bits
Try
Stored

 

Now chop the exponent and the mantissa independently, and watch the levels move:

Chopping bits off a float. The blue bars are 3,000 weights. Every gold tick is a value that the float format can hold, and each weight rounds to the nearest one. The usual 32-bit float has 8 exponent bits and 23 mantissa bits.

 

Two things stand out:

So weights cluster, and a fixed rule wastes levels on the places where they are not. The next intuitive step is to put the levels where the weights are, so that each level serves about the same number of weights. That is what Lloyd’s algorithm does: send every weight to its nearest level, move each level to the average of the weights it got, and repeat. Try it on a lumpy cloud of values, and then on a bell curve:

Finding the levels with Lloyd's algorithm. The bars count 4,000 values. Gold ticks are the levels, and the dashed lines between them cut the number line into the zone each level serves. Each step sends every value to its nearest level, then moves each level to the average of the values it got. The bars compare the error with a tiny float of the same number of bits, and the line shows the error after each step.

 

The bits slider sets how many levels there are: 2 bits give 4 levels, 3 give 8, 4 give 16. The levels settle where the values crowd, and the error drops. A list of levels like this is called a codebook: to store a weight you keep the index of its entry, and to read it back you look the index up.Here a codebook is a list of levels: 16 numbers, and each weight rounds to one of them independently of the others. That is scalar quantization, and so is every step of this article. The rotation in step 2 only changes the coordinates the rounding happens in. Vector quantizers instead replace a whole group of weights with one entry from a codebook of points; QuIP# is an example.

Detour: other ways to quantize

Quantization methods differ on several independent axes, and this article covers one combination of them. In short:

Axis Options What this article does
What is stored per number A low-precision float (fp16, bf16, fp8, FP4), or a small integer index An index, into a codebook
Level spacing Evenly spaced, geometric (floats), or fitted to the weights (Lloyd–Max, NF4) Evenly spaced at first, then fitted
Scalar or vector One number at a time, or a whole group per codebook entry Scalar
Scale One per tensor, per row, or per group One per group of 64
Preprocessing None, or a rotation (QuIP#, QuaRot) A Hadamard rotation
What is quantized Weights only, or activations and the attention cache too Weights only
When After training (post-training), or during it (quantization-aware training) After training, with GPTQ rounding for SmolLM2

Where to read more:

The trouble with outliers

Now that the levels fit the weights, real weight matrices bring a complication. Most weights are small, yet a few are much larger. These outliers are what much of the quantization literature is about. Try them: add an outlier to 40 weights, and compare levels spread evenly from the smallest weight to the largest with levels fitted by Lloyd’s algorithm:

Rounding 40 weights to a few levels. Dots are the weights, ticks are the levels, and each line shows where a weight rounds to. The levels are either fitted to the weights with Lloyd's algorithm or spread evenly from the smallest weight to the largest. More bits mean more levels and a smaller error, but a bigger model. Click a dot to see the actual bits of that weight, before and after.

 

The next step limits the damage.

Step 1: cut each row into groups

From here to step 3, the figures use the simplest levels: evenly spaced from −m-m to +m+m, where mm is the largest weight in absolute value (the “scale”). With 2b2^b levels the gap between neighbours is gap=2m2b−1\mathrm{gap} = \dfrac{2m}{2^b - 1}, and level number kk, counting from 0, is −m+k⋅gap-m + k\cdot\mathrm{gap}. That one number mm is stored alongside, so the levels can be turned back into real values. Evenly spaced levels make the effect of groups and rotation easiest to see; step 3 brings the fitted levels back.

Detour: what exactly is a group, and why along a row?

A group is a run of consecutive columns inside one row. It never spans rows, and two rows never share a scale. For a weight matrix WW with out\mathrm{out} rows and in\mathrm{in} columns, and a group size GG:

  • Row ii, which is one output of the layer, is cut into in/G\mathrm{in}/G groups, left to right.
  • Group gg of row ii holds the weights Wi, gGW_{i,\,gG} to Wi, gG+G−1W_{i,\,gG+G-1}.
  • It has its own scale si,gs_{i,g}, so the whole matrix stores out×in/G\mathrm{out}\times\mathrm{in}/G scales.
  row i:   W[i,0] … W[i,G−1] | W[i,G] … W[i,2G−1] | …
           └──── group 0 ────┘ └───── group 1 ─────┘
              scale s[i,0]         scale s[i,1]

Why along a row. A layer turns the incoming numbers xx (the activations) into output ii with a dot product along row ii:

yi  =  ∑jWij xjy_i \;=\; \sum_j W_{ij}\,x_j

Each weight is stored as a scale and a level index, Wij≈si,g cidxijW_{ij} \approx s_{i,g}\,c_{\mathrm{idx}_{ij}}, where g=⌊j/G⌋g=\lfloor j/G\rfloor is the group that column jj falls in and cc is the list of levels. Put that into the dot product and group the terms by gg:

yi  =  ∑gsi,g(∑j ∈ gcidxij xj)y_i \;=\; \sum_g s_{i,g}\Bigl(\sum_{j\,\in\,g} c_{\mathrm{idx}_{ij}}\,x_j\Bigr)

Inside the brackets every term shares the same scale, so the scale comes out of the sum. The inner sum uses only small level values and the activations, and the scale is multiplied in once per group instead of once per weight. Step 2 adds the rotation HH, which acts on the GG activations of one column block xgx_g:

yi  =  ∑gsi,g (ci,g⋅Hxg)y_i \;=\; \sum_g s_{i,g}\,\bigl(\mathbf{c}_{i,g}\cdot H x_g\bigr)

Here ci,g\mathbf{c}_{i,g} is the vector of the GG levels chosen in group gg of row ii. The rotated activations HxgH x_g do not depend on ii, so they are computed once per column block and reused by every row. That works because every row cuts its columns at the same places.

Other layouts. Groups could be shaped differently, and the same formula shows what each shape does:

Layout Can the scale come out of the sum? Notes
Run along a row (used here) Yes, one multiply per group Contiguous in memory, and the rotated activations are shared by all rows
Strip down a column No: each term of the sum comes from a different strip with its own scale One multiply per weight, and the reads are scattered
Square tile (some formats use 128×128) Yes, once per row of the tile Far fewer scales, but one outlier spoils a much larger block. This project did not measure it

To get a feel for what grouping buys, try it on a toy matrix of 12 rows and 64 columns:

Why weights are grouped. Each cell is one weight (blue negative, red positive). White lines separate the rows and the groups within each row, and every row has its own groups and scales. The number printed on each group is its scale, its largest weight (Whistle uses the group's length instead, as we will see). A gold outline marks an outlier, and a group that contains one is outlined in the scales heatmap.

 

An outline marks a group that contains an outlier.

Step 2: rotate the group

Grouping stops an outlier from spoiling its whole row, but inside its own group it still stretches the levels for the other 63 weights.

A spike costs the whole group

Here are the groups from the figure above, split into those that contain an outlier and the rest. Each dot is one group, and the tick marks the average:

Groups with an outlier lose more. It follows the matrix and the groups per row you chose above, and "New matrix", "Groups with outliers" and "Outlier size" change it too.

 

Groups of two weights, as points

To visualize axes, let’s consider the smallest case: groups of just 2 weights. Below is a row of weights cut into groups of two, like one row of the toy matrix above, drawn larger. The groups run from a short one, where one weight is much bigger than the other, to a long one, where the two weights are equal:

Groups of two weights. Each group has a number and a color, so that you can follow it through the figures below.
Groups

The first group is short, and one of its two weights is much bigger than the other. The last is long, and its two weights are equal. The groups in between change gradually from one to the other, and the number of groups goes up in twos. In the first half one weight dominates: those groups are spiky (triangles). In the second half the weights are alike: those groups are flat (circles).

Weights, with each group's scale printed on it

Scale of each group, its largest weight (the more gold, the larger)

Rounding error, after rounding each weight to one of 4 levels (the redder, the larger)

Each group is just two numbers, w1 and w2, so it can also be read as a point on a plane: the first weight is its horizontal coordinate and the second weight is its vertical one. Here are the same groups drawn that way, with the same numbers and colors:

Groups as points. Each marker is one group, placed at (w1, w2): triangles are the first half (closer to an axis), circles the second half (closer to a diagonal). Change the number of groups above and the markers move.

Quantizing the points

Now let’s see what quantization does to these points:

Quantizing groups of two weights. Each group has its own scale, as in step 1, so each gets its own square in its own color, drawn to just reach its largest weight. Each grid point is a possible stored value, and the label next to each solid marker gives its group's stored value: the two level numbers in binary, 2 bits per weight. A trail of shadows leads back to the group's real weights. The selected group's levels are numbered along its square. Click a marker to select a group.

Here is how much each group loses to rounding, the length of its trail as a share of the group's length, split into the first half (closer to an axis) and the second half (closer to a diagonal):

 

Rotating the points

Now rotate the groups and watch what happens to each half:

Rotating groups of two weights. Dashed outlines mark where the groups started, filled markers are where the rotation has taken them, and solid outlines are where each group is stored.
Rotation

 

And here is how much each group loses to rounding at the current angle, split into the first half (spiky) and the second half (flat). Turn the angle and watch the two rows move:

 

The rotation slider is only an animation: in practice a rotation is applied once, all at once.

Detour: how is this different from PCA?

If you have met principal component analysis (PCA), this may look familiar, because PCA is also a rotation. But what we are trying to do is exactly the opposite of PCA. PCA looks for the angle that makes data points more spiky, so that most of the information sits in a few coordinates. We look for the angle that flattens them instead:

PCA What we want
Goal Make groups spikier: line the axes up with the directions the data varies along most, so the energy piles into a few coordinates Make groups flatter: turn spikes away from the axes, so the energy spreads over all coordinates
Good for Compressing, by dropping the small coordinates Rounding, where one dominant coordinate stretches the levels for all the others
In short The best angle to concentrate the data The best angle to flatten it

Here are both, applied to the same 24 groups of 8 weights. PCA piles the energy into a few coordinates, and a flattening rotation spreads it out:

PCA and a flattening rotation on the same groups. Each row of a grid is one group of 8 weights (blue negative, red positive). The bars show how the total energy, the sum of squares, is divided among the 8 coordinates. Pick a data set, and compare how each rotation changes the largest coordinate and the rounding error of a group.

 

Here is the same contrast in three dimensions, where you can watch it happen:

  • Each dot is a group of three weights, red when its 2-bit rounding error is large and blue when it is small.
  • The flattening rotation is a 60° turn about the diagonal direction (1, 1, 1), the flattest rotation of three weights (the detour on three weights at the end of this step shows it).
  • Choose a data set and a rotation, then drag the slider from 0 to 100% of that rotation. “Spin the view” turns the camera so you can see the cloud from all sides.
PCA and a flattening rotation, in 3D. Gold dashed lines are the cloud's principal axes, the directions PCA turns into the coordinate axes. The readout gives the average rounding error and how the energy is divided among the three coordinates. Drag the background to look around.

 

The slider is only an animation. A real PCA step applies one matrix, fitted to the data, all at once. The flattening rotation is the 60° turn about (1, 1, 1).

PCA and a flattening rotation do opposite jobs. When the data has no preferred direction (the first data set), PCA’s axes are close to arbitrary, so it can help or hurt by chance. When the weights move together (the second), PCA does exactly what compression wants and exactly what rounding does not: it makes every group spikier.

Spiky or flat? Telling them apart

So a rotation helps spiky groups and hurts flat ones. That raises a natural question: how do we tell which kind a group is?

Detour: measuring how spiky a group is

Two numbers are commonly used. For a group of nn weights w1,…,wnw_1, \dots, w_n with length ∥w∥\lVert w\rVert:

  • Spikiness s=max⁡j∣wj∣∥w∥s = \dfrac{\max_j |w_j|}{\lVert w\rVert}: the largest weight as a share of the group’s length. It is the quantity on the curves above. It runs from 1/n1/\sqrt{n} (all weights the same size) to 1 (a single weight).
  • Crest factor c=max⁡j∣wj∣rms(w)=snc = \dfrac{\max_j |w_j|}{\mathrm{rms}(w)} = s\sqrt{n}, where rms(w)=∥w∥/n\mathrm{rms}(w) = \lVert w\rVert/\sqrt{n} is the typical size of a weight. It says how many typical weights the biggest one is worth, so it is easier to compare across group sizes.

To decide what counts as large, compare with groups of ordinary weights, which behave like Gaussian numbers. Here is what 20,000 random Gaussian groups give:

Group size nn Flat s=1/ns = 1/\sqrt{n} Typical ss: mean 99th percentile Typical cc: mean 99th percentile
2 0.71 0.90 1.00 1.27 1.41
8 0.35 0.65 0.89 1.84 2.52
64 0.125 0.33 0.45 2.61 3.60
128 0.088 0.25 0.34 2.84 3.89

A reasonable rule of thumb is to say a group of 64 has an outlier if its crest factor is above about 4: a weight more than four times the typical size, rarer than 1 group in 100 among ordinary ones. The threshold is a choice. The useful readout is the share of groups above it, measured layer by layer.

  • In two dimensions the crest factor can never exceed 2\sqrt{2}, so a rule like this has no bite. Our groups are spiky or flat only because of where we placed them.
  • With more weights the gap between a flat group and a spike grows, which is why outliers only become a sharp problem for bigger groups.

Which rotation?

Many rotations would flatten a spike, as the figures above show. Whistle uses one that needs no multiplications and no stored matrix, because a simple rule generates it: the Walsh–Hadamard matrix.Rotating before quantizing is a published technique. QuIP# (Tseng et al., 2024) uses a randomized Hadamard transform on the weights, with random signs that my version does not add. QuaRot (Ashkboos et al., 2024) rotates a whole LLM with Hadamard matrices to remove outliers for 4-bit inference. Our groups so far have had two weights, but Whistle’s have 64 or 128, so we need one for bigger groups too.

It is a square matrix whose entries are all +1 or −1, divided by n\sqrt{n} so that it keeps lengths. The 2×2 one is

H2=12[111−1]H_2 = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 & 1 \\ 1 & -1 \end{bmatrix}

and each bigger one is four copies of the previous one, with the bottom-right copy negated:

H2n=12[HnHnHn−Hn]H_{2n} = \frac{1}{\sqrt{2}}\begin{bmatrix} H_n & H_n \\ H_n & -H_n \end{bmatrix}

That gives a matrix for every size that doubles from 2: 4, 8, 16, and so on up to 64 and 128. Here are the entries, before the scaling:

Building a Walsh–Hadamard matrix by doubling. Blue is +1, red is −1. Pick a size, then click a row to see what it computes.

 

Two properties make this the right tool:

To see the first property at work: orange is a group of weights as they come, with some outliers mixed in, and blue is the same group after rotation. The histograms below the bars show where the values sit:

What rotation does to a group. Each bar is one weight. Make the group bigger and watch one tall spike turn into many short bars of random sign, then add outliers and watch the pile in the middle turn into a bell. Both rows share one scale, so the heights are directly comparable.

 

Why and when the Hadamard matrix flattens best

The figures above showed that rotating a spiky group can help. Here is what makes the Hadamard rotation the one to use, and when it works.

Why: a lone spike of any size ends up, after a Hadamard rotation, equally spread over all nn outputs, so each output holds 1/n1/\sqrt{n} of its length. The more weights share the rotation, the flatter it gets:

largest coordinatelength  =  1n\frac{\text{largest coordinate}}{\text{length}} \;=\; \frac{1}{\sqrt{n}}
Weights in the group Largest coordinate after rotating a lone spike
1 (nothing to spread over) 100% of the length
3 (no Hadamard matrix exists for this size; see the detour on three weights) at best 58%
8 35%
64 12.5% (1/8)
128 8.8%

In two dimensions the Hadamard matrix is exactly the 45° rotation with its two outputs swapped, so the two-weight picture shows when it helps. Here it is again, with that angle marked:

The Hadamard angle. Two kinds of group, as before, but now arranged symmetrically: the spiky groups (triangles) are spread evenly on both sides of an axis, and the flat groups (circles) on both sides of a diagonal. That way the Hadamard matrix, marked on the curves and shown next to the rotation matrix, sits exactly at the bottom of the spiky curve. Rotate:
Groups
Rotation

 

In two dimensions the match is exact: the Hadamard matrix is the 45° rotation with its two outputs swapped. Swapping outputs does not change the rounding error, because the grid is the same either way.

When does 45° (the Hadamard matrix) give the least spikiness? Only for certain groups, and the two curves show which:
  • Spiky groups, the triangles. They start near an axis, as far from 45° as possible, so turning by 45° brings them to the flat direction. The gold curve is lowest at 45°.
  • Flat groups, the circles. They start near a diagonal, so they are already flat, and 45° turns them into spikes. The blue curve peaks at 45°.
With 64 weights the same holds. A single big weight is a spike along one axis, and the Hadamard matrix spreads it evenly over all 64 outputs. The groups it makes spikier are the ones that already look like one of its rows: a group whose 64 weights are all equal becomes a single spike. Weights with one large outlier are far more common than that, which is why Whistle uses it.

For other group sizes the flattest a rotation can be is set by the length: each row of the matrix has length 1, so its largest entry is at least 1/n1/\sqrt{n}. The Hadamard matrix reaches that bound whenever it exists:

Group size nn Smallest largest entry Which matrix gets there
2 0.71 The Hadamard matrix, exactly: the 45° rotation above
3 0.67 No Hadamard matrix of this size exists. The 60° rotation in the detour on three weights is the flattest we found (a numerical search agrees)
4 0.5 The 4×4 Hadamard matrix
8 0.35 The 8×8 Hadamard matrix
64 0.125 The 64×64 Hadamard matrix, which is why groups of 64 (or 128) are a natural size
Detour: what about the 3D case (3 weights per group)?

Everything so far used groups of two weights, and Whistle’s groups have 64 or 128. Three weights is the case in between that you can still draw. It comes with a catch: a Hadamard matrix can’t be built for three weights. The doubling rule only makes sizes 2, 4, 8, 16 and so on, and in general a matrix of +1 and −1 entries with perpendicular rows exists only for sizes 1, 2 and multiples of 4. So for three weights we use the flattest rotation we know instead: a turn about the diagonal direction (1,1,1)(1, 1, 1), which is best at 60°.

Here are some groups of three weights, laid out like the two-weight ones. They run from a short group where one weight dominates to a long group where all three weights are equal:

Groups of three weights. Each group has a number and a color. The first half, where one weight dominates, are spiky (triangles). The second half, where the weights are alike, are flat (circles).
Groups

Weights, with each group's scale printed on it

Scale of each group, its largest weight (the more gold, the larger)

Rounding error, after rounding each weight to one of 4 levels (the redder, the larger)

Three weights make a point in 3D space, one weight per axis. Each group gets a cube of levels, 4 per axis, that just reaches its largest weight. Here is the same setup as for two weights, with the groups rotated about the diagonal:

Rotating groups of three weights. Dashed outlines mark where the groups started, filled markers are where the rotation has taken them, and solid outlines are where each group is stored. Drag the background to look around, click a marker to select a group, or switch on "spin the view" and move the slider while everything turns.
Rotation

 

And here is how much each group loses to rounding at the current angle, split into the first half (spiky) and the second half (flat):

 

The slider is only an animation: in practice a rotation is applied once, all at once.

Two things differ from the two-weight picture:

  • The flat groups sit on the diagonal that the rotation turns about, so they hardly move, and the rotation does not hurt them here the way it did in 2D.
  • At 60° the rotation matrix has entries of ±2/3 and ±1/3, which is as flat as any 3×3 rotation can be. That flattens a lone spike to two thirds of its length, a smaller gain than for bigger groups, where a Hadamard matrix takes it down to 1/n1/\sqrt{n}.

A whole matrix, rotated

Here is the toy matrix from step 1 again, with every group rotated by the same Hadamard matrix. The left half of each heatmap is the matrix as it was, and the right half is the same weights after rotation. Both halves use the plain scheme from step 1, with 16 evenly spaced levels up to the group’s largest weight:

The same matrix, before and after rotating each group. Each cell is one weight (blue negative, red positive), and white lines separate the groups. A gold outline marks an outlier on the left, and the group that contains it is outlined in the scales heatmap on both sides.

Scale of each group, its largest weight (the more gold, the larger)

Rounding error after 4 bits, per weight (redder is larger)

So the rotation is settled: one Hadamard matrix, applied to every group. Next come the levels.

Step 3: one codebook for every group

Per-group codebooks are expensive to store

The fitted levels from the start of this article come with a cost that we have not counted yet: the codebook has to be stored too. Evenly spaced levels are free in that respect, since the scale alone fixes all of them. Fitted levels are not, because every level is a number. And each group of weights has its own values, so each group would get its own codebook. Let us see what that costs.

Take a group of 64 weights, which is the group size I used for SmolLM2 (Whistle uses 128). Its 64 weights hardly spread evenly: most sit near zero and a few lie far out. So we want the fitted levels for this group, which are not evenly spaced.

To see such a set, pick two of the 64 weights at random, say the 21st and the 58th, and plot them as one point. The levels of the group, applied to both axes, make a grid around it. Then take the weights at the same two positions in another group: a different point, and a different set of levels, because the other group has its own weights. Add groups and levels below and watch the table:

Every group has its own levels. Each colored point is the 21st and 58th weight of one group of 64, and the grid of the same color is built from that group's levels, which are not evenly spaced. The numbers are made up for illustration.

 

The table keeps growing: one row for every group, one column for every level. Each number takes 32 bits, and a group’s 64 weights are stored as indices into its row. A full table is therefore a large part of what a group costs (at 16 levels, two thirds of it).

One shared codebook

That table is what the rest of this step gets rid of. The way out is to make every group look the same, so that one shared row serves them all. What “the same” has to mean is worked out in the next two parts. Then each group stores only its indices and one scale, and the codebook is stored once for the whole model. Here is what one group stores in each scheme, drawn in proportion. Move the sliders to see what the price depends on:

What is stored for one group. The same three kinds of data: the indices (blue), the group's scale (gold) and a codebook (red).

 

Detour: the arithmetic behind it

Take groups of GG weights, bb-bit indices (so 2b2^b levels), and cc-bit numbers (c=32c = 32 by default in the figure). Anything stored once for the whole model is negligible, so what counts is what is stored for every group:

shared codebook:b+cGfresh codebook per group:b+2b cG\begin{aligned} \text{shared codebook:}\quad & b + \frac{c}{G} \\ \text{fresh codebook per group:}\quad & b + \frac{2^b\,c}{G} \end{aligned}
  • The shared scheme pays for one scale per group. The fresh scheme pays for 2b2^b levels per group instead, which is 2b2^b times as much: with b=4b = 4, sixteen times.
  • The fresh codebook’s cost per weight equals the indices’ own bb bits when G=2bc/bG = 2^b c / b, which is 128 for b=4b = 4 and c=32c = 32. Below that size, the codebooks outweigh the data they describe.
  • To keep the fresh codebooks under 10% of the indices, the group would need G≥10⋅2bc/bG \ge 10 \cdot 2^b c / b, which is 1,280 weights for b=4b = 4 and c=32c = 32. But groups have to stay small for step 1 to work: an outlier spoils its whole group, so a big group means more weights spoiled.
  • The same logic rules out a different rotation for every group. A dense G×GG \times G matrix costs G2cG^2 c bits per group, which is GcG c bits per weight, always far more than bb. A single Hadamard matrix, generated by a rule, costs 0.

So sharing is what makes small groups affordable: each group may only pay for a single scale.

What shape do the values have?

Trained weights are usually roughly bell-shaped: they start out as random Gaussian numbers, and training adds up many small nudges. Here are the real weights of SmolLM2. Each group is first divided by its length, so that every group is measured on the same scale. Under the plot, the readout gives the kurtosis of the values, which is 3 for a bell curve:Kurtosis measures how heavy the tails are. It is the average of x4x^4 when the values xx are measured in standard deviations: a bell curve scores 3, a sharper peak with long tails scores more, and values packed into a narrow range score less.

SmolLM2's weights as a histogram. Each bar counts the weights in a narrow range of values, measured in standard deviations (σ) after dividing each group by its length. The dashed curve is an ideal bell curve.

 

For groups of 64, the attention and feed-forward weights come close to a bell curve, with a kurtosis of 3.2 and 3.1 (a bell curve has 3).The embedding is the exception, with a kurtosis of 5.2: a sharper peak and longer tails.

Detour: does the rotation make them even more bell-shaped?

As one would expect, yes. Each rotated weight is a mix of all the weights in its group, so by the central limit theorem the values pile up in a bell curve even when the originals do not. Here are the same weights before and after rotating each group:

SmolLM2's weights, before and after rotation. The same weights appear in both panels; the right one has each group multiplied by the Hadamard matrix. Switch on the log scale to see the tails, where the counts are tiny, or switch off the division to see what the groups' different sizes do.

 

For groups of 64:

  • Rotation trims the tails and evens out the layers. Every part lands at a kurtosis of 2.9. That is exactly what a random direction in 64 dimensions gives (3G/(G+2)3G/(G+2)), so it cannot get more bell-like than that. It sits a little below the 3 of a normal distribution because a group of fixed length cannot hold extreme values. The share of weights beyond 3σ falls from 0.50% to 0.20%, and the largest value from 7.9σ to 5.5σ.
  • It helps most where the weights are least bell-shaped. The embedding goes from a kurtosis of 5.2 to 2.9. The same holds for groups with an outlier, as in the toy examples earlier. So the rotation is less a source of the bell shape than a guarantee of it.
  • Without dividing by the length, the picture is much worse. Groups differ a lot in size, so the values are a mixture of many bell curves of different widths, which has heavy tails (kurtosis 5.4, largest value 51σ). Rotating helps (3.7, 25σ) but cannot make the groups alike: it mixes the weights inside a group and leaves the group’s size alone. That is why the length is divided out first.
  • Each group, not just the pile. A histogram of all the weights together does not prove that each group is bell-shaped, and 64 values could not prove it anyway. What can be checked is whether the groups differ from bell-shaped ones by more than chance would. The kurtosis of a rotated group of 64 has a 10th, 50th and 90th percentile of 2.33, 2.81 and 3.59 in SmolLM2, and 2.33, 2.81 and 3.60 for groups drawn from a true normal distribution. The shared codebook also needs only the pooled shape to be right: if each group were given its own best 16 levels (fitted to those same 64 values, so this flatters it), the rounding error would fall from 9.6% to 7.2%, at the cost of 12 bits per weight instead of 4.5.
  • Small groups show little. In a group of 8 no value can exceed 8=2.8σ\sqrt{8} = 2.8\sigma, so there are no tails to trim. Slide the group size up to see the effect grow.

Placing the levels

Knowing the shape settles two things:

Putting it together, the step does this:

The levels come from Lloyd’s algorithm, exactly as in the figure near the start, only run on a bell curve instead of a lumpy cloud: switch that figure to “bell curve” and step through it to see where the levels settle.

Now the same idea on the two-weight groups from step 2. The length of each group is divided out, so the dots sit on a circle of length 1, and one grid is shared by every group:

One fixed grid for every group. A group's scale is now its length, and its weights are what is left after dividing it out. The gold square is the one grid all groups share. It is sized here for a perfectly flat group, with both weights 0.71, while Whistle's grid is Gaussian-shaped and built for groups of 64. Rotate the groups and watch the arrows: they shrink for the triangles (the spiky first half) as those get flatter, and grow for the circles (the flat second half):
Groups

Weights with the length divided out, and each group's length (its scale) printed on it

Scale of each group, its length (the more gold, the larger)

Rounding error, in units of the group's length (the redder, the larger)

Rotation

 

Step 4: never rebuild the weights

Steps 1 to 3 made the weights small. This step keeps them small while the model runs, and makes using them fast.

A layer computes dot products of the weights with the incoming numbers (the activations). Because the rotation HH is orthogonal, rotating both sides leaves the dot product unchanged:

w⋅x  =  (Hw)⋅(Hx)  ≈  ℓ∑kcidxk (Hx)k\mathbf{w}\cdot\mathbf{x} \;=\; (H\mathbf{w})\cdot(H\mathbf{x}) \;\approx\; \ell\sum_k c_{\mathrm{idx}_k}\,(H\mathbf{x})_k

Here w\mathbf{w} is a group of weights, x\mathbf{x} the matching activations, ℓ\ell the group’s length, cc the codebook, and idxk\mathrm{idx}_k the index stored for weight kk. The first equality is exact. The ≈\approx is where the rounding enters: the stored level only approximates the kk-th entry of HwH\mathbf{w}, divided by ℓ\ell.

So in practice:

Here is the shortcut with actual numbers. Both routes give the same answer, but rotating the activations once is much cheaper when many rows share them:

Rotate the activations, not the weights. The top shows one group of 8 weights and the 8 activations they meet. The bottom counts the operations for groups of 64 when a matrix has many rows.

 

The rotation itself is cheap thanks to the fast Walsh–Hadamard transform. Because the matrix is built by doubling, it can be applied in log⁡2n\log_2 n rounds of add-and-subtract on pairs, the same butterfly pattern as the FFT, with no multiplications:

The fast transform on 8 numbers. Each round replaces pairs (a, b) with (a + b, a − b). Step through the rounds.

 

Taking stock: what the steps buy, and what they cost

All four steps are in place. Here is the whole pipeline on one group, then its gains and its price:

Quantizing one group of 32 weights. Dots are the values to store, ticks the available levels, and lines show where each value rounds to. Turn rotation and the codebook off to compare with plain evenly spaced levels; the average over 500 random groups is the number to watch.

 

What rotation and the codebook buy, measured on random groups of 64 and on SmolLM2’s real layers:

What it costs to store:

The shared parts are almost free: the codebook is 16 numbers, 64 bytes for the whole model, and the Hadamard matrix costs nothing, because the doubling rule generates it. The alternatives that step 3 ruled out are not. Per weight, for groups of 64 and 32-bit numbers:

What is stored Bits per weight SmolLM2
32-bit floats, for reference 32 538 MB
What we do: 4-bit indices, one scale per group, one Hadamard matrix and one codebook for everything 4.5 76 MB
A fresh 16-level codebook for every group instead of the shared one (it replaces the scale: 16 numbers instead of 1) 12 202 MB
A different dense random rotation for every group, instead of the Hadamard matrix (64×6464\times 64 numbers each) about 2,050 about 34 GB

Other choices cost other amounts. Try a few and compare with the iPod’s 256 MB of RAM:

What a model costs to store. 134.5 million weights, each taking a number of bits, plus one 32-bit scale for every group.

 

Rounding smarter: GPTQ

One more refinement is for SmolLM2 only. It changes only which level each weight rounds to, so the stored format and the kernel stay the same.

Perplexity on held-out chats, where lower is better:

SmolLM2-135M stored as Size Perplexity
32-bit floats 538 MB 7.2
Before: evenly spaced 4-bit layers, 8-bit embedding 98 MB 10.1
4-bit codebook, nearest level 76 MB 9.6
4-bit codebook, GPTQ rounding 76 MB 8.1–8.9
3-bit codebook 59–66 MB 29–81, unusable

What it adds up to

A weight now costs 4.5 bits instead of 32: a 4-bit index, plus a 32-bit scale shared by 64 weights. SmolLM2 shrank from 538 MB to 76 MB, and the codebook costs 64 bytes for the whole model. Each step earned its place:

What the numbers do and do not show:

To go further, the margin notes link the papers behind each step. The reference implementation (cq.py) and the scripts behind the histograms are in the project’s repository. The story of getting these models to answer questions on the iPod picks up where this article stops.