Linear Algebra

Linear algebraVectorsBeginner22 min

By: Anacodic Team

TL;DR — Almost every number a model produces comes out of one operation: matrix multiplication. A layer is a matrix, applying it is a matmul, and running a batch is the same matmul with more rows — which is why batching is nearly free per item and why hardware is built for this one operation. Two things follow that you will use constantly. The shape rule (n×k) @ (k×m) → (n×m) explains the single most common bug in ML code, and it is always a forgotten transpose or a missing batch dimension. And the cost is exactly n·k·m, so doubling model width quadruples the work while doubling the batch only doubles it. Norms are the other half: L2 is ordinary length, and dividing it out is precisely what turns a dot product into cosine similarity. It stops being enough when the question is about curvature or optimisation — that's calculus, not this.

1. Simple explanation

Linear algebra has a reputation for being about abstract spaces. For AI engineering it is much more concrete: it is the bookkeeping for doing the same arithmetic to a lot of numbers at once.

A vector is a list of numbers — an embedding, a set of features, a row of data. A matrix is a grid, and the useful way to read one is as a transformation: feed it a vector, get a different vector out, usually with a different number of components.

That transformation is what a neural network layer is. Not a metaphor — a layer's weights are literally a matrix, and "running the layer" is literally multiplying by it. Stack a few dozen with a nonlinearity between each and you have a model.

Analogy — a recipe scaled up. A recipe converts quantities of ingredients into quantities of dishes: a fixed table of conversion factors. Cooking for one person applies the table once. Cooking for four does not need a different recipe — it needs the same table applied to a bigger list, and doing four at once is barely more work than doing one because you set up the kitchen only once. That is exactly why models process batches: the weights (the recipe) are fetched once and reused across every row, and fetching them is the expensive part.

2. Diagram

A LAYER IS A MATRIX. APPLYING IT IS A MATMUL.

  input (1 x 3)        weights (3 x 2)         output (1 x 2)
  [2.0 1.0 0.5]   @    [ 0.5  -0.2 ]      =   [1.8  0.3]
                       [ 1.0   0.3 ]
                       [-0.4   0.8 ]
        ^                    ^                      ^
   3 features          3 in -> 2 out           2 features


A BATCH IS THE SAME MATMUL WITH MORE ROWS

  (4 x 3)          @   (3 x 2)      =    (4 x 2)
  [ 2.0  1.0  0.5]                       [ 1.80  0.30]
  [ 0.0  3.0  1.0]     unchanged         [ 2.60  1.70]
  [ 1.5 -1.0  2.0]      weights          [-1.05  1.00]
  [ 0.2  0.2  0.2]                       [ 0.22  0.18]
       ^                                       ^
  stack inputs                    every row computed independently
  as rows                         -- weights fetched ONCE for all four


THE SHAPE RULE — memorise this one line

     (n x k) @ (k x m)  ->  (n x m)
          ^^^^^^^^
       INNER dims must match; they vanish. Outer dims survive.

  (3x2) @ (4x3)  ->  ERROR: 2 != 4
                     almost always a forgotten transpose,
                     or a missing batch dimension


COST = n * k * m

       n x k @ k x m       multiply-adds
   1 x  768 @  768 x  768        589,824
  32 x  768 @  768 x  768     18,874,368
   1 x 4096 @ 4096 x 4096     16,777,216
  32 x 4096 @ 4096 x 4096    536,870,912

  double the WIDTH (k, m) -> 4x the work
  double the BATCH (n)    -> 2x the work, same weights

3. How it works

3.1 Matrix multiplication, and the one rule that matters

C = A @ B means: entry C[i][j] is the dot product of row i of A with column j of B.

Which gives the shape rule directly — you can only pair a row with a column if they are the same length:

  (n x k) @ (k x m) -> (n x m)

The inner dimensions must match and then disappear; the outer ones survive. Write the shapes down when debugging and the error becomes obvious. It is worth noting that matmul is not commutative: A @ B and B @ A are different operations and usually one of them is a shape error, which is itself a useful signal that you have something transposed.

3.2 Why batching is nearly free

Stack four inputs as four rows and multiply by the same weight matrix. Each output row depends only on its own input row — the results are identical to running them one at a time, which §4 verifies.

So why bother? Because the expensive part of the operation on real hardware is moving the weights, not the arithmetic. A (1×4096) @ (4096×4096) matmul reads 16 million weights to do 16 million multiply-adds — a terrible ratio. The same weights at batch 32 do 32× the arithmetic for the same weight read.

That is the whole reason inference servers batch requests, and it is why a single request costs far more per token than one in a full batch. See ML Inference Systems.

3.3 The cost asymmetry to keep in your head

  cost = n * k * m

Three consequences worth internalising:

change	effect
Double the batch (`n`)	2× work, weights reused
Double the width (`k` and `m`)	4× work
Double the depth (more layers)	2× work

Model width is quadratic in cost, which is why parameter counts grow so much faster than capability, and why width is the first thing compressed. Attention adds a separate quadratic term in sequence length — see Transformers Deep Dive.

3.4 Norms: how big is a vector, and why divide it out

  L2 (Euclidean) : sqrt(sum of squares)   -- ordinary length
  L1 (Manhattan) : sum of absolute values -- total displacement

L2 is the default and the one that matters for retrieval, because it is the denominator in cosine similarity. Dividing by both lengths is what removes magnitude and leaves only direction — so a long repetitive document cannot outrank a short precise one just by being bigger.

Normalising means scaling a vector to length 1. Once every vector is normalised, cosine similarity and the plain dot product are the same computation, and the dot product is cheaper — which is why many embedding APIs return normalised vectors and many vector databases default to dot-product scoring. Cosine and dot product themselves are covered in Embeddings and Cosine Similarity and Vector Search for Retrieval; this article is about the machinery underneath them.

L1 shows up in regularisation, where it drives weights to exactly zero and produces sparsity, in a way L2 does not.

3.5 Shape errors, and how to read them fast

The most common bug in ML code is a shape mismatch, and it has a small number of causes:

A forgotten transpose. You have (k×n) where (n×k) was wanted.
A missing batch dimension. A single input is (3,) when the code wants (1, 3).
Row-vector versus column-vector confusion, which is the same problem wearing a different hat.
A dimension that silently broadcast and produced the wrong answer without erroring — the dangerous one, because it doesn't crash.

The habit that fixes all of them: write the expected shape next to each line as a comment, and assert it in tests. Shape assertions are the cheapest bug-catching in numerical code.

3.6 Where this stops being enough

This article covers the operations you use; it does not cover why training works. Gradients, the chain rule and optimisation are calculus — see Calculus. Eigenvalues and decompositions (SVD, PCA) matter for dimensionality reduction and for analysing what a matrix does geometrically, and are a separate topic. And the practical performance story on real hardware is dominated by memory layout, cache behaviour and kernel choice, none of which is visible in the arithmetic here — a naive Python matmul and an optimised one differ by orders of magnitude while computing identical numbers.

4. The math

4.1 The operations

  dot(u, v)  = sum_i u_i * v_i                     two vectors -> a scalar
  (A @ B)[i][j] = sum_t A[i][t] * B[t][j]          the definition

  shape:  (n x k) @ (k x m) -> (n x m)
  cost :  n * k * m multiply-adds

  L2(v) = sqrt(sum_i v_i^2)
  L1(v) = sum_i |v_i|
  normalise(v) = v / L2(v)                         -> length exactly 1

4.2 Worked example

A layer with 3 input features and 2 outputs — so its weight matrix is (3×2).

one input  (1, 3) @ weights (3, 2) -> (1, 2)
  [[1.8, 0.3]]

The same weights on a batch of four:

four inputs (4, 3) @ weights (3, 2) -> (4, 2)
  [2.0, 1.0, 0.5]  -> [1.8, 0.3]
  [0.0, 3.0, 1.0]  -> [2.6, 1.7]
  [1.5, -1.0, 2.0] -> [-1.05, 1.0]
  [0.2, 0.2, 0.2]  -> [0.22, 0.18]

The weights did not change and the first row's answer is identical to running it alone. Batching adds rows, nothing else.

4.3 The shape rule, when it bites

  W @ batch raises: shape mismatch: (3x2) @ (4x3) -- 2 != 4

(3×2) @ (4×3): the inner dimensions are 2 and 4, so there is no valid pairing. In real code this is a transpose you forgot, and the message tells you exactly which pair disagreed.

4.4 Cost, and where it goes

       n x k @ k x m   multiply-adds
     1 x 768 @ 768 x 768         589,824
    32 x 768 @ 768 x 768      18,874,368
  1 x 4096 @ 4096 x 4096      16,777,216
  32 x 4096 @ 4096 x 4096     536,870,912

Read the two single-row entries: going from width 768 to 4096 — a factor of 5.3 — multiplies the work by 28×, because both k and m grew. Then read down the column: batch 1 to 32 multiplies it by exactly 32, and reuses the same weights throughout.

4.5 Norms

  [3.0, 4.0]                 L2  5.000   L1  7.000
  [1.0, 1.0, 1.0, 1.0]       L2  2.000   L1  4.000
  [10.0, 0.0]                L2 10.000   L1 10.000

  normalising [3,4] -> [0.6, 0.8], length 1.000000

The classic 3-4-5 triangle confirms L2 is ordinary distance. Note the second row: four components of size 1 give L2 = 2 but L1 = 4 — L1 grows with the number of nonzero components, which is exactly the property that makes it produce sparsity when used as a penalty.

5. Real code

"""Matrix multiplication is the whole computation. Shapes are the whole debugging."""


def matmul(A, B):
    """(n x k) @ (k x m) -> (n x m). The inner dimensions must agree."""
    n, k = len(A), len(A[0])
    k2, m = len(B), len(B[0])
    if k != k2:
        raise ValueError(f"shape mismatch: ({n}x{k}) @ ({k2}x{m}) -- {k} != {k2}")
    return [[sum(A[i][t] * B[t][j] for t in range(k)) for j in range(m)]
            for i in range(n)]


def shape(M):
    return (len(M), len(M[0]))


# A 'layer' is a matrix. Applying it to an input is a matmul. That is all a
# neural network does, repeatedly, with a nonlinearity between.
#                    3 input features -> 2 output features
W = [[0.5, -0.2],
     [1.0,  0.3],
     [-0.4, 0.8]]

x = [[2.0, 1.0, 0.5]]                      # ONE input, as a 1x3 row
print(f"one input  {shape(x)} @ weights {shape(W)} -> {shape(matmul(x, W))}")
print(f"  {matmul(x, W)}")

# The same weights applied to a BATCH: stack inputs as rows. Nothing else changes.
batch = [[2.0, 1.0, 0.5],
         [0.0, 3.0, 1.0],
         [1.5, -1.0, 2.0],
         [0.2, 0.2, 0.2]]
out = matmul(batch, W)
print(f"\nfour inputs {shape(batch)} @ weights {shape(W)} -> {shape(out)}")
for row_in, row_out in zip(batch, out):
    print(f"  {row_in} -> {[round(v, 2) for v in row_out]}")
print("  ...the weights did not change. Batching is ONE bigger matmul, which is")
print("  why hardware built for matmuls makes batching almost free per item.")

print("\nSHAPE ERRORS ARE THE #1 BUG, AND THE RULE IS ONE LINE")
print("  (n x k) @ (k x m) -> (n x m)   the INNER dimensions must match")
try:
    matmul(W, batch)                        # (3x2) @ (4x3): 2 != 4
except ValueError as e:
    print(f"  W @ batch raises: {e}")
print("  Read a stack trace by writing the shapes down. It is almost always")
print("  a transpose you forgot, or a batch dimension you did not add.")

print("\nCOST: why the matmul dominates everything")
print(f"  {'n x k @ k x m':>22} {'multiply-adds':>15}")
for n, k, m in [(1, 768, 768), (32, 768, 768), (1, 4096, 4096), (32, 4096, 4096)]:
    print(f"  {f'{n} x {k} @ {k} x {m}':>22} {n*k*m:>15,}")
print("  cost = n*k*m. Doubling the model width QUADRUPLES the work;")
print("  doubling the batch only DOUBLES it -- and reuses the same weights.")

print("\nNORMS: 'how big is this vector', and what each one is for")


def l2(v):
    return sum(x * x for x in v) ** 0.5


def l1(v):
    return sum(abs(x) for x in v)


for v in ([3.0, 4.0], [1.0, 1.0, 1.0, 1.0], [10.0, 0.0]):
    print(f"  {str(v):<26} L2 {l2(v):>6.3f}   L1 {l1(v):>6.3f}")
print("  L2 is ordinary length -- it is the denominator in cosine similarity,")
print("  which is how a vector's SIZE gets divided out so only DIRECTION counts.")
print("  (see the embeddings and vector-search articles for that use)")

unit = [x / l2([3.0, 4.0]) for x in [3.0, 4.0]]
print(f"\n  normalising [3,4] -> {[round(u,3) for u in unit]}, length {l2(unit):.6f}")
print("  Once every vector has length 1, cosine similarity and the dot product")
print("  are the SAME computation -- and the dot product is cheaper.")

assert shape(matmul(x, W)) == (1, 2)
assert shape(matmul(batch, W)) == (4, 2)
# Batching does not change any individual result.
assert matmul([batch[0]], W)[0] == out[0]
# The inner dimensions must agree, and mismatches raise rather than silently pad.
try:
    matmul(W, batch)
    raise AssertionError("should have raised")
except ValueError:
    pass
# Cost is exactly n*k*m multiply-adds.
assert 32 * 4096 * 4096 == 536_870_912
# A normalised vector has unit length, which is what makes cosine == dot.
assert abs(l2(unit) - 1.0) < 1e-12
assert abs(l2([3.0, 4.0]) - 5.0) < 1e-12
print("\nall assertions passed")

# Output:
#   one input  (1, 3) @ weights (3, 2) -> (1, 2)
#     [[1.8, 0.3]]
#
#   four inputs (4, 3) @ weights (3, 2) -> (4, 2)
#     [2.0, 1.0, 0.5] -> [1.8, 0.3]
#     [0.0, 3.0, 1.0] -> [2.6, 1.7]
#     [1.5, -1.0, 2.0] -> [-1.05, 1.0]
#     [0.2, 0.2, 0.2] -> [0.22, 0.18]
#     ...the weights did not change. Batching is ONE bigger matmul, which is
#     why hardware built for matmuls makes batching almost free per item.
#
#   SHAPE ERRORS ARE THE #1 BUG, AND THE RULE IS ONE LINE
#     (n x k) @ (k x m) -> (n x m)   the INNER dimensions must match
#     W @ batch raises: shape mismatch: (3x2) @ (4x3) -- 2 != 4
#     Read a stack trace by writing the shapes down. It is almost always
#     a transpose you forgot, or a batch dimension you did not add.
#
#   COST: why the matmul dominates everything
#             n x k @ k x m   multiply-adds
#       1 x 768 @ 768 x 768         589,824
#      32 x 768 @ 768 x 768      18,874,368
#     1 x 4096 @ 4096 x 4096      16,777,216
#     32 x 4096 @ 4096 x 4096     536,870,912
#     cost = n*k*m. Doubling the model width QUADRUPLES the work;
#     doubling the batch only DOUBLES it -- and reuses the same weights.
#
#   NORMS: 'how big is this vector', and what each one is for
#     [3.0, 4.0]                 L2  5.000   L1  7.000
#     [1.0, 1.0, 1.0, 1.0]       L2  2.000   L1  4.000
#     [10.0, 0.0]                L2 10.000   L1 10.000
#     L2 is ordinary length -- it is the denominator in cosine similarity,
#     which is how a vector's SIZE gets divided out so only DIRECTION counts.
#     (see the embeddings and vector-search articles for that use)
#
#     normalising [3,4] -> [0.6, 0.8], length 1.000000
#     Once every vector has length 1, cosine similarity and the dot product
#     are the SAME computation -- and the dot product is cheaper.
#
#   all assertions passed

Pure Python so the arithmetic is visible. In practice you use NumPy or a framework, where the same A @ B runs through an optimised kernel orders of magnitude faster while computing identical numbers.

6. Real-world example

A team's embedding search started returning slightly wrong neighbours after a refactor. Not catastrophically wrong — plausible results, ranked oddly — which is the hardest kind of bug to notice.

Nothing raised an error. The refactor had changed how query vectors were assembled, and a single query was now being passed as shape (768,) where the previous code produced (1, 768). NumPy broadcast it against the document matrix without complaint and computed something: numerically valid, silently a different operation.

Two properties made it slow to find. It didn't crash — a shape mismatch that raises is a good day, and this one broadcast instead. And the results were plausible, because the wrong operation still produced numbers in a sensible range with a sensible ordering, just not the right one.

They found it by adding shape assertions at the boundaries — assert q.shape == (1, dim) before the matmul — which turned a silent misbehaviour into an immediate failure. The regression test that now guards it is three lines and checks a known query returns a known neighbour.

The general lesson: the shape errors that raise are the safe ones. Broadcasting is a convenience that converts some bugs into wrong answers rather than exceptions, so assert your shapes explicitly at the points where data enters and leaves a computation.

7. Interview questions companies actually ask

Q1. What does a neural network layer actually compute? A matrix multiplication, plus a bias, plus a nonlinearity. The weights are a matrix of shape (inputs × outputs), and applying the layer is input @ W. Stacking layers means repeating that with a nonlinearity between — without the nonlinearity, a stack of matmuls collapses into a single matmul and the depth buys nothing.

Q2. State the shape rule for matrix multiplication. (n×k) @ (k×m) → (n×m): the inner dimensions must match and they disappear, the outer ones survive. It's not commutative — A @ B and B @ A are different operations and usually one of them is a shape error, which is a useful hint that something is transposed.

Q3. Why do inference servers batch requests? Because the expensive part is moving the weights, not the arithmetic. A single-row matmul reads the entire weight matrix to do comparatively little work; batch 32 does 32× the arithmetic for the same weight read. Results are unchanged — each row is computed independently — so batching is nearly free per item and is why a solo request costs more per token than one in a full batch.

Q4. If you double the model width, what happens to compute? It quadruples, because cost is n·k·m and both k and m grew. Doubling the batch only doubles it, and doubling depth only doubles it. That asymmetry is why width is the first dimension people compress and why parameter counts grow faster than capability.

Q5. What's the relationship between dot product and cosine similarity? Cosine is the dot product divided by both vectors' L2 norms, which removes magnitude and compares direction only. If your vectors are already normalised to length 1, the two are identical computations and the dot product is cheaper — which is why many embedding APIs return normalised vectors and vector databases default to dot-product scoring.

Q6. Difference between L1 and L2 norms, and when does it matter? L2 is Euclidean length, the square root of the sum of squares; L1 is the sum of absolute values. L2 is what cosine similarity divides by. L1 matters in regularisation, where it pushes weights to exactly zero and produces sparsity — L2 shrinks weights toward zero without reaching it.

Q7. Your matmul doesn't error but the results are wrong. What do you check? Broadcasting. A missing batch dimension can broadcast into a valid-but-different operation that produces plausible numbers instead of an exception. Assert shapes explicitly at the boundaries of a computation; the errors that raise are the safe ones, and the ones that quietly broadcast are what take a week to find.

8. When to use / tradeoffs

You need this when:

Reading or debugging model code — shape errors are the most common bug
Reasoning about compute cost, batch size or model width
Working with embeddings, similarity or any retrieval system
Deciding what to compress when a model won't fit

You don't need more than this when:

Consuming an API — the shapes are the provider's problem
The framework handles the arithmetic and nothing is misbehaving

Situation	Why it breaks	Do this instead
Shape mismatch error	Inner dimensions disagree	Write shapes down; find the transpose
No error, wrong results	Silent broadcasting	Assert shapes at boundaries
Single requests are slow/expensive	Weight read not amortised	Batch, if latency allows
"Just make the model wider"	Cost is quadratic in width	Depth, or a smaller width with more data
Comparing raw dot products	Length affects the ranking	Normalise, or use cosine
Naive Python matmul in a hot path	Orders of magnitude slower	NumPy or a framework kernel
Question is about gradients	This is calculus, not linear algebra	See the calculus article

Honest limits. The implementation in §5 is pure Python triple-loop matmul, chosen so the arithmetic is legible; it is thousands of times slower than an optimised kernel and computes identical numbers, which is itself the point that performance lives in memory layout rather than in the formula. The cost model counts multiply-adds and ignores everything that actually dominates real hardware — memory bandwidth, cache behaviour, kernel fusion, precision. It also treats a layer as a bare matmul, omitting the bias and the nonlinearity that make stacking meaningful. And it covers none of the decomposition side of linear algebra — eigenvalues, SVD, PCA — which matter for dimensionality reduction and for understanding what a matrix does geometrically rather than how to apply one.

A layer is a matrix; applying it is a matmul. That single operation is where nearly all model compute goes.
Shape rule: (n×k) @ (k×m) → (n×m). Inner dims must match and vanish; outer dims survive. Matmul is not commutative.
A batch is the same matmul with more rows. Results per row are identical — verified in §4.
Batching is nearly free per item because the expensive part is reading the weights, and a batch reuses them.
Cost = n·k·m. Doubling width quadruples work; doubling batch only doubles it.
L2 is ordinary length and is the denominator in cosine similarity — dividing it out is what makes ranking length-invariant.
Once vectors are normalised, cosine and dot product are the same computation, and dot is cheaper.
L1 grows with the number of nonzero components, which is why it produces sparsity as a penalty.
Shape errors that raise are the safe ones. Silent broadcasting produces plausible wrong answers — assert shapes.
Gradients and optimisation are calculus; decompositions are a separate topic.

Calculus — gradients and optimisation, the other half of how models train
Logarithms, Exponents & the Log Scale — the other maths the loss function needs
Probability & Statistics Foundations — what the outputs of these matmuls become
Embeddings and Cosine Similarity — vectors as meaning, and cosine in use
Vector Search for Retrieval — why length must be divided out
Transformers Deep Dive — attention as matmuls, plus its own quadratic term
Model Compression: Quantization, Distillation and Pruning — shrinking the matrices that §3.3 says dominate
ML Inference Systems — batching as a serving decision

Resources

Strang, G. — Introduction to Linear Algebra, and the MIT 18.06 lectures — the standard course, free online, and unusually good on matrices-as-transformations: https://ocw.mit.edu/courses/18-06-linear-algebra-spring-2010/
3Blue1Brown — Essence of Linear Algebra — the geometric intuition for what a matrix does, in about three hours: https://www.3blue1brown.com/topics/linear-algebra
Goodfellow, Bengio & Courville — Deep Learning, Ch. 2 — exactly the subset of linear algebra that deep learning uses, free online: https://www.deeplearningbook.org/contents/linear_algebra.html
NumPy — broadcasting rules, which is where the §6 failure comes from: https://numpy.org/doc/stable/user/basics.broadcasting.html
Petersen & Pedersen — The Matrix Cookbook — a reference for identities you will need and not want to re-derive: https://www.math.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf

← Previous: Information Theory