Part 6: Enter the Matrix

Before moving on to more complex models, we need to take a brief tangent into the world of matrices, or — more specifically — linear algebra.

Matrices are a powerful tool for representing and manipulating data, and they form the backbone of many machine learning algorithms.

Linear algebra and ancient problems

Linear algebra can be quite an intimidating term for those without a mathematical background, but at its core, it’s just a way of representing and manipulating data in a structured way.

Its roots go back to ancient times, where people arrived at practical systems for solving everyday problems, and when you look at it through that lens, it becomes much more intuitive.

As a simple example, let’s say you are a farmer in ancient Babylon and you made the following two purchases last year:

1. 3 bags of wheat and 2 bags of barley for 10 silver coins
2. 1 bag of wheat and 4 bags of barley for 11 silver coins

Now you need to budget for the next year, which means you need to work out the cost of a single bag of wheat and a single bag of barley.

If you stare at those transactions for a while, you might notice that you can scale the second purchase to make the number of bags of wheat equal, and that allows you to focus on just the barley:

1. 3 bags of wheat and 2 bags of barley for 10 silver coins
2. 3 bags of wheat and 12 bags of barley for 33 silver coins (← Scaled by 3)

We can now basically ignore (eliminate) wheat and look at the cost of barley. When you add 10 more bags of barley, you pay 23 more silver coins.

So a bag of barley costs 23/10 = 2.3 silver coins. We can use that information to work out the cost of a bag of wheat, which is (10 - (2 * 2.3)) / 3 = 1.8 silver coins.

More formally, you’d write this as follows, using x and y for the cost of wheat and barley respectively:

Our two purchases:
3x + 2y = 10 (first purchase)
1x + 4y = 11 (second purchase)

We scale the second equation by 3 to eliminate `x`:
3x + 2y = 10
3x + 12y = 33 (second purchase scaled by 3)

Now we can remove `x` and solve for `y`:
2y - 12y = 10 - 33
-10y = -23
y = 2.3

This leaves us with the first equation to solve for `x`:
3x + 2(2.3) = 10
3x + 4.6 = 10
3x = 10 - 4.6
3x = 5.4
x = 5.4 / 3 = 1.8

These are linear equations, and the process we just went through is called solving a system of linear equations.

The reason this is relevant to machine learning is that we can represent input data as linear equations (the data points are the equations), and we can use linear algebra to solve for the unknowns — the xs and ys are the model’s parameters.

Everything beyond that is simply a question of scaling up — we can solve larger numbers of equations with more unknowns by using matrices and vectors to do it.

Scalars, Vectors, and Matrices

First, let’s recap our understanding thus far, then we can look at extending our knowledge to matrices.

Scalars

A scalar is a single number, like 5 or -3.14.

You can think of a scalar as a number on a number line — i.e. a point in one-dimensional space.

In general terms, we’d use a scalar to represent a single quantity or magnitude, such as the temperature in a room, the weight of an object, or the speed of a car.

Vectors

A vector is an ordered list of numbers, which can be thought of as a point in multi-dimensional space. For example, a 2D vector can be represented as:

const vector2D = [3, 4]; // Represents the point (3, 4) in 2D space

In a 2D space, a ‘point’ can be thought of in different ways, all correct:

• a line
• a direction
• a magnitude

This can be confusing at first, and is worth sitting with for a moment.

More generally, you can think of a vector as being able to group a quality with one or more dimensions, such as:

• the concept of ‘speed’ in a given direction (velocity)
• change of speed in a given direction (acceleration)
• the cost of a jar of peanut butter in different stores (price vector)

Dot Products

Another example, let’s say we have a shopping list of items we want to buy, and the cost of each item is represented as a vector:

const itemCost = [
   1.5, // apple
   0.5, // banana
   0.75 // orange
]

Now let’s take another vector that contains the quantity of each item we want to buy.

const quantities = [
   2, // 2 apples
   3, // 3 bananas
   1, // 1 orange
]

We can multiply the price of each item by the quantity we want to buy, and sum the results to get the total cost.

2 apples at 1.5 each = 3 
3 bananas at 0.5 each = 1.5
1 orange at 0.75 each = 0.75

Total cost = 3 + 1.5 + 0.75 = 5.

Or, programmatically:

const totalCost = itemCost.reduce(
  (acc, price, index) => acc + price * quantities[index], 0
)

You may recognise this operation as a dot product.

Dot products contain within them a very useful concept, which is the relative importance, or weight of each item in the vector. In our example above, the price of each item is weighted by the quantity we want to buy.

In a neuron, the weights are the parameters that determine how much influence each input has on the output. Same idea.

Matrices

Matrices extend this concept. Let’s imagine you have a number of vectors representing nutritional information for different foods. You can start with a table:

Each food is a row on this particular table, and each row (food) can be thought of as a vector:

const peanutButter = [8, 16, 6] // protein, fat, carbs
const jelly = [0, 0, 14] // protein, fat, carbs
const bread = [4, 2, 20] // protein, fat, carbs

The table could be expressed as a multidimensional array — which is all a matrix is:

const foodMatrix = [
  peanutButter,
  jelly,
  bread
]
// To get the nutritional info for a food, we need to grab the columns
const proteinContent = foodMatrix.map(row => row[0]) // [8, 0, 4]

So a matrix is just a way to group together a number of vectors so we can organise our thoughts and calculations.

Using vectors and matrices to solve problems

Let’s build on these examples. Let’s say we have a vector that represents the quantity of each food in a recipe, and we want to calculate the total nutritional content of the recipe.

const recipeQuantities = [
  2, // 2 servings of peanut butter
  1, // 1 serving of jelly
  2, // 2 servings of bread
]

// Protein, fat, carbs for each food
const peanutButter = [8, 16, 6] 
const jelly = [0, 0, 14] 
const bread = [4, 2, 20]

// Macronutrients in peanut butter, jelly, and bread
const protein = [8, 0, 4] 
const fat = [16, 0, 2] 
const carbs = [6, 14, 20] 

const nutritionalContent = [
  protein, fat, carbs
].map((macroNutrient) => dot(macroNutrient, recipeQuantities))

That operation — applying the dot product to each row of the matrix — is called a matrix-vector product, or mat-vec-mul for short. It’s such a common operation that we can write a helper for it that can live alongside dot:

function matVecMul(matrix, vector) {
  return matrix.map((row) => dot(row, vector))
}

Efficiency, matrices and GPUs

One big reason to use matrices is that each row of the matrix is an independent calculation. It doesn’t matter if you calculate protein, fat or carbs first.

This means the matrix calculation itself represents a sort of async promise — we can split out all the rows and calculate them in parallel, independently of each other, then assemble the results at the end.

The piece of hardware that makes this possible is the GPU (graphics processing unit). This is a type of computer chip that was designed to handle computer displays, particularly focusing on the requirements around gaming and video, where millions of pixels on a screen need to be updated very rapidly and accurately.

This is in contrast to the normal kind of chip in a computer — the CPU (central processing unit) — which handles one task at a time, but is extremely fast.

We can write matrix operations such as matVecMul in GPU code in a way that allows the protein row to be calculated at the same time as fat and carbs, and then assemble the result at the end.

The ability to calculate in parallel is a big deal when we start to scale up to large numbers of calculations, which is exactly what we do in machine learning.

Matrix multiplication

By extension, we can dot product two matrices together. This is matrix multiplication, or matMul.

This allows us to unlock an extra dimension for the data. Consider:

Matrix-Vector multiplication

• recipe (vector of ingredient quantities)
• ingredients’ nutritional values (matrix)
• output: nutritional values for the recipe (another vector — protein, fat, carbs)

Matrix multiplication

• meals with nutritional value (matrix of recipes x nutritional values)
• relative quantities per person per meal (matrix of person x recipe quantities)
• Output: nutritional content per person for each meal (person x nutritional values)

Let’s work through this.

It is an upscale of a basic linear operation, i.e.

const pancakeProtein = 6
const numberOfPancakesEaten = 3
const totalProtein = pancakeProtein * numberOfPancakesEaten // 18

…But scaled to include fats and carbs, to include multiple meals, and to include multiple people.

If we add the other macros, and extra person, we can start to see how this builds up:

// protein, fat, carbs
const pancakeMacros = [6, 2, 20] 
// Dad eats 3, Mum eats 2, Child eats 1
const pancakesPerPerson = [3, 2, 1] 

const macrosPerPerson = pancakesPerPerson.map((person) => 
  pancakeMacros.map(macro => macro * person)
)
// Gives:
[ 
  // protein, fat,  carbs 
  [18,      6,      60], // Dad
  [12,      4,      40], // Mum
  [6,       2,      20], // Child
]

Notice first that the ‘way round’ you do this calculation matters. If we had done pancakeMacros.map(...) first, we would have ended up with a matrix that was the other way round:

pancakeMacros.map((macro) => 
  pancakesPerPerson.map(person => macro * person)
)
// Gives:
[ 
  // Dad, Mum, Child
  [18, 12, 6], // protein
  [6, 4, 2], // fat
  [60, 40, 20], // carbs
]

That’s fine, but it affects the access of the data: do you want Dad’s protein to live at result[dad][protein] or result[protein][dad]? We don’t want to have to transpose it to get it into the right shape if we can avoid it.

Now, let’s scale a bit further by adding the two additional meals from the original example.

Now we can line up the servings per person:

And we can do the same calculation as before, but now we have to loop through and cross-reference each meal’s macros (the columns on the first table) with each person’s servings (the rows on the second table).

We can make this easier to conceptualise by transposing one of the tables, for example the first:

Now we can just dot product the rows from both tables. Dad’s protein is just the dot product of the protein row and Dad’s servings row:

const proteinRow = [6, 4, 10] // protein for each meal
const dadServings = [2, 1, 1] // servings for each meal
const dadProtein = dot(proteinRow, dadServings) // 26

Let’s write a quick transpose function to make this easier to work with:

function transpose(matrix) {
  // Create a new matrix with the dimensions swapped
  const numberOfRows = matrix.length
  const numberOfCols = matrix[0].length
  const transposed = Array.from(
   { length: numberOfCols },
   () => Array.from({ length: numberOfRows })
  )
  // Fill it in by swapping rows and columns
  matrix.forEach((row, i) => row.forEach((col,j) => transposed[j][i] = col))

  return transposed
}

Now we can just dot product row by row:

// Servings: person x recipe
const mealsPerPerson = [
  [2, 1, 1], // Dad
  [1, 2, 0], // Mum
  [0, 1, 2], // Child
  [1, 1, 1], // Gran
]

// Meals - recipe x protein, fat, carbs
const mealsMacros = [
  [6, 2, 20], // Pancakes
  [4, 2, 20], // Beans on Toast
  [10, 5, 30], // Spaghetti
]

// Transpose the mealsMacros matrix so we can dot product the rows
const mealsMacros_T = transpose(mealsMacros)

// Now we can dot product each person's servings with the macros for each meal
const macrosPerPerson = mealsPerPerson.map(
  personServings => mealsMacros_T.map(
    macro => dot(personServings, macro)
  )
)

Let’s wrap up by writing a matMul helper that pulls this together:

/**
  Performs a dot product of the rows from matrix A
  with the columns of matrix B.
*/
function matMul(A, B) {
  let B_T = transpose(B)
  return A.map(
    row => B_T.map(
      col => dot(row, col)
    )
  )
}

Important to note that our function here uses transpose to make the code easier to read and conceptualise. The key thing to retain is that the rows of A are dotted with the columns of B.

Shape

Notice the shapes of the matrices we multiplied there.

Shape is a big deal with matrices, so let’s take a moment to understand it.

When we multiply matrices, we dot product the rows of one with the columns of the other.

This means the lengths of the row and column must match, or the dot product won’t work. For example:

[4 rows x 3 cols] ⋅ [3 rows x 3 cols] (Works!)
[4 rows x 3 cols] ⋅ [4 rows x 3 cols] (Doesn’t work!)

That matching row and column (‘meals’ in the example) then disappears during the dot product operation. No ‘meals’ dimension in the output.

The dimensions of the output matrix will be the remaining row and column length (‘people’ and ‘macros’).

You can eyeball a mat-mul to check if it will work and to understand the shape of the output by ‘removing’ the inner dimensions of the two matrices — the outer dimensions are the ones the output will have:

Application in machine learning

A neuron is a single linear equation - $y = w·x + b$. You may already have realised we can represent a layer of neurons as a matrix.

const weights = [
  [0.2, 0.5, 0.1], // neuron 1
  [0.4, 0.3, 0.6], // neuron 2
  [0.7, 0.8, 0.9], // neuron 3
]
const inputs = [1, 2, 3] // x1, x2, x3
const biases = [0.1, 0.2, 0.3] // b1, b2, b3

const output = relu(
  matVecMul(weights, inputs).map((sum, i) => sum + biases[i])
)

Nonlinearities revisited

You’ll recall that we need a nonlinearity like relu to avoid subsequent layers collapsing into a single linear equation, and we can now see why.

Let’s start with two layers of neurons — a hidden layer and an output layer, each with their own weights (W1 and W2):

const W1 = [[1,-2],[1,1]]
const W2 = [[1,1],[2,0]]
const x = [1,2]

const hiddenLayer = matVecMul(W1, x) // [-3, 3]
const outputLayer = matVecMul(W2, hiddenLayer) // [0, -6]

We can simplify this by substituting in the hidden layer:

const outputLayer2 = matVecMul(W2, matVecMul(W1, x)) // [0, -6]

Before we continue, we need to discuss an important property of matrix multiplication: associativity.

Associativity

There is an important insight here: matrix multiplication is associative.

Remember that mathematical expressions are just a chain of operations.

Associativity means that you can group operations in any way you like and the result will be the same.

For example, you could group your getting dressed operations in any way you like:

$(trousers → socks) → shoes = trousers → (socks → shoes)$

However, grouping often does matter:

$juice + (milk + cereal) ≠ (juice + milk) + cereal$

Note that this is not about the order. If we got dressed in a different order, it wouldn’t work, however we grouped the operations:

$shoes → trousers → socks ≠ trousers → socks → shoes$

Matrices are like this, too - we can group them in any way we like, but we can’t change the order of the operations.

$W2 ⋅ (W1 ⋅ x) = (W2 ⋅ W1) ⋅ x$ (grouping change — works!)
$W2 ⋅ W1 ⋅ x ≠ W1 ⋅ W2 ⋅ x$ (order change — doesn’t work!)

Collapsing the layers

So back to our example — for us, that means:

• matVecMul(W2, matVecMul(W1, x)) equals matVecMul(matMul(W2, W1), x).

We can therefore just express the two layers as a single, new matrix, W3:

const W3 = matMul(W2, W1) // [[2, -1], [2, -4]]

…and we get the same result:

const outputLayer3 = matVecMul(W3, x) // [0, -6] - Same result!

We just lost our hidden layer, because it collapsed into a single matrix along with the weights of the output layer.

Adding a nonlinearity

Now let’s separate the layers with a nonlinear function like relu, which clips any negative value to zero:

const hiddenLayer = matVecMul(W1, x) // [-3, 3]
const activated = relu(hiddenLayer) // [0, 3] - the -3 gets clipped!
const outputLayer = matVecMul(W2, activated) // [3, 0]

Now we can’t collapse the two layers, because the relu sits between them and breaks the chain.

// with relu, we get a different result -- the layers no longer collapse
matVecMul(W2, relu(matVecMul(W1, x))) // [3, 0]
matVecMul(matMul(W2, W1), x)           // [0, -6] -- not the same!

Training

In the example from Ancient Babylon at the start of the chapter, we took the prices and quantities of wheat and barley and solved for the unknowns — the cost of an individual bag of wheat (x) or barley (y).

Our two purchases:
3x + 2y = 10 (first purchase)
1x + 4y = 11 (second purchase)
...
x = 1.8, y = 2.3

We can reframe that as:

• x and y are the unknowns (the weights of the model)
• the purchases are the inputs (the data)

We know the answers (10 coins, 11 coins, quantities of wheat and barley), and we work backwards to find the unknowns (price of a bag). This is similar to how training works using gradient descent, except it is an iterative process where we make a guess and then refine it iteratively using loss.

The forward pass works in the opposite direction: we now have the prices of the bags, so can calculate new purchases given the quantities of wheat and barley.

Now that layers are expressible as matrices, we no longer need to think about looping back through individual neurons to update their weights like we did before.

The efficiency of matrix multiplication really comes into play when we train models, since we need to do forward passes and backward passes for each loop of training.

Geometric intuition

A big conceptual leap is to take some linear equations, or a vector, or a table of data (matrix), and to plot it on a graph — i.e., give it a dimensional shape.

It’s not an immediately intuitive step to take (I don’t know if any Babylonian farmers had the idea of doing this to visualise wheat prices), but it allows us to visualise the relationships between the data points.

When we use this visualisation we can perform geometry on the data, which is a very powerful lever.

Earlier, we touched on the idea of a vector having ‘magnitude’ (length). If you think about this for a moment, it sounds mad: how can something like ‘3 bags of wheat and 2 bags of barley have a length?

The answer is that we can plot $x = 3, y = 2$ in a 2-dimensional graph, and use a triangle to calculate its length:

[Callback magnitude illustration - ED]

Why is this useful? On its own it may not be, but when we start to look at the relationships between vectors, it becomes very useful.

Similarity

Let’s imagine two vectors, const a = [2,2] and const b = [0,4].

We can imagine the ‘shadow’ of a on b, and drop a perpendicular line to it to form a right angle triangle:

This ‘shadow’ is referred to as the projection of a onto b.

As we will see shortly, the dot product of these two vectors turns out to be the length of the shadow, scaled by the length of the vector it is projected onto.

So in a very real way, we can visualise the dot product geometrically as a sort of overlap between two vectors.

The missing piece is cosine, so let’s tackle that.

Using cosine

Cosine is a mathematical function that gives us a number between -1 and 1 for how closed or open an angle is between two lines, or how aligned two directions are:

• 0 when they are perpendicular (referred to as orthogonal)
• 1 when they are pointing in the same direction
• -1 when they are pointing in opposite directions.

It doesn’t matter how long the lines are, just the angle between them. This angle is called θ (theta).

We have all the puzzle pieces now:

• we can use cosine to measure the angle between two vectors
• we can drop a line from one vector to the other to form a triangle
• the base of that triangle (the projection) indicates alignment

As the two vectors change in length and direction, the angle between them and the length of the shadow also change in a predictable way.

In other words, for any two vectors, there is a relationship between:

• the cosine of the angle between them
• their magnitude (length)
• their projection (or ‘shadow’ on to each other)

That relationship is the dot product:

a · b = length of a × length of b × cos θ

Below is a visualisation you can play with to see how the dot product changes as you change the angle between two vectors:

So we can start to see that the idea of dot products and weights actually has a visual, geometric interpretation — the more aligned two vectors are, the more they influence each other.

Conclusion

We will build on the ideas we have covered here in later chapters, but the key takeaways are:

• Matrices are a way to organise and manipulate data, and they are the backbone of many machine learning algorithms.
• Matrix multiplication is a powerful operation that allows us to combine multiple neurons into a single layer, and to perform calculations on those layers in parallel.
• The geometric interpretation of vectors and matrices allows us to visualise the relationships between data points and to measure their similarity and alignment.

Next, we will build a model architecture that addresses the limitations of the Convolutional Neural Network (CNN), and we will see how the concepts of linear algebra and matrix multiplication come into play.