Mathematics for Machine Learning - Day 5
Tags: learning, machinelearning, tutorial, beginners
Published: 2024-07-12 14:27:26

![Machine Learning Math Meme](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rfkqv96ildlgfdubbd34.jpg)


## Disclaimer
Today's topic will still be split between today and tomorrow. Since you might tell from the meme, I don't know if I'm a genius, an idiot, or that's just how math maths. Usually it's the second one.

Today we'll discuss
1. Elementary Transformations
2. An example of using _row-echelon form (REF)_ to help us find the particular and general solutions
3. A missing connection between previous discussion.

Tomorrow, I'll write about what's _row-echelon form (REF)_ even is and what's the difference between using gaussian elimination to create a upper triangle matrix. Honestly, I'm just curious because that's what REF seems to me...

## Elementary Transformation
Basically, it transforms the matrix into a simpler form without changing the result. We can transform it in a few different ways
1. Exchange of two equations (rows in the matrix representing the system of equations)
2. Multiplication or divisions of an equation (row) with a constant.
3. Addition or subtractions of two equations (rows)

### Example
#### System of equations

{% katex %}
\begin{align*}
-2x_1 + 4x_2 - 2x_3 - 1x_4 + 4x_5 &= -3 \\\
4x_1 - 8x_2 + 3x_3 - 3x_4 + 1x_5 &= 2 \\\
1x_1 - 2x_2 + 1x_3 - 1x_4 + 1x_5 &= 0 \\\
1x_1 - 2x_2 + 0x_3 - 3x_4 + 4x_5 &= a
\end{align*}
{% endkatex %}

#### Matrix-vector multiplication form
Let's divide the variables and the constants!

{% katex %}
\begin{pmatrix}
-2 & 4 & -2 & -1 & 4 \\\
4 & -8 & 3 & -3 & 1 \\\
1 & -2 & 1 & -1 & 1 \\\
1 & -2 & 0 & -3 & 4
\end{pmatrix}
\begin{pmatrix}
x_1 \\\
x_2 \\\
x_3 \\\
x_4 \\\
x_5
\end{pmatrix}
=
\begin{pmatrix}
-3 \\\
2 \\\
0 \\\
a
\end{pmatrix}
{% endkatex %}

## Augmented matrix form
Finally some new UI design of the matrix, looks more elegant this way.

{% katex %}
\left[
\begin{array}{ccccc|c}
-2 & 4 & -2 & -1 & 4 & -3 \\\
4 & -8 & 3 & -3 & 1 & 2 \\\
1 & -2 & 1 & -1 & 1 & 0 \\\
1 & -2 & 0 & -3 & 4 & a
\end{array}
\right]
{% endkatex %}

### Now what?
Now we'll transform the augmented matrix using elementary transformation to becoming a _REF_!

Before we continue, remember! R stands for row and the notation will be Rx -> Ry, which means the value of Rx will be put in row Ry. Now let's continue.

#### Swap R1 and R3

{% katex %}
\overset{R_1 \leftrightarrow R_3}{\left[
\begin{array}{ccccc|c}
1 & -2 & 1 & -1 & 1 & 0 \\\
4 & -8 & 3 & -3 & 1 & 2 \\\
-2 & 4 & -2 & -1 & 4 & -3 \\\
1 & -2 & 0 & -3 & 4 & a
\end{array}
\right]}
{% endkatex %}


#### Multiple Row Multiplication

{% katex %}
\overset{\begin{array}{l}
R_2 - 4R_1 \rightarrow R_2 \\\
R_3 + 2R_1 \rightarrow R_3 \\\
R_4 - R_1 \rightarrow R_4
\end{array}}{\left[
\begin{array}{ccccc|c}
1 & -2 & 1 & -1 & 1 & 0 \\\
0 & 0 & -1 & 1 & -3 & 2 \\\
0 & 0 & 0 & -3 & 6 & -3 \\\
0 & 0 & -1 & -2 & 3 & a
\end{array}
\right]}
{% endkatex %}

#### Subtract R4 with R2 and R3

{% katex %}
\overset{R_4 - R_2 - R_3 \rightarrow R_4}{\left[
\begin{array}{ccccc|c}
1 & -2 & 1 & -1 & 1 & 0 \\\
0 & 0 & -1 & 1 & -3 & 2 \\\
0 & 0 & 0 & -3 & 6 & -3 \\\
0 & 0 & 0 & 0 & 0 & a - 2 + 3
\end{array}
\right]}
{% endkatex %}

#### Scaling

{% katex %}
\overset{\begin{array}{l}
-R_2 \rightarrow R_2 \\\
-\frac{1}{3}R_3 \rightarrow R_3
\end{array}}{\left[
\begin{array}{ccccc|c}
1 & -2 & 1 & -1 & 1 & 0 \\\
0 & 0 & 1 & -1 & 3 & -2 \\\
0 & 0 & 0 & 1 & -2 & 1 \\\
0 & 0 & 0 & 0 & 0 & a + 1
\end{array}
\right]}
{% endkatex %}

---

### Okay Terra, you made me read all of this complex mumbo jumbo, what do I do now?
First off, you need to relax. Secondly, Given that a+1 = 0 we can create a solution for `a = -1`!

But... I need to confess, the authors didn't provide a step by step solution to their answers :( But I did create one for my solution :D so you be the judge if I'm correct or not by proving it yourselves as well!

#### Particular solution
Remember on the previous chapter we used a type of identity matrix? well, there isn't one here and since on the pages I'm reading on didn't have the steps and only the solution, I'll show you know I did it :D

##### Find Ax = B

{% katex %}
B = \begin{pmatrix}
0 \\\
-2 \\\
1 \\\
0
\end{pmatrix} 
{% endkatex %}

and assume that:
{% katex %}
x_1 = a \\\
x_2 = b \\\
x_3 = c \\\
x_4 = d \\\
x_5 = e
{% endkatex %}

We can split the result to form an equation like this, where each x is associated to a specific column.

{% katex %}
\begin{pmatrix}
0 \\\
-2 \\\
1 \\\
0
\end{pmatrix} = a
\begin{pmatrix}
1 \\\
0 \\\
0 \\\
0
\end{pmatrix} + b
\begin{pmatrix}
-2 \\\
0 \\\
0 \\\
0
\end{pmatrix} + c
\begin{pmatrix}
1 \\\
1 \\\
0 \\\
0
\end{pmatrix} + \underbrace{d
\begin{pmatrix}
-1 \\\
-1 \\\
1 \\\
0
\end{pmatrix} + e
\begin{pmatrix}
1 \\\
-3 \\\
-2 \\\
0
\end{pmatrix}}_{\text{Let's focus on this part}}
{% endkatex %}

##### Let's assume that d and e are equal to 1 and add both of these to see what we get

{% katex %}
1
\begin{pmatrix}
-1 \\\
-1 \\\
1 \\\
0
\end{pmatrix} + 1
\begin{pmatrix}
1 \\\
-3 \\\
-2 \\\
0
\end{pmatrix} = 
\begin{pmatrix}
0 \\\
2 \\\
-1 \\\
0
\end{pmatrix}
{% endkatex %}

Notice that the result of adding these result in the negative value of the result we're searching for?

{% katex %}
\begin{pmatrix}
0 \\\
2 \\\
-1 \\\
0
\end{pmatrix} = -1
\begin{pmatrix}
0 \\\
-2 \\\
1 \\\
0
\end{pmatrix}
{% endkatex %}

##### Finding d and e
Assume:

{% katex %}
x = \begin{pmatrix}
-1 \\\
-1 \\\
1 \\\
0
\end{pmatrix}
{% endkatex %}

and

{% katex %}
y = \begin{pmatrix}
1 \\\
-3 \\\
-2 \\\
0
\end{pmatrix}
{% endkatex %}

Given:

{% katex %}
dx + ey = -(x + y)
{% endkatex %}

{% katex %}
dx + x + ey + y = 0
{% endkatex %}

{% katex %}
x(d+1) + y(e+1) = 0
{% endkatex %}

Since we know that x and y are not zero, the only way to fulfill this equation is by having (d+1) and (e+1) = 0.

So, both

{% katex %}
d = -1 \quad \text{and} \quad e = -1
{% endkatex %}

#### Particular Solution Result
Great! now we can conclude that the particular solution is:

{% katex %}
x \text{ (My answer)} = \begin{pmatrix}
0 \\\
0 \\\
0 \\\
-1 \\\
-1
\end{pmatrix}
{% endkatex %}


Though... there's a small caveat. This isn't the author's answer, it's mine, I tried finding the solution in the sentences or maybe I slipped a page or two, but I think it's after going in-depth to _row echelon form_ first, so I'm not sure about the method used to solve it.

{% katex %}
x \text{ (Author's answer)} = \begin{pmatrix}
2 \\\
0 \\\
-1 \\\
1 \\\
0
\end{pmatrix}
{% endkatex %}

#### General Solutions
I won't go into depth in this one, considering I also got the answer of Ax = 0 different. For more context, I'm using the same method as the yesterday with the lambda is from column 4 and column 5 (Since only those two contain the most non-zero column) and the column will become -1.

I'm really not sure if this is just math being math or I'm doing something wrong!

##### My answer

{% katex %}
\left\( x \in \mathbb{R}^5 \mid x = 
\begin{pmatrix}
0 \\\
0 \\\
0 \\\
-1 \\\
-1
\end{pmatrix} + \lambda_1 
\begin{pmatrix}
1 \\\
0 \\\
-\frac{5
}{2} \\\
0 \\\
\frac{1}{2}
\end{pmatrix} + \lambda_2 
\begin{pmatrix}
-2 \\\
0 \\\
5 \\\
2 \\\
-1
\end{pmatrix}, \; \lambda_1, \lambda_2 \in \mathbb{R} \right\)
{% endkatex %}

##### Author's answer

{% katex %}
\left\( x \in \mathbb{R}^5 \mid x = 
\begin{pmatrix}
2 \\\
0 \\\
-1 \\\
1 \\\
0
\end{pmatrix} + \lambda_1 
\begin{pmatrix}
2 \\\
1 \\\
0 \\\
0 \\\
0
\end{pmatrix} + \lambda_2 
\begin{pmatrix}
2 \\\
0 \\\
-1 \\\
2 \\\
1
\end{pmatrix}, \; \lambda_1, \lambda_2 \in \mathbb{R} \right\)
{% endkatex %}

### Ending note:
That's the end, I spent too much time racking my brain about the answer and might as well post it since I might get an answer on what kind of behavior is this.

---

## Acknowledgement
I can't overstate this: I'm truly grateful for this book being open-sourced for everyone. Many people will be able to learn and understand machine learning on a fundamental level. Whether changing careers, demystifying AI, or just learning in general, this book offers immense value even for _fledgling composer_ such as myself. So, Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong, thank you for this book.

Source:
Deisenroth, M. P., Faisal, A. A., &#38; Ong, C. S. (2020). Mathematics for Machine Learning. Cambridge: Cambridge University Press.
https://mml-book.com