Linear Regression

English translation

Imports

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as func
import torch.optim as opt

from google.colab import drive
drive.mount('/content/drive')

# Fix the Seed
torch.manual_seed(1)

Mounted at /content/drive

<torch._C.Generator at 0x7f7d89bf3070>

1 Data loading

x	y
1	1
2	2
3	3

# Enter Data manually
x_train = torch.FloatTensor([[1],[2],[3]])
y_train = torch.FloatTensor([[1],[2],[3]])

print(x_train)
print(x_train.shape)
print(y_train)
print(y_train.shape)

tensor([[1.],
        [2.],
        [3.]])
torch.Size([3, 1])
tensor([[1.],
        [2.],
        [3.]])
torch.Size([3, 1])

2 Hypothesis and Cost

$y_{hypo} = H(x) = w \cdot x + b$

Assuming linearity, the equations above must hold.
For 3 data points, $\begin{bmatrix} y_1 \\ y_2 \\ y_3 \end{bmatrix} = w \cdot \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} + b$
- Find $w$ and $b$ that satisfy this equation.
Use mean squared error as the cost function: $\frac{1}{N} \sum_{n=1}^{n=N} (y_{hypo, (n)} - y_{train, (n)})^2$

# Initialize the Model (input dim, output dim)
model = nn.Linear(1, 1)

# Define the Hypothesis
y_hypo = model(x_train)
print(list(model.parameters()))
print(y_hypo)

# Compute Cost(Mean Sqaure Error)
cost = func.mse_loss(y_hypo, y_train)
print(cost)

[Parameter containing:
tensor([[0.9812]], requires_grad=True), Parameter containing:
tensor([-0.4231], requires_grad=True)]
tensor([[0.5581],
        [1.5393],
        [2.5206]], grad_fn=<AddmmBackward>)
tensor(0.2124, grad_fn=<MseLossBackward>)

3 Cost Optimization

Use stochastic gradient descent as the optimizer.

# Configure the Optimizer (set learning rate = 0.01)
optimizer = opt.SGD(model.parameters(), lr=0.01)

# Repeat
for epoch in range(1000):

  # Compute Cost / mse_loss(predicted value, true value)
  y_hypo = model(x_train)
  cost = func.mse_loss(y_hypo, y_train)

  # Update the model using cost
  optimizer.zero_grad()
  cost.backward()
  optimizer.step()

  # Print intermediate results every 100 iterations
  if epoch % 100 == 99:
    params = list(model.parameters())
    w = params[0].item()
    b = params[1].item()
    print('Epoch {:4d}/{} w: {:.3f} b: {:.3f} Cost: {:.6f}'.format(
        epoch+1, 1000, w, b, cost.item()
    ))

Epoch  100/1000 w: 1.006 b: -0.014 Cost: 0.000026
Epoch  200/1000 w: 1.005 b: -0.011 Cost: 0.000016
Epoch  300/1000 w: 1.004 b: -0.008 Cost: 0.000010
Epoch  400/1000 w: 1.003 b: -0.007 Cost: 0.000006
Epoch  500/1000 w: 1.002 b: -0.005 Cost: 0.000004
Epoch  600/1000 w: 1.002 b: -0.004 Cost: 0.000002
Epoch  700/1000 w: 1.001 b: -0.003 Cost: 0.000001
Epoch  800/1000 w: 1.001 b: -0.003 Cost: 0.000001
Epoch  900/1000 w: 1.001 b: -0.002 Cost: 0.000001
Epoch 1000/1000 w: 1.001 b: -0.002 Cost: 0.000000

4 Multivariable Linear Regression

$y_{hypo} = H(\textbf{x}) = \textbf{W} \cdot \textbf{x} + b$

Under the linearity assumption, the equation above must hold: $y_{hypo} = \begin{bmatrix} w_{1} && w_{2} && w_{3} \end{bmatrix} \cdot \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} + \begin{bmatrix} b \end{bmatrix}$
- Find $w$ and $b$ that minimize the MSE cost.

# Use Numpy to load a csv file as an ndarray
dataset = np.loadtxt(
    '/content/drive/MyDrive/Colab Notebooks/courses/AI_Appl/data_linear_regression.csv', 
    delimiter=',', 
    dtype=np.float32)
# Shuffle the order randomly
np.random.shuffle(dataset)

# Convert to a torch tensor
x_train = torch.FloatTensor(dataset[:,:-1])
y_train = torch.FloatTensor(dataset[:,[-1]])

# Linear model
model = nn.Linear(3,1)
print(list(model.parameters()))
# Configure the Optimizer
optimizer = opt.SGD(model.parameters(), lr=0.00001)

# Repeat
for epoch in range(1000):
  y_hypo = model(x_train)
  cost = func.mse_loss(y_hypo, y_train)

  # model update
  optimizer.zero_grad()
  cost.backward()
  optimizer.step()

  # Print intermediate results every 100 iterations
  if epoch % 100 == 99:
    print('Epoch {:4d}/{} Cost: {:.6f}'.format(epoch+1, 1000, cost.item()))

print(y_hypo-y_train)

[Parameter containing:
tensor([[ 0.2975, -0.2548, -0.1119]], requires_grad=True), Parameter containing:
tensor([0.2710], requires_grad=True)]
Epoch  100/1000 Cost: 15.271611
Epoch  200/1000 Cost: 14.740535
Epoch  300/1000 Cost: 14.242667
Epoch  400/1000 Cost: 13.775784
Epoch  500/1000 Cost: 13.337808
Epoch  600/1000 Cost: 12.926823
Epoch  700/1000 Cost: 12.541004
Epoch  800/1000 Cost: 12.178720
Epoch  900/1000 Cost: 11.838418
Epoch 1000/1000 Cost: 11.518639
tensor([[ 2.1901],
        [ 1.7932],
        [-9.6721],
        [-0.0190],
        [-1.1170],
        [-0.3340],
        [ 2.1892],
        [-3.1836],
        [ 1.1443],
        [ 0.3135],
        [ 4.1560],
        [ 0.3398],
        [ 1.6441],
        [ 3.6337],
        [ 4.9286],
        [-6.1586],
        [ 1.3795],
        [-6.1406],
        [-1.1264],
        [-3.2643],
        [ 2.1824],
        [ 3.9121],
        [ 0.3311],
        [ 0.6157],
        [ 1.0512]], grad_fn=<SubBackward0>)

5 Assignment

Solve a matrix equation
- Solve the following matrix equation using linear regression.
  - Choose a suitable learning rate and run about 1000 epochs.
  - Compare with the solution using the pseudoinverse.
- Hint: Rewrite it as y = wx.
  - Note that x is unknown in Ax=B, whereas w is unknown in y=wx.
  - Look up how to use nn.Linear() to remove b from the linear model. $Ax=B$ $A = \begin{bmatrix}0 & 1 \\ 1 & 1 \\ 2 & 1 \\ 3 & 1 \end{bmatrix}$ $B = \begin{bmatrix}-1 \\ 0.2 \\ 0.9 \\ 2.1 \end{bmatrix}$