Linear Regression

English translation

Imports

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as func
import torch.optim as opt

from google.colab import drive
drive.mount('/content/drive')

# Fix the Seed
torch.manual_seed(1)

Mounted at /content/drive

<torch._C.Generator at 0x7f729acdb070>

1 Data loading

x	y
1	1
2	2
3	3

# Enter Data manually
x_train = torch.FloatTensor([[1],[2],[3]])
y_train = torch.FloatTensor([[1],[2],[3]])

print(x_train)
print(x_train.shape)
print(y_train)
print(y_train.shape)

tensor([[1.],
        [2.],
        [3.]])
torch.Size([3, 1])
tensor([[1.],
        [2.],
        [3.]])
torch.Size([3, 1])

2 Hypothesis and Cost

$y_{hypo} = H(x) = w \cdot x + b$

Assuming linearity, the equations above must hold.
For 3 data points, $\begin{bmatrix} y_1 \\ y_2 \\ y_3 \end{bmatrix} = w \cdot \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} + b$
- Find $w$ and $b$ that satisfy this equation.
Use mean squared error as the cost function: $\frac{1}{N} \sum_{n=1}^{n=N} (y_{hypo, (n)} - y_{train, (n)})^2$

# Initialize the Model (input dim, output dim)
model = nn.Linear(1, 1)

# Define the Hypothesis
y_hypo = model(x_train)
print(list(model.parameters()))
print(y_hypo)

# Compute Cost(Mean Sqaure Error)
cost = func.mse_loss(y_hypo, y_train)
print(cost)

[Parameter containing:
tensor([[0.5153]], requires_grad=True), Parameter containing:
tensor([-0.4414], requires_grad=True)]
tensor([[0.0739],
        [0.5891],
        [1.1044]], grad_fn=<AddmmBackward0>)
tensor(2.1471, grad_fn=<MseLossBackward0>)

3 Cost Optimization

Use stochastic gradient descent as the optimizer.

# Configure the Optimizer (set learning rate = 0.01)
optimizer = opt.SGD(model.parameters(), lr=0.01)

# Repeat
for epoch in range(1000):

  # Compute Cost / mse_loss(predicted value, true value)
  y_hypo = model(x_train)
  cost = func.mse_loss(y_hypo, y_train)

  # Update the model using cost
  optimizer.zero_grad()
  cost.backward()
  optimizer.step()

  # Print intermediate results every 100 iterations
  if epoch % 100 == 99:
    params = list(model.parameters())
    w = params[0].item()
    b = params[1].item()
    print('Epoch {:4d}/{} w: {:.3f} b: {:.3f} Cost: {:.6f}'.format(
        epoch+1, 1000, w, b, cost.item()
    ))

Epoch  100/1000 w: 1.006 b: -0.014 Cost: 0.000026
Epoch  200/1000 w: 1.005 b: -0.011 Cost: 0.000016
Epoch  300/1000 w: 1.004 b: -0.008 Cost: 0.000010
Epoch  400/1000 w: 1.003 b: -0.007 Cost: 0.000006
Epoch  500/1000 w: 1.002 b: -0.005 Cost: 0.000004
Epoch  600/1000 w: 1.002 b: -0.004 Cost: 0.000002
Epoch  700/1000 w: 1.001 b: -0.003 Cost: 0.000001
Epoch  800/1000 w: 1.001 b: -0.003 Cost: 0.000001
Epoch  900/1000 w: 1.001 b: -0.002 Cost: 0.000001
Epoch 1000/1000 w: 1.001 b: -0.002 Cost: 0.000000

4 Multivariable Linear Regression

$y_{hypo} = H(\textbf{x}) = \textbf{W} \cdot \textbf{x} + b$

Under the linearity assumption, the equation above must hold: $y_{hypo} = \begin{bmatrix} w_{1} && w_{2} && w_{3} \end{bmatrix} \cdot \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} + \begin{bmatrix} b \end{bmatrix}$
- Find $w$ and $b$ that minimize the MSE cost.

# Use Numpy to load a csv file as an ndarray
dataset = np.loadtxt(
    '/content/drive/MyDrive/Colab Notebooks/courses/ML_Exp/data_linear_regression.csv', 
    delimiter=',', 
    dtype=np.float32)
# Shuffle the order randomly
np.random.shuffle(dataset)

# Convert to a torch tensor
x_train = torch.FloatTensor(dataset[:,:-1])
y_train = torch.FloatTensor(dataset[:,[-1]])

# Linear model
model = nn.Linear(3,1)

# Configure the Optimizer
optimizer = opt.SGD(model.parameters(), lr=0.00001)

# Repeat
for epoch in range(1000):
  y_hypo = model(x_train)
  cost = func.mse_loss(y_hypo, y_train)

  # model update
  optimizer.zero_grad()
  cost.backward()
  optimizer.step()

  # Print intermediate results every 100 iterations
  if epoch % 100 == 99:
    print('Epoch {:4d}/{} Cost: {:.6f}'.format(epoch+1, 1000, cost.item()))

print(y_hypo-y_train)

Epoch  100/1000 Cost: 7.438078
Epoch  200/1000 Cost: 7.359958
Epoch  300/1000 Cost: 7.286556
Epoch  400/1000 Cost: 7.217497
Epoch  500/1000 Cost: 7.152554
Epoch  600/1000 Cost: 7.091449
Epoch  700/1000 Cost: 7.033942
Epoch  800/1000 Cost: 6.979786
Epoch  900/1000 Cost: 6.928793
Epoch 1000/1000 Cost: 6.880751
tensor([[ 2.6862],
        [-3.3903],
        [-0.6640],
        [ 1.5607],
        [ 0.4769],
        [ 4.4378],
        [-0.0459],
        [-1.7203],
        [ 1.9515],
        [-0.2119],
        [ 0.2966],
        [ 0.6019],
        [ 0.5383],
        [ 2.6749],
        [-8.3141],
        [ 2.5375],
        [-0.4138],
        [-0.9415],
        [-1.0623],
        [ 2.7567],
        [-0.3143],
        [ 2.6057],
        [-3.2611],
        [-3.1444],
        [ 1.7440]], grad_fn=<SubBackward0>)

5 Assignment

Solve a matrix equation
- Solve the following matrix equation using linear regression.
  - Choose a suitable learning rate and run about 1000 epochs.
  - Compare with the solution using the pseudoinverse.
- Hint: Rewrite it as y = wx.
  - Note that x is unknown in Ax=B, whereas w is unknown in y=wx.
  - Look up how to use nn.Linear() to remove b from the linear model. $Ax=B$ $A = \begin{bmatrix}0 & 1 \\ 1 & 1 \\ 2 & 1 \\ 3 & 1 \end{bmatrix}$ $B = \begin{bmatrix}-1 \\ 0.2 \\ 0.9 \\ 2.1 \end{bmatrix}$