Linear Regression

Learning objectives

Examining how error decreases together with the resulting prediction equation helps you assess why a learning rate or bias is needed.

Sensor calibration with bias

Starting conditions

The provided code generates synthetic data by adding small noise to y=1.8x+.7 at $x\in\{0,.5\}$,…,3.5. Use seed=7.

Procedure

1. Run the provided 200-epoch baseline example and plot the loss curve and prediction line.
2. Record the learned w and b separately from the data generation coefficients in a table.
3. Given the noise, explain whether the learned coefficients should exactly match the data generation coefficients.

Results to retain

• Plots of the data, prediction line, and loss, plus a coefficient table.

import numpy as np
import matplotlib.pyplot as plt
import torch
from torch import nn
torch.manual_seed(7)
x=torch.arange(0,4,.5).reshape(-1,1)
y=1.8*x+.7+.15*torch.randn_like(x)
def fit_regression(lr=.05,bias=True,epochs=200):
    torch.manual_seed(7);model=nn.Linear(1,1,bias=bias);opt=torch.optim.SGD(model.parameters(),lr=lr);history=[]
    for _ in range(epochs):
        loss=((model(x)-y)**2).mean();opt.zero_grad();loss.backward();opt.step();history.append(loss.item())
    return model,history
model,history=fit_regression()
print('weight',model.weight.detach().numpy(),'bias',model.bias.detach().numpy())

Record results and self-check

Model:

w:

b:

Training MSE:

Prediction for $x=0$:

Points to check

• Did you check that the prediction equation is wx+b?
• Did you recalculate the final loss using the final model?

Comparing learning rates for the same model

Starting conditions

Fix bias=True and epoch=200, and compare lr=.01,.05,.1.

Procedure

1. Fix the initial seed and data for all runs.
2. Record the final MSE and the loss at epochs 1, 20, and 200.
3. Distinguish the smallest final loss from the fastest initial decrease.

Results to retain

• A 3-row table and 1 loss curve.

for lr in [.01,.05,.1]:
    m,h=fit_regression(lr=lr)
    print(lr,h[0],h[19],((m(x)-y)**2).mean().item())
# TODO: Record the coefficients and Loss plot.

Record results and self-check

lr:

Loss@1:

Loss@20:

Final MSE:

w:

b:

Points to check

• Did you initialize a new model for each condition?
• Are gradients reset at every iteration?

Error remaining without bias

Starting conditions

Using the same data, keep lr=.05 and epoch=200, and compare bias=True/False.

Procedure

1. Plot the two prediction lines in the same figure.
2. Record the predictions at $x\in\{0,.25,1.25,2.75\}$ in a table.
3. Explain the difference between increasing training iterations and allowing bias in the model.

Results to retain

• Plots for the two models, a prediction table, and a brief interpretation.

query=torch.tensor([[0.],[.25],[1.25],[2.75]])
for use_bias in [True,False]:
    m,h=fit_regression(bias=use_bias);print(use_bias,m(query).detach().reshape(-1))

Record results and self-check

Use bias:

Training MSE:

$x=0$:

$x=.25$:

$x=1.25$:

$x=2.75$:

Points to check

• Did you avoid mistaking the generated data for a problem without bias?
• Did you distinguish model assumptions from optimization conditions?