Linear Regression · Learning

Topics covered

Examining how error decreases together with the resulting prediction equation helps you assess why a learning rate or bias is needed.

Overall workflow

Inputs and targets → Linear equation → Loss and gradient → Prediction error

Prediction equations and training

The Linear Regression prediction equation is ŷ=wx+b. Update w and b in the direction that reduces loss. Without bias, the prediction at input 0 is also constrained to 0, making sensor offsets difficult to represent.

$$\hat{y}=wx+b$$

Why read loss curves?

If the learning rate is too large, loss may increase or oscillate; if it is too small, loss may not decrease sufficiently within the allotted iterations. Change only one condition while keeping the data, initial values, and iteration count the same to isolate the cause.

Worked example

The provided code generates synthetic data by adding small noise to y=1.8x+.7 at $x\in\{0,.5\}$,…,3.5. Use seed=7.

import numpy as np
import matplotlib.pyplot as plt
import torch
from torch import nn
torch.manual_seed(7)
x=torch.arange(0,4,.5).reshape(-1,1)
y=1.8*x+.7+.15*torch.randn_like(x)
def fit_regression(lr=.05,bias=True,epochs=200):
    torch.manual_seed(7);model=nn.Linear(1,1,bias=bias);opt=torch.optim.SGD(model.parameters(),lr=lr);history=[]
    for _ in range(epochs):
        loss=((model(x)-y)**2).mean();opt.zero_grad();loss.backward();opt.step();history.append(loss.item())
    return model,history
model,history=fit_regression()
print('weight',model.weight.detach().numpy(),'bias',model.bias.detach().numpy())

weight [[1.8055699]] bias [0.72974294]

Reference example

View more examples

Reference materials