NiiL

Find a line by reducing prediction error

English translation

AI Applications · Week 6

This week's goal · Examine the connections between data, predictions, Loss, Gradients, and updates.

Run: Save a copy to your Drive in Colab, then run Code Cells from the top using ▶ or Shift+Enter. Double-click Text Cells marked ⟦Enter your response⟧ to edit them.

From a known line equation to learning

Knowing a line's slope and intercept lets you compute outputs from inputs. Building a model from data reverses this process: use inputs and measurements to find the slope and intercept explaining their relationship.

This exercise simplifies the problem by fixing the intercept at 0. The only value to find is slope ww, and the prediction equation is ŷ=wx\hat y=wx. This assumption allows only lines through the origin.

Changing the slope changes the prediction error for each data point. This learning process combines those errors into one MSE Loss and adjusts the slope gradually to reduce Loss. The Gradient helps determine the update direction.

More training does not automatically change a model's assumptions. If the relationship does not pass through the origin, adjusting only the slope cannot easily eliminate all errors. Compare direct computation with iterative training to examine what was found within the chosen model.

Learning by reducing prediction error

Hypothesis maps inputs to predictions and is computed as model(x) in code. MSE Loss is the Cost measuring prediction errors against targets. nn.Linear stores Weight and Bias; SGD adjusts them gradually opposite to the Gradient. This baseline fixes Bias at 0 to focus on one Weight. Omitting Bias is a condition of this example, not a rule for all Linear Regression.

1. First decide which line to find

General Linear Regression is y≈wx+by\approx wx+b. This assignment uses bias=False for y≈wxy\approx wx, which passes through the origin. Even on the same data, including or excluding b changes the problem being solved.

First decide which line to find

nn.Linear(1,1,bias=False) maps one input feature to one output. Matching x and y to (N,1) prevents unintended broadcasting.

2. Combine multiple errors into one Loss

MSE is the mean squared difference between predictions and targets across samples. A small loss means small prediction errors on this data. Keep the pre-training plot and loss as a comparison baseline.

Combine multiple errors into one Loss

Squaring gives larger errors more weight. It also prevents errors with opposite signs from canceling in the mean.

3. Update gradually using the Gradient

First predict with model(x) and compare with targets to compute Loss. Clear previous Gradients with zero_grad, then compute current Loss Gradients with backward. Finally, step uses those Gradients to update Weight. Predicting again with the updated Weight begins the next training step.

Update gradually using the Gradient

Gradients accumulate by default. In a typical single Batch update, clear previous Gradients. A fair Learning Rate comparison starts from identical initial parameters and uses the same iteration count.

4. Compare iterative training with direct computation

A linear least-squares problem can also be solved with the Pseudoinverse. Comparing pinv(A)@B with iterative training shows what learning seeks. Alongside the results and input matrix shapes, record predictions minus targets (Residuals).

Compare iterative training with direct computation

(AᵀA)⁻¹Aᵀ applies when the columns of A are independent. pinv can compute the Pseudoinverse even when they are not. Depending on the Learning Rate, iteration count, and numerical error, the iterative solution may differ slightly from the direct solution.

Check your understanding

  1. What values should go into the code, and what does each row and column of the resulting array represent? If calculating physical quantities, have you also checked the units?
  2. Can you identify which parts to rerun when you change one condition?
  3. Do the results match your expectations? If they differ, can you explain why using the output values or plots?

Official PyTorch Tutorials

Practice materials