Find a line by reducing prediction error

Create your own copy to run, edit, and record results.

• Name: ⟦Enter here⟧
• Student ID: ⟦Enter here⟧

Self-study material

• Run Code Cells: ▶ or Shift+Enter, in order from the top.
• Edit a Text Cell: double-click → write your response → Shift+Enter.
• Response cell: enter your own code.
• After changing values: rerun that cell and the relevant cells below it.

1. Regression through the origin

A's first column is input x; the 1 in its second column is a constant for Bias. This baseline problem fixes Bias, so use only the first column as A_prime. Hypothesis is model(A_prime), and Cost is MSE.

A. Baseline example · Read and run first

import sys, numpy as np, matplotlib.pyplot as plt
import torch
from torch import nn
import torch.nn.functional as F
torch.manual_seed(42)
torch.set_num_threads(2)
print('Python:', sys.version.split()[0], '| PyTorch:', torch.__version__)
A=torch.tensor([[0.,1.],[1.,1.],[2.,1.],[3.,1.]])
B=torch.tensor([[-1.],[0.2],[0.9],[2.1]])
A_prime=A[:,0:1]; B_prime=B
print(A_prime.shape,B_prime.shape)
plt.scatter(A_prime.numpy(),B_prime.numpy());plt.xlabel('x');plt.ylabel('y');plt.show()

B. Hands-on activities

Each row of A contains an input value and the constant 1.

1. In the data cell, print A and A_prime side by side and mark the selected columns.
2. Explain the constant column's role in $\hat y=wx$ and $\hat y=wx+b$. Choose the expression corresponding to this model's bias=False.
3. Compare the Target of the sample with input $x=0$ to the values the model can produce, and state whether there is an error that training cannot eliminate.

Your prediction may describe a shape or an increasing/decreasing trend rather than an exact number. Use the example above to write code; answer explanation-only items in the Text Cell below.

# ✍ Write your own: add your code below this cell.
# You may add Code Cells if the code gets long.

C. Your record · Double-click this Text Cell to edit

• Conditions checked or changed: ⟦Enter your response⟧
• Prediction before execution: ⟦Enter your response⟧
• Value checked in the actual output/plot: ⟦Enter your response⟧
• Difference from prediction and explanation: ⟦Enter your response⟧

2. Training and records

train_linear(lr=0.01, epochs=1000) creates and trains a new model with the same Seed. For Learning Rate comparisons, change only lr and keep epochs fixed. Gradients point toward increasing Loss; SGD updates Weight in the opposite direction.

A. Baseline example · Read and run first

def train_linear(lr=0.01,epochs=1000):
    torch.manual_seed(42);model=nn.Linear(1,1,bias=False)
    opt=torch.optim.SGD(model.parameters(),lr=lr);history=[]
    for _ in range(epochs):
        pred=model(A_prime);loss=F.mse_loss(pred,B_prime)
        opt.zero_grad();loss.backward();opt.step();history.append(loss.item())
    return model,history
model,history=train_linear()
print('w:',model.weight.detach(),'MSE:',F.mse_loss(model(A_prime),B_prime).item())
plt.plot(history);plt.xlabel('Epoch');plt.ylabel('MSE');plt.grid();plt.show()

B. Hands-on activities

1. Call train_linear with Learning Rates $\eta\in\{0.001,0.01,0.1\}$, each for 1000 epochs. Keep the same Seed and model initialization inside the function.
2. Plot the three Loss curves together and tabulate the final Weight and MSE for each Learning Rate. Explain the rate of Loss decrease with reference to the curves.
3. In train_linear, immediately after loss.backward(), add code that prints model.weight.grad only during the first epoch. Relate the Gradient sign to the Weight update direction and explain why zero_grad() is needed before the next epoch.

Your prediction may describe a shape or an increasing/decreasing trend rather than an exact number. Use the example above to write code; answer explanation-only items in the Text Cell below.

# ✍ Write your own: add your code below this cell.
# You may add Code Cells if the code gets long.

C. Your record · Double-click this Text Cell to edit

• Conditions checked or changed: ⟦Enter your response⟧
• Prediction before execution: ⟦Enter your response⟧
• Value checked in the actual output/plot: ⟦Enter your response⟧
• Difference from prediction and explanation: ⟦Enter your response⟧

3. Compare with the direct solution

pinv computes the general Pseudoinverse.

A. Baseline example · Read and run first

w_pseudo=torch.linalg.pinv(A_prime)@B_prime
w_gradient=model.weight.detach().T
print('pinv:',w_pseudo,'gradient:',w_gradient,'difference:',(w_pseudo-w_gradient).abs())
plt.scatter(A_prime.numpy(),B_prime.numpy(),label='data')
plt.plot(A_prime.numpy(),model(A_prime).detach().numpy(),label='fitted');plt.legend();plt.show()
assert torch.allclose(w_pseudo,w_gradient,atol=1e-3)

B. Hands-on activities

1. Print the values and Determinant of A_prime.T @ A_prime and determine whether its Inverse exists.
2. In a three-row table, record torch.linalg.inv(A_prime.T @ A_prime) @ A_prime.T @ B_prime, torch.linalg.pinv(A_prime) @ B_prime, and the Weight trained with Learning Rate 0.01 for 1000 epochs. Also record the absolute difference between the directly computed solution and the training result.
3. For $x\in\{0,1,2,3\}$, record Input, Target, prediction, and prediction−Target in one table.

Your prediction may describe a shape or an increasing/decreasing trend rather than an exact number. Use the example above to write code; answer explanation-only items in the Text Cell below.

# ✍ Write your own: add your code below this cell.
# You may add Code Cells if the code gets long.

C. Your record · Double-click this Text Cell to edit

• Conditions checked or changed: ⟦Enter your response⟧
• Prediction before execution: ⟦Enter your response⟧
• Value checked in the actual output/plot: ⟦Enter your response⟧
• Difference from prediction and explanation: ⟦Enter your response⟧

Practice through repetition · Reason before running

Summarize the three conditions compared in the preceding activities in this table. You do not need to repeat experiments already completed. If comparisons are insufficient, supplement them by changing just one condition.

# ✍ Code for repeated experiments

Optional exercise · Find the cause of an error and fix the code

When model and input dtypes differ

1. After training the model, add a Code Cell at the bottom of the notebook and run the following code.

   print("model:", model.weight.dtype)
   for debug_input in [A_prime.double(), A_prime.float()]:
       print("Input:", debug_input.shape, debug_input.dtype)
       try:
           with torch.no_grad():
               print("Output:", model(debug_input).shape)
       except (RuntimeError, ValueError) as error:
           print(type(error).__name__, str(error))

2. In a Text Cell, tabulate the dtype of each input and whether execution succeeded, and explain the cause in one sentence in terms of the model's dtype. Input values and shapes are identical in both calls.

Check before saving

1. Check the response fields, outputs, and plots, then run all cells in order in a fresh runtime.
2. Save to your Drive, then download using .ipynb.

Include code, actual outputs and plots, and your own interpretation. Submit through KLAS; check the semester schedule and latest announcements for the deadline.

Filename: Name_StudentID_Week6.ipynb.