NiiL

Separate training and evaluation in image classification

English translation

Create your own copy to run, edit, and record results.

Self-study material

Local setup

1. MNIST and execution scale

Keep QUICK_RUN=True for this exercise, using 1200 training samples and 300 evaluation samples. The first run downloads MNIST and requires internet access. When training on the full dataset, also record the sample count used.

A. Baseline example · Read and run first

import sys, numpy as np, matplotlib.pyplot as plt
import torch
from torch import nn
import torch.nn.functional as F
torch.manual_seed(42)
torch.set_num_threads(2)
print('Python:', sys.version.split()[0], '| PyTorch:', torch.__version__)
import torchvision
from torchvision import datasets,transforms
from torch.utils.data import DataLoader,Subset
QUICK_RUN=True # Change yourself: False for the full dataset
train_data=datasets.MNIST('mnist_data',train=True,download=True,transform=transforms.ToTensor())
test_data=datasets.MNIST('mnist_data',train=False,download=True,transform=transforms.ToTensor())
train_used=Subset(train_data,range(1200)) if QUICK_RUN else train_data
test_used=Subset(test_data,range(300)) if QUICK_RUN else test_data
train_loader=DataLoader(train_used,batch_size=64,shuffle=True,generator=torch.Generator().manual_seed(42))
test_loader=DataLoader(test_used,batch_size=128,shuffle=False)
print('Mode:', 'QUICK' if QUICK_RUN else 'FULL','train:',len(train_used),'test:',len(test_used))
img,label=train_data[0];print(img.shape,label);plt.imshow(img.squeeze(),cmap='gray');plt.show()

B. Hands-on activities

  1. Run the data preparation cell. Print the image shape and Label from train_data[0] and describe the image's three axes.
  2. Iterate through all of train_loader and collect image counts per Batch in a list. Record the first Batch size, last Batch size, and total Batch count.
  3. Calculate the expected last Batch size from len(train_used) and Batch Size 64 and compare it with the printed result.

Your prediction may describe a shape or an increasing/decreasing trend rather than an exact number. Use the example above to write code; answer explanation-only items in the Text Cell below.

# ✍ Write your own: add your code below this cell.
# You may add Code Cells if the code gets long.

C. Your record · Double-click this Text Cell to edit

2. Train and evaluate the CNN

Evaluation does not update parameters.

A. Baseline example · Read and run first

class CNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1=nn.Sequential(nn.Conv2d(1,32,3,padding=1),nn.ReLU(),nn.MaxPool2d(2))
        self.layer2=nn.Sequential(nn.Conv2d(32,64,3,padding=1),nn.ReLU(),nn.MaxPool2d(2))
        self.layer3=nn.Sequential(nn.Conv2d(64,128,3,padding=1),nn.ReLU(),nn.MaxPool2d(2,padding=1))
        self.fc1=nn.Linear(128*4*4,625);self.relu=nn.ReLU();self.dropout=nn.Dropout();self.fc2=nn.Linear(625,10)
    def forward(self,x):
        x=self.layer3(self.layer2(self.layer1(x)));x=x.flatten(1)
        return self.fc2(self.dropout(self.relu(self.fc1(x))))
model=CNN();optimizer=torch.optim.Adam(model.parameters(),lr=.001)
epochs=2 if QUICK_RUN else 5
loss_history=[];accuracy_history=[]
for epoch in range(epochs):
    model.train();total_loss=0.;correct=0;count=0
    for xb,yb in train_loader:
        logits=model(xb);loss=F.cross_entropy(logits,yb)
        optimizer.zero_grad();loss.backward();optimizer.step()
        total_loss+=loss.item()*len(yb);correct+=(logits.argmax(1)==yb).sum().item();count+=len(yb)
    loss_history.append(total_loss/count);accuracy_history.append(correct/count)
    print(epoch+1,loss_history[-1],accuracy_history[-1])
model.eval();correct=0;count=0;examples=[];mistakes=[]
with torch.no_grad():
    for xb,yb in test_loader:
        pred=model(xb).argmax(1);correct+=(pred==yb).sum().item();count+=len(yb)
        for image,truth,guess in zip(xb,yb,pred):
            item=(image.squeeze(),truth.item(),guess.item())
            if len(examples)<10:examples.append(item)
            if truth!=guess and len(mistakes)<5:mistakes.append(item)
print('TEST:',correct/count,'samples:',count)

B. Hands-on activities

  1. Run the data cell with QUICK_RUN=True, followed by the CNN training and evaluation cells. Record the training epochs, training image count, and evaluation image count.
  2. Find model.train() in the training code and model.eval() and torch.no_grad() in the evaluation code. Explain in one sentence each whether it controls Dropout behavior or Gradient recording.

To compare with the full dataset, change to QUICK_RUN=False and rerun from the data cell.

Your prediction may describe a shape or an increasing/decreasing trend rather than an exact number. Use the example above to write code; answer explanation-only items in the Text Cell below.

# ✍ Write your own: add your code below this cell.
# You may add Code Cells if the code gets long.

C. Your record · Double-click this Text Cell to edit

3. Plots and misclassifications

Inspect actual images alongside predictions.

A. Baseline example · Read and run first

fig,axes=plt.subplots(1,2,figsize=(9,3));axes[0].plot(loss_history);axes[1].plot(accuracy_history)
for ax,yl in zip(axes,['train loss','train accuracy']):ax.set(xlabel='Epoch',ylabel=yl);ax.grid()
plt.show()
for title,items in [('test examples',examples),('misclassified',mistakes)]:
    if not items:print(title,': none in evaluated subset');continue
    fig,axes=plt.subplots(1,len(items),figsize=(2*len(items),2),squeeze=False)
    for ax,(image,truth,guess) in zip(axes[0],items):ax.imshow(image,cmap='gray');ax.set_title(f'T:{truth} P:{guess}');ax.axis('off')
    fig.suptitle(title);plt.show()

B. Hands-on activities

  1. Compare the actual and predicted Labels in the visualizations of examples and mistakes against each image. For correct and incorrect examples, record which digit was predicted as which, and describe noticeable features in misclassified images.
  2. Alongside the recorded Test Accuracy, write the number of evaluation images. Distinguish evaluation on a subset under QUICK_RUN=True from evaluation on the entire Test Set.

Your prediction may describe a shape or an increasing/decreasing trend rather than an exact number. Use the example above to write code; answer explanation-only items in the Text Cell below.

# ✍ Write your own: add your code below this cell.
# You may add Code Cells if the code gets long.

C. Your record · Double-click this Text Cell to edit

Practice through repetition · Reason before running

Summarize the three conditions compared in the preceding activities in this table. You do not need to repeat experiments already completed. If comparisons are insufficient, supplement them by changing just one condition.

Changed condition Predicted shape/value/result Actual result Cause of any error
⟦Enter your response 1⟧ ⟦Enter your response⟧ ⟦Enter your response⟧ ⟦Enter your response⟧
⟦Enter your response 2⟧ ⟦Enter your response⟧ ⟦Enter your response⟧ ⟦Enter your response⟧
⟦Enter your response 3⟧ ⟦Enter your response⟧ ⟦Enter your response⟧ ⟦Enter your response⟧ 
# ✍ Code for repeated experiments

Optional exercise · Find the cause of an error and fix the code

Evaluating the same image twice with Dropout enabled

  1. After training and evaluating the CNN, add a Code Cell at the bottom of the notebook and run the following code.

    debug_image = test_used[0][0].unsqueeze(0)
for training in [True, False]:
    model.train(training)
    with torch.no_grad():
        first = model(debug_image)
        second = model(debug_image)
    print("train" if training else "eval",
          "maximum difference:", (first - second).abs().max().item())
model.eval()

  2. In a Text Cell, record the maximum absolute difference for each mode and explain why the same image produces different outputs in terms of Dropout. This code evaluates without computing Gradients or updating weights and ends in eval mode.

Check before saving

  1. Check the response fields, outputs, and plots, then run all cells in order in a fresh runtime.
  2. Save to your Drive, then download using .ipynb.

Include code, actual outputs and plots, and your own interpretation. Submit through KLAS; check the semester schedule and latest announcements for the deadline.

Filename: Name_StudentID_Week10.ipynb.