NiiL

Distinguish probability from classification criteria

English translation

AI Applications · Week 7

This week's goal · Explain why changing the threshold can change decisions for the same model.

Run: Save a copy to your Drive in Colab, then run Code Cells from the top using ▶ or Shift+Enter. Double-click Text Cells marked ⟦Enter your response⟧ to edit them.

Compute probabilities and Loss from Logits

Applying Sigmoid to the Logit computed by Linear yields a probability between 0 and 1. BCEWithLogitsLoss combines Sigmoid and Binary Cross Entropy, so feed it Logits directly during training. Use sigmoid to display probabilities or make Threshold decisions. Each CSV sample has 8 features, so the input shape is (N,8).

Why learn Sigmoid and Loss separately but compute them together?

Sigmoid converts scores into probabilities, and Binary Cross Entropy evaluates the difference between those probabilities and targets. Separating these stages clarifies prediction and error computation. In implementation, BCEWithLogitsLoss combines both operations in a numerically stable form, so do not apply Sigmoid again before calling it. The two conceptually distinct stages are computed by one function.

PyTorch BCEWithLogitsLoss

1. From numerical prediction to binary decisions

The linear layer's output z is an unrestricted score (logit). sigmoid(z) converts it to a value between 0 and 1. Comparing this with a threshold gives the final decision, 0 or 1.

From numerical prediction to binary decisions

A probability-like output and the final class are different values. If z=0z=0, then σ(z)=0.5\sigma(z)=0.5. A threshold of 0.5 gives the same decision as z>=0.

2. Check Input and Target shapes

This CSV contains 8 input features and a final label column. Set x to (N,8) and y to (N,1), both float. First print the binary labels to confirm they are 0 and 1.

Check Input and Target shapes

Preprocessing is part of the model. Compute the mean and standard deviation for Standardization from the training data and apply the same values to new data. Training Accuracy here indicates how many training samples were classified correctly. It does not imply the same accuracy on unseen data.

3. Train with Binary Cross Entropy

This exercise uses numerically stable BCEWithLogitsLoss. Feed the linear layer's logits directly to the training Loss, and apply sigmoid when displaying results. Conceptually, it combines sigmoid and binary cross entropy.

Train with binary cross entropy

Applying sigmoid again before BCEWithLogitsLoss supplies incorrect inputs. BCELoss, in contrast, requires probabilities between 0 and 1. Check the function together with its expected output range.

4. Control what changes

Do not retrain when comparing thresholds 0.3 and 0.5. To compare Learning Rates 0.001 and 0.1, create fresh models with the same Initial State and train for the same epoch count. Match other conditions so the Threshold comparison changes only the decision criterion, and the Learning Rate comparison changes only the Weight update scale during training.

Control what changes

A threshold with high Accuracy is not best for every purpose. Checking which misclassifications increased helps explain the decision criterion.

Check your understanding

  1. What values should go into the code, and what does each row and column of the resulting array represent? If calculating physical quantities, have you also checked the units?
  2. Can you identify which parts to rerun when you change one condition?
  3. Do the results match your expectations? If they differ, can you explain why using the output values or plots?

Official PyTorch Tutorials

Practice materials