Logistic Regression · Learning

English translation

Topics covered

Distinguish the model that calculates probabilities from the criterion that triggers the final alarm to adjust the costs of misses and false alarms.

Overall workflow

Logit → Sigmoid probability → Threshold → Misses and false alarms

Score, probability, and decision

Applying sigmoid to logit z gives p=1/(1+exp(−z)). Probability and the final decision are different. Lowering the threshold yields more positive predictions but can also increase false alarms.

$p=\frac{1}{1+e^{-z}}$

Values passed to the loss function

Pass logits before Sigmoid to BCEWithLogitsLoss. Evaluate the training loss separately from the costs of misses and false alarms in deployment.

Class 1 is positive. TP counts samples with actual 1 and predicted 1; FN, actual 1 and predicted 0; FP, actual 0 and predicted 1; TN, actual 0 and predicted 0.

$\mathrm{Accuracy}=\frac{TP+TN}{TP+FP+FN+TN}$ : The fraction of all samples predicted correctly.
$\mathrm{Precision}=\frac{TP}{TP+FP}$ : The fraction of samples predicted as 1 that are actually 1.
$\mathrm{Recall}=\frac{TP}{TP+FN}$ : The fraction of actual 1 samples identified as 1.

When a denominator is 0, record the display rule you used alongside the results.

Worked example

The inputs are the provided 12×2 synthetic features; samples whose two features sum to more than 1 belong to Class 1.

import numpy as np
import matplotlib.pyplot as plt
import torch
from torch import nn
X=torch.tensor([[-2.,-1.],[-1.,-2.],[-1.,0.],[0.,-1.],[1.,1.],[1.,2.],[2.,1.],[2.,2.],[-2.,1.],[1.,-2.],[0.,2.],[2.,0.]])
y=(X.sum(dim=1)>1).float().reshape(-1,1)
def fit_binary(weights=None):
    torch.manual_seed(7);m=nn.Linear(2,1);opt=torch.optim.SGD(m.parameters(),lr=.1);history=[]
    for _ in range(200):
        losses=nn.functional.binary_cross_entropy_with_logits(m(X),y,reduction='none')
        loss=(losses if weights is None else losses*weights).mean()
        opt.zero_grad();loss.backward();opt.step();history.append(loss.item())
    return m,history
model,history=fit_binary();prob=torch.sigmoid(model(X)).detach().reshape(-1).numpy();truth=y.numpy().reshape(-1).astype(int)
print(prob)

[0.00286324 0.00291552 0.09087328 0.09238355 0.95395184 0.9918115
 0.99195766 0.99861526 0.08938526 0.09391632 0.95314825 0.9547425 ]

Reference example

View more examples

Reference materials