[Week 8] Starting Classification: Building a Logistic Regression Model 🎯

Welcome to Week 8!
Last time, we addressed a regression problem that predicts continuous numerical values
using a Linear Regression model.

Today, we explore another major Machine Learning task: Classification.
The goal is to assign given data to one of the specified categories.

Implement and evaluate Logistic Regression, a basic but important classification model,
using scikit-learn.

🤔 Module 1: Why Is Linear Regression Insufficient? (Classification)

What is classification? - Assigning data to one of a predefined set of categories.

• Examples: Is an email spam? (Binary classification.) Is the animal in a photo a dog, cat, or bird? (Multiclass classification.)

Limitations of Linear Regression:

• Linear Regression is designed to predict continuous numerical values. ($y = Wx + b$)\
• Predictions may fall within $(-\infty, +\infty)$.\
• Applied to classification (e.g., 0 or 1), it can produce meaningless values outside the categories,
  such as 0.5, 1.7, or -0.2.

Classification therefore requires a new model that restricts outputs to a particular range (e.g., between 0 and 1).

# Import libraries
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression

# Generate synthetic classification data (pass/fail based on study hours)
# Study hours (feature X)
study_hours = np.array(
    [[1], [2], [3], [4], [5], [6], [7], [8], [9], [10]]
    )
# Pass status (ground truth y, 0: fail, 1: pass)
pass_fail = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1, 1])

# Try training a Linear Regression model
linear_model = LinearRegression()
linear_model.fit(study_hours, pass_fail)

# Visualize the results
plt.scatter(study_hours, pass_fail,
            c=pass_fail, cmap='coolwarm', s=100,
            label='Actual Data (0:Fail, 1:Pass)')
# Prediction line from the Linear Regression model
plt.plot(study_hours, linear_model.predict(study_hours),
         color='black', label='Linear Regression Prediction')

plt.title("Classification by regression")
plt.xlabel("Study time")
plt.ylabel("Pass or not (0 or 1)")
plt.yticks([0, 1]) # Fix the y-axis ticks at 0 and 1
plt.legend()
plt.grid(True)
plt.show()

print("결과 해석:")
print("- 예측 직선이 y=0과 y=1 범위를 크게 벗어납니다.")
print("- 예를 들어 공부 시간이 1시간일 때 예측값은 0보다 작고,")
print("  10시간일 때는 1보다 큰 값이 나옵니다.")
print("=> 즉, 분류 문제에 직접 적용하기 부적합합니다.")

<Figure size 640x480 with 1 Axes>

결과 해석:
- 예측 직선이 y=0과 y=1 범위를 크게 벗어납니다.
- 예를 들어 공부 시간이 1시간일 때 예측값은 0보다 작고,
  10시간일 때는 1보다 큰 값이 나옵니다.
=> 즉, 분류 문제에 직접 적용하기 부적합합니다.

Module 2: Logistic Regression and the Sigmoid ($\sigma$) Function

Logistic Regression (Logistic Regression):

• Despite 'Regression' in its name, this model is actually used for classification.\
• Pass the Linear Regression prediction $(z = Wx+b)$ through a Sigmoid function
  to convert the output into a probability between 0 and 1.

Sigmoid Function:

• $\sigma(z) = \frac{1}{1 + e^{-z}}$ (where $z = Wx+b$)\
• $e$ is Euler's number (approximately 2.718), and $e^{-z}$ approaches 0 as $z$ increases,
  and approaches infinity as $z$ decreases.\
• Because of this property, when $z$ becomes very large, the denominator becomes $1+0=1$, giving $\sigma(z) \approx 1$,
  and when $z$ becomes very small, the denominator becomes $1+\infty=\infty$, giving $\sigma(z) \approx 0$.\
• This is why the sigmoid function always forms an S-shaped curve between 0 and 1.

# Define the sigmoid function
def sigmoid(z):
  return 1 / (1 + np.exp(-z))

# Generate z values over the range (-10 ~ 10)
z = np.linspace(-10, 10, 100)
# Calculate the sigmoid function for each z value
sigma_z = sigmoid(z)

# Plot the sigmoid function
plt.plot(z, sigma_z)
plt.title("Sigmoid Function")
plt.xlabel("z (Linear Regression Output: Wx + b)")
plt.ylabel("Sigmoid Output (Probability)")
plt.grid(True)
# Mark the point where probability is 0.5 (z=0) with lines
plt.axhline(y=0.5, color='red', linestyle='--',
            label='Decision Boundary (Prob=0.5)')
plt.axvline(x=0, color='gray', linestyle=':')
plt.legend()
plt.show()

print("결과 해석:")
print("- 선형 회귀의 결과(z)가 아무리 커져도 확률은 1에 수렴합니다.")
print("- 결과(z)가 아무리 작아져도 확률은 0에 수렴합니다.")
print("- 결과(z)가 0일 때, 확률은 정확히 0.5가 됩니다.")

<Figure size 640x480 with 1 Axes>

결과 해석:
- 선형 회귀의 결과(z)가 아무리 커져도 확률은 1에 수렴합니다.
- 결과(z)가 아무리 작아져도 확률은 0에 수렴합니다.
- 결과(z)가 0일 때, 확률은 정확히 0.5가 됩니다.

Decision Boundary:

• Typically, probabilities of 0.5 or higher are classified as 1 and those below 0.5 as 0.\
• The point corresponding to this criterion (probability=0.5, or $z=0$) is called the decision boundary.

Cost Function:

• Linear Regression used MSE (mean squared error),
  whereas Logistic Regression uses the Cross-Entropy cost function, which is better suited to probability prediction.\
• Cross-Entropy measures how far the model's predicted probabilities differ from the ground truth (0 or 1),
  and guides training by imposing a larger penalty for incorrect predictions.
  (The formula is omitted because it is beyond this course's scope.)

🩺 Module 3: Logistic Regression Modeling Step by Step (Breast Cancer Diagnosis)

Now solve a real classification problem using scikit-learn.
Using the breast cancer diagnosis dataset, build a model that classifies tumors as malignant or benign from their features.

Follow the 5-step Machine Learning workflow again.

[Supplement] NumPy Indexing [:, 0:1] Explained

Before feeding data into the scikit-learn model,
let's take another detailed look at the X = cancer.data[:, 0:1] code we use.

1. Shape of cancer.data - The breast cancer dataset contains multiple samples (tumors),
each with multiple features (radius, texture, etc.).

• Thus, cancer.data has Shape (number of samples, number of features) and is a 2-dimensional array (Matrix).
  • Example: (569, 30) -> 569 tumor samples, each with 30 feature values

2. [:, 0] vs [:, 0:1]

• [:, 0]: "For every row, select just one column, at Index 0."
  • Result: [value1, value2, ..., value569] -> 1-dimensional array (Vector), Shape: (569,)
  • When the result is a single column, NumPy reduces its dimensionality and returns a 1-dimensional array.
• [:, 0:1]: "For every row, select columns starting at Index 0 and ending before Index 1."
  • Result: [[value1], [value2], ..., [value569]] -> 2-dimensional array (Matrix), Shape: (569, 1)
  • With slicing (:), NumPy preserves the original 2-dimensional structure even when selecting just one column.

Conclusion: For scikit-learn models, input X must always be a 2-dimensional array with shape (sample count, feature count).
Even when selecting one feature, use slicing such as [:, 0:1] to preserve dimensions.

# Import additional required libraries
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression # Change the model!
from sklearn.model_selection import train_test_split
# The metrics module contains performance metrics.
from sklearn.metrics import (accuracy_score, precision_score,
                             recall_score, f1_score,
                             confusion_matrix, classification_report)

# 1. Prepare data (breast cancer dataset)
cancer = load_breast_cancer()

# Use only 'mean radius' as a 2D array from the available features
X = cancer.data[:, 0:1]
y = cancer.target       # 0: Malignant, 1: Benign

# Explore the data (omitted here, but important in actual analysis)
print(f"데이터셋 크기 (샘플 수, 특징 수): {X.shape}")
print(f"타겟 종류 및 개수: {np.unique(y, return_counts=True)}")

데이터셋 크기 (샘플 수, 특징 수): (569, 1)
타겟 종류 및 개수: (array([0, 1]), array([212, 357]))

# 2. Split data into 'practice problems' and a 'real exam' at a 7:3 ratio
# stratify=y : Preserve the original class proportions (the ratio of 0 and 1) when splitting
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

print(f"Train 데이터 크기: {X_train.shape}")
print(f"Test 데이터 크기: {X_test.shape}")

Train 데이터 크기: (398, 1)
Test 데이터 크기: (171, 1)

# 3. Create a Logistic Regression model object
model = LogisticRegression()

# Train the model on 'practice problems' (fit)
# Internally, it calculates probabilities using the sigmoid function,
# then compares them with the ground truth (0 or 1) to find the optimal W, b.
model.fit(X_train, y_train)

print("로지스틱 회귀 모델 학습 완료!")

로지스틱 회귀 모델 학습 완료!

✅ Module 4: Evaluating Classification Models with Confusion Matrices

Step 4: Evaluate Performance

To understand classification performance, go beyond 'how many were correct (Accuracy)'
and examine 'how' predictions were right or wrong.

Confusion Matrix (Confusion Matrix):

• A table comparing the model's predictions with the actual ground truth.\
• Binary classification (0/1) is represented by a 2x2 matrix.

Let's understand this through the example of a doctor's cancer diagnosis.

• Actual: Whether the patient actually has cancer
• Predicted: Whether a doctor diagnosed cancer based on test results

1. TN (True Negative): A doctor correctly diagnoses a healthy (Negative) person as healthy (Negative)

2. FP (False Positive): The doctor incorrectly diagnoses someone who is actually healthy (Negative) as having cancer (Positive)
   (Telling a healthy person they have cancer - Type I Error)

3. FN (False Negative): For someone who actually has cancer (Positive), the doctor incorrectly diagnoses them as healthy (Negative)
   (Missing a cancer patient - Type II Error)

4. TP (True Positive): A doctor correctly diagnoses an actual cancer patient (Positive) as having cancer (Positive)

Key Performance Metrics:

1. Precision (Precision): Among people diagnosed by the doctor as having cancer (TP + FP), the fraction who actually have cancer (TP)\

1. $Precision = \frac{TP}{TP + FP}$\
2. How precise was the doctor's diagnosis? (FP must be reduced.)

2. Recall (Sensitivity): Among actual cancer patients (TP + FN), the proportion correctly diagnosed with cancer (TP)\

1. $Recall = \frac{TP}{TP + FN}$\
2. How completely did the doctor identify cancer patients? (Reduce FN)

3. Accuracy: The fraction of all predictions that are correct (TP + TN)\

1. $Accuracy = \frac{TP + TN}{TP + TN + FP + FN}$\
2. The most intuitive metric, but it can distort performance on imbalanced data (e.g., 99% normal, 1% defective).

4. F1 Score: The harmonic mean of Precision and Recall. Used to consider both metrics in balance.\

1. $F_1 = 2 \times \frac{Precision \times Recall}{Precision + Recall}$\
2. F1 Score is high when both Precision and Recall are high.

# Make predictions on the 'real exam' data
y_pred = model.predict(X_test)

# 1. Calculate and display the Confusion Matrix
# TN  FP
# FN  TP
conf_matrix = confusion_matrix(y_test, y_pred)
print("--- Confusion Matrix ---")
print(conf_matrix)
tn, fp, fn, tp = conf_matrix.ravel() # Assign each value to a separate variable

--- Confusion Matrix ---
[[51 13]
 [11 96]]

# Visualize a heatmap using seaborn
import seaborn as sns

plt.figure(figsize=(6, 5))
sns.heatmap(conf_matrix, annot=True, fmt='d', cmap='Blues',
            xticklabels=['Predicted Malignant(0)', 'Predicted Benign(1)'],
            yticklabels=['Actual Malignant(0)', 'Actual Benign(1)'])
plt.xlabel('Predicted Label')
plt.ylabel('True Label')
plt.title('Confusion Matrix Visualization')
plt.show()

<Figure size 600x500 with 2 Axes>

# 2. Calculate the main performance metrics
accuracy = accuracy_score(y_test, y_pred)
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
f1 = f1_score(y_test, y_pred)

print("\n--- 주요 성능 지표 ---")
print(f"정확도 (Accuracy): {accuracy:.4f}")   # (TP + TN) / Total
print(f"정밀도 (Precision): {precision:.4f}")  # TP / (TP + FP)
print(f"재현율 (Recall): {recall:.4f}")  # TP / (TP + FN)
print(f"F1 점수 (F1 Score): {f1:.4f}")  # Harmonic mean of Precision and Recall

--- 주요 성능 지표 ---
정확도 (Accuracy): 0.8596
정밀도 (Precision): 0.8807
재현율 (Recall): 0.8972
F1 점수 (F1 Score): 0.8889

# 3. Use classification_report() (summarize everything at once)
# support: Actual sample count for each class
print("\n--- Classification Report ---")
print(classification_report(y_test, y_pred,
                            target_names=['Malignant(0)', 'Benign(1)']))

print("\n해석:")
print("- Malignant(0) Precision = 0.82: 암(악성) 진단한 경우, 82%는 실제 암이었습니다.")
print("- Malignant(0) Recall = 0.80: 실제 암(악성) 환자 중 80%를 찾아냈습니다. (20%는 놓침)")
print("- Benign(1) Precision = 0.88: 정상(양성) 진단한 경우, 88%는 실제 정상이었습니다.")
print("- Benign(1) Recall = 0.90: 실제 정상(양성) 환자 중 90%를 찾아냈습니다. (10%는 놓침)")

--- Classification Report ---
              precision    recall  f1-score   support

Malignant(0)       0.82      0.80      0.81        64
   Benign(1)       0.88      0.90      0.89       107

    accuracy                           0.86       171
   macro avg       0.85      0.85      0.85       171
weighted avg       0.86      0.86      0.86       171


해석:
- Malignant(0) Precision = 0.82: 암(악성) 진단한 경우, 82%는 실제 암이었습니다.
- Malignant(0) Recall = 0.80: 실제 암(악성) 환자 중 80%를 찾아냈습니다. (20%는 놓침)
- Benign(1) Precision = 0.88: 정상(양성) 진단한 경우, 88%는 실제 정상이었습니다.
- Benign(1) Recall = 0.90: 실제 정상(양성) 환자 중 90%를 찾아냈습니다. (10%는 놓침)

# Visualize results (including the decision boundary)
# Scatter plot of Test data (colored by ground truth)
plt.scatter(X_test, y_test, c=y_test, cmap='coolwarm',
            s=50, marker='o', alpha=0.1, label='Actual Class')
# Add small points colored by model predictions to identify incorrect predictions
plt.scatter(X_test, y_pred, c=y_pred, cmap='coolwarm',
            s=10, marker='x', label='Predicted Class')

# Calculate and plot the decision boundary (probability=0.5)
# Find where z = Wx + b = 0 -> x = -b / W
decision_boundary = -model.intercept_ / model.coef_[0]
plt.axvline(x=decision_boundary, color='black', linestyle='--',
            label='Decision Boundary')

plt.title("Test Data")
plt.xlabel("Mean Radius")
plt.ylabel("Class (0:Malignant, 1:Benign)")
plt.yticks([0, 1])
plt.legend()
plt.grid(True)
plt.show()

print(f"\n결정 경계: 평균 반지름 약 {decision_boundary[0]:.2f}")
print("=> 이 모델은 평균 반지름이 이 값보다 크면 0(악성), 작으면 1(양성)으로 예측합니다.")

<Figure size 640x480 with 1 Axes>

결정 경계: 평균 반지름 약 14.71
=> 이 모델은 평균 반지름이 이 값보다 크면 0(악성), 작으면 1(양성)으로 예측합니다.

Step 5: Inspect Predicted Probabilities and Make Predictions

Logistic Regression calculates probabilities internally.
predict() returns the final classification (0 or 1),
whereas predict_proba() lets you inspect the probability of belonging to each class (0 and 1).

Use this to perform Step 5: prediction.

# Inspect predicted probabilities for the first 5 Test samples
# Results have the form [probability of 0, probability of 1].
probabilities = model.predict_proba(X_test[:5])
print("--- 첫 5개 Test 데이터의 예측 확률 ---")
print("[ 0(악성)일 확률, 1(양성)일 확률 ]")
print(np.round(probabilities, 3)) # Round to 3 decimal places

# Inspect final predictions for the first 5 Test samples
predictions = model.predict(X_test[:5])
print("\n--- 첫 5개 Test 데이터의 최종 예측 ---")
print(predictions)

# Predict on new tumor data
# A tumor with mean radius 10
new_tumor_small = np.array([[10]])
pred_small_prob = model.predict_proba(new_tumor_small)
pred_small = model.predict(new_tumor_small)
print(f"\n--- 평균 반지름 10인 종양 ---")
print(f" 예측 확률 [악성, 양성]: {np.round(pred_small_prob[0], 3)}")
print(f" 최종 예측: {'양성(Benign)' if pred_small[0] == 1 else '악성(Malignant)'}")

# A tumor with mean radius 20
new_tumor_large = np.array([[20]])
pred_large_prob = model.predict_proba(new_tumor_large)
pred_large = model.predict(new_tumor_large)
print(f"\n--- 평균 반지름 20인 종양 ---")
print(f" 예측 확률 [악성, 양성]: {np.round(pred_large_prob[0], 3)}")
print(f" 최종 예측: {'양성(Benign)' if pred_large[0] == 1 else '악성(Malignant)'}")

--- 첫 5개 Test 데이터의 예측 확률 ---
[ 0(악성)일 확률, 1(양성)일 확률 ]
[[0.267 0.733]
 [0.147 0.853]
 [0.052 0.948]
 [0.989 0.011]
 [0.44  0.56 ]]

--- 첫 5개 Test 데이터의 최종 예측 ---
[1 1 1 0 1]

--- 평균 반지름 10인 종양 ---
 예측 확률 [악성, 양성]: [0.008 0.992]
 최종 예측: 양성(Benign)

--- 평균 반지름 20인 종양 ---
 예측 확률 [악성, 양성]: [0.996 0.004]
 최종 예측: 악성(Malignant)