Course materials Β· Introduction to Machine Learning

[Week 10] Solving Nonlinear Problems: Decision Tree and Random Forest 🌳🌲

English translation

Welcome to Week 10!
So far, we have learned Linear Regression and Logistic Regression, both linear models.
These models attempted to explain relationships in data with a single 'line' or 'plane'.

Real-world data often cannot be explained by a straight line alone,
having more complex nonlinear relationships.

Today, we will learn powerful models for nonlinear problems:
Decision Trees and their extension, Random Forests,
and implement them ourselves with scikit-learn.

🌳 Module 1: Decision Trees - Splitting with Questions Like Twenty Questions

What Is a Decision Tree?

Example Data: Deciding whether to visit an amusement park

Weather Weekend status Go
Sunny Weekend Yes
Sunny Weekday Yes
Cloudy Weekend Yes
Rainy Weekend No
Rainy Weekday No

Goal: Find the best question for splitting the data!
➑️ Make groups as pure as possible (only one type per group)

Step 1: Calculate Impurity in the Current State (Root Node)

Step 2: Evaluate Candidate First Questions

Candidate Question 1: "What Is the Weather?"

Candidate Question 2: "Is It a Weekend?"

Step 3: Choose the Best Question

Step 4: Complete and Visualize the Tree

The code below visualizes this final tree structure and explains key terms.

# Visualize the tree using graphviz
import graphviz

# Define the tree structure (DOT language)
dot_source = """
digraph WeatherTree {
    node [shape=box, style="filled, rounded", fillcolor="#e8f4f8"];
    edge [fontname="Helvetica"];

    // 루트 λ…Έλ“œ
    0 [label="λ‚ μ”¨λŠ”?", fillcolor="#a1d6e2"];

    // 리프 λ…Έλ“œ
    1 [label="κ°„λ‹€ (Yes)\n(λ§‘μŒ)", fillcolor="#7dd3a2"];
    2 [label="κ°„λ‹€ (Yes)\n(흐림)", fillcolor="#7dd3a2"];
    3 [label="μ•ˆκ°„λ‹€ (No)\n(λΉ„)", fillcolor="#f8a7a7"];

    // 가지 (Branch)
    0 -> 1 [label="λ§‘μŒ"];
    0 -> 2 [label="흐림"];
    0 -> 3 [label=" λΉ„ "];
}
"""
graph = graphviz.Source(dot_source)
graph # Display the visualization directly in Colab.
<graphviz.sources.Source at 0x7e59e45aaed0>

Key Decision Tree Terms (See the Figure Above):

A Decision Tree repeatedly splits data by finding questions that maximize information gain at each step,
building its final prediction rules.

🌸 Module 2: Step-by-Step Decision Tree Modeling (Iris Data)

Now build a Decision Tree model on the Iris dataset using scikit-learn.
Use the two features petal length and petal width
to classify all three varieties (setosa, versicolor, virginica),
a Multiclass Classification problem.

Follow the 5-step Machine Learning workflow again.

# Import required libraries
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
# Import DecisionTreeClassifier.
from sklearn.tree import DecisionTreeClassifier
# Import evaluation metric functions.
from sklearn.metrics import (accuracy_score,
                             confusion_matrix,
                             classification_report)

# 1. Load data
iris = load_iris()

# Features (X): Petal length (index 2), petal width (index 3) -> Use 2 features!
X = iris.data[:, 2:4]
# Ground truth (y): Iris species (0, 1, 2)
y = iris.target
feature_names = iris.feature_names[2:]
        # 'petal length (cm)', 'petal width (cm)'
target_names = iris.target_names
        # 'setosa', 'versicolor', 'virginica'

# 2. Explore data (visualization)
plt.figure(figsize=(8, 6))
# Plot each species (y) with a different color/marker
for i, target_name in enumerate(target_names):
    plt.scatter(X[y == i, 0], X[y == i, 1], label=target_name)

plt.title("Iris data (feature)")
plt.xlabel(feature_names[0])
plt.ylabel(feature_names[1])
plt.legend()
plt.grid(True)
plt.show()

print("κ²°κ³Ό 해석:")
print("- Setosa(νŒŒλž‘μƒ‰)λŠ” λͺ…ν™•νžˆ κ΅¬λΆ„λ©λ‹ˆλ‹€.")
print("- Versicolor(주황색)와 Virginica(녹색)λŠ” 일뢀 겹쳐 λ³΄μž…λ‹ˆλ‹€.")
print("- 직선 ν•˜λ‚˜λ‘œλŠ” μ„Έ ν’ˆμ’…μ„ μ™„λ²½νžˆ λ‚˜λˆ„κΈ° μ–΄λ €μ›Œ λ³΄μž…λ‹ˆλ‹€.")
<Figure size 800x600 with 1 Axes>
μ‹€ν–‰ κ²°κ³Ό
κ²°κ³Ό 해석:
- Setosa(νŒŒλž‘μƒ‰)λŠ” λͺ…ν™•νžˆ κ΅¬λΆ„λ©λ‹ˆλ‹€.
- Versicolor(주황색)와 Virginica(녹색)λŠ” 일뢀 겹쳐 λ³΄μž…λ‹ˆλ‹€.
- 직선 ν•˜λ‚˜λ‘œλŠ” μ„Έ ν’ˆμ’…μ„ μ™„λ²½νžˆ λ‚˜λˆ„κΈ° μ–΄λ €μ›Œ λ³΄μž…λ‹ˆλ‹€.

Next, split the data into practice problems and an actual exam.
Since this is multiclass classification, use the stratify option.

# Split data into 'practice problems' and a 'real exam' at a 7:3 ratio
# Use stratify=y for this multiclass classification problem
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

print(f"Train 데이터 크기: {X_train.shape}")
print(f"Test 데이터 크기: {X_test.shape}")
Train 데이터 크기: (105, 2)
Test 데이터 크기: (45, 2)

Now create a Decision Tree model and train it.

Main hyperparameters of DecisionTreeClassifier:

# 1. Create a Decision Tree model object
# random_state: Fix the random seed to obtain identical results
# max_depth: Limit maximum tree depth (None allows unlimited growth)
# -> An important hyperparameter for preventing overfitting (explained later)
model_dt = DecisionTreeClassifier(random_state=42)

# 2. Train the model on 'practice problems' (fit)
# Internally, it finds optimal questions (features/thresholds)
# and builds a tree structure that splits the data.
model_dt.fit(X_train, y_train)

print("κ²°μ • 트리 λͺ¨λΈ ν•™μŠ΅ μ™„λ£Œ!")
κ²°μ • 트리 λͺ¨λΈ ν•™μŠ΅ μ™„λ£Œ!

Visualize the trained Decision Tree's internal structure
to see which questions the model uses to classify data.
(Requires the graphviz library)

import graphviz
from sklearn.tree import export_graphviz
# Visualize the trained tree structure
dot_data = export_graphviz(model_dt, out_file=None,
                           feature_names=feature_names,
                           class_names=target_names,
                           filled=True, rounded=True,
                           special_characters=True)
graph = graphviz.Source(dot_data)
graph # Display the visualization directly in Colab.
<graphviz.sources.Source at 0x7e59b4f63020>

Visualize how the trained tree partitions the data space (decision boundaries).
This can reveal signs of Overfitting.

Overfitting occurs when a model becomes too specialized in practice problems
and performs poorly on the actual exam.
If a Decision Tree grows without limits, it can learn even noise in the training data,
producing very complex decision boundaries and becoming prone to Overfitting.

# Define a decision boundary visualization function (reuse the previous lecture's code)
def plot_decision_boundary(model, X, y, target_names, feature_names, title):
    plt.figure()
    x_min, x_max = X[:, 0].min() - 0.5, X[:, 0].max() + 0.5
    y_min, y_max = X[:, 1].min() - 0.5, X[:, 1].max() + 0.5
    xx, yy = np.meshgrid(np.arange(x_min, x_max, 0.02),
                         np.arange(y_min, y_max, 0.02))
    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    plt.contourf(xx, yy, Z, cmap='coolwarm', alpha=0.8)
    for i, target_name in enumerate(target_names):
        plt.scatter(X[y == i, 0], X[y == i, 1], label=target_name,
                    edgecolors='k')
    plt.title(title)
    plt.xlabel(feature_names[0])
    plt.ylabel(feature_names[1])
    plt.legend()
    plt.show()

# Inspect the trained Decision Tree's decision boundary using Train data
plot_decision_boundary(model_dt, X_train, y_train, target_names,
                       feature_names, "Decision Boundary (Train Data)")

print("κ²°κ³Ό 해석:")
print("- μ—°μŠ΅ 문제(Train Data)λ₯Ό 잘 λΆ„λ₯˜ν•˜κΈ° μœ„ν•΄")
print("  μ ˆλ¬˜ν•œ κ²°μ • 경계가 λ§Œλ“€μ–΄μ‘ŒμŠ΅λ‹ˆλ‹€.")
print("- 과적합(Overfitting)의 κ°€λŠ₯성이 μžˆμŠ΅λ‹ˆλ‹€.")
<Figure size 640x480 with 1 Axes>
μ‹€ν–‰ κ²°κ³Ό
κ²°κ³Ό 해석:
- μ—°μŠ΅ 문제(Train Data)λ₯Ό 잘 λΆ„λ₯˜ν•˜κΈ° μœ„ν•΄
  μ ˆλ¬˜ν•œ κ²°μ • 경계가 λ§Œλ“€μ–΄μ‘ŒμŠ΅λ‹ˆλ‹€.
- 과적합(Overfitting)의 κ°€λŠ₯성이 μžˆμŠ΅λ‹ˆλ‹€.

# Evaluate the Decision Tree model using Test data
y_pred_dt = model_dt.predict(X_test)

accuracy_dt = accuracy_score(y_test, y_pred_dt)
report_dt = classification_report(y_test, y_pred_dt, target_names=target_names)

print("--- κ²°μ • 트리 μ„±λŠ₯ 평가 (Test Data) ---")
print(f"정확도: {accuracy_dt:.4f}")
print("\nClassification Report:")
print(report_dt)

# Visualize the decision boundary using Test data
plot_decision_boundary(model_dt, X_test, y_test, target_names,
                       feature_names, "Decision Boundary (Test Data)")
--- κ²°μ • 트리 μ„±λŠ₯ 평가 (Test Data) ---
정확도: 0.9333

Classification Report:
              precision    recall  f1-score   support

      setosa       1.00      1.00      1.00        15
  versicolor       0.88      0.93      0.90        15
   virginica       0.93      0.87      0.90        15

    accuracy                           0.93        45
   macro avg       0.93      0.93      0.93        45
weighted avg       0.93      0.93      0.93        45


<Figure size 640x480 with 1 Axes>
μ‹€ν–‰ κ²°κ³Ό

🌲 Module 3: Overfitting and Random Forest

A Limitation of Decision Trees: Overfitting (Overfitting)

Ways to Reduce Overfitting:

Random Forest: The Power of Ensembles

Imagine a Decision Tree as an all-purpose doctor who diagnoses every patient alone.
This doctor gives perfect diagnoses (100 points) for previously seen patients (training data),
but may become so specialized in those experiences that new patients with slightly different symptoms (test data)
receive incorrect diagnoses (overfitting).

Random Forest addresses this problem
by gathering multiple 'slightly different specialists' instead of one all-purpose doctor
to form a Committee of Doctors.
➑️ The number of doctors is the n_estimators parameter of RandomForestClassifier.

How Do We Create 'Slightly Different' Doctors? (Randomness!)

  1. Different Patient Experience (Data Sampling β€” Bagging):
    • From all patient records (the original data), randomly sample patient records with replacement
      and give them to each doctor.
    • A doctor may see a particular patient multiple times and never see some others.
    • This gives each doctor experience with slightly different patient groups.
    • ➑️ The bootstrap=True parameter (the default) means this method is used.
  2. Limit the Symptoms Considered (Random Feature Selection):
    • Whenever a doctor makes a diagnosis (asks a question),
      randomly limit the types of symptoms (features) they can consider.
    • For example, one doctor must decide using only 'temperature' and 'cough',
      while another uses only 'headache' and 'fatigue'.
    • This gives each doctor more expertise in particular symptom combinations.
    • ➑️ The max_features parameter controls this step.
      (Default: n_features\sqrt{n\_features})

Diagnosis Process (Prediction):

Why Is It More Effective?

Now create and train a Random Forest model
and compare its performance with the Decision Tree model.

# Import the Random Forest model
from sklearn.ensemble import RandomForestClassifier

# 1. Create and train a Random Forest model
# n_estimators: Number of trees (doctors). More is better, but increases computation time.
# random_state: Seed for reproducible results.
model_rf = RandomForestClassifier(n_estimators=15, random_state=42)
model_rf.fit(X_train, y_train)
plot_decision_boundary(model_rf, X_train, y_train, target_names,
                       feature_names, "Random Forest Decision Boundary (Train Data)")

# 2. Evaluate the Random Forest model using Test data
y_pred_rf = model_rf.predict(X_test)
accuracy_rf = accuracy_score(y_test, y_pred_rf)
report_rf = classification_report(y_test, y_pred_rf, target_names=target_names)

print("--- 랜덀 포레슀트 μ„±λŠ₯ 평가 (Test Data) ---")
print(f"정확도: {accuracy_rf:.4f}")
print("\nClassification Report:")
print(report_rf)

# Visualize the decision boundary using Test data
plot_decision_boundary(model_rf, X_test, y_test, target_names,
                       feature_names, "Random Forest Decision Boundary (Test Data)")

print("\n비ꡐ:")
print(f"- κ²°μ • 트리 정확도: {accuracy_dt:.4f}")
print(f"- 랜덀 포레슀트 정확도: {accuracy_rf:.4f}")
print("\n=> κ³Όμ ν•©μ˜ λŠλ‚Œμ΄ μ—†μ§€λŠ” μ•Šμ§€λ§Œ, 정확도가 μ•½κ°„ ν–₯μƒλ˜μ—ˆμŠ΅λ‹ˆλ‹€.")
<Figure size 640x480 with 1 Axes>
μ‹€ν–‰ κ²°κ³Ό
--- 랜덀 포레슀트 μ„±λŠ₯ 평가 (Test Data) ---
정확도: 0.9556

Classification Report:
              precision    recall  f1-score   support

      setosa       1.00      1.00      1.00        15
  versicolor       0.93      0.93      0.93        15
   virginica       0.93      0.93      0.93        15

    accuracy                           0.96        45
   macro avg       0.96      0.96      0.96        45
weighted avg       0.96      0.96      0.96        45


<Figure size 640x480 with 1 Axes>
μ‹€ν–‰ κ²°κ³Ό
비ꡐ:
- κ²°μ • 트리 정확도: 0.9333
- 랜덀 포레슀트 정확도: 0.9556

=> κ³Όμ ν•©μ˜ λŠλ‚Œμ΄ μ—†μ§€λŠ” μ•Šμ§€λ§Œ, 정확도가 μ•½κ°„ ν–₯μƒλ˜μ—ˆμŠ΅λ‹ˆλ‹€.

Random Forest provides information about how much each feature contributed to model predictions:
Feature Importance.

# (Reference) Inspect feature importance
# Stored in the model_rf object's feature_importances_ attribute
importances = model_rf.feature_importances_
print("\n--- νŠΉμ§• μ€‘μš”λ„ (Feature Importances) ---")
for name, importance in zip(feature_names, importances):
    print(f"{name}: {importance:.4f}")
print("\n=> 이 λ¬Έμ œμ—μ„œλŠ” κ½ƒμžŽ λ„ˆλΉ„(petal width)κ°€ ν’ˆμ’… λΆ„λ₯˜μ—")
print("   κ½ƒμžŽ 길이(petal length)보닀 μ•½κ°„ 더 μ€‘μš”ν•œ νŠΉμ§•μž„μ„ μ•Œ 수 μžˆμŠ΅λ‹ˆλ‹€.")
--- νŠΉμ§• μ€‘μš”λ„ (Feature Importances) ---
petal length (cm): 0.5217
petal width (cm): 0.4783

=> 이 λ¬Έμ œμ—μ„œλŠ” κ½ƒμžŽ λ„ˆλΉ„(petal width)κ°€ ν’ˆμ’… λΆ„λ₯˜μ—
   κ½ƒμžŽ 길이(petal length)보닀 μ•½κ°„ 더 μ€‘μš”ν•œ νŠΉμ§•μž„μ„ μ•Œ 수 μžˆμŠ΅λ‹ˆλ‹€.

✨ Wrap-up

Today we learned powerful tools for nonlinear problems:
Decision Trees and Random Forests.

Next time, we will explore a method that finds hidden structure in data without correct answers:
Unsupervised Learning, represented by K-Means Clustering.