[Week 2] A History of Machine Learning: The Emergence of Machines That Learn Their Own Features 🚀

Welcome to Week 2!

Last time, we learned the basic types of Machine Learning and 'linear models'.

Today, we will go one step further: > "Why has Deep Learning become mainstream in modern Machine Learning?"

Let us travel back in time to find the answer to this question.

Today's goal is to understand the key ideas that overcame the limitations of each era,

The goal is to gain a natural sense of why Deep Learning is needed.

⛰️ Module 1: 'Limitations' of Traditional Machine Learning

• Scenario: Imagine building a model to distinguish the digits '7' and '1'.
• Question: To help the model distinguish '7' from '1', > what should we tell it?
• Traditional Approach: Data Features are defined by humans and supplied to the model.
  • "The digit '7' has a horizontal stroke."
  • "The digit '1' has a long vertical line."
  • "How densely are pixels concentrated in the image center?"
• Limitations
  • Expertise required: Creating good features is difficult without domain expertise.
  • Enormous effort: As data grows more complex, manually accounting for every feature becomes almost impossible.
    • (e.g., distinguishing dog and cat photos)
  • Performance limits: Model performance is difficult to generalize across all problems.

Then,

Could machines also learn useful features on their own from data?

This question is the starting point of Deep Learning.

# Import libraries
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_digits

# Hide unnecessary warnings
import warnings
warnings.filterwarnings(action='ignore')

# Load the digit images built into scikit-learn (0–9)
digits = load_digits()

# Filter for only the digits '1' and '7'
X = digits.data[(digits.target == 1) | (digits.target == 7)]
y = digits.target[(digits.target == 1) | (digits.target == 7)]
y[y == 1] = 0  # Assign digit 1 to class 0
y[y == 7] = 1  # Assign digit 7 to class 1

# Feature Engineering: A human defines two features directly.
# Feature 1: Sum of all image pixel values (brightness)
feature1 = X.sum(axis=1)
# Feature 2: Ratio of pixel sums in the top and bottom halves of the image
feature2 = X[:, :32].sum(axis=1) / (X[:, 32:].sum(axis=1) + 1e-6)

# Visualize the data using 2 human-designed features
plt.figure(figsize=(8, 6))
plt.scatter(feature1[y==0], feature2[y==0], label='Digit 1')
plt.scatter(feature1[y==1], feature2[y==1], label='Digit 7')
plt.title("Data Distribution By Heuristics")
plt.xlabel("Feature 1")
plt.ylabel("Feature 2")
plt.legend()
plt.show()
print("두 특징으로 어느정도 구분이 되는 것 같지만, 겹치는 부분이 많아 보입니다.")

<Figure size 800x600 with 1 Axes>

두 특징으로 어느정도 구분이 되는 것 같지만, 겹치는 부분이 많아 보입니다.

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression

# Human-designed feature data
X_features = np.c_[feature1, feature2]

# Split into training/test data
(X_train, X_test,
 y_train, y_test) = train_test_split(X_features, y,
                                     test_size=0.3,
                                     random_state=42)

# Train a Logistic Regression model
model = LogisticRegression()
model.fit(X_train, y_train)

# Check the results
accuracy = model.score(X_test, y_test)
print(f"사람이 만든 특징을 사용한 모델의 정확도: {accuracy * 100:.2f}%")
print("\n나쁘지 않은 성능이지만, 더 좋은 특징을 찾으면 성능이 얼마나 더 오를까요?")
print("\n이처럼 모델의 성능이 '사람의 직관(Heuristics)'에 의존하는 것이,")
print(f"그리고 그 한계가 뚜렷한 것이, 전통적인 머신러닝의 한계였습니다.")

사람이 만든 특징을 사용한 모델의 정확도: 58.72%

나쁘지 않은 성능이지만, 더 좋은 특징을 찾으면 성능이 얼마나 더 오를까요?

이처럼 모델의 성능이 '사람의 직관(Heuristics)'에 의존하는 것이,
그리고 그 한계가 뚜렷한 것이, 전통적인 머신러닝의 한계였습니다.

❄️ Module 2: The Birth of Artificial Neural Networks and the First 'AI Winter'

• Perceptron (1957)
  • The first artificial Neural Network model, inspired by neurons in the brain.
  • It received multiple inputs and produced one output, with a structure nearly identical to the linear model from Week 1.
• Great Expectations: It inspired excitement: "Machines can now think like brains!"
• Critical limitation: The perceptron was shown to be unable to solve problems that are not linearly separable.
  • XOR problem: A classic nonlinear problem in which a single straight line can never separate the two groups, as shown below.
• The First AI Winter: The period when support and interest in artificial Neural Network research declined sharply after this limitation became known.

# Generate XOR problem data
X_xor = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y_xor = np.array([0, 1, 1, 0]) # Output 1 for different inputs and 0 for identical inputs

# Visualize the XOR data
plt.figure(figsize=(8, 6))
plt.scatter(X_xor[:, 0], X_xor[:, 1], c=y_xor, cmap='coolwarm', s=200)
plt.title("XOR Problem: Can we find the line for classification?")
plt.xlabel("Input 1")
plt.ylabel("Input 2")
plt.grid(True)
plt.show()
print("어떤 방향으로 직선을 그어도, 빨간 점과 파란 점을 완벽하게 나눌 수 없습니다.")

<Figure size 800x600 with 1 Axes>

어떤 방향으로 직선을 그어도, 빨간 점과 파란 점을 완벽하게 나눌 수 없습니다.

# Attempt to solve XOR using the linear model (Logistic Regression) from Week 1
xor_model = LogisticRegression()
xor_model.fit(X_xor, y_xor)

# Check the results
accuracy = xor_model.score(X_xor, y_xor)
print(f"선형 모델의 XOR 문제 정확도: {accuracy * 100:.2f}%")
print("\n모델이 '찍기'와 다름없는 성능을 보여줍니다. 이는 선형 모델의 명백한 한계입니다.")

선형 모델의 XOR 문제 정확도: 50.00%

모델이 '찍기'와 다름없는 성능을 보여줍니다. 이는 선형 모델의 명백한 한계입니다.

🔥 Module 3: Ideas That Thawed the AI Winter

The AI winter lasted a long time, but a few researchers kept seeking solutions.

• Solution: "Stacking multiple layers of perceptrons can solve nonlinear problems!"
  • Multilayer Perceptron (MLP)
  • This is a basic structure of modern Deep Learning.
  • It combines multiple 'lines' to form 'curves' or 'regions'.
• A New Problem: Multilayer models were too complex to train using existing methods.
• Breakthrough: Backpropagation (1986)
  • Propagate the error between the model's final prediction and the true answer > from the Back toward the front (Propagate),
  • sequentially adjusting each neuron's connection strengths (weights).

It finally became possible to train Deep Neural Networks.

from sklearn.neural_network import MLPClassifier

# Create an MLP model (2 hidden layers, with 4-4 neurons)
mlp_model = MLPClassifier(hidden_layer_sizes=(4,4,),
                          activation='relu', solver='adam',
                          random_state=42, max_iter=1000)

# Train the MLP on the XOR data
mlp_model.fit(X_xor, y_xor)

# Check the results
accuracy = mlp_model.score(X_xor, y_xor)
print(f"MLP 모델의 XOR 문제 정확도: {accuracy * 100:.2f}%")
print("\n여러 개의 선을 조합하는 MLP는 비선형적인 XOR 문제를 완벽하게 해결했습니다!")

MLP 모델의 XOR 문제 정확도: 100.00%

여러 개의 선을 조합하는 MLP는 비선형적인 XOR 문제를 완벽하게 해결했습니다!

# Visualize the decision boundary learned by the MLP
plt.figure(figsize=(8, 6))
xx, yy = np.meshgrid(np.linspace(-0.5, 1.5, 100),
                     np.linspace(-0.5, 1.5, 100))
Z = mlp_model.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)

plt.contourf(xx, yy, Z, cmap='coolwarm', alpha=0.8) # Show the decision boundary with background colors
plt.scatter(X_xor[:, 0], X_xor[:, 1],
            c=y_xor, cmap='coolwarm',
            s=200, edgecolors='k')
plt.title("Classification by MLP")
plt.show()
print("MLP는 그림처럼 직선이 아닌, 복잡한 형태의 경계선을 만들어낼 수 있습니다.")

<Figure size 800x600 with 1 Axes>

MLP는 그림처럼 직선이 아닌, 복잡한 형태의 경계선을 만들어낼 수 있습니다.

💥 Module 4: The Big Bang of Deep Learning

Despite the invention of Backpropagation, > Deep Learning still could not solve real-world problems,

It did not become mainstream.

At the time, nobody knew why, but

Two main reasons were discovered later.

1. Slow computers (CPU)
2. Insufficient training data

The decisive event that brought all this to light occurred in 2012.

• The ImageNet competition
  • Classifying more than one million images into 1000 categories,
  • it was the largest and most challenging image recognition competition of its time.
• The arrival of AlexNet (2012, Professor Geoffrey Hinton's team)
  • A model built from Convolutional Neural Networks (CNNs).
  • GPU parallel processing dramatically accelerated training.
  • It won by a large margin, astonishing the world.
• A Paradigm Shift
  • This event shifted the Machine Learning research paradigm: > from human-designed Feature Engineering to Deep Learning, where models learn features themselves.
  • This complete shift marked the full beginning of the Deep Learning era.

# Solve Module 1's digit '1' versus '7' problem using Deep Learning (MLP).
# This time, do not use the human-designed features (feature1, feature2).
# Use all 64 raw image pixel values (X) directly.
# The problem is simple enough that a CNN is unnecessary.

# Split into training/test data
(X_train_orig,
 X_test_orig,
 y_train_orig,
 y_test_orig) = train_test_split(X, y, test_size=0.3, random_state=42)

# Create a deeper MLP model
deep_model = MLPClassifier(hidden_layer_sizes=(100, 50),
                           random_state=42, max_iter=500)

# Train directly on raw pixel data
deep_model.fit(X_train_orig, y_train_orig)

# Check the results
accuracy = deep_model.score(X_test_orig, y_test_orig)
print("모듈 1에서 사용했던 데이터와 동일한 데이터입니다.")
print(f"모듈 1에서 사람이 만든 특징 기반 모델의 정확도: 58.72%")
print(f"딥러닝 모델(MLP)의 정확도: {accuracy * 100:.2f}%")
print("\n✨ 결과: 딥러닝 모델은 64개 픽셀의 복잡한 관계 속에서 '스스로' 최적의 특징을 학습하여,")
print(f"사람이 만든 특징보다 더 높은 성능을 보여줍니다!")

모듈 1에서 사용했던 데이터와 동일한 데이터입니다.
모듈 1에서 사람이 만든 특징 기반 모델의 정확도: 58.72%
딥러닝 모델(MLP)의 정확도: 100.00%

✨ 결과: 딥러닝 모델은 64개 픽셀의 복잡한 관계 속에서 '스스로' 최적의 특징을 학습하여,
사람이 만든 특징보다 더 높은 성능을 보여줍니다!