Text Classification
English translation
Text classification is a natural language processing task that assigns labels or classes to text and has many practical applications.
Sentiment analysis is one of the most common text classification tasks, assigning labels such as ๐positive and ๐negative to text sequences.
In this material, you will learn:
- Use the IMDb dataset and open-source pretrained models to classify movie reviews as positive or negative.
- This material uses a DistillBert-based pretrained model: jzonthemtn/distilbert-imdb
0 Install libraries
Using Hugging Face Transformers and Datasets
! pip install transformers datasets
Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.46.3) Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (3.1.0) Requirement already satisfied: evaluate in /usr/local/lib/python3.10/dist-packages (0.4.3) Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.16.1) Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.26.3) Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.26.4) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (24.2) Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.2) Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2024.9.11) Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.32.3) Requirement already satisfied: tokenizers<0.21,>=0.20 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.20.3) Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.5) Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.66.6) Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (17.0.0) Requirement already satisfied: dill<0.3.9,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.3.8) Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets) (2.2.2) Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets) (3.5.0) Requirement already satisfied: multiprocess<0.70.17 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.70.16) Requirement already satisfied: fsspec<=2024.9.0,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]<=2024.9.0,>=2023.1.0->datasets) (2024.9.0) Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets) (3.11.9) Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (2.4.4) Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.1) Requirement already satisfied: async-timeout<6.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (4.0.3) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (24.2.0) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.5.0) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (6.1.0) Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (0.2.1) Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.18.3) Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.23.2->transformers) (4.12.2) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.4.0) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.10) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.2.3) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2024.8.30) Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2) Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.2) Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.2) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)
Model inference
CPU / CUDA GPU: Automatically select an available device.
import torch
pipeline_device = 0 if torch.cuda.is_available() else -1
1. Dataset
Load the IMDB Dataset (movie website reviews).
Dataset reference paper
from datasets import load_dataset
imdb = load_dataset('imdb')/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: The secret `HF_TOKEN` does not exist in your Colab secrets. To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session. You will be able to reuse this secret in all of your notebooks. Please note that authentication is recommended but still optional to access public models or datasets. warnings.warn(
README.md: 0%| | 0.00/7.81k [00:00<?, ?B/s]
train-00000-of-00001.parquet: 0%| | 0.00/21.0M [00:00<?, ?B/s]
test-00000-of-00001.parquet: 0%| | 0.00/20.5M [00:00<?, ?B/s]
unsupervised-00000-of-00001.parquet: 0%| | 0.00/42.0M [00:00<?, ?B/s]
Generating train split: 0%| | 0/25000 [00:00<?, ? examples/s]
Generating test split: 0%| | 0/25000 [00:00<?, ? examples/s]
Generating unsupervised split: 0%| | 0/50000 [00:00<?, ? examples/s]
Print the first data item - data format: Dictionary ({key:value})
- text: list of review texts
- label: list of manually assigned target labels (0: negative, 1: positive)
imdb['test'][0:3]{'text': ['I love sci-fi and am willing to put up with a lot. Sci-fi movies/TV are usually underfunded, under-appreciated and misunderstood. I tried to like this, I really did, but it is to good TV sci-fi as Babylon 5 is to Star Trek (the original). Silly prosthetics, cheap cardboard sets, stilted dialogues, CG that doesn\'t match the background, and painfully one-dimensional characters cannot be overcome with a \'sci-fi\' setting. (I\'m sure there are those of you out there who think Babylon 5 is good sci-fi TV. It\'s not. It\'s clichรฉd and uninspiring.) While US viewers might like emotion and character development, sci-fi is a genre that does not take itself seriously (cf. Star Trek). It may treat important issues, yet not as a serious philosophy. It\'s really difficult to care about the characters here as they are not simply foolish, just missing a spark of life. Their actions and reactions are wooden and predictable, often painful to watch. The makers of Earth KNOW it\'s rubbish as they have to always say "Gene Roddenberry\'s Earth..." otherwise people would not continue watching. Roddenberry\'s ashes must be turning in their orbit as this dull, cheap, poorly edited (watching it without advert breaks really brings this home) trudging Trabant of a show lumbers into space. Spoiler. So, kill off a main character. And then bring him back as another actor. Jeeez! Dallas all over again.',
"Worth the entertainment value of a rental, especially if you like action movies. This one features the usual car chases, fights with the great Van Damme kick style, shooting battles with the 40 shell load shotgun, and even terrorist style bombs. All of this is entertaining and competently handled but there is nothing that really blows you away if you've seen your share before.<br /><br />The plot is made interesting by the inclusion of a rabbit, which is clever but hardly profound. Many of the characters are heavily stereotyped -- the angry veterans, the terrified illegal aliens, the crooked cops, the indifferent feds, the bitchy tough lady station head, the crooked politician, the fat federale who looks like he was typecast as the Mexican in a Hollywood movie from the 1940s. All passably acted but again nothing special.<br /><br />I thought the main villains were pretty well done and fairly well acted. By the end of the movie you certainly knew who the good guys were and weren't. There was an emotional lift as the really bad ones got their just deserts. Very simplistic, but then you weren't expecting Hamlet, right? The only thing I found really annoying was the constant cuts to VDs daughter during the last fight scene.<br /><br />Not bad. Not good. Passable 4.",
"its a totally average film with a few semi-alright action sequences that make the plot seem a little better and remind the viewer of the classic van dam films. parts of the plot don't make sense and seem to be added in to use up time. the end plot is that of a very basic type that doesn't leave the viewer guessing and any twists are obvious from the beginning. the end scene with the flask backs don't make sense as they are added in and seem to have little relevance to the history of van dam's character. not really worth watching again, bit disappointed in the end production, even though it is apparent it was shot on a low budget certain shots and sections in the film are of poor directed quality"],
'label': [0, 0, 0]}
2 Inference using a pretrained model
Load a pretrained model.
model_name = 'jzonthemtn/distilbert-imdb'
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(model_name)The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.
0it [00:00, ?it/s]
config.json: 0%| | 0.00/615 [00:00<?, ?B/s]
pytorch_model.bin: 0%| | 0.00/268M [00:00<?, ?B/s]
Try inference on a simple example.
Using pipeline() is convenient.
device = 0 is an argument supplied when using a GPU.
sentiment-analysis is an arbitrarily chosen pipeline name.
from transformers import pipeline
classifier = pipeline('sentiment-analysis', model=model_name, device=pipeline_device)
text = "This was a masterpiece. Not completely faithful to the books, but enthralling from beginning to end. Might be my favorite of the three."
#text = "This was so bad!"
#text = "This was so good!"
classifier(text)tokenizer_config.json: 0%| | 0.00/360 [00:00<?, ?B/s]
vocab.txt: 0%| | 0.00/232k [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/711k [00:00<?, ?B/s]
special_tokens_map.json: 0%| | 0.00/125 [00:00<?, ?B/s]
[{'label': 'LABEL_1', 'score': 0.9958142638206482}]
3 Evaluate the model
Accuracy
import numpy as np
def classification_accuracy(predictions, labels):
return {'accuracy': float(np.mean(np.asarray(predictions) == np.asarray(labels)))}
def compute_metrics(eval_pred):
predictions, labels = eval_pred
predictions = np.argmax(predictions, axis=1)
return classification_accuracy(predictions, labels)
Downloading builder script: 0%| | 0.00/4.20k [00:00<?, ?B/s]
Randomly sample examples from the Test dataset and evaluate on them.
test_list = imdb['test']
test_sample_size = 20
rng = np.random.default_rng()
random_indices = rng.choice(len(test_list), size=test_sample_size, replace=False)
sampled_data = test_list.select(random_indices)
test_predictions = classifier(list(sampled_data['text']), max_length=512, truncation=True)
predicted_labels = [label['label'] for label in test_predictions]
mapped_labels = [model.config.label2id[label] for label in predicted_labels]
reference_labels = list(sampled_data['label'])
print(classification_accuracy(mapped_labels, reference_labels))
print(reference_labels)
print(mapped_labels){'accuracy': 1.0}
[1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0]
[1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0]
4 Assignment
a) Compare different pretrained models publicly available on Hugging Face.
b) Find a Korean sentence classification model using another Dataset, create a Test dataset of your own, and evaluate it.
ko_sentiment_data = load_dataset('sepidmnorozy/Korean_sentiment')train.csv: 0%| | 0.00/3.29M [00:00<?, ?B/s]
dev.csv: 0%| | 0.00/127k [00:00<?, ?B/s]
test.csv: 0%| | 0.00/239k [00:00<?, ?B/s]
Generating train split: 0%| | 0/36000 [00:00<?, ? examples/s]
Generating validation split: 0%| | 0/1333 [00:00<?, ? examples/s]
Generating test split: 0%| | 0/2667 [00:00<?, ? examples/s]
ko_sentiment_data['test'][0:3]{'label': [0, 1, 1],
'text': ['์ด ์ํ๋ ์๋ฆ๋ค์ด ์์๋ฏธ๋ก ์ฐ๋ ๊ธฐ๊ฐ์ ๋ด์ฉ์ ๊ฐ์ถ๊ณ ๋ฉ์ง ์ํ๋ก ์์ฅ์ ํ๊ณ ์๋ค. ๋๋ ์ด ์ํ๊ฐ ๋ง๋ค๋ค๊ฐ ๋ง ์ํ๋ผ๊ณ ์๊ฐํ๋ค.',
'์๋ฉ ใ
ใ
์์ด์ธ๋งจ์ด ์ฌ๊ธฐ๋์ค๋์ค์ ใ
ใ
๋ฐ์ ์ด๋ค์',
'์ฌ์ ์ง์ง ๊ฐ๊ท์ฝ๋ค ใ
ใ
ใ
ใ
']}model_name = 'WhitePeak/bert-base-cased-Korean-sentiment'
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(model_name)
#classifier = pipeline('sentiment-analysis', model=model_name, device=pipeline_device)
classifier = pipeline('text-classification', model=model_name, device=pipeline_device)
classifier(ko_sentiment_data['test'][3]['text'])[{'label': 'LABEL_1', 'score': 0.9814258813858032}]test_list = ko_sentiment_data['test']
test_sample_size = 100
rng = np.random.default_rng()
random_indices = rng.choice(len(test_list), size=test_sample_size, replace=False)
sampled_data = test_list.select(random_indices)
test_predictions = classifier(list(sampled_data['text']), max_length=512, truncation=True)
predicted_labels = [label['label'] for label in test_predictions]
mapped_labels = [model.config.label2id[label] for label in predicted_labels]
reference_labels = list(sampled_data['label'])
print(classification_accuracy(mapped_labels, reference_labels))
print(reference_labels)
print(mapped_labels){'accuracy': 0.77}
[0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0]
[0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0]