Text Classification
텍스트 분류는 자연어 처리의 일종으로, 텍스트에 레이블 또는 클래스를 지정하는 작업이며, 다양한 실용적인 응용 분야에서 텍스트 분류사용한다.
가장 많이 사용되는 텍스트 분류 사례들 중 하나는 감성 분석으로, 텍스트 시퀀스에 🙂긍정, 🙁부정과 같은 label을 지정해 분류한다..
이 강의자료에서 학습할 내용은:
- IMDb 데이터셋을 이용해, 오픈소스로 공유된 사전학습 모델을 이용해 영화 리뷰가 긍정적인지 부정적인지 판단합니다.
- 이 강의자료에서는 DistillBert 기반의 사전학습모델을 사용합니다. jzonthemtn/distilbert-imdb
0 라이브러리 설치
Hugging Face Transformers·Datasets 활용
! pip install transformers datasets
Requirement already satisfied: transformers in /usr/local/lib/python3.10/dist-packages (4.46.3) Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (3.1.0) Requirement already satisfied: evaluate in /usr/local/lib/python3.10/dist-packages (0.4.3) Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers) (3.16.1) Requirement already satisfied: huggingface-hub<1.0,>=0.23.2 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.26.3) Requirement already satisfied: numpy>=1.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (1.26.4) Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from transformers) (24.2) Requirement already satisfied: pyyaml>=5.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (6.0.2) Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers) (2024.9.11) Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers) (2.32.3) Requirement already satisfied: tokenizers<0.21,>=0.20 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.20.3) Requirement already satisfied: safetensors>=0.4.1 in /usr/local/lib/python3.10/dist-packages (from transformers) (0.4.5) Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers) (4.66.6) Requirement already satisfied: pyarrow>=15.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (17.0.0) Requirement already satisfied: dill<0.3.9,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.3.8) Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets) (2.2.2) Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets) (3.5.0) Requirement already satisfied: multiprocess<0.70.17 in /usr/local/lib/python3.10/dist-packages (from datasets) (0.70.16) Requirement already satisfied: fsspec<=2024.9.0,>=2023.1.0 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]<=2024.9.0,>=2023.1.0->datasets) (2024.9.0) Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets) (3.11.9) Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (2.4.4) Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.3.1) Requirement already satisfied: async-timeout<6.0,>=4.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (4.0.3) Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (24.2.0) Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.5.0) Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (6.1.0) Requirement already satisfied: propcache>=0.2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (0.2.1) Requirement already satisfied: yarl<2.0,>=1.17.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets) (1.18.3) Requirement already satisfied: typing-extensions>=3.7.4.3 in /usr/local/lib/python3.10/dist-packages (from huggingface-hub<1.0,>=0.23.2->transformers) (4.12.2) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.4.0) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (3.10) Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2.2.3) Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers) (2024.8.30) Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2.8.2) Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.2) Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets) (2024.2) Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->datasets) (1.16.0)
Model inference
CPU / CUDA GPU: 사용 가능한 장치를 자동으로 선택합니다.
import torch
pipeline_device = 0 if torch.cuda.is_available() else -1
1. Dataset
IMDB(영화사이트 리뷰) Dataset 가져오기.
Dataset 참조논문
from datasets import load_dataset
imdb = load_dataset('imdb')/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: The secret `HF_TOKEN` does not exist in your Colab secrets. To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session. You will be able to reuse this secret in all of your notebooks. Please note that authentication is recommended but still optional to access public models or datasets. warnings.warn(
README.md: 0%| | 0.00/7.81k [00:00<?, ?B/s]
train-00000-of-00001.parquet: 0%| | 0.00/21.0M [00:00<?, ?B/s]
test-00000-of-00001.parquet: 0%| | 0.00/20.5M [00:00<?, ?B/s]
unsupervised-00000-of-00001.parquet: 0%| | 0.00/42.0M [00:00<?, ?B/s]
Generating train split: 0%| | 0/25000 [00:00<?, ? examples/s]
Generating test split: 0%| | 0/25000 [00:00<?, ? examples/s]
Generating unsupervised split: 0%| | 0/50000 [00:00<?, ? examples/s]
첫 번째 데이터 출력해보기 - 데이터 형태는 Dictionary({key:value})
- text: 리뷰 텍스트 리스트
- label: 사람이 직접 분류한 정답 label(0: 부정, 1: 긍정) 리스트
imdb['test'][0:3]{'text': ['I love sci-fi and am willing to put up with a lot. Sci-fi movies/TV are usually underfunded, under-appreciated and misunderstood. I tried to like this, I really did, but it is to good TV sci-fi as Babylon 5 is to Star Trek (the original). Silly prosthetics, cheap cardboard sets, stilted dialogues, CG that doesn\'t match the background, and painfully one-dimensional characters cannot be overcome with a \'sci-fi\' setting. (I\'m sure there are those of you out there who think Babylon 5 is good sci-fi TV. It\'s not. It\'s clichéd and uninspiring.) While US viewers might like emotion and character development, sci-fi is a genre that does not take itself seriously (cf. Star Trek). It may treat important issues, yet not as a serious philosophy. It\'s really difficult to care about the characters here as they are not simply foolish, just missing a spark of life. Their actions and reactions are wooden and predictable, often painful to watch. The makers of Earth KNOW it\'s rubbish as they have to always say "Gene Roddenberry\'s Earth..." otherwise people would not continue watching. Roddenberry\'s ashes must be turning in their orbit as this dull, cheap, poorly edited (watching it without advert breaks really brings this home) trudging Trabant of a show lumbers into space. Spoiler. So, kill off a main character. And then bring him back as another actor. Jeeez! Dallas all over again.',
"Worth the entertainment value of a rental, especially if you like action movies. This one features the usual car chases, fights with the great Van Damme kick style, shooting battles with the 40 shell load shotgun, and even terrorist style bombs. All of this is entertaining and competently handled but there is nothing that really blows you away if you've seen your share before.<br /><br />The plot is made interesting by the inclusion of a rabbit, which is clever but hardly profound. Many of the characters are heavily stereotyped -- the angry veterans, the terrified illegal aliens, the crooked cops, the indifferent feds, the bitchy tough lady station head, the crooked politician, the fat federale who looks like he was typecast as the Mexican in a Hollywood movie from the 1940s. All passably acted but again nothing special.<br /><br />I thought the main villains were pretty well done and fairly well acted. By the end of the movie you certainly knew who the good guys were and weren't. There was an emotional lift as the really bad ones got their just deserts. Very simplistic, but then you weren't expecting Hamlet, right? The only thing I found really annoying was the constant cuts to VDs daughter during the last fight scene.<br /><br />Not bad. Not good. Passable 4.",
"its a totally average film with a few semi-alright action sequences that make the plot seem a little better and remind the viewer of the classic van dam films. parts of the plot don't make sense and seem to be added in to use up time. the end plot is that of a very basic type that doesn't leave the viewer guessing and any twists are obvious from the beginning. the end scene with the flask backs don't make sense as they are added in and seem to have little relevance to the history of van dam's character. not really worth watching again, bit disappointed in the end production, even though it is apparent it was shot on a low budget certain shots and sections in the film are of poor directed quality"],
'label': [0, 0, 0]}2 사전 학습 모델을 이용한 추론
사전 학습된 모델 불러오기.
model_name = 'jzonthemtn/distilbert-imdb'
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(model_name)The cache for model files in Transformers v4.22.0 has been updated. Migrating your old cache. This is a one-time only operation. You can interrupt this and resume the migration later on by calling `transformers.utils.move_cache()`.
0it [00:00, ?it/s]
config.json: 0%| | 0.00/615 [00:00<?, ?B/s]
pytorch_model.bin: 0%| | 0.00/268M [00:00<?, ?B/s]
간단한 예시 추론해보기.
pipeline()를 이용하면 편리하다.
device = 0은 GPU를 사용할 때 입력하는 인자이다.
sentiment-analysis는 pipeline 이름을 임의로 정한것이다.
from transformers import pipeline
classifier = pipeline('sentiment-analysis', model=model_name, device=pipeline_device)
text = "This was a masterpiece. Not completely faithful to the books, but enthralling from beginning to end. Might be my favorite of the three."
#text = "This was so bad!"
#text = "This was so good!"
classifier(text)tokenizer_config.json: 0%| | 0.00/360 [00:00<?, ?B/s]
vocab.txt: 0%| | 0.00/232k [00:00<?, ?B/s]
tokenizer.json: 0%| | 0.00/711k [00:00<?, ?B/s]
special_tokens_map.json: 0%| | 0.00/125 [00:00<?, ?B/s]
[{'label': 'LABEL_1', 'score': 0.9958142638206482}]3 모델 평가하기
Accuracy
import numpy as np
def classification_accuracy(predictions, labels):
return {'accuracy': float(np.mean(np.asarray(predictions) == np.asarray(labels)))}
def compute_metrics(eval_pred):
predictions, labels = eval_pred
predictions = np.argmax(predictions, axis=1)
return classification_accuracy(predictions, labels)
Downloading builder script: 0%| | 0.00/4.20k [00:00<?, ?B/s]
Test dataset으로 부터 임의의 샘플을 random하게 추출하고, 이를 이용해 평가한다.
test_list = imdb['test']
test_sample_size = 20
rng = np.random.default_rng()
random_indices = rng.choice(len(test_list), size=test_sample_size, replace=False)
sampled_data = test_list.select(random_indices)
test_predictions = classifier(list(sampled_data['text']), max_length=512, truncation=True)
predicted_labels = [label['label'] for label in test_predictions]
mapped_labels = [model.config.label2id[label] for label in predicted_labels]
reference_labels = list(sampled_data['label'])
print(classification_accuracy(mapped_labels, reference_labels))
print(reference_labels)
print(mapped_labels){'accuracy': 1.0}
[1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0]
[1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0]
4 Assignment
a) Hugging Face에 공개된 다양한 사전학습 모델들을 바꿔가며 비교해본다.
b) 다른 Dataset을 이용한 한글 문장 분류 모델을 찾아보고, 임의로 Test dataset을 만들어 평가해본다.
ko_sentiment_data = load_dataset('sepidmnorozy/Korean_sentiment')train.csv: 0%| | 0.00/3.29M [00:00<?, ?B/s]
dev.csv: 0%| | 0.00/127k [00:00<?, ?B/s]
test.csv: 0%| | 0.00/239k [00:00<?, ?B/s]
Generating train split: 0%| | 0/36000 [00:00<?, ? examples/s]
Generating validation split: 0%| | 0/1333 [00:00<?, ? examples/s]
Generating test split: 0%| | 0/2667 [00:00<?, ? examples/s]
ko_sentiment_data['test'][0:3]{'label': [0, 1, 1],
'text': ['이 영화는 아름다운 영상미로 쓰레기같은 내용을 감추고 멋진 영화로 위장을 하고있다. 나는 이 영화가 만들다가 만 영화라고 생각한다.',
'워메 ㅋㅋ 아이언맨이 여기나오는줄은 ㅋㅋ반전이네요',
'여자 진짜 개귀엽다 ㅋㅋㅋㅋ']}model_name = 'WhitePeak/bert-base-cased-Korean-sentiment'
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(model_name)
#classifier = pipeline('sentiment-analysis', model=model_name, device=pipeline_device)
classifier = pipeline('text-classification', model=model_name, device=pipeline_device)
classifier(ko_sentiment_data['test'][3]['text'])[{'label': 'LABEL_1', 'score': 0.9814258813858032}]test_list = ko_sentiment_data['test']
test_sample_size = 100
rng = np.random.default_rng()
random_indices = rng.choice(len(test_list), size=test_sample_size, replace=False)
sampled_data = test_list.select(random_indices)
test_predictions = classifier(list(sampled_data['text']), max_length=512, truncation=True)
predicted_labels = [label['label'] for label in test_predictions]
mapped_labels = [model.config.label2id[label] for label in predicted_labels]
reference_labels = list(sampled_data['label'])
print(classification_accuracy(mapped_labels, reference_labels))
print(reference_labels)
print(mapped_labels){'accuracy': 0.77}
[0, 1, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0]
[0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0]