Course materials · AI Applications

Text Classification and Sentiment Analysis

English translation

AI Applications · Week 14

This week's goal · Explain how data changes from tokenization to prediction.

Run: Save a copy to your Drive in Colab, then run Code Cells from the top using ▶ or Shift+Enter. Double-click Text Cells marked ⟦Enter your response⟧ to edit them.

1. The process of classifying sentence sentiment

To classify sentence sentiment, first represent words numerically and combine their information to represent the entire sentence. Follow Sentence → Tokens and Indices → Embedding → Sentence vector → Sentiment Classification. Inspect inputs and outputs at each stage using 12 short English Reviews with targets 0: negative, 1: neutral, and 2: positive.

Processing stage Code used Results to check
Convert Text into a computable form lower().split() and vocab Word list and Index of each word
Batch sentences of different lengths encode() Integer Tensor (N,S) including PAD
Represent words as vectors nn.Embedding Floating-point Tensor (N,S,E)
Combine information from the entire sentence Mean Pooling with a Mask One vector per sentence, (N,E)
Determine a sentence's Class nn.Linear and argmax (N,3) Logits and one prediction per sentence

N is the number of sentences, S is the sentence length after padding within a Batch, and E is the word vector size. Here, use E=16.

Concepts from the IMDB example and this exercise

IMDB Review Classification in the lecture video illustrates assigning one Label to an entire sentence, accepting varying lengths such as movie reviews. Here, Digitalize means converting text into computable indices, Vocabulary is a mapping between words and indices, and Embedding is a Lookup Table retrieving trainable vectors by index. These roles are needed to feed sentences to a Neural Network even when tools change.

Text processing tools have bundled recurring tasks such as reading Datasets, Tokenization, and Vocabulary construction. TorchText, which served this role, is no longer under development, so using the package combinations in older examples in current environments has limitations. Here, you will write preprocessing with Python string operations and dictionaries and inspect the results at each stage. Official TorchText notice

The exercise data is reduced to 12 short Reviews with three sentiment Labels so you can change sentence lengths and words and inspect Tokens, PAD, and predictions. This process also provides a basis for extending data processing and models to long reviews such as IMDB. Mean Pooling is a simple model chosen to show how word vectors form a sentence vector. Data and model complexity were chosen for the learning objective, independently of library support.

2. What changes from Weeks 12 and 13?

Compare with a short example of the next-character prediction covered in Weeks 12 and 13. With hello, the target is the next character at each position. For Input hell, Target is ello. In the sentence classification example, the entire sentence great battery receives one sentiment Label.

Comparison Weeks 12 and 13: next-character prediction Week 14: sentence classification
One Token One character One whitespace-separated word
Answer Next-character Index at each time step (N,S) Sentiment Index per sentence (N,)
Model output Character scores at each time step (N,S,C) Sentiment score for each sentence (N,3)
How context is processed RNN, LSTM, and GRU process in order Summarize sentences by averaging word vectors

RNN, LSTM, and GRU can accumulate context in State while reading words in order. Mean Pooling averages word vectors into one sentence vector. Both approaches support sentence classification; here, start with Mean Pooling because its computations are easy to inspect. Reordering words later reveals the information lost through this simplification.

3. Convert a sentence to Indices

great battery is split into the Token list ['great', 'battery']. vocab is a vocabulary assigning an integer index to each word. Larger indices do not imply more positive or negative sentiment.

The following vocabulary is illustrative. Print vocab to check the actual indices in the Notebook.

Token Illustrative Index Role
PAD 0 Empty positions added to match lengths
UNK 1 Word absent from the vocabulary
great 2 Actual word
battery 3 Actual word
bad 4 Actual word

Batching great battery and bad gives Indices [[2,3],[4,0]]. An unseen word such as excellent battery becomes [1,3]. PAD marks a position with no word to read; UNK means a word exists but is unknown.

Read three preprocessing lines
  • text.lower().split() converts text to lowercase and splits words on whitespace.
  • vocab.get(word, 1) looks up an index in the vocabulary and returns UNK index 1 if it is absent.
  • Append 0 to shorter sentences to match sentence lengths within a Batch.

This Tokenizer is for simple English examples. It does not separate punctuation automatically, so battery and battery! are processed differently. First write a sentence using training vocabulary words, then add one unknown word or punctuation mark to inspect the difference.

Build the Vocabulary once from the training data. Rebuilding it for new predictions could make the same Index refer to a different word, so reuse the existing vocabulary.

4. Embedding looks up vectors by word index

nn.Embedding(len(vocab),16) is a table storing a vector of length 16 for each word. Given Index Tensor (N,S), it retrieves the corresponding rows and returns (N,S,16). Use integer dtype torch.long for Indices.

The One-hot representation in Weeks 12 and 13 was fixed, with 1 only at the character's position. More word types require longer One-hot vectors. Embedding learns a fixed-length vector for each word. This is why the word index retrieves a row rather than being used in multiplication. Here, Embedding vectors start from initial values and are trained with the classifier to reduce Loss. Creating nn.Embedding alone does not mean it already knows word meanings.

Does the same word have the same vector?

In this Embedding Layer, the same word Index always refers to the same row. Thus, at the same training step, the same vector is retrieved regardless of position in a sentence. When training updates the table, the word's vector also changes.

5. Create sentence vectors excluding PAD

For three words, sum three vectors and divide by 3. Even if one PAD increases the length to 4, the actual word count is 3. Even with a PAD vector of 0, dividing by 4 reduces the mean.

For example, if actual word vectors are [2,0] and [0,2] and PAD is [0,0], the correct mean is [1,1]. Counting PAD as well changes it to [2/3,2/3].

mask = (ids != 0).unsqueeze(-1)       # Real token: True, PAD: False
embedded = model.embedding(ids)      # (N,S,E)
pooled = (embedded * mask).sum(1) / mask.sum(1).clamp_min(1)

sum(1) sums along the word axis within each sentence. unsqueeze(-1) makes the Mask (N,S,1) so the same selection applies to every component of a word vector. The result is (N,E), one vector per sentence.

To observe the Padding effect

Among the baseline training sentences, screen size okay has three words, so one PAD is added when it is batched with a four-word sentence. The Notebook comparison uses great battery and great battery bright screen, which differ in length. Check whether applying the Mask makes the result for the same short sentence identical when processed alone and when batched with a longer sentence.

clamp_min(1) prevents a denominator of 0. It does not turn an empty sentence into a meaningful one. In this example, encode() handles an empty sentence as one UNK.

6. From sentence vectors to sentiment decisions

Linear(16,3) maps sentence vectors to Logits for three Classes. During training, feed Logits and target Indices to CrossEntropyLoss. Use softmax to display probability-form results and argmax to select the highest-scoring Class. This is the same method as Week 8 Multiclass Classification.

Update Embedding and Linear Parameters together, while keeping the preprocessing vocabulary and Label meanings fixed. Training Accuracy on this data indicates how well the model classifies the 12 training sentences.

Examine Mean Pooling limitations

great battery and battery great have the same sum of word vectors and therefore the same sentence vector. This model cannot distinguish differences in meaning due to word order. Compare their Logits directly. Comparing this with why the RNN/LSTM in Weeks 12 and 13 processed inputs in order clarifies this limitation.

Mapping unknown words to UNK does not reveal their meanings. Check the actual Tokens and Indices as well as predictions.

What should you check when preprocessing tools change?

Read code by function roles rather than function names. With a new Tokenizer, first check how it splits great battery! into Tokens. If the Vocabulary changes, check the PAD and UNK indices and the number of Embedding rows, and verify that the model output still consists of per-sentence Class scores.

Even when implementation tools change, the core remains: convert sentences into computable representations and predict Labels from them. Beyond finding a new API, you should be able to explain how it transforms the input and verify the results. This is why you print Tokens, Indices, and shapes at each stage of this exercise.

Check your understanding

  1. Can you explain why there is one target per sentence, unlike in Weeks 12 and 13?
  2. Can you distinguish PAD and UNK and track shapes through each stage?
  3. Can you explain why this model cannot distinguish word order and what Training Accuracy means?

PyTorch Embedding · PyTorch CrossEntropyLoss

Practice materials