Course materials · AI Applications

Remember previous characters to predict the next one

English translation

AI Applications · Week 12

This week's goal · Convert characters to Indices, feed them to an RNN in order, and inspect changes in Hidden State and predictions.

Run: Save a copy to your Drive in Colab, then run Code Cells from the top using ▶ or Shift+Enter. Double-click Text Cells marked ⟦Enter your response⟧ to edit them.

From classifying one character to prediction using earlier context

Illustrative example · The short string below shows why earlier context is needed. Distinguish it from the Notebook's actual training sentences.

In Week 8, you computed scores for multiple categories and compared them with the target category. Next-character prediction also classifies by selecting one candidate character. However, the current character alone may be insufficient.

Look at the two occurrences of l in hello. The target following the first l is l, and that following the second l is o. A model given only the current character is asked to produce different targets for the same Input.

An RNN carries previously read information forward as a numerical vector called the Hidden State. Thus, reading the same l can yield different predictions when earlier context differs. The addition here is earlier information for prediction, rather than new categories of classification results.

This is also why the exercise aligns Input and Target characters and checks predictions at each time step. Hidden State is a summary learned for prediction, not memory guaranteed to store all previous context accurately.

1. Use earlier characters to predict the next character

In Week 10, you predicted one digit Label per image. Now read characters in order and predict the next character at each position. An RNN's Hidden State is a vector summarizing information read so far numerically and carrying it to the next time step.

For example, to train on hello, form pairs as follows.

Time point Character currently being read Next character to predict
0 h e
1 e l
2 l l
3 l o

Input is hell, and Target is ello. Even for the same l, different earlier context can produce a different next-character target. An RNN computes a new Hidden State from the current character and previous Hidden State.

Does computation proceed in order even when an entire sentence is supplied at once?

Although the code uses one line, model(X), the RNN repeatedly applies the same Cell in Sequence order internally. This is a unidirectional RNN, so it does not read ahead beyond the current position. Hidden State is a learned numerical summary rather than storage retaining every character verbatim.

2. Character → Index → One-hot

chars is a sorted list of unique characters; vocab maps characters to Indices. Indices start at 0, and their magnitude does not imply semantic order. A One-hot vector has 1 only at that character's position and 0 elsewhere.

For example, with vocabulary ['a','b','c'], b has Index 1 and One-hot [0,1,0]. With E character types, One-hot length is also E. This week, use fixed One-hot representations instead of nn.Embedding.

Character indices can differ between vocabularies

Character indices can depend on the ordering of the vocabulary. Rather than entering indices manually, create the Input and Target using vocab, and convert predictions back to characters using the same chars. For vocabulary size C, valid target Indices range from 0 to C−1.

3. Group sentences into one Batch

In this exercise, howareyou, whats up?, and iamgreat. each have 9 characters. Spaces, question marks, and periods count as characters. Shifting the Input–Target pair by one character gives S=8. The current code uses the shortest of the three sentence lengths. If you change lengths later, check whether anything is truncated.

Value shape Meaning
ids (3,8) 3 sentences × 8 character Indices
X (3,8,E) Input with each Index converted to One-hot
Y (3,8) Next-character Index to predict at each position

batch_first=True means placing the Batch axis first. Check the actual E in the exercise output. X is floating-point, and Y is torch.long.

4. Convert RNN Hidden States into character scores

The model is RNN → Linear. Linear converts RNN output (N,S,H) into per-character Logits (N,S,C). H is Hidden Size, and C is the number of character types. This exercise uses H=32.

The lecture video example sets Hidden Size equal to the number of character types, interpreting each time step's output directly as character scores. This simply illustrates converting next-character prediction into Classification.

The exercise builds an internal representation with Hidden Size H=32, then Linear converts it into C character scores. This separation lets you choose the memory representation size independently of the character vocabulary size. It extends a design producing character scores directly into one separating internal representation from classification.

Why flatten for Loss computation?

Treat each position of each sentence as one classification problem. Flatten Logit (N,S,C) to (N×S,C) and Target (N,S) to (N×S,), then feed both to CrossEntropyLoss. Flatten both Tensors in the same order to align predictions with targets. This applies Week 8 classification to all time steps.

output holds Hidden States at all time steps, and h_n holds the final-time-step State. Since the task predicts the next character at every position, use output from all time steps.

5. Read predictions before and after training

Compare the pre-training and post-training text shown in the Notebook alongside the Target. Record whether Loss decreased and which positions changed. When replacing some Input characters with characters from the vocabulary, inspect the same model's response without retraining.

Is this the same as generating a sentence?

This code supplies the actual Input character at each time step and predicts the next one. It differs from automatic generation that feeds predicted characters back as Input to continue a long sentence. Correctly predicting the training text does not establish good generation of new sentences.

To feed only the beginning, slice the Sequence axis as in model(X[:, :4, :]). Keep the model's final axis size unchanged.

Check your understanding

  1. Can you explain the difference between character Indices and One-hot representations?
  2. Can you explain why RNN output needs Linear and the shape of the final Logits?
  3. Can you distinguish successful next-character prediction from the ability to generate new sentences?

PyTorch RNN

Practice materials