Course materials · AI Applications

Split long text to predict the next character

English translation

AI Applications · Week 13

This week's goal · Understand how to create short training examples from long text and compare RNN, LSTM, and GRU predictions on the same data.

Run: Save a copy to your Drive in Colab, then run Code Cells from the top using ▶ or Shift+Enter. Double-click Text Cells marked ⟦Enter your response⟧ to edit them.

1. Change the memory structure for the same prediction problem

Arrange hello! in order as h → e → l → l → o → !. Such an ordered series of data is a Sequence. Here, one character is the input at one time step. Spaces and punctuation marks also count as one character each.

As in Week 12, predict the next character at each position from earlier characters. Keep the rules for constructing Input and Target, the Linear output, and the Loss computation, and replace the RNN Cell with LSTM and GRU for comparison.

An RNN updates its Hidden State at every time step. As a Sequence grows longer, early information and learning signals can weaken through repeated transformations. LSTM and GRU use Gates to learn how much previous and new information to incorporate.

2. Create short training examples from long text

Illustrative example · The short string hello! below explains Windows and the Input–Target relationship. In the exercise, apply the same rules to the longer original text provided in the Notebook.

Long text is also a Sequence of characters. To make it manageable, split it into fixed-length segments to create multiple training examples. A contiguous segment taken from the original text is called a Window. A Window is a data segment, not the name of a model such as an RNN.

First read just three characters

Choose the first three characters hel of hello! as Input. The model predicts the immediately following character at each position, so the Target is ell.

Source index   0   1   2   3   4   5
Source char    h   e   l   l   o   !
               └── hel ──┘           Input: indices 0–2
                   └── ell ──┘       Target: indices 1–3

Read       h → e → l
Predict    e   l   l

The first output predicts e after h, the second predicts l after he, and the third predicts l after hel. There is one target at each position, rather than just one after reading all three characters. Input and Target both have length 3.

Move the starting point to create the next example

Now move the starting point two positions and read from original position 2. Input becomes llo, and Target becomes lo!.

Training example Starting position Input: read three characters Target: the next character at each position
First 0 hel ell
Second 2 llo lo!

Creating examples by moving the segment's starting position is called a Sliding Window. Here, Window Size=3 is one example's input length, and Stride=2 is the step between starting positions. The two segments overlap because both include original position 2.

Distinguish two shifts. The Target always starts one character after the Input because this is next-character prediction. Stride is the step taken to reach the next training example. Setting Stride to 2 does not turn it into prediction two characters ahead.

What if Stride changes to 1?

Using the same hello! and length 3, create three pairs, hel→ell, ell→llo, and llo→lo!, at starting positions 0, 1, and 2. A smaller Stride increases overlap and the number of training examples in this example.

At position 3, Input can be sliced as lo!, but no character follows the final ! to serve as a target. Therefore, this segment is not used as a training example. One Input–Target pair of length S requires S+1 consecutive characters.

Group examples into a Batch

Stacking the two examples above as rows gives the following. Distinguish reading one row from left to right from stacking different examples vertically.

             Input              Target
Example 1    h  e  l             e  l  l
Example 2    l  l  o             l  o  !
             ──────→ Time order

There are N=2 examples, each of length S=3. Converting each character into a One-hot vector of length E gives Input (2,3,E). Target stores only one character index per position, giving (2,3). In this exercise, all created examples are grouped into one Batch for training.

Use length 20 and step size 5 in the exercise

The Notebook segments a longer English text using the same rules. window_size=20 means 20 input characters, and window_stride=5 means starting positions 0, 5, 10, …. Each Target starts one character after its Input. Check actual example count N and character type count E in the output.

Value shape Meaning of each axis
One-hot Input X (N,20,E) Number of examples · Number of characters to read · Character representation size
Next-character Target Y (N,20) Number of examples · Target character index at each position
Final Logits for each model (N,20,C) Number of examples · Each position · Per-character scores
Does overlapping text also carry model memory forward?

Overlapping text segments and carrying model State forward are separate issues. Within an example, State built while reading characters on the left passes to the next character. This exercise processes examples separately without passing the final State of one example to the next.

If overlapping examples are randomly split for evaluation, character segments seen during training may also appear in the evaluation data. Do not interpret this as performance on independent new text. The values compared here are Training Accuracy on the examples used for training.

3. Read LSTM and GRU Gates

A Gate multiplies each component by a value between 0 and 1 to determine how much to retain. Learned Parameters compute Gate values; a person does not choose them with if statements for each sentence.

Structure State maintained Role of Gates
RNN Hidden State Compute a new State from the previous State and current Input
LSTM Hidden State and Cell State Forget: how much previous Cell State to retain; Input: how much new information to add; Output: how much to expose as Hidden State
GRU Hidden State Reset: how much past information contributes to computing the candidate State; Update: how the previous State and candidate are mixed
Think through a single Gate value

If one component of the previous Cell State is 2 and the Forget Gate is 0.9, 1.8 remains before adding new information. With Gate 0.1, 0.2 remains. LSTM then adds new information controlled by the Input Gate. This numerical example illustrates the flow; actual Gates are vectors.

LSTM Cell State and Hidden State are different values. GRU controls information within Hidden State without a separate Cell State. Do not assume the three models always have the same performance ranking.

4. Compare three models under the same conditions

fit_char('RNN', ...), fit_char('LSTM', ...), and fit_char('GRU', ...) train fresh models on the same data. Compare with Hidden Size=32, 300 epochs, Learning Rate=0.01, and Adam. First run each once and record Loss, Training Accuracy, and predicted text for the first Window side by side.

Even with the same Seed for initialization, model structures and Parameter counts differ. This experiment compares structures under matched training conditions; it does not determine the best model for every situation. Fitting the same original text well does not establish understanding of long context.

What you can check without repeating all training

Trained models are stored in results. Printing each model's output, final State, and predicted text requires no retraining. Fill the comparison record with the three models' earlier results. Change the Window size for comparison, using identical data for all three models.

5. Steps for inspecting return values

First create the model in exercise Section 2, then inspect its return values. For a unidirectional model with 1 Layer, they are as follows.

Structure output at all time steps Final State
RNN / GRU (N,S,32) h_n: (1,N,32)
LSTM (N,S,32) h_n, c_n: each (1,N,32)

batch_first=True arranges Input and output as (N,S,...). The first axis of the final State represents Layers and directions, so it remains (1,N,32). Character scores are obtained by applying Linear to this output.

Check your understanding

  1. Can you explain the rule for creating next-character Targets and the State differences among RNN, LSTM, and GRU?
  2. Can you distinguish Window length from the step between Windows?
  3. Can you explain observed results without generalizing model superiority from one experiment's Accuracy?

PyTorch LSTM · PyTorch GRU

Practice materials