NiiL

Split long text to predict the next character

English translation

Create your own copy to run, edit, and record results.

Self-study material

1. Create short training examples from long text

Read hello! in order as h → e → l → l → o → !. Such an ordered series is a Sequence. One character is one time step, and a space also counts as a character.

A fixed-length segment taken from the original text is called a Window. When reading hel with length 3, the next-character targets at each position are ell. h is followed by e, he by l, and hel by l. There is a target at every position, not just the last.

Example Starting position in the original text Input Target
1 0 hel ell
2 2 llo lo!

Window Size=3 is the input length; Stride=2 is the step between starting positions of successive examples. The Target is always one character after the Input. Stride 2 does not mean predicting two characters ahead. Examples 1 and 2 both include original text position 2.

Stacking two examples as rows creates a Batch. There are N=2 rows and S=3 characters per row. Representing each character as a One-hot vector of length E gives Input shape (2,3,E). State passes to the next character within a row, but not to another row.

A-1. Check segmentation rules with a short string

# Distinguish one example's input length from the step to the next example.
demo_text = "hello!"
demo_size, demo_stride = 3, 2
for start in range(0, len(demo_text) - demo_size, demo_stride):
    inp = demo_text[start:start + demo_size]
    target = demo_text[start + 1:start + demo_size + 1]
    print(f"시작 위치 {start}: Input={inp!r}, Target={target!r}")

A-2. Apply the same rules to long text

Now set input length to 20 and step size to 5. Starting positions are 0, 5, 10, …, and each Target starts one character after the Input. One extra character beyond the input length is needed to include the final target. Group all created examples into one Batch, and inspect the output for actual N, S, and E.

import sys, numpy as np, matplotlib.pyplot as plt
import torch
from torch import nn
import torch.nn.functional as F
torch.manual_seed(42)
torch.set_num_threads(2)
print('Python:', sys.version.split()[0], '| PyTorch:', torch.__version__)
sentences=["if you want to build a ship, don't drum up people together to", "collect wood and don't assign them tasks and work, but rather", "teach them to long for the endless immensity of the sea."]
sentence=' '.join(sentences);chars=sorted(set(sentence));vocab={c:i for i,c in enumerate(chars)}
window_size = 20
window_stride = 5
starts = range(0, len(sentence) - window_size, window_stride)
ids = torch.tensor([[vocab[c] for c in sentence[i:i+window_size]] for i in starts])
Y = torch.tensor([[vocab[c] for c in sentence[i+1:i+window_size+1]] for i in starts])
X = F.one_hot(ids, num_classes=len(chars)).float()
print('X:', X.shape, 'Y:', Y.shape)
for i in range(2):
    print('input:', ''.join(chars[j] for j in ids[i]))
    print('target:', ''.join(chars[j] for j in Y[i]))
assert X.shape[:2] == Y.shape

B. Hands-on activities

  1. In the first example cell, print the two Input–Target pairs created by segmenting hello! with length 3 and Stride 2. Put each pair's Input and Target in two rows and check the one-character shift within each column.
  2. Apply the same markings to the first two Input–Target pairs in the long-text data cell. Record how many characters separate their starting positions.
  3. Match each axis of X.shape and Y.shape to the number of Windows NN, length SS, and character representation size EE.

Your prediction may describe a shape or an increasing/decreasing trend rather than an exact number. Record explanations in the Text Cell below.

# ✍ Write your own: add your code below this cell.
# You may add Code Cells if the code gets long.

C. Your record · Double-click this Text Cell to edit

2. Compare RNN, LSTM, and GRU

Compare RNN, LSTM, and GRU using the same Window data and training conditions. Each model has Hidden Size=32 and feeds output at all time steps through Linear to produce character scores. Only LSTM returns a Cell State alongside the Hidden State. Training results are stored in results, so no retraining is needed to fill the table or inspect States.

A. Baseline example · Read and run first

class CharModel(nn.Module):
    def __init__(self,kind,vocab_size,hidden=32):
        super().__init__();self.rnn=getattr(nn,kind)(vocab_size,hidden,batch_first=True);self.fc=nn.Linear(hidden,vocab_size)
    def forward(self,x):
        output,state=self.rnn(x);return self.fc(output)
def fit_char(kind,X,Y,epochs=300,lr=.01):
    torch.manual_seed(42);m=CharModel(kind,X.shape[-1]);opt=torch.optim.Adam(m.parameters(),lr=lr);losses=[]
    for epoch in range(epochs):
        logits=m(X);loss=F.cross_entropy(logits.reshape(-1,X.shape[-1]),Y.reshape(-1))
        opt.zero_grad();loss.backward();opt.step();losses.append(loss.item())
        if epoch%100==0:print(kind,epoch,loss.item())
    with torch.no_grad():pred=m(X).argmax(-1)
    return m,losses,pred
results={}
for kind in ['RNN','LSTM','GRU']:
    m,h,p=fit_char(kind,X,Y,epochs=300,lr=.01);results[kind]=(m,h,p)
    print(kind,'accuracy:',(p==Y).float().mean().item());print(''.join(chars[i] for i in p[0]))
    plt.plot(h,label=kind)
plt.legend();plt.xlabel('Epoch');plt.ylabel('Loss');plt.grid();plt.show()
with torch.no_grad():output,(h_n,c_n)=results['LSTM'][0].rnn(X)
print('LSTM output:',output.shape,'h_n:',h_n.shape,'c_n:',c_n.shape)
for kind in ['RNN', 'GRU']:
    with torch.no_grad(): output, h_n = results[kind][0].rnn(X)
    print(kind, 'output:', output.shape, 'h_n:', h_n.shape)

B. Hands-on activities

  1. Run the model training cell. In one table, record the final Loss, Training Accuracy, and predicted text for the first Window for each RNN, LSTM, and GRU. Use identical data, epochs, and Learning Rate for all three models.
  2. Run the State output cell and tabulate the shapes of output, h_n, and c_n for each model. Mark States that are not returned as “None.”
  3. Explain the difference between the Sequence axis of output and the final-time-step representation in h_n, and describe the role of c_n, which LSTM additionally returns.

Your prediction may describe a shape or an increasing/decreasing trend rather than an exact number. Record explanations in the Text Cell below.

# ✍ Write your own: add your code below this cell.
# You may add Code Cells if the code gets long.

C. Your record · Double-click this Text Cell to edit

Practice through repetition · Reason before running

Summarize the RNN, LSTM, and GRU results compared earlier in this table. You do not need to train the same models three more times. Check that the data, epochs, and Learning Rate were the same.

Changed condition Predicted shape/value/result Actual result Cause of any error
⟦Enter your response 1⟧ ⟦Enter your response⟧ ⟦Enter your response⟧ ⟦Enter your response⟧
⟦Enter your response 2⟧ ⟦Enter your response⟧ ⟦Enter your response⟧ ⟦Enter your response⟧
⟦Enter your response 3⟧ ⟦Enter your response⟧ ⟦Enter your response⟧ ⟦Enter your response⟧ 
# ✍ Code for repeated experiments

Optional exercise · Find the cause of an error and fix the code

When character Logits and Targets are flattened along mismatched axes

  1. After training the LSTM, add a Code Cell at the bottom of the notebook and run the following code.

    with torch.no_grad():
    debug_logits = results['LSTM'][0](X)
debug_flat_logits = debug_logits.reshape(-1, debug_logits.shape[-1])
print("Logits:", debug_logits.shape, "flattened:", debug_flat_logits.shape)
for debug_target in [Y, Y.reshape(-1)]:
    print("Target:", debug_target.shape)
    try:
        print("Loss:", F.cross_entropy(debug_flat_logits, debug_target).item())
    except (RuntimeError, ValueError) as error:
        print(type(error).__name__, str(error))

  2. In a Text Cell, tabulate both Target shapes and execution success. Explain what each row of flattened Logits predicts and the axis order needed to list the corresponding Targets.

Check before saving

  1. Check the response fields, outputs, and plots, then run all cells in order in a fresh runtime.
  2. Save to your Drive, then download using .ipynb.

Include code, actual outputs and plots, and your own interpretation. Submit through KLAS; check the semester schedule and latest announcements for the deadline.

Filename: Name_StudentID_Week13.ipynb.