NiiL

Remember previous characters to predict the next one

English translation

Create your own copy to run, edit, and record results.

Self-study material

1. Construct a character Batch

Fix vocabulary order and shift inputs and targets by one position.

Character Indices start at 0. The One-hot length and final Class count equal the character vocabulary size; Hidden Size is chosen separately. Each of the three sentences contains 9 characters including spaces and punctuation. Shifting by one position gives Input and Target of 8 characters each. For howareyou, use howareyo → owareyou.

A. Baseline example · Read and run first

import sys, numpy as np, matplotlib.pyplot as plt
import torch
from torch import nn
import torch.nn.functional as F
torch.manual_seed(42)
torch.set_num_threads(2)
print('Python:', sys.version.split()[0], '| PyTorch:', torch.__version__)
sentences=['howareyou','whats up?','iamgreat.']
chars=sorted(set(''.join(sentences)));vocab={c:i for i,c in enumerate(chars)}
length=min(map(len,sentences))-1
ids=torch.tensor([[vocab[c] for c in s[:length]] for s in sentences])
Y=torch.tensor([[vocab[c] for c in s[1:length+1]] for s in sentences])
X=F.one_hot(ids,num_classes=len(chars)).float()
print('X:',X.shape,'Y:',Y.shape)
print('input:',sentences[0][:length],'target:',sentences[0][1:length+1])

B. Hands-on activities

  1. Run the data cell and print the Input and Target characters in two rows for each of the three sentences. Check that each Target character is immediately after the Input character in the same column in the original text.
  2. Print X.shape and Y.shape. Label each axis with the appropriate meaning: sentence count, character count, or Vocabulary size.
  3. Print the Vocabulary's character list and record the positions containing spaces and punctuation.

Your prediction may describe a shape or an increasing/decreasing trend rather than an exact number. Record explanations in the Text Cell below.

# ✍ Write your own: add your code below this cell.
# You may add Code Cells if the code gets long.

C. Your record · Double-click this Text Cell to edit

2. Train an RNN

Predict characters at every time step.

A. Baseline example · Read and run first

class CharModel(nn.Module):
    def __init__(self,kind,vocab_size,hidden=32):
        super().__init__();self.rnn=getattr(nn,kind)(vocab_size,hidden,batch_first=True);self.fc=nn.Linear(hidden,vocab_size)
    def forward(self,x):
        output,state=self.rnn(x);return self.fc(output)
def fit_char(kind,X,Y,epochs=300,lr=.01):
    torch.manual_seed(42);m=CharModel(kind,X.shape[-1]);opt=torch.optim.Adam(m.parameters(),lr=lr);losses=[]
    for epoch in range(epochs):
        logits=m(X);loss=F.cross_entropy(logits.reshape(-1,X.shape[-1]),Y.reshape(-1))
        opt.zero_grad();loss.backward();opt.step();losses.append(loss.item())
        if epoch%100==0:print(kind,epoch,loss.item())
    with torch.no_grad():pred=m(X).argmax(-1)
    return m,losses,pred
torch.manual_seed(42)
untrained = CharModel('RNN', X.shape[-1])
with torch.no_grad(): before_pred = untrained(X).argmax(-1)
print('Before training:', ''.join(chars[i] for i in before_pred[0]))
model,losses,pred=fit_char('RNN',X,Y)
for target,guess in zip(Y,pred):print('target:',''.join(chars[i] for i in target),'prediction:',''.join(chars[i] for i in guess))
plt.plot(losses);plt.title('RNN training loss');plt.xlabel('Epoch');plt.ylabel('Loss');plt.grid();plt.show()

B. Hands-on activities

  1. Compare before_pred[0] printed by the training cell and post-training pred[0] character by character against Y[0]. Record incorrect positions before and after training separately.
  2. Feed X[:, :4, :], X[:, :6, :], and X[:, :8, :] separately to the trained model. Run within torch.no_grad() without retraining.
  3. Record input Sequence length, full input shape, and full output shape in a three-row table. Mark the output axis that changes with length.

Your prediction may describe a shape or an increasing/decreasing trend rather than an exact number. Record explanations in the Text Cell below.

# ✍ Write your own: add your code below this cell.
# You may add Code Cells if the code gets long.

C. Your record · Double-click this Text Cell to edit

Practice through repetition · Reason before running

Summarize the three conditions compared in the preceding activities in this table. You do not need to repeat experiments already completed. If comparisons are insufficient, supplement them by changing just one condition.

Changed condition Predicted shape/value/result Actual result Cause of any error
⟦Enter your response 1⟧ ⟦Enter your response⟧ ⟦Enter your response⟧ ⟦Enter your response⟧
⟦Enter your response 2⟧ ⟦Enter your response⟧ ⟦Enter your response⟧ ⟦Enter your response⟧
⟦Enter your response 3⟧ ⟦Enter your response⟧ ⟦Enter your response⟧ ⟦Enter your response⟧ 
# ✍ Code for repeated experiments

Optional exercise · Find the cause of an error and fix the code

When Batch and Sequence axes are swapped

  1. After training the RNN with three sentences, add a Code Cell at the bottom of the notebook and run the following code.

    for debug_input in [X, X.transpose(0, 1)]:
    with torch.no_grad():
        debug_output = model(debug_input)
    print("Input:", debug_input.shape, "Output:", debug_output.shape,
          "Target:", Y.shape)

  2. In a Text Cell, record the Batch size and Sequence length interpreted by the model in each call. With batch_first=True, the first axis is Batch. Explain which input treats each of the three sentences as one Sequence, and why the other call also runs without an error.

Check before saving

  1. Check the response fields, outputs, and plots, then run all cells in order in a fresh runtime.
  2. Save to your Drive, then download using .ipynb.

Include code, actual outputs and plots, and your own interpretation. Submit through KLAS; check the semester schedule and latest announcements for the deadline.

Filename: Name_StudentID_Week12.ipynb.