NiiL

Week 13 · Split long text to predict the next character

English translation

Before practice

01 How to train on long data

If an entire long text is difficult to process at once, how could you organize an exercise that reads segments and predicts the next character?

Description
  • Use consecutive Windows as Input
  • Set the next character at each position as Target. Example: abcd → bcde

02 Remember necessary information

Information near the beginning of a sentence may be needed for predictions much later. What problem could arise if earlier information is continually overwritten?

Description
  • Control how much information is retained or updated
  • Compare performance and cost with the same data and conditions. More complex structures are not always better

Practice

1. Create short training examples from long text

  1. In the first example cell, print the two Input–Target pairs created by segmenting hello! with length 3 and Stride 2. Put each pair's Input and Target in two rows and check the one-character shift within each column.
  2. Apply the same markings to the first two Input–Target pairs in the long-text data cell. Record how many characters separate their starting positions.
  3. Match each axis of X.shape and Y.shape to the number of Windows NN, length SS, and character representation size EE.

2. Compare RNN, LSTM, and GRU

  1. Run the model training cell. In one table, record the final Loss, Training Accuracy, and predicted text for the first Window for each RNN, LSTM, and GRU. Use identical data, epochs, and Learning Rate for all three models.
  2. Run the State output cell and tabulate the shapes of output, h_n, and c_n for each model. Mark States that are not returned as “None.”
  3. Explain the difference between the Sequence axis of output and the final-time-step representation in h_n, and describe the role of c_n, which LSTM additionally returns.

Submission and saving

Include code, actual outputs and plots, and your own interpretation. Submit through KLAS; check the semester schedule and latest announcements for the deadline.

Filename: Name_StudentID_Week13.ipynb.

Restart the runtime, run all code from the top, then save.

Optional practice

When character Logits and Targets are flattened along mismatched axes

  1. After training the LSTM, add a Code Cell at the bottom of the notebook and run the following code.
    with torch.no_grad():
    debug_logits = results['LSTM'][0](X)
debug_flat_logits = debug_logits.reshape(-1, debug_logits.shape[-1])
print("Logits:", debug_logits.shape, "flattened:", debug_flat_logits.shape)
for debug_target in [Y, Y.reshape(-1)]:
    print("Target:", debug_target.shape)
    try:
        print("Loss:", F.cross_entropy(debug_flat_logits, debug_target).item())
    except (RuntimeError, ValueError) as error:
        print(type(error).__name__, str(error))
  2. In a Text Cell, tabulate both Target shapes and execution success. Explain what each row of flattened Logits predicts and the axis order needed to list the corresponding Targets.