DQN

Compare state representations in a Q-table and a neural network on the same Queue problem, and evaluate the learned policy against rule-based policies.

Run the baseline Queue example with NumPy. Compare policies on seed=100…104, which was not used for training.

PyTorch is required. This is a short synthetic experiment; its configuration does not guarantee learning performance.

Public environment activities use gymnasium and stable-baselines3. gymnasium[classic-control] is required only for visualization.

If the required package is missing, run %pip install control==0.10.2 gymnasium==1.3.0 stable-baselines3==2.9.2 in a Colab or Jupyter code cell.

Throughput and cost of equipment and communication resources

An average of 2 items arrive at the synthetic queue each time step. Increasing service capacity reduces queue length but increases energy cost.

Observation·State

Queue length $q=0$…20. Observation $q/20$.

Action

Choose a processing capacity of 0, 2, or 4.

Reward·Constraint

−(next queue length + 0.2×capacity² + 10×dropped items). Time limit at 100 steps.

Comparison baseline

Constant capacity of 2, and a threshold policy based on queue length.

Metrics to check

Evaluation return, mean queue length, mean capacity², and dropped-item count.

Design energy storage operation

A research design case distinguishing the roles of price, load, and stored energy before connecting measured data.

Observation·State

SOC, load, price, and time. Observations must exclude future information.

Action

Charging and discharging power. Decide first between discrete levels and a continuous range.

Reward·Constraint

Electricity and degradation costs. Specify rules for SOC and power limits.

Comparison baseline

Time-of-day rules and, if possible, optimization using the same information.

Metrics to check

Total cost, constraint violations, and maximum grid power draw. Split train/test data by date.

Connect one key change to actual policy metrics instead of adding more full retraining runs.

Starting conditions

• Environment: Queue.
• Network: $1\to32\to3$.
• Optimizer: Adam, lr=.001.
• Discount Factor: $\gamma=.95$.
• Replay: 3000. Batch: 32.
• Training budget: 1800 transitions.

Compare how Q-tables and neural networks represent actions for each state, and evaluate the learned policy.

Procedure

1. Check the performance of Queue's rule-based policies first, then run the provided training function.
2. Plot training loss and selected actions for $q=0$…20.
3. Evaluate return, queue length, cost, and dropped items on seed=100…104, which was not used for training.

Results to retain

• Loss and actions by state plots, and an evaluation table comparing with the rule-based policy.

Starting conditions

SB3 DQN, CartPole-v1, and 2048 transitions. Compare with random and rule-based policies in the same environment, without directly comparing scores with the earlier Queue results.

Procedure

1. Map the public DQN's policy, replay size, target update interval, and learning start to the preceding implementation.
2. Run the provided short training and evaluate deterministically on seed=100…102.
3. Compare episode lengths and returns of the learned policy, Random, and Rule, and state the evidence for judging whether the training budget is sufficient.

Results to retain

• An implementation configuration mapping table and a common evaluation table.

Starting conditions

Based on the preceding experiments, choose Queue, Motor, or one of your research problems and design an experiment for real application.

Procedure

1. Define state, observation, action, reward, termination, and time limit in one table. Record each variable's units and the information you can actually obtain.
2. Write 3 expected transitions numerically and check whether rewards for good and bad actions match your intent.
3. Choose one rule-based comparison baseline, one evaluation condition, and one failure case of greatest concern. Also list the required measurements or simulator.

Results to retain

• MDP design table, 3 transition examples, and comparative experiment plan.
• If real research data are unavailable, design using synthetic conditions from an application case.

You may continue with the research problem and design table selected earlier. Revisit existing entries and transition examples using the concepts learned in this activity, and update changed entries or reasons for retaining them.

• SB3 · DQN
• CartPole · environment definition