NiiL

Public environments and experiment records · Learning

English translation

Execution conditions

Run the baseline Queue example with NumPy. Compare policies on seed=100…104, which was not used for training.

Public environment activities use gymnasium and stable-baselines3. gymnasium[classic-control] is required only for visualization.

If the required package is missing, run %pip install control==0.10.2 gymnasium==1.3.0 stable-baselines3==2.9.2 in a Colab or Jupyter code cell.

Topics covered

One line of termination handling can change the training target. Start by checking environment boundaries and fair policy evaluation.

Overall workflow

Environment and observations → policy and actions → transitions and rewards → evaluation under common conditions

Unit of interaction

reset returns the initial observation and info; step returns the new observation, reward, terminated, truncated, and info. Distinguish actual termination from collection time limits to calculate training targets correctly.

Transitions to reuse

Store the observation, action, reward, next observation, and termination type together. Episode boundaries, environment ID, and seed are needed to interpret the transitions later.

Worked example

Run the reset → step → termination flow using the Queue example. Use seed=7, action 1 (capacity 2), and 100 steps.

"""Synthetic service queue. An action chooses capacity, including its energy cost."""
import numpy as np

class Queue:
    def reset(self, seed=0):
        self.rng = np.random.default_rng(seed)
        self.q, self.t = 0, 0
        return self.q

    def step(self, action):
        capacity = (0, 2, 4)[int(action)]
        arriving = int(self.rng.poisson(2))
        raw = max(0, self.q - capacity) + arriving
        dropped = max(0, raw - 20)
        self.q = min(20, raw)
        self.t += 1
        reward = -(self.q + .2 * capacity**2 + 10 * dropped)
        return self.q, reward, False, self.t >= 100, {'queue': self.q, 'capacity': capacity, 'dropped': dropped}

def evaluate(policy, seeds=range(100, 105)):
    rows = []
    for seed in seeds:
        env = Queue(); q = env.reset(seed)
        total = waiting = energy = dropped = 0
        while True:
            q, r, terminated, truncated, info = env.step(policy(q))
            total += r; waiting += q; energy += info['capacity']**2; dropped += info['dropped']
            if terminated or truncated: break
        rows.append([total, waiting / env.t, energy / env.t, dropped])
    return np.array(rows)

policies = {'constant': lambda q: 1, 'threshold': lambda q: 0 if q == 0 else (1 if q < 5 else 2)}
for name, policy in policies.items():
    print(name, evaluate(policy).mean(axis=0))

env=Queue();q=env.reset(7)
for k in range(5):
    nq,r,te,tr,info=env.step(1);print(k,q,1,r,nq,te,tr);q=nq

constant [-921.2      8.052    4.       3.6  ]
threshold [-387.4     2.77    5.52    0.  ]
0 0 1 -3.8 3 False False
1 3 1 -3.8 3 False False
2 3 1 -4.8 4 False False
3 4 1 -3.8 3 False False
4 3 1 -3.8 3 False False

Official documentation and data

Reference example

View more examples