Belief-State Q-Learning for Dynamic Channel Selection in Cognitive Radio Networks: A Simulation Study

Dynamic spectrum access requires a secondary user (SU) to repeatedly choose which of several primary-user (PU) channels to sense and access under channel occupancy that is correlated in time (a “restless” process) and a cost for switching channels. This paper presents a simulation-based performance evaluation of channel-selection policies for cognitive radio (CR) networks, modeling each channel as an
independent two-state (idle/busy) Markov chain. As its principal contribution, the paper implements and benchmarks a belief state Q-learning agent — which discretizes a continuously propagated per-channel occupancy belief into a tabular state and learns a channel-selection policy with no knowledge of the
underlying transition probabilities — against four baselines (Random, Round-Robin, frequentist ε-greedy, and UCB1) and a model-based myopic belief policy given the true channel statistics. Averaged over 12 independent runs of 20,000 time slots with 5 heterogeneous channels, the proposed Q-learning agent achieves 74.6% steady-state throughput, versus 71.1% for UCB1, 69.2% for ε-greedy, and 66.7%/67.3% for Random/Round-Robin, and comes within 4.1 percentage points of the model-based myopic policy (77.8%). The paper further quantifies cumulative regret against a genie oracle, the throughput/switching-rate trade-off as a function of an explicit channel-switching cost, sensitivity to the number of channels,
the effect of the exploration schedule on convergence, and a state-space-coverage limitation of the tabular approach. All results are produced by Python-based Monte Carlo simulation (NumPy); no physical RF measurements were taken.