Follower-Aware Reward Shaping for Leader–Follower Formation Control via Dueling Deep Q-Networks

Abstract

This paper proposes a follower-aware reward shaping strategy for leader–follower formation control of non-holonomic mobile robots, implemented within a hybrid Dueling Deep Q-Network (Dueling DQN) architecture. The leader employs a Dueling DQN enhanced with Prioritized Experience Replay, Frame Stacking, and a novel follower-aware safety penalty trained under a Curriculum Learning regime with progressive Domain Randomization. The follower tracks a virtual target located behind the leader using a proportional–derivative controller, thereby decoupling computational complexity between the agents. Validated in an 8 × 8 m environment featuring a 2.0 m inner gap and a 1.0 m lateral exit door (d_collision = 0.50 m), the proposed system achieves 100% success across three fixed-geometry configurations (500 episodes each) and 74.0% in a procedural generalization test with randomized obstacle placement. Comprehensive ablation studies confirm that the follower-aware reward shaping strategy is the single most impactful component, outweighing established techniques such as Prioritized Experience Replay and dueling decomposition. The findings provide empirical evidence that explicitly encoding cooperative safety constraints into the leader’s policy yields emergent path-planning behaviors that account for the follower’s kinematic limitations, offering a practical pathway toward robust multi-robot coordination in cluttered environments.