Skip to main content
Back to timeline
arXivSource publication:

Agents that see only total demand can still learn equilibrium: a 0.332 deviation gap in a ten-prosumer microgrid

Synopsis

In Markov games where each reward depends only on an agent's own action and an aggregate of others' actions, a new decentralized algorithm lets every agent use a broadcast aggregate to evaluate the immediate rewards of all candidate actions and to forecast their next losses, without observing opponents' individual actions or the transition kernel. It yields a high-probability regret bound that depends on prediction error while keeping the standard worst-case order; in a ten-prosumer microgrid with stochastic renewables and shared storage, its final unilateral deviation gap is 0.332, below three learning baselines, with favorable user rewards and high renewable utilization.

Source-provided article image: Exploiting Aggregate Feedback for Decentralized Equilibrium Learning in Markov Games: A Smart Grid Application
arXiv

Interpretation

Agents can learn a game equilibrium from a public aggregate signal even without knowing opponents' individual decisions, instead of relying only on feedback from the action they actually played. Earlier decentralized learning algorithms for Markov games mainly used bandit feedback, seeing only the reward of the selected action and not exploiting the extra information in aggregate observations. In the ten-prosumer microgrid experiment, the trained algorithm reached a unilateral deviation gap of 0.332, below three learning baselines: Mao-style V-learning, its adaptive-stage variant, and a bandit-feedback ablation that discards the full immediate-loss feedback.

Each agent uses its known local reward function and the observed aggregate to compute the immediate loss of every candidate action, then forecasts the next immediate loss from historical aggregates, while continuation values are still learned only from the selected action's state transitions. This hybrid-feedback design turns immediate loss into full feedback while keeping bandit-style estimation for the unknown transition kernel, so no opponent-specific information or knowledge of the kernel is needed. The algorithm splits learning into two timescales: a slow scale adapts stage lengths to aggregate fluctuations and performs optimistic value estimation, and a fast scale updates policies with aggregation-predictive Exp3-IX inside each stage; in the microgrid experiment its normalized gap-AUC was lower than that of Mao-style V-learning.

Perspective

The demonstrated setting is a ten-prosumer, five-period microgrid simulation with stochastic renewables and shared storage, trained with five matched random seeds and evaluated over 300 independent test episodes; the theoretical guarantee applies to a certified output policy built from historical policies, while the reported deviation gap is an offline diagnostic of the current policy.

The simulation uses a half-scaled stage-end bonus, so its empirical results do not instantiate the theorem for the unscaled bonus and do not imply a worst-case convergence-rate improvement; the theoretical guarantee applies to the certified output policy, while the directly learned policies were only observed to have relatively small deviation gaps and favorable rewards.