Effective Value Function Factorization and Exploration in Multi-Agent Reinforcement Learning
Open Access DepositedThe evolution of computer science has profoundly impacted decision-making processes across diverse domains, culminating in the development of artificial intelligence (AI) and, more specifically, reinforcement learning (RL). RL, inspired by behavioral psychology, involves training agents to make sequential decisions through interactions with their environment, aiming to maximize cumulative rewards. While early RL algorithms were effective in single-agent scenarios, real-world problems often require coordination among multiple interacting entities. This realization has driven the advancement of multi-agent reinforcement learning (MARL), where multiple agents learn and make decisions within a shared environment. MARL introduces significant challenges, such as managing the exponentially growing joint state-action space and ensuring stable learning in dynamic, multi-agent settings. The field of multi-agent reinforcement learning (MARL) has seen substantial advancements, in addressing the challenges of coordinating multiple agents in complex environments. This dissertation explores several novel methodologies within the paradigm of Centralized Training with Decentralized Execution (CTDE) to enhance the representational capacity of value functions, optimize experience replay mechanisms, and improve off-policy evaluation in MARL. This dissertation addresses the complexities of MARL by introducing novel methods and frameworks that enhance value function representation and coordination among agents. One significant contribution is the development of PAC, a new architecture for assisted value factorization using counterfactual predictions. PAC leverages a counterfactual baseline to marginalize an agent's potential optimal action, generating assistive information that expands the representational ability of value functions. This approach directly optimizes the factorization of the joint action-value function, \(Q_{tot}\), through a novel counterfactual loss, ensuring relevant and optimal assistive information. By decoupling decision-making from value function networks and maximizing entropy, PAC maintains the Centralized Training and Decentralized Execution (CTDE) paradigm while encouraging exploration. Empirical results demonstrate PAC's superior performance and faster convergence on complex tasks requiring significant coordination among agents. Simultaneously, the dissertation explores improvements in the experience replay mechanism, which is crucial for off-policy RL. The experience replay mechanism allows the utilization of historical experiences stored in a replay buffer for training, significantly enhancing policy learning and stability. While prioritized experience replay (PER) has shown success in single-agent RL by prioritizing experiences based on the temporal-difference (TD) error, its extension to MARL presents unique challenges. Standard sampling strategies in MARL do not adequately reflect the dynamics caused by multi-agent interactions, leading to suboptimal training. To address this, the dissertation proposes MAC-PO, a method that formulates MARL prioritized experience replay as a regret minimization problem over sampling weights. By minimizing policy regret, optimal sampling weights are derived, considering the interactions among all agents, leading to improved training efficiency and performance. In summary, this dissertation contributes to the field of MARL whose innovations collectively address key challenges in RL, paving the way for more effective and efficient learning algorithms in complex multi-agent environments. The findings underscore the importance of advanced representation and coordination techniques in overcoming the inherent difficulties of MARL, offering valuable insights and methodologies for future research and applications.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.