Decision-Making via Optimization in Challenging Environment: Bayesian and Multi-Agent Reinforcement Learning Perspectives
Open Access DepositedDecision-making is essential in many real-world problems, where the goal is to select actions that optimize long-term outcomes under uncertainty. From tuning hyperparameters to devising control strategies in robotics, decision-making often involves optimizing complex, unknown functions that are costly to evaluate or too intricate to model analytically. Achieving efficient decision-making requires advanced approaches to excel in uncertain environments, making decisions iteratively based on partial information. Under this purpose, this dissertation explores decision-making by optimizing unknown functions via Bayesian optimization (BO) and investigates optimized decision-making in the multi-agent reinforcement learning (MARL) domain. For BO, we address two areas: finding local optima in multimodal black-box functions and optimizing functions with discrete spatiotemporal data inputs. In MARL, we minimize regret to improve decision-making, focusing on finding important transitions in replay buffers and optimal projection of an unconstrained mixing function onto monotonic function classes, allowing revealing better policies and enhancing decision-making efficiency. The first part of this dissertation introduces two new Bayesian optimization algorithms in different settings to achieve efficient decision-making via optimizing the unknown functions. In the starting chapter, we introduce a multimodal BO framework to discover both local and global optima in expensive multimodal objective functions. Current methods primarily focus on finding a single global optimum. However, in many real-world problems, multiple optimal solutions are needed due to practical constraints such as resource limitations or physical restrictions. By identifying multiple optima, alternative solutions can be implemented when needed, ensuring optimal system performance. Furthermore, in the next chapter, we propose a novel BO framework by extending the standard BO algorithm to handle discrete point data inputs using a doubly stochastic process to model the unknown function. This design is based on maximum a posteriori inference for Gaussian Cox processes, utilizing the Laplace approximation and kernel transformations to improve computational tractability. This method provides both a functional posterior and a covariance estimate for the latent intensity function. To further enhance efficiency, we introduce a Nyström approximation to reduce the computational overhead and handle the situation where the explicit Mercer’s expansion does not exist. The second part of this dissertation introduces two optimization methods for better decision-making in multi-agent reinforcement learning (MARL). In this chapter, we focus on optimizing prioritized sampling weights for experience replay based on the unknown policy regret minimization problem, an area yet to be explored, and propose a new algorithm, MAC-PO. Minimizing policy regret reduces the gap between the current policy and an optimal nominal policy, thereby improving prioritization in multi-agent tasks and enhancing decision-making. In the last main chapter, we discuss finding the optimal projection of an unconstrained mixing function onto monotonic function classes and propose ReMIX, which casts this optimal projection problem for value function factorization as regret minimization over projection weights of different transitions. This optimization problem can be relaxed and solved using the Lagrangian multiplier method to obtain the optimal projection weights in a closed form, thereby enhancing the monotonic value function factorization for MARL decision-making tasks. Overall, this dissertation presents new optimization techniques to enhance decision-making in BO and MARL. These contributions advance decision-making in complex, uncertain environments, with broad implications for machine learning systems and real-world environmental and social analyses.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.