Data-driven Online Network Optimization through Reinforcement Learning
Open AccessIn the past decade, the Internet has expanded and developed rapidly. With more users getting access to the Internet, many new applications appeared to fulfill the ever-growing needs of users -- from personal entertainment to the world economy. These new applications continuously bring challenges to the whole Internet ecosystem, targeting the computation resource, reliability, latency, energy, etc. Various problems are proposed to resolve these challenges, including data caching, data rescheduling/prefetching, service placement, etc. Aiming to solve each proposed problem, existing works utilize varied techniques to optimize network performance in different aspects. Traditional optimization methods take advantage of the awareness of problem models. With the mathematical model, the optimal decision can be chosen at each system state to guarantee the maximum reward. For simple optimization objectives that can be easily calculated using the mathematical models, the model-based methods proved their efficiencies for many research fields. However, when solving real-world network optimizations, with the growth of the problem complexities, e.g., the optimization objective is jointly decided by multiple system metrics, the amount of time and space to obtain the calculation result could be extremely large. Thus, the utilization of novel model-free techniques, such as machine learning, is motivated. This dissertation aims to solve the optimization quests for applications on networks using model-free methods. In other words, the optimizer does not require full information on the problem models. This model-free optimization task can be resolved by a state-of-the-art solution -- Deep Reinforcement Learning (DRL). We concluded the advantages of DRL as: (i) it is able to explore the state transitions automatically and self-improve the decision making, thus an accurate problem model is not required; (ii) utilizing neural networks, the system state spaces can be vast so that complex problems involving multiple state variables can be easily optimized; (iii) with the trained neural network, DRL consumes fixed time span and storage space, which assures the scalability of problem applications. In this dissertation, we use various methods to optimize the performances for different network applications, including mobile advertisements, video streaming, and service tree placement. The optimized targets include energy consumption, data cache hit ratio, data stream process stall, multi-user quality of experience, etc. We model the problems into decision-making processes -- not limited to Markov Decision Processes (MDP), so that DRL could be applied. It is worth mentioning that since the neural networks have limited output layer sizes, we develop novel methods to break down large action spaces so that the scalabilities of problems are guaranteed. By implementing each optimization problem on our designed testbeds, the proposed DRL optimizers showed great potentials and outperformed existing baseline decision-making policies.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.