Novel Strategies for Responsive Load Balancing in Cloud Applications
Open Access DepositedRecent years have witnessed a substantial surge in cloud adoption, fueled by architectural shifts and the emergence of applications tailored for edge devices. Cloud-deployed applications, typically following the microservices architecture pattern, break down large monolithic structures into smaller, independent components known as microservices. While this approach enhances development flexibility and deployment agility, it introduces an internal network within the application. Consequently, load balancing becomes imperative among microservices, creating both a bottleneck and a potential single point of failure in communication. To address these challenges, the distributed client-side load balancer has been introduced. This architectural shift prompts the adaptation of load balancing algorithms from single-node centralized configurations to the distributed client-side paradigm. However, this transition poses challenges, particularly for dynamic algorithms like Least Connection. While derived from well established and even nearly optimal algorithms like Join-the-Shortest-Queue (JSQ), these algorithms mostly depend on the availability of accurate and current backend state information, which is missing in distributed client side algorithms. This discrepancy hampers its effectiveness in distributed client-side scenarios. In this work, we establish that load balancing algorithms, transposed from the centralized load balancer world, fail to hold up the same properties and actually hamper performance. We recognize that these challenges are unique and the obvious solution is to regain the information lost. The most common approaches for that are to either build centralized state or to pass messages between the load balancers. However, both those solutions become nearly impossible to scale to applications spanning tens of thousands of microservices over data centers spread geographically. Thus we propose that "distributed client-side load balancing" is a distinct subfield within the broader domain of load balancing that requires further exploration. More specifically, in this work we:• Build algorithms that use feedback mechanisms to rectify the information loss in distributed load balancing that most algorithms rely on. • Focus on minimizing response time latency which enables us to dramatically reduce tail latency and response time spread. • These algorithms significantly improve the system’s ability to quickly utilize freshly added resources while completely avoiding herding. • Enable the system as a whole to quickly react to overload conditions. Our results show that for most applications types, our algorithms can reduce response time variability over current state-of-the-art by 2-4 times and 99th percentile latency by about 2 times. Moreover, a new wave of applications operates on edge devices, where one set generates inputs interpreted by cloud-based services, triggering actions on other edge devices. Such applications demand asynchronous connectivity while adhering to predefined time-bound response requirements. Traditional asynchronous application designs, in the absence of any temporal bound on message delivery, struggle to ensure response time guarantees. While it is easy to introduce load balancing here to extract better performance, most of these applications need to maintain a state in the flow of their messages. Hence the goal in this part of our work is find a balance between loss of performance and the cost of rebuilding state in the flow of messages. Towards that goal, we propose and discuss:• Methods that allow us to incorporate synchrony within an asyn- chronous network. • Mechanisms to rebuild state split from moving streams between multiple leaders, that is required for the above. • Algorithm that can handle dynamically changing load whose charac- teristics are unknown ahead of time. Our results in this area show that 99th percentile latency can be reduced by as much as 73%. More importantly, we found that in our experiments the latency profile remained nearly constant when varying the service cost, whereas Apache Kafka’s latency increased super-linearly.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.
| Thumbnail | Title | Date Uploaded | Visibility | Actions |
|---|---|---|---|---|
|
|
Bhattacharya_gwu_0075A_16935.pdf | 2025-04-09 | Open Access |
|