Intelligent Approaches to Improve the Performance of Cloud Storage System by Optimizing Edge Data Transmission and Storage
Open AccessThe emergence of IoT and edge computing will result in smart mobile devices, connected cars, sensors etc. generating large volumes of data, and thus data processing which is used to be done in cloud is being pushed toward edge devices. According to recent market analysis results, the data stored in the edge-facing IoT devices reach 5.9 ZB by 2021 and 43% of such data will be processed in edge devices. The life cycle of data at network edge can be described in three stages, and each stage has its own difficulties in edge computing technologies: (1) processing data in edge devices, which can not only make full use of edge resources, but also sharply reduce data transmission burden between edge devices and the faraway cloud centers; (2) in data transmission, it is hard to optimize the transmission routing paths and the optimization can be easily influenced by interferences; (3) after data transmission, the data stored in the cloud centers is not safe forever, andit is necessary to develop effective methods to monitor the status of cloud servers and set accurate anomaly detection alarms and repair destroyed data quickly and entirely. Thus, it is important to develop intelligent approaches to improve the performance of data processing in edge devices, data transmission effectiveness in routing, and anomaly detection and data repair ability in cloud centers.Firstly, we discuss the stage of processing data in edge stages. Main challenges of this stage includes huge data amount to be processed, and limited edge storage and computing resources. Reportedly, monthly global mobile data traffic will be 30.6 exabytes by 2020. Facing the huge amount of IoT data, existed deduplication researches mainly focus on compressing datafor each device[42, 118], neglecting making use of duplicates among different IoT devices. In this dissertation, since massive data generated in edge devices tends to have large duplication ratio due to chronological and spatial correlation, we propose a novel system for collaborative Edge-Facilitated Deduplication (EF-dedup) wherein we partition edge nodes into disjoint clusters and perform decentralized deduplication within each cluster, optimizing network cost and storage space trade-off. We evaluate the performance of our system in edge nodes on OpenStack and AWS by using real-world IoT data, demonstrating that EF-dedup achieves much better deduplication throughput than other algorithms.Secondly, we talk about the stage of data transmission. Main challenges of this stage includes how to optimize the data transmission and the interferences in this process. IoT devices include home appliances, surveillance cameras, monitoring sensors, actuators, displays, vehicles, and so on, thus the various kinds of IoT devices make it difficult to build stable cooperations among them. Furthermore, the data transmission rate for IoT devices are relatively tiny, varying from250Kbps to 54Mbps, which are the common rates through IEEE 802.15.4, IEEE 802.11, NB-IoT, or ZigBee. The variety and limited transmission rate cause network interferences and bottlenecks. Existed researches focus on estimating network inferences or optimizing network resources on ideally perfect network, but we jointly analyze the influence of different kinds of network inferences on optimizing network resources. In this dissertation, we use Projected Gradient Algorithm(PGA) applied in distributed NUM problem to prove that, under arbitrary, bounded inference errors, distributed traffic optimization algorithms based on the Lagrangian can still converge using running averages of the primal and dual variables.We quantify the optimality gap and validate our theoretic results using extensive NS2 simulations with real world scenarios and showed it worked well in all scenarios.Finally, after data is sent to the data center, it does not mean the data is safe forever, and it is necessary to set accurate anomaly detection for servers in data center and do fast data repair after servers are crashed. For anomaly detection, it is challenging to make precise anomaly predictions by using various kinds of metrics of cloud servers. Existed works only use few metrics to estimate anomaly status for servers in data centers. In this dissertation, we present a machine learning basedtwo-layer system to predict anomaly statuses of cloud servers and launch alarms. We use Amazon CloudWatch service to collect monitoring metrics from Amazon EC2 instances, and use them to predict abnormalities in the future. The first layer is a LSTM(Long short-term memory) network, which is used to predict the future metric vectors. The second layer is to compare the predicted future metrics vectors with current ones, and learn to judge whether abnormalities will appear. The novelty of our design includes that our system could adapt to a much wider range of prediction problems than existed algorithms, and the two-layer structure could extend alarming time and increase prediction accuracy as well. We compare the performance of our proposed system with other popular machine learning algorithms, and the accuracy of our system is much better than others.For data repair, in newly appeared erasure-coded storage systems, it is challenging to optimize the data repair process due to complicated problems includes repair data choices, repair data transmission optimization, and repair task management. Few existed works only consider one or two problems from data selection for repair, data repair deadlines, and bandwidth allocation. In this dissertation, we propose our research on how to repair data efficiently when data is destroyedin erasure-coded storage data centers. We develop an online algorithm to maximize the number of tasks that successfully meet deadlines, under the constraints of data placement, network topology and available bandwidth. The problem is challenging because we need to jointly solve a chunk selection problem that determines the (erasure-coded) chunks used to generate background traffic, a bandwidth allocation problem, and a scheduling problem that schedules tasks with respect to their deadlines. The novelty of our work is our proposed algorithm jointly considers current network topology, source selection and bandwidth constraints. We evaluate our proposed algorithm extensively in both simulation and real experiments in an OpenStack cluster, and demonstrate it has much better performance than other existed algorithms.In conclusion, in this dissertation, we propose various novel models, algorithms and systems to conquer these complicated problems at the same time, prove their theoretical performance, and evaluate the performance in simulations and real-world data and systems. These unique approaches include minimizing storage space, optimizing network resources, network interference analysis, and anomaly prediction, and many of them solve multiple problems simultaneously and even needto balance the conflicts among different tasks. The proposed algorithms and systems are evaluated in both simulations and real-world data and nodes.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.