Towards Formalizing Data-Driven Decision-Making from Big Data: A Systematic Multi-Criteria Decision-Making Approach in Online Controlled Experiments
Open AccessUser-intensive software systems such as web and mobile applications are defined as systems that serve and interact with an increasingly large number of users. Inefficient launch decisions in user-intensive systems in domains such as social media, information retrieval and e-commerce can lead to dramatic loss in the goal metrics of these highly scalable applications, and therefore impact potentially billions of users. Due to the complexity of user-intensive systems, engineers rely heavily on A/B testing (i.e., online controlled experiments) to evaluate and measure the impact of new changes. However, little attention has been paid to improving the empirical process of making launch decisions based on the A/B testing results. In this paper, we propose a framework to address this issue. We present a Multi-Criteria Decision-Making (MCDM) framework that uses A/B Testing results to provide launch decisions analysis, as a complementary tool to assist decision-making. The framework includes modules for 1) configuration setup, 2) Criteria Weighting, 3) pairwise comparison between criteria and alternatives, 4) Analysis of Alternatives using MCDM and produces launch decisions based on the A/B testing results. Experimental results from publicly available dataset that compares well-known and widely applied MDCM methods shows that a good combination of the Analysis of Alternative method (such as TOPSIS-Vector, MMOORA, and VIKOR) and Criteria Weighting method (such as Standard Deviation and AHP) in the framework led to more effective launch decision-making. This indicates that the empirical decision-making process from analyzing big data could be formalized in engineering development.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.