Characterization of continuous experimentation in software engineering: Expressions, models, and strategies

• Published in: Science of computer programming Vol. 229; p. 102961
• Main Authors: Erthal, Vladimir M., de Souza, Bruno P., dos Santos, Paulo Sérgio M., Travassos, Guilherme H.
• Format: Journal Article
• Language: English
• Published: Elsevier B.V 01-07-2023
• Subjects: A/b testing; Continuous experimentation; Controlled experiments; Data-driven development; Evidence-based software engineering; A/b testing; Continuous experimentation; Data-driven development; Controlled experiments; Evidence-based software engineering
• ISSN: 0167-6423; 1872-7964
• DOI: 10.1016/j.scico.2023.102961

Continuous Experimentation (CE) has become increasingly popular across industry and academic communities. Major software organizations use CE to increase their revenue by adding value to end-users, and researchers are investigating the CE adoption process and usage to expand its success. Given this rapid evolution, observing a shared understanding of CE definitions, processes, and experiment strategies is difficult, potentially jeopardizing new implementations and focused research efforts.
To characterize CE from the perspective of its definitions, processes, and strategies for experimentation available in the technical literature and to evolve the understanding perspectives for “continuous experimentation” and “data-driven development” definitions.
To select and analyze sources of information in the technical literature dealing with different aspects of continuous experimentation through a Literature Study using an ad hoc search improved with snowballing (backward and forward). Organize the findings into new perspectives for CE definitions, processes, and experiment strategies.
It was possible to identify many different definitions, processes, and experimental strategies used to describe CE in the 72 analyzed empirical papers, making it difficult to decide on their combination to be applied in a real software development project. Therefore, it has been proposed to evolve the CE understanding perspective, to categorize its experiment strategies, and to offer a combined development process for CE combining parts of other processes. Besides, conjectural requirements have been identified, which can contribute to better differentiating requirements and hypotheses in the CE context.
Likely, a better understanding of CE is still missing. It can contribute towards organizing a common taxonomy to facilitate the possible choices for the experiment strategies. Therefore, there is space for more investigations on its applicability and value in different categories of software systems, despite all the advancements of CE and its promotion in developing modern software systems.
•It largely extends and brings new contributions regarding the paper presented at CibSe2022.•It offers an organized evidence-based characterization of Continuous Experimentation in Software Engineering.•It presents a Combined Process for Continuous Experimentation in Software Engineering.•It raises the discussions on new perspectives on Continuous Experimentation in Software Engineering.