MaRe: a MapReduce-Oriented Framework for Processing Big Data with Application Containers

• Main Authors: Capuccini, Marco, Dahlö, Martin, Toor, Salman, Spjuth, Ola
• Format: Journal Article
• Language: English
• Published: 07-08-2018
• Subjects: Computer Science - Distributed, Parallel, and Cluster Computing
• DOI: 10.48550/arxiv.1808.02318

Background. Life science is increasingly driven by Big Data analytics, and
the MapReduce programming model has been proven successful for data-intensive
analyses. However, current MapReduce frameworks offer poor support for reusing
existing processing tools in bioinformatics pipelines. Further, these
frameworks do not have native support for application containers, which are
becoming popular in scientific data processing.
Results. Here we present MaRe, a programming model with an associated
open-source implementation, which introduces support for application containers
in MapReduce. MaRe is based on Apache Spark and Docker, the MapReduce framework
and container engine that have collected the largest open source community,
thus providing interoperability with the cutting-edge software ecosystem. We
demonstrate MaRe on two data-intensive applications in life science, showing
ease of use and scalability.
Conclusions. MaRe enables scalable data-intensive processing in life science
with MapReduce and application containers. When compared with current best
practices, that involve the use of workflow systems, MaRe has the advantage of
providing data locality, ingestion from heterogeneous storage systems and
interactive processing. MaRe is generally-applicable and available as open
source software.