Accumulating computational resource usage of genomic data analysis workflow to optimize cloud computing instance selection

• Published in: Gigascience Vol. 8; no. 4
• Main Authors: Ohta, Tazro, Tanjo, Tomoya, Ogasawara, Osamu
• Format: Journal Article
• Language: English
• Published: United States Oxford University Press 01-04-2019
• Subjects: Bioinformatics; Cloud Computing; Computational Biology - methods; Computer applications; Containers; Data analysis; Genomic analysis; Genomics - methods; High-Throughput Nucleotide Sequencing; Run time (computers); Software; Transcriptomes; Workflow; cloud computing; Common Workflow Language; high-throughput nucleotide sequencing
• ISSN: 2047-217X; 2047-217X
• DOI: 10.1093/gigascience/giz052

Container virtualization technologies such as Docker are popular in the bioinformatics domain because they improve the portability and reproducibility of software deployment. Along with software packaged in containers, the standardized workflow descriptors Common Workflow Language (CWL) enable data to be easily analyzed on multiple computing environments. These technologies accelerate the use of on-demand cloud computing platforms, which can be scaled according to the quantity of data. However, to optimize the time and budgetary restraints of cloud usage, users must select a suitable instance type that corresponds to the resource requirements of their workflows.
We developed CWL-metrics, a utility tool for cwltool (the reference implementation of CWL), to collect runtime metrics of Docker containers and workflow metadata to analyze workflow resource requirements. To demonstrate the use of this tool, we analyzed 7 transcriptome quantification workflows on 6 instance types. The results revealed that choice of instance type can deliver lower financial costs and faster execution times using the required amount of computational resources.
CWL-metrics can generate a summary of resource requirements for workflow executions, which can help users to optimize their use of cloud computing by selecting appropriate instances. The runtime metrics data generated by CWL-metrics can also help users to share workflows between different workflow management frameworks.