A hybrid parallel DEM approach with workload balancing based on HSFC

• Published in: Engineering computations Vol. 33; no. 8; pp. 2264 - 2287
• Main Authors: Cintra, Diogo Tenório, Willmersdorf, Ramiro Brito, Lyra, Paulo Roberto Maciel, Lira, William Wagner Matos
• Format: Journal Article
• Language: English
• Published: Bradford Emerald Group Publishing Limited 07-11-2016
• Subjects: Access control; Communication; Computation; Computer simulation; Decomposition; Discrete element method; Domain decomposition methods; Environment models; Hilbert curve; Hilbert space; Iterative methods; Kinematics; Message passing; Methodology; Methods; Parallel processing; Recursive methods; Scale models; Simulation; Software; Workload; Workloads; Hybrid parallelization; HSFC; High-performance computing; DEM
• ISSN: 0264-4401; 1758-7077
• DOI: 10.1108/EC-01-2016-0019

Purpose
The purpose of this paper is to present a methodology of hybrid parallelization applied to the discrete element method that combines message-passing interface and OpenMP to improve computational performance. The scheme is based on mapping procedures based on Hilbert space-filling curves (HSFC).
Design/methodology/approach
The methodology uses domain decomposition strategies to distribute the computation of large-scale models in a cluster. It also partitions the workload of each subdomain among threads. This additional procedure aims to reach higher computational performance by adjusting the usage of message-passing artefacts and threads. The main objective is to reduce the communication among processes. The work division by threads employs HSFC in order to improve data locality and to avoid related overheads. Numerical simulations presented in this work permit to evaluate the proposed method in terms of parallel performance for models that contain up to 3.2 million particles.
Findings
Distinct partitioning algorithms were used in order to evaluate the local decomposition scheme, including the recursive coordinate bisection method and a topological scheme based on METIS. The results show that the hybrid implementations reach better computational performance than those based on message passing only, including a good control of load balancing among threads. Case studies present good scalability and parallel efficiencies.
Originality/value
The proposed approach defines a configurable execution environment for numerical models and introduces a combined scheme that improves data locality and iterative workload balancing.