Algorithm and Architecture Independent Benchmarking with SEAK

• Published in: 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS) pp. 63 - 72
• Main Authors: Tallent, Nathan R., Manzano, Joseph B., Gawande, Nitin A., Seunghwa Kang, Kerbyson, Darren J., Hoisie, Adolfy, Cross, Joseph K.
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01-05-2016
• Subjects: Benchmark testing; bottleneck specification; Computer architecture; constraining problem; functional; Kernel; Measurement; mission interface; Parallel processing; Software algorithms; tradeoff evaluation
• ISSN: 1530-2075
• DOI: 10.1109/IPDPS.2016.25

Many applications of high performance embedded computing are constrained by performance or power bottlenecks. We designed a new benchmark suite, the Suite for Embedded Applications and Kernels (SEAK), (a) to capture these bottlenecks in a way that encourages creative solutions, and (b) to facilitate rigorous tradeoff evaluation for their solutions. To avoid biases toward existing solutions, both algorithms and architecture are variables. Thus, each benchmark has a mission-centric (abstracted from a particular algorithm) and goal-oriented (functional) specification. To encourage solutions that are any combination of software or hardware, we use an end-user black-box evaluation. To inform procurement decisions, evaluations capture tradeoffs between performance, power, accuracy, size, and weight. We call our benchmarks future proof because they remain useful despite shifting algorithmic/architectural preferences. To create both concise and precise mission-centric specifications, we introduce two distinct benchmark classes. This paper describes the SEAK suite and presents an evaluation of sample solutions that highlights power and performance tradeoffs.