A Fundamental Tradeoff Between Computation and Communication in Distributed Computing

• Published in: IEEE transactions on information theory Vol. 64; no. 1; pp. 109 - 128
• Main Authors: Songze Li, Maddah-Ali, Mohammad Ali, Qian Yu, Avestimehr, A. Salman
• Format: Journal Article
• Language: English
• Published: New York IEEE 01-01-2018; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Benchmark testing; coded multicasting; coded TeraSort; Coding; Communication; Computation; computation-communication tradeoff; Computer networks; Distributed computing; Distributed databases; Distributed processing; Electrical engineering; Electronic mail; Encoding; Information theory; Load distribution (forces); Lower bounds; MapReduce; Nodes; Optimization; Stress concentration; Tradeoffs
• ISSN: 0018-9448; 1557-9654
• DOI: 10.1109/TIT.2017.2756959

How can we optimally trade extra computing power to reduce the communication load in distributed computing? We answer this question by characterizing a fundamental tradeoff between computation and communication in distributed computing, i.e., the two are inversely proportional to each other. More specifically, a general distributed computing framework, motivated by commonly used structures like MapReduce, is considered, where the overall computation is decomposed into computing a set of "Map" and "Reduce" functions distributedly across multiple computing nodes. A coded scheme, named "coded distributed computing" (CDC), is proposed to demonstrate that increasing the computation load of the Map functions by a factor of r (i.e., evaluating each function at r carefully chosen nodes) can create novel coding opportunities that reduce the communication load by the same factor. An information-theoretic lower bound on the communication load is also provided, which matches the communication load achieved by the CDC scheme. As a result, the optimal computation-communication tradeoff in distributed computing is exactly characterized. Finally, the coding techniques of CDC is applied to the Hadoop TeraSort benchmark to develop a novel CodedTeraSort algorithm, which is empirically demonstrated to speed up the overall job execution by 1.97× -3.39×, for typical settings of interest.