Optimizing network performance of computing pipelines in distributed environments

Qishi Wu, Yi Gu, Mengxia Zhu, Nageswara S.V. Rao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

15 Scopus citations

Abstract

Supporting high performance computing pipelines over wide-area networks is critical to enabling large-scale distributed scientific applications that require fast responses for interactive operations or smooth flows for data streaming. We construct analytical cost models for computing modules, network nodes, and communication links to estimate the computing times on nodes and the data transport times over connections. Based on these time estimates, we present the Efficient Linear Pipeline Configuration method based on dynamic programming that partitions the pipeline modules into groups and strategically maps them onto a set of selected computing nodes in a network to achieve minimum end-to-end delay or maximum frame rate. We implemented this method and evaluated its effectiveness with experiments on a large set of simulated application pipelines and computing networks. The experimental results show that the proposed method outperforms the Streamline and Greedy algorithms. These results, together with polynomial computational complexity, make our method a potential scalable solution for large practical deployments.

Original languageEnglish
Title of host publicationIPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM
DOIs
StatePublished - 10 Sep 2008
EventIPDPS 2008 - 22nd IEEE International Parallel and Distributed Processing Symposium - Miami, FL, United States
Duration: 14 Apr 200818 Apr 2008

Publication series

NameIPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM

Other

OtherIPDPS 2008 - 22nd IEEE International Parallel and Distributed Processing Symposium
CountryUnited States
CityMiami, FL
Period14/04/0818/04/08

    Fingerprint

Cite this

Wu, Q., Gu, Y., Zhu, M., & Rao, N. S. V. (2008). Optimizing network performance of computing pipelines in distributed environments. In IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM [4536465] (IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM). https://doi.org/10.1109/IPDPS.2008.4536465