Optimizing network performance of computing pipelines in distributed environments

Qishi Wu, Yi Gu, Michelle Zhu, Nageswara S.V. Rao

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

15 Citations (Scopus)

Abstract

Supporting high performance computing pipelines over wide-area networks is critical to enabling large-scale distributed scientific applications that require fast responses for interactive operations or smooth flows for data streaming. We construct analytical cost models for computing modules, network nodes, and communication links to estimate the computing times on nodes and the data transport times over connections. Based on these time estimates, we present the Efficient Linear Pipeline Configuration method based on dynamic programming that partitions the pipeline modules into groups and strategically maps them onto a set of selected computing nodes in a network to achieve minimum end-to-end delay or maximum frame rate. We implemented this method and evaluated its effectiveness with experiments on a large set of simulated application pipelines and computing networks. The experimental results show that the proposed method outperforms the Streamline and Greedy algorithms. These results, together with polynomial computational complexity, make our method a potential scalable solution for large practical deployments.

Original languageEnglish
Title of host publicationIPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM
DOIs
StatePublished - 10 Sep 2008
EventIPDPS 2008 - 22nd IEEE International Parallel and Distributed Processing Symposium - Miami, FL, United States
Duration: 14 Apr 200818 Apr 2008

Publication series

NameIPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM

Other

OtherIPDPS 2008 - 22nd IEEE International Parallel and Distributed Processing Symposium
CountryUnited States
CityMiami, FL
Period14/04/0818/04/08

Fingerprint

Network performance
Pipelines
Wide area networks
Dynamic programming
Telecommunication links
Computational complexity
Polynomials
Costs
Experiments

Cite this

Wu, Q., Gu, Y., Zhu, M., & Rao, N. S. V. (2008). Optimizing network performance of computing pipelines in distributed environments. In IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM [4536465] (IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM). https://doi.org/10.1109/IPDPS.2008.4536465
Wu, Qishi ; Gu, Yi ; Zhu, Michelle ; Rao, Nageswara S.V. / Optimizing network performance of computing pipelines in distributed environments. IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM. 2008. (IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM).
@inproceedings{6d651836ac9f49e3a11f9e62148630f8,
title = "Optimizing network performance of computing pipelines in distributed environments",
abstract = "Supporting high performance computing pipelines over wide-area networks is critical to enabling large-scale distributed scientific applications that require fast responses for interactive operations or smooth flows for data streaming. We construct analytical cost models for computing modules, network nodes, and communication links to estimate the computing times on nodes and the data transport times over connections. Based on these time estimates, we present the Efficient Linear Pipeline Configuration method based on dynamic programming that partitions the pipeline modules into groups and strategically maps them onto a set of selected computing nodes in a network to achieve minimum end-to-end delay or maximum frame rate. We implemented this method and evaluated its effectiveness with experiments on a large set of simulated application pipelines and computing networks. The experimental results show that the proposed method outperforms the Streamline and Greedy algorithms. These results, together with polynomial computational complexity, make our method a potential scalable solution for large practical deployments.",
author = "Qishi Wu and Yi Gu and Michelle Zhu and Rao, {Nageswara S.V.}",
year = "2008",
month = "9",
day = "10",
doi = "10.1109/IPDPS.2008.4536465",
language = "English",
isbn = "9781424416943",
series = "IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM",
booktitle = "IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM",

}

Wu, Q, Gu, Y, Zhu, M & Rao, NSV 2008, Optimizing network performance of computing pipelines in distributed environments. in IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM., 4536465, IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM, IPDPS 2008 - 22nd IEEE International Parallel and Distributed Processing Symposium, Miami, FL, United States, 14/04/08. https://doi.org/10.1109/IPDPS.2008.4536465

Optimizing network performance of computing pipelines in distributed environments. / Wu, Qishi; Gu, Yi; Zhu, Michelle; Rao, Nageswara S.V.

IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM. 2008. 4536465 (IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM).

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

TY - GEN

T1 - Optimizing network performance of computing pipelines in distributed environments

AU - Wu, Qishi

AU - Gu, Yi

AU - Zhu, Michelle

AU - Rao, Nageswara S.V.

PY - 2008/9/10

Y1 - 2008/9/10

N2 - Supporting high performance computing pipelines over wide-area networks is critical to enabling large-scale distributed scientific applications that require fast responses for interactive operations or smooth flows for data streaming. We construct analytical cost models for computing modules, network nodes, and communication links to estimate the computing times on nodes and the data transport times over connections. Based on these time estimates, we present the Efficient Linear Pipeline Configuration method based on dynamic programming that partitions the pipeline modules into groups and strategically maps them onto a set of selected computing nodes in a network to achieve minimum end-to-end delay or maximum frame rate. We implemented this method and evaluated its effectiveness with experiments on a large set of simulated application pipelines and computing networks. The experimental results show that the proposed method outperforms the Streamline and Greedy algorithms. These results, together with polynomial computational complexity, make our method a potential scalable solution for large practical deployments.

AB - Supporting high performance computing pipelines over wide-area networks is critical to enabling large-scale distributed scientific applications that require fast responses for interactive operations or smooth flows for data streaming. We construct analytical cost models for computing modules, network nodes, and communication links to estimate the computing times on nodes and the data transport times over connections. Based on these time estimates, we present the Efficient Linear Pipeline Configuration method based on dynamic programming that partitions the pipeline modules into groups and strategically maps them onto a set of selected computing nodes in a network to achieve minimum end-to-end delay or maximum frame rate. We implemented this method and evaluated its effectiveness with experiments on a large set of simulated application pipelines and computing networks. The experimental results show that the proposed method outperforms the Streamline and Greedy algorithms. These results, together with polynomial computational complexity, make our method a potential scalable solution for large practical deployments.

UR - http://www.scopus.com/inward/record.url?scp=51049114925&partnerID=8YFLogxK

U2 - 10.1109/IPDPS.2008.4536465

DO - 10.1109/IPDPS.2008.4536465

M3 - Conference contribution

SN - 9781424416943

T3 - IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM

BT - IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM

ER -

Wu Q, Gu Y, Zhu M, Rao NSV. Optimizing network performance of computing pipelines in distributed environments. In IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM. 2008. 4536465. (IPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM). https://doi.org/10.1109/IPDPS.2008.4536465