Transport-support workflow composition and optimization for big data movement in high-performance networks

Daqing Yun, Chase Q. Wu, Michelle M. Zhu

Research output: Contribution to journalArticle

4 Scopus citations

Abstract

High-performance networks (HPNs) are being increasingly developed and deployed to support the transfer of big data. However, such HPN-based technologies and services have not been fully utilized as their use often requires considerable networking and system domain knowledge and many application users are even not aware of their existence. This work develops an integrated solution to discover system and network resources and compose end-to-end paths for big data movement. We first develop profiling and modeling approaches to characterize various types of resources distributed in end systems, edge segments, and backbone networks. A comprehensive set of performance metrics and network parameters are considered in different phases including device deployment, circuit setup, and data transfer. Based on these profiles and models, we then formulate a class of transport-support workflow optimization problems to compose the best end-to-end path that meets various performance requirements. We prove this problem to be NP-complete and design pseudo-polynomial optimal algorithms. We conduct extensive simulations to evaluate the proposed algorithms in comparison with a greedy approach, and also carry out real-life experiments across different network segments in production HPNs to evaluate the validity of the constructed cost models and illustrate the efficacy of the proposed transport solution.

Original languageEnglish
Article number8000332
Pages (from-to)3656-3670
Number of pages15
JournalIEEE Transactions on Parallel and Distributed Systems
Volume28
Issue number12
DOIs
StatePublished - 1 Dec 2017

Keywords

  • Big data transfer
  • high-performance networks
  • performance modeling
  • workflow optimization

Fingerprint Dive into the research topics of 'Transport-support workflow composition and optimization for big data movement in high-performance networks'. Together they form a unique fingerprint.

  • Cite this