Transport-support workflow composition and optimization for big data movement in high-performance networks

Daqing Yun, Chase Q. Wu, Michelle Zhu

Research output: Contribution to journalArticle

4 Citations (Scopus)

Abstract

High-performance networks (HPNs) are being increasingly developed and deployed to support the transfer of big data. However, such HPN-based technologies and services have not been fully utilized as their use often requires considerable networking and system domain knowledge and many application users are even not aware of their existence. This work develops an integrated solution to discover system and network resources and compose end-to-end paths for big data movement. We first develop profiling and modeling approaches to characterize various types of resources distributed in end systems, edge segments, and backbone networks. A comprehensive set of performance metrics and network parameters are considered in different phases including device deployment, circuit setup, and data transfer. Based on these profiles and models, we then formulate a class of transport-support workflow optimization problems to compose the best end-to-end path that meets various performance requirements. We prove this problem to be NP-complete and design pseudo-polynomial optimal algorithms. We conduct extensive simulations to evaluate the proposed algorithms in comparison with a greedy approach, and also carry out real-life experiments across different network segments in production HPNs to evaluate the validity of the constructed cost models and illustrate the efficacy of the proposed transport solution.

Original languageEnglish
Article number8000332
Pages (from-to)3656-3670
Number of pages15
JournalIEEE Transactions on Parallel and Distributed Systems
Volume28
Issue number12
DOIs
StatePublished - 1 Dec 2017

Fingerprint

Network performance
Chemical analysis
Data transfer
Polynomials
Networks (circuits)
Big data
Costs
Experiments

Keywords

  • Big data transfer
  • high-performance networks
  • performance modeling
  • workflow optimization

Cite this

@article{99bfe74a0bfe448ba06bb9464c5acd1e,
title = "Transport-support workflow composition and optimization for big data movement in high-performance networks",
abstract = "High-performance networks (HPNs) are being increasingly developed and deployed to support the transfer of big data. However, such HPN-based technologies and services have not been fully utilized as their use often requires considerable networking and system domain knowledge and many application users are even not aware of their existence. This work develops an integrated solution to discover system and network resources and compose end-to-end paths for big data movement. We first develop profiling and modeling approaches to characterize various types of resources distributed in end systems, edge segments, and backbone networks. A comprehensive set of performance metrics and network parameters are considered in different phases including device deployment, circuit setup, and data transfer. Based on these profiles and models, we then formulate a class of transport-support workflow optimization problems to compose the best end-to-end path that meets various performance requirements. We prove this problem to be NP-complete and design pseudo-polynomial optimal algorithms. We conduct extensive simulations to evaluate the proposed algorithms in comparison with a greedy approach, and also carry out real-life experiments across different network segments in production HPNs to evaluate the validity of the constructed cost models and illustrate the efficacy of the proposed transport solution.",
keywords = "Big data transfer, high-performance networks, performance modeling, workflow optimization",
author = "Daqing Yun and Wu, {Chase Q.} and Michelle Zhu",
year = "2017",
month = "12",
day = "1",
doi = "10.1109/TPDS.2017.2732987",
language = "English",
volume = "28",
pages = "3656--3670",
journal = "IEEE Transactions on Parallel and Distributed Systems",
issn = "1045-9219",
publisher = "IEEE Computer Society",
number = "12",

}

Transport-support workflow composition and optimization for big data movement in high-performance networks. / Yun, Daqing; Wu, Chase Q.; Zhu, Michelle.

In: IEEE Transactions on Parallel and Distributed Systems, Vol. 28, No. 12, 8000332, 01.12.2017, p. 3656-3670.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Transport-support workflow composition and optimization for big data movement in high-performance networks

AU - Yun, Daqing

AU - Wu, Chase Q.

AU - Zhu, Michelle

PY - 2017/12/1

Y1 - 2017/12/1

N2 - High-performance networks (HPNs) are being increasingly developed and deployed to support the transfer of big data. However, such HPN-based technologies and services have not been fully utilized as their use often requires considerable networking and system domain knowledge and many application users are even not aware of their existence. This work develops an integrated solution to discover system and network resources and compose end-to-end paths for big data movement. We first develop profiling and modeling approaches to characterize various types of resources distributed in end systems, edge segments, and backbone networks. A comprehensive set of performance metrics and network parameters are considered in different phases including device deployment, circuit setup, and data transfer. Based on these profiles and models, we then formulate a class of transport-support workflow optimization problems to compose the best end-to-end path that meets various performance requirements. We prove this problem to be NP-complete and design pseudo-polynomial optimal algorithms. We conduct extensive simulations to evaluate the proposed algorithms in comparison with a greedy approach, and also carry out real-life experiments across different network segments in production HPNs to evaluate the validity of the constructed cost models and illustrate the efficacy of the proposed transport solution.

AB - High-performance networks (HPNs) are being increasingly developed and deployed to support the transfer of big data. However, such HPN-based technologies and services have not been fully utilized as their use often requires considerable networking and system domain knowledge and many application users are even not aware of their existence. This work develops an integrated solution to discover system and network resources and compose end-to-end paths for big data movement. We first develop profiling and modeling approaches to characterize various types of resources distributed in end systems, edge segments, and backbone networks. A comprehensive set of performance metrics and network parameters are considered in different phases including device deployment, circuit setup, and data transfer. Based on these profiles and models, we then formulate a class of transport-support workflow optimization problems to compose the best end-to-end path that meets various performance requirements. We prove this problem to be NP-complete and design pseudo-polynomial optimal algorithms. We conduct extensive simulations to evaluate the proposed algorithms in comparison with a greedy approach, and also carry out real-life experiments across different network segments in production HPNs to evaluate the validity of the constructed cost models and illustrate the efficacy of the proposed transport solution.

KW - Big data transfer

KW - high-performance networks

KW - performance modeling

KW - workflow optimization

UR - http://www.scopus.com/inward/record.url?scp=85028940166&partnerID=8YFLogxK

U2 - 10.1109/TPDS.2017.2732987

DO - 10.1109/TPDS.2017.2732987

M3 - Article

AN - SCOPUS:85028940166

VL - 28

SP - 3656

EP - 3670

JO - IEEE Transactions on Parallel and Distributed Systems

JF - IEEE Transactions on Parallel and Distributed Systems

SN - 1045-9219

IS - 12

M1 - 8000332

ER -