On optimization of scientific workflows to support streaming applications in distributed network environments

Qishi Wu; Yi Gu; Xukang Lu; Mengxia Zhu; Patrick Brown; Wuyin Lin; Yangang Liu

doi:10.1109/WORKS.2010.5671851

On optimization of scientific workflows to support streaming applications in distributed network environments

Qishi Wu
, Yi Gu
, Xukang Lu
, Mengxia Zhu
, Patrick Brown
, Wuyin Lin
, Yangang Liu

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

Large-scale data-intensive streaming applications in various science fields feature complex DAG-structured workflows comprised of distributed computing modules with intricate inter-module dependencies. Supporting such workflows in high-performance network environments and optimizing their throughput are crucial to collaborative scientific exploration and discovery. We formulate workflow mapping as a frame rate optimization problem and propose an efficient heuristic solution, which is integrated into the Condor-based Scientific Workflow Automation and Management Platform (SWAMP) in place of Condor's default mapping scheme. The SWAMP system is also augmented with several new components to improve the workflow management process. The performance superiority of the proposed solution is verified using both simulations and a real-life scientific workflow for climate modeling deployed in a distributed heterogeneous network environment.

Original language	English
Title of host publication	2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010
DOIs	https://doi.org/10.1109/WORKS.2010.5671851
State	Published - 2010
Event	2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010 - New Orleans, LA, United States Duration: 14 Nov 2010 → 14 Nov 2010

Publication series

Name	2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010

Other

Other	2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010
Country/Territory	United States
City	New Orleans, LA
Period	14/11/10 → 14/11/10

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 13 Climate Action

Keywords

Distributed workflow
Frame rate
Grid computing

Access to Document

10.1109/WORKS.2010.5671851

Cite this

Wu, Q., Gu, Y., Lu, X., Zhu, M., Brown, P., Lin, W., & Liu, Y. (2010). On optimization of scientific workflows to support streaming applications in distributed network environments. In 2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010 Article 5671851 (2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010). https://doi.org/10.1109/WORKS.2010.5671851

@inproceedings{b6b45023133447a889a524138d6fab0c,

title = "On optimization of scientific workflows to support streaming applications in distributed network environments",

abstract = "Large-scale data-intensive streaming applications in various science fields feature complex DAG-structured workflows comprised of distributed computing modules with intricate inter-module dependencies. Supporting such workflows in high-performance network environments and optimizing their throughput are crucial to collaborative scientific exploration and discovery. We formulate workflow mapping as a frame rate optimization problem and propose an efficient heuristic solution, which is integrated into the Condor-based Scientific Workflow Automation and Management Platform (SWAMP) in place of Condor's default mapping scheme. The SWAMP system is also augmented with several new components to improve the workflow management process. The performance superiority of the proposed solution is verified using both simulations and a real-life scientific workflow for climate modeling deployed in a distributed heterogeneous network environment.",

keywords = "Distributed workflow, Frame rate, Grid computing",

author = "Qishi Wu and Yi Gu and Xukang Lu and Mengxia Zhu and Patrick Brown and Wuyin Lin and Yangang Liu",

year = "2010",

doi = "10.1109/WORKS.2010.5671851",

language = "English",

isbn = "9781424489893",

series = "2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010",

booktitle = "2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010",

note = "2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010 ; Conference date: 14-11-2010 Through 14-11-2010",

}

Wu, Q, Gu, Y, Lu, X, Zhu, M, Brown, P, Lin, W & Liu, Y 2010, On optimization of scientific workflows to support streaming applications in distributed network environments. in 2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010., 5671851, 2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010, 2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010, New Orleans, LA, United States, 14/11/10. https://doi.org/10.1109/WORKS.2010.5671851

On optimization of scientific workflows to support streaming applications in distributed network environments. / Wu, Qishi; Gu, Yi; Lu, Xukang et al.
2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010. 2010. 5671851 (2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - On optimization of scientific workflows to support streaming applications in distributed network environments

AU - Wu, Qishi

AU - Gu, Yi

AU - Lu, Xukang

AU - Zhu, Mengxia

AU - Brown, Patrick

AU - Lin, Wuyin

AU - Liu, Yangang

PY - 2010

Y1 - 2010

N2 - Large-scale data-intensive streaming applications in various science fields feature complex DAG-structured workflows comprised of distributed computing modules with intricate inter-module dependencies. Supporting such workflows in high-performance network environments and optimizing their throughput are crucial to collaborative scientific exploration and discovery. We formulate workflow mapping as a frame rate optimization problem and propose an efficient heuristic solution, which is integrated into the Condor-based Scientific Workflow Automation and Management Platform (SWAMP) in place of Condor's default mapping scheme. The SWAMP system is also augmented with several new components to improve the workflow management process. The performance superiority of the proposed solution is verified using both simulations and a real-life scientific workflow for climate modeling deployed in a distributed heterogeneous network environment.

AB - Large-scale data-intensive streaming applications in various science fields feature complex DAG-structured workflows comprised of distributed computing modules with intricate inter-module dependencies. Supporting such workflows in high-performance network environments and optimizing their throughput are crucial to collaborative scientific exploration and discovery. We formulate workflow mapping as a frame rate optimization problem and propose an efficient heuristic solution, which is integrated into the Condor-based Scientific Workflow Automation and Management Platform (SWAMP) in place of Condor's default mapping scheme. The SWAMP system is also augmented with several new components to improve the workflow management process. The performance superiority of the proposed solution is verified using both simulations and a real-life scientific workflow for climate modeling deployed in a distributed heterogeneous network environment.

KW - Distributed workflow

KW - Frame rate

KW - Grid computing

UR - https://www.scopus.com/pages/publications/78751488983

U2 - 10.1109/WORKS.2010.5671851

DO - 10.1109/WORKS.2010.5671851

M3 - Conference contribution

AN - SCOPUS:78751488983

SN - 9781424489893

T3 - 2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010

BT - 2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010

T2 - 2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010

Y2 - 14 November 2010 through 14 November 2010

ER -

On optimization of scientific workflows to support streaming applications in distributed network environments

Abstract

Publication series

Other

UN SDGs

Keywords

Access to Document

Other files and links

Fingerprint

Cite this