A distributed workflow management system with case study of real-life scientific applications

Qishi Wu, Michelle Zhu, Yi Gu, Xukang Lu, Patrick Brown, Michael A. Reuter, Stephen D. Miller

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Supporting large-scale scientific workflows in distributed network environments and optimizing their performances are crucial to the success of collaborative scientific discovery. We develop a generic scientific workflow platform, referred to as SciFlow, which constitutes a flexible framework to facilitate the distributed execution and management of scientific workflows and incorporates a class of workflow mapping schemes to achieve optimal end-to-end performances. The functionalities of SciFlow are provided and its interactions with other tools or systems are enabled through web services for easy access over standard Internet protocols while being independent of different platforms and programming languages. The performance superiority of SciFlow over existing workflow mapping schemes and management systems is illustrated by extensive simulations and is further verified by large-scale experiments on real-life scientific workflows through effective system implementation and deployment in distributed network environments.

Original languageEnglish
Title of host publication30th IEEE International Performance Computing and Communications Conference, IPCCC 2011
DOIs
StatePublished - 1 Dec 2011
Event30th IEEE International Performance, Computing and Communications Conference, IPCCC 2011 - Orlando, FL, United States
Duration: 17 Nov 201119 Nov 2011

Publication series

NameConference Proceedings of the IEEE International Performance, Computing, and Communications Conference

Other

Other30th IEEE International Performance, Computing and Communications Conference, IPCCC 2011
CountryUnited States
CityOrlando, FL
Period17/11/1119/11/11

Fingerprint

Internet protocols
Computer programming languages
Web services
Experiments

Keywords

  • distributed computing
  • performance modeling
  • scientific workflow
  • workflow system

Cite this

Wu, Q., Zhu, M., Gu, Y., Lu, X., Brown, P., Reuter, M. A., & Miller, S. D. (2011). A distributed workflow management system with case study of real-life scientific applications. In 30th IEEE International Performance Computing and Communications Conference, IPCCC 2011 [6108109] (Conference Proceedings of the IEEE International Performance, Computing, and Communications Conference). https://doi.org/10.1109/PCCC.2011.6108109
Wu, Qishi ; Zhu, Michelle ; Gu, Yi ; Lu, Xukang ; Brown, Patrick ; Reuter, Michael A. ; Miller, Stephen D. / A distributed workflow management system with case study of real-life scientific applications. 30th IEEE International Performance Computing and Communications Conference, IPCCC 2011. 2011. (Conference Proceedings of the IEEE International Performance, Computing, and Communications Conference).
@inproceedings{405b7e7f08744597a8ab541582704756,
title = "A distributed workflow management system with case study of real-life scientific applications",
abstract = "Supporting large-scale scientific workflows in distributed network environments and optimizing their performances are crucial to the success of collaborative scientific discovery. We develop a generic scientific workflow platform, referred to as SciFlow, which constitutes a flexible framework to facilitate the distributed execution and management of scientific workflows and incorporates a class of workflow mapping schemes to achieve optimal end-to-end performances. The functionalities of SciFlow are provided and its interactions with other tools or systems are enabled through web services for easy access over standard Internet protocols while being independent of different platforms and programming languages. The performance superiority of SciFlow over existing workflow mapping schemes and management systems is illustrated by extensive simulations and is further verified by large-scale experiments on real-life scientific workflows through effective system implementation and deployment in distributed network environments.",
keywords = "distributed computing, performance modeling, scientific workflow, workflow system",
author = "Qishi Wu and Michelle Zhu and Yi Gu and Xukang Lu and Patrick Brown and Reuter, {Michael A.} and Miller, {Stephen D.}",
year = "2011",
month = "12",
day = "1",
doi = "10.1109/PCCC.2011.6108109",
language = "English",
isbn = "9781467300100",
series = "Conference Proceedings of the IEEE International Performance, Computing, and Communications Conference",
booktitle = "30th IEEE International Performance Computing and Communications Conference, IPCCC 2011",

}

Wu, Q, Zhu, M, Gu, Y, Lu, X, Brown, P, Reuter, MA & Miller, SD 2011, A distributed workflow management system with case study of real-life scientific applications. in 30th IEEE International Performance Computing and Communications Conference, IPCCC 2011., 6108109, Conference Proceedings of the IEEE International Performance, Computing, and Communications Conference, 30th IEEE International Performance, Computing and Communications Conference, IPCCC 2011, Orlando, FL, United States, 17/11/11. https://doi.org/10.1109/PCCC.2011.6108109

A distributed workflow management system with case study of real-life scientific applications. / Wu, Qishi; Zhu, Michelle; Gu, Yi; Lu, Xukang; Brown, Patrick; Reuter, Michael A.; Miller, Stephen D.

30th IEEE International Performance Computing and Communications Conference, IPCCC 2011. 2011. 6108109 (Conference Proceedings of the IEEE International Performance, Computing, and Communications Conference).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - A distributed workflow management system with case study of real-life scientific applications

AU - Wu, Qishi

AU - Zhu, Michelle

AU - Gu, Yi

AU - Lu, Xukang

AU - Brown, Patrick

AU - Reuter, Michael A.

AU - Miller, Stephen D.

PY - 2011/12/1

Y1 - 2011/12/1

N2 - Supporting large-scale scientific workflows in distributed network environments and optimizing their performances are crucial to the success of collaborative scientific discovery. We develop a generic scientific workflow platform, referred to as SciFlow, which constitutes a flexible framework to facilitate the distributed execution and management of scientific workflows and incorporates a class of workflow mapping schemes to achieve optimal end-to-end performances. The functionalities of SciFlow are provided and its interactions with other tools or systems are enabled through web services for easy access over standard Internet protocols while being independent of different platforms and programming languages. The performance superiority of SciFlow over existing workflow mapping schemes and management systems is illustrated by extensive simulations and is further verified by large-scale experiments on real-life scientific workflows through effective system implementation and deployment in distributed network environments.

AB - Supporting large-scale scientific workflows in distributed network environments and optimizing their performances are crucial to the success of collaborative scientific discovery. We develop a generic scientific workflow platform, referred to as SciFlow, which constitutes a flexible framework to facilitate the distributed execution and management of scientific workflows and incorporates a class of workflow mapping schemes to achieve optimal end-to-end performances. The functionalities of SciFlow are provided and its interactions with other tools or systems are enabled through web services for easy access over standard Internet protocols while being independent of different platforms and programming languages. The performance superiority of SciFlow over existing workflow mapping schemes and management systems is illustrated by extensive simulations and is further verified by large-scale experiments on real-life scientific workflows through effective system implementation and deployment in distributed network environments.

KW - distributed computing

KW - performance modeling

KW - scientific workflow

KW - workflow system

UR - http://www.scopus.com/inward/record.url?scp=84863419736&partnerID=8YFLogxK

U2 - 10.1109/PCCC.2011.6108109

DO - 10.1109/PCCC.2011.6108109

M3 - Conference contribution

AN - SCOPUS:84863419736

SN - 9781467300100

T3 - Conference Proceedings of the IEEE International Performance, Computing, and Communications Conference

BT - 30th IEEE International Performance Computing and Communications Conference, IPCCC 2011

ER -

Wu Q, Zhu M, Gu Y, Lu X, Brown P, Reuter MA et al. A distributed workflow management system with case study of real-life scientific applications. In 30th IEEE International Performance Computing and Communications Conference, IPCCC 2011. 2011. 6108109. (Conference Proceedings of the IEEE International Performance, Computing, and Communications Conference). https://doi.org/10.1109/PCCC.2011.6108109