Automation and management of scientificworkflows in distributed network environments

Qishi Wu, Mengxia Zhu, Xukang Lu, Patrick Brown, Yunyue Lin, Yi Gu, Fei Cao, Michael A. Reuter

Research output: Chapter in Book/Report/Conference proceedingConference contribution

18 Scopus citations

Abstract

Large-scale computation-intensive applications in various science fields feature complex DAG-structured workflows comprised of distributed computing modules with intricate inter-module dependencies. Supporting such workflows in heterogeneous network environments and optimizing their end-to-end performance are crucial to the success of large-scale collaborative scientific applications. We design and develop a generic Scientific Workflow Automation and Management Platform (SWAMP), which contains a set of easy-to-use computing and networking toolkits for application scientists to conveniently assemble, execute, monitor, and control complex computing workflows in distributed network environments. The current version of SWAMP integrates the graphical user interface of Kepler to compose abstract workflows and employs Condor DAGMan for workflow dispatch and execution. SWAMP provides a web-based user interface to automate and manage workflow executions and uses a special workflow mapper to optimize the end-to-end workflow performance. A case study of the workflow for Spallation Neutron Source datasets in real networks is presented to show the efficacy of the proposed platform.

Original languageEnglish
Title of host publicationProceedings of the 2010 IEEE International Symposium on Parallel and Distributed Processing, Workshops and Phd Forum, IPDPSW 2010
DOIs
StatePublished - 2 Jul 2010
Event2010 IEEE International Symposium on Parallel and Distributed Processing, Workshops and Phd Forum, IPDPSW 2010 - Atlanta, GA, United States
Duration: 19 Apr 201023 Apr 2010

Publication series

NameProceedings of the 2010 IEEE International Symposium on Parallel and Distributed Processing, Workshops and Phd Forum, IPDPSW 2010

Other

Other2010 IEEE International Symposium on Parallel and Distributed Processing, Workshops and Phd Forum, IPDPSW 2010
CountryUnited States
CityAtlanta, GA
Period19/04/1023/04/10

Keywords

  • Distributed computing
  • Scientific workflow
  • Workflow management

Fingerprint Dive into the research topics of 'Automation and management of scientificworkflows in distributed network environments'. Together they form a unique fingerprint.

  • Cite this

    Wu, Q., Zhu, M., Lu, X., Brown, P., Lin, Y., Gu, Y., Cao, F., & Reuter, M. A. (2010). Automation and management of scientificworkflows in distributed network environments. In Proceedings of the 2010 IEEE International Symposium on Parallel and Distributed Processing, Workshops and Phd Forum, IPDPSW 2010 [5470720] (Proceedings of the 2010 IEEE International Symposium on Parallel and Distributed Processing, Workshops and Phd Forum, IPDPSW 2010). https://doi.org/10.1109/IPDPSW.2010.5470720