On optimization of scientific workflows to support streaming applications in distributed network environments

Qishi Wu, Yi Gu, Xukang Lu, Mengxia Zhu, Patrick Brown, Wuyin Lin, Yangang Liu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

Large-scale data-intensive streaming applications in various science fields feature complex DAG-structured workflows comprised of distributed computing modules with intricate inter-module dependencies. Supporting such workflows in high-performance network environments and optimizing their throughput are crucial to collaborative scientific exploration and discovery. We formulate workflow mapping as a frame rate optimization problem and propose an efficient heuristic solution, which is integrated into the Condor-based Scientific Workflow Automation and Management Platform (SWAMP) in place of Condor's default mapping scheme. The SWAMP system is also augmented with several new components to improve the workflow management process. The performance superiority of the proposed solution is verified using both simulations and a real-life scientific workflow for climate modeling deployed in a distributed heterogeneous network environment.

Original languageEnglish
Title of host publication2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010
DOIs
StatePublished - 1 Dec 2010
Event2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010 - New Orleans, LA, United States
Duration: 14 Nov 201014 Nov 2010

Publication series

Name2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010

Other

Other2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010
CountryUnited States
CityNew Orleans, LA
Period14/11/1014/11/10

Keywords

  • Distributed workflow
  • Frame rate
  • Grid computing

Fingerprint Dive into the research topics of 'On optimization of scientific workflows to support streaming applications in distributed network environments'. Together they form a unique fingerprint.

  • Cite this

    Wu, Q., Gu, Y., Lu, X., Zhu, M., Brown, P., Lin, W., & Liu, Y. (2010). On optimization of scientific workflows to support streaming applications in distributed network environments. In 2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010 [5671851] (2010 5th Workshop on Workflows in Support of Large-Scale Science, WORKS 2010). https://doi.org/10.1109/WORKS.2010.5671851