A workflow-based network advisor for data movement with End-to-End performance optimization

Patrick Brown, Michelle Zhu, Qishi Wu, Daqing Yun, Jason Zurawski

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations

Abstract

Next-generation eScience applications often generate large amounts of simulation, experimental, or observational data that must be shared and managed by collaborative organizations. Advanced networking technologies and services have been rapidly developed and deployed to facilitate such massive data transfer. However, these technologies and services have not been fully utilized mainly because their use typically requires significant system-And network-related domain knowledge that most appli-cation users lack, and in many cases users may even not be aware of their existence. By leveraging the functionalities of an existing data movement advising utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization. WINDMA integrates three major components: resource discovery, data movement, and status monitoring, and supports the sharing of common data movement workflows through account and database man-Agement. This system provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. We demonstrate the efficacy of the proposed transport-support WINDMA system in several use cases based on its implementation and deployment in wide-Area networks.

Original languageEnglish
Title of host publicationProceedings - 2012 SC Companion
Subtitle of host publicationHigh Performance Computing, Networking Storage and Analysis, SCC 2012
Pages73-81
Number of pages9
DOIs
Publication statusPublished - 1 Dec 2012
Event2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012 - Salt Lake City, UT, United States
Duration: 10 Nov 201216 Nov 2012

Publication series

NameProceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012

Other

Other2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012
CountryUnited States
CitySalt Lake City, UT
Period10/11/1216/11/12

    Fingerprint

Cite this

Brown, P., Zhu, M., Wu, Q., Yun, D., & Zurawski, J. (2012). A workflow-based network advisor for data movement with End-to-End performance optimization. In Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012 (pp. 73-81). [6495804] (Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012). https://doi.org/10.1109/SC.Companion.2012.22