A workflow-based network advisor for data movement with End-to-End performance optimization

Patrick Brown, Michelle Zhu, Qishi Wu, Daqing Yun, Jason Zurawski

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

1 Citation (Scopus)

Abstract

Next-generation eScience applications often generate large amounts of simulation, experimental, or observational data that must be shared and managed by collaborative organizations. Advanced networking technologies and services have been rapidly developed and deployed to facilitate such massive data transfer. However, these technologies and services have not been fully utilized mainly because their use typically requires significant system-And network-related domain knowledge that most appli-cation users lack, and in many cases users may even not be aware of their existence. By leveraging the functionalities of an existing data movement advising utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization. WINDMA integrates three major components: resource discovery, data movement, and status monitoring, and supports the sharing of common data movement workflows through account and database man-Agement. This system provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. We demonstrate the efficacy of the proposed transport-support WINDMA system in several use cases based on its implementation and deployment in wide-Area networks.

Original languageEnglish
Title of host publicationProceedings - 2012 SC Companion
Subtitle of host publicationHigh Performance Computing, Networking Storage and Analysis, SCC 2012
Pages73-81
Number of pages9
DOIs
StatePublished - 1 Dec 2012
Event2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012 - Salt Lake City, UT, United States
Duration: 10 Nov 201216 Nov 2012

Other

Other2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012
CountryUnited States
CitySalt Lake City, UT
Period10/11/1216/11/12

Fingerprint

Intelligent networks
Wide area networks
Data transfer
Positive ions
Monitoring

Cite this

Brown, P., Zhu, M., Wu, Q., Yun, D., & Zurawski, J. (2012). A workflow-based network advisor for data movement with End-to-End performance optimization. In Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012 (pp. 73-81). [6495804] https://doi.org/10.1109/SC.Companion.2012.22
Brown, Patrick ; Zhu, Michelle ; Wu, Qishi ; Yun, Daqing ; Zurawski, Jason. / A workflow-based network advisor for data movement with End-to-End performance optimization. Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012. 2012. pp. 73-81
@inproceedings{0ff13b16b41948d98f8ec33de23bd7dc,
title = "A workflow-based network advisor for data movement with End-to-End performance optimization",
abstract = "Next-generation eScience applications often generate large amounts of simulation, experimental, or observational data that must be shared and managed by collaborative organizations. Advanced networking technologies and services have been rapidly developed and deployed to facilitate such massive data transfer. However, these technologies and services have not been fully utilized mainly because their use typically requires significant system-And network-related domain knowledge that most appli-cation users lack, and in many cases users may even not be aware of their existence. By leveraging the functionalities of an existing data movement advising utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization. WINDMA integrates three major components: resource discovery, data movement, and status monitoring, and supports the sharing of common data movement workflows through account and database man-Agement. This system provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. We demonstrate the efficacy of the proposed transport-support WINDMA system in several use cases based on its implementation and deployment in wide-Area networks.",
author = "Patrick Brown and Michelle Zhu and Qishi Wu and Daqing Yun and Jason Zurawski",
year = "2012",
month = "12",
day = "1",
doi = "10.1109/SC.Companion.2012.22",
language = "English",
isbn = "9780769549569",
pages = "73--81",
booktitle = "Proceedings - 2012 SC Companion",

}

Brown, P, Zhu, M, Wu, Q, Yun, D & Zurawski, J 2012, A workflow-based network advisor for data movement with End-to-End performance optimization. in Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012., 6495804, pp. 73-81, 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012, Salt Lake City, UT, United States, 10/11/12. https://doi.org/10.1109/SC.Companion.2012.22

A workflow-based network advisor for data movement with End-to-End performance optimization. / Brown, Patrick; Zhu, Michelle; Wu, Qishi; Yun, Daqing; Zurawski, Jason.

Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012. 2012. p. 73-81 6495804.

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

TY - GEN

T1 - A workflow-based network advisor for data movement with End-to-End performance optimization

AU - Brown, Patrick

AU - Zhu, Michelle

AU - Wu, Qishi

AU - Yun, Daqing

AU - Zurawski, Jason

PY - 2012/12/1

Y1 - 2012/12/1

N2 - Next-generation eScience applications often generate large amounts of simulation, experimental, or observational data that must be shared and managed by collaborative organizations. Advanced networking technologies and services have been rapidly developed and deployed to facilitate such massive data transfer. However, these technologies and services have not been fully utilized mainly because their use typically requires significant system-And network-related domain knowledge that most appli-cation users lack, and in many cases users may even not be aware of their existence. By leveraging the functionalities of an existing data movement advising utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization. WINDMA integrates three major components: resource discovery, data movement, and status monitoring, and supports the sharing of common data movement workflows through account and database man-Agement. This system provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. We demonstrate the efficacy of the proposed transport-support WINDMA system in several use cases based on its implementation and deployment in wide-Area networks.

AB - Next-generation eScience applications often generate large amounts of simulation, experimental, or observational data that must be shared and managed by collaborative organizations. Advanced networking technologies and services have been rapidly developed and deployed to facilitate such massive data transfer. However, these technologies and services have not been fully utilized mainly because their use typically requires significant system-And network-related domain knowledge that most appli-cation users lack, and in many cases users may even not be aware of their existence. By leveraging the functionalities of an existing data movement advising utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization. WINDMA integrates three major components: resource discovery, data movement, and status monitoring, and supports the sharing of common data movement workflows through account and database man-Agement. This system provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. We demonstrate the efficacy of the proposed transport-support WINDMA system in several use cases based on its implementation and deployment in wide-Area networks.

UR - http://www.scopus.com/inward/record.url?scp=84876536614&partnerID=8YFLogxK

U2 - 10.1109/SC.Companion.2012.22

DO - 10.1109/SC.Companion.2012.22

M3 - Conference contribution

SN - 9780769549569

SP - 73

EP - 81

BT - Proceedings - 2012 SC Companion

ER -

Brown P, Zhu M, Wu Q, Yun D, Zurawski J. A workflow-based network advisor for data movement with End-to-End performance optimization. In Proceedings - 2012 SC Companion: High Performance Computing, Networking Storage and Analysis, SCC 2012. 2012. p. 73-81. 6495804 https://doi.org/10.1109/SC.Companion.2012.22