Exploring the optimal strategy for large-scale data movement in high-performance networks

Patrick Brown, Michelle Zhu, Qishi Wu, Daqing Yun, Jason Zurawski

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Advanced networking technologies and services have been rapidly developed and deployed to facilitate bulk data transfer so as to support next-generation eScience applications. However, these technologies and services have not been fully utilized due to the knowledge lack of scientific domain experts. By leveraging the functionalities of an existing data movement advising utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization. WINDMA provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. Efficacy of WINDMA has been demonstrated in several use cases based on its implementation and deployment in wide-area networks.

Original languageEnglish
Title of host publication2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012
Pages181-182
Number of pages2
DOIs
StatePublished - 1 Dec 2012
Event2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012 - Austin, TX, United States
Duration: 1 Dec 20123 Dec 2012

Other

Other2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012
CountryUnited States
CityAustin, TX
Period1/12/123/12/12

Fingerprint

Intelligent networks
Network performance
Wide area networks
Data transfer

Keywords

  • eScience applications
  • High-performance networks
  • large data transfer

Cite this

Brown, P., Zhu, M., Wu, Q., Yun, D., & Zurawski, J. (2012). Exploring the optimal strategy for large-scale data movement in high-performance networks. In 2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012 (pp. 181-182). [6407676] https://doi.org/10.1109/PCCC.2012.6407676
Brown, Patrick ; Zhu, Michelle ; Wu, Qishi ; Yun, Daqing ; Zurawski, Jason. / Exploring the optimal strategy for large-scale data movement in high-performance networks. 2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012. 2012. pp. 181-182
@inproceedings{bd98a77369d942b0b2595b1e703c37d8,
title = "Exploring the optimal strategy for large-scale data movement in high-performance networks",
abstract = "Advanced networking technologies and services have been rapidly developed and deployed to facilitate bulk data transfer so as to support next-generation eScience applications. However, these technologies and services have not been fully utilized due to the knowledge lack of scientific domain experts. By leveraging the functionalities of an existing data movement advising utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization. WINDMA provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. Efficacy of WINDMA has been demonstrated in several use cases based on its implementation and deployment in wide-area networks.",
keywords = "eScience applications, High-performance networks, large data transfer",
author = "Patrick Brown and Michelle Zhu and Qishi Wu and Daqing Yun and Jason Zurawski",
year = "2012",
month = "12",
day = "1",
doi = "10.1109/PCCC.2012.6407676",
language = "English",
isbn = "9781467348812",
pages = "181--182",
booktitle = "2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012",

}

Brown, P, Zhu, M, Wu, Q, Yun, D & Zurawski, J 2012, Exploring the optimal strategy for large-scale data movement in high-performance networks. in 2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012., 6407676, pp. 181-182, 2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012, Austin, TX, United States, 1/12/12. https://doi.org/10.1109/PCCC.2012.6407676

Exploring the optimal strategy for large-scale data movement in high-performance networks. / Brown, Patrick; Zhu, Michelle; Wu, Qishi; Yun, Daqing; Zurawski, Jason.

2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012. 2012. p. 181-182 6407676.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Exploring the optimal strategy for large-scale data movement in high-performance networks

AU - Brown, Patrick

AU - Zhu, Michelle

AU - Wu, Qishi

AU - Yun, Daqing

AU - Zurawski, Jason

PY - 2012/12/1

Y1 - 2012/12/1

N2 - Advanced networking technologies and services have been rapidly developed and deployed to facilitate bulk data transfer so as to support next-generation eScience applications. However, these technologies and services have not been fully utilized due to the knowledge lack of scientific domain experts. By leveraging the functionalities of an existing data movement advising utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization. WINDMA provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. Efficacy of WINDMA has been demonstrated in several use cases based on its implementation and deployment in wide-area networks.

AB - Advanced networking technologies and services have been rapidly developed and deployed to facilitate bulk data transfer so as to support next-generation eScience applications. However, these technologies and services have not been fully utilized due to the knowledge lack of scientific domain experts. By leveraging the functionalities of an existing data movement advising utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization. WINDMA provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. Efficacy of WINDMA has been demonstrated in several use cases based on its implementation and deployment in wide-area networks.

KW - eScience applications

KW - High-performance networks

KW - large data transfer

UR - http://www.scopus.com/inward/record.url?scp=84874305124&partnerID=8YFLogxK

U2 - 10.1109/PCCC.2012.6407676

DO - 10.1109/PCCC.2012.6407676

M3 - Conference contribution

AN - SCOPUS:84874305124

SN - 9781467348812

SP - 181

EP - 182

BT - 2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012

ER -

Brown P, Zhu M, Wu Q, Yun D, Zurawski J. Exploring the optimal strategy for large-scale data movement in high-performance networks. In 2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012. 2012. p. 181-182. 6407676 https://doi.org/10.1109/PCCC.2012.6407676