Exploring the optimal strategy for large-scale data movement in high-performance networks

Patrick Brown; Mengxia Zhu; Qishi Wu; Daqing Yun; Jason Zurawski

doi:10.1109/PCCC.2012.6407676

Exploring the optimal strategy for large-scale data movement in high-performance networks

Patrick Brown, Mengxia Zhu, Qishi Wu, Daqing Yun, Jason Zurawski

Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Advanced networking technologies and services have been rapidly developed and deployed to facilitate bulk data transfer so as to support next-generation eScience applications. However, these technologies and services have not been fully utilized due to the knowledge lack of scientific domain experts. By leveraging the functionalities of an existing data movement advising utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization. WINDMA provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. Efficacy of WINDMA has been demonstrated in several use cases based on its implementation and deployment in wide-area networks.

Original language	English
Title of host publication	2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012
Pages	181-182
Number of pages	2
DOIs	https://doi.org/10.1109/PCCC.2012.6407676
State	Published - 1 Dec 2012
Event	2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012 - Austin, TX, United States Duration: 1 Dec 2012 → 3 Dec 2012

Other

Other	2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012
Country/Territory	United States
City	Austin, TX
Period	1/12/12 → 3/12/12

Keywords

eScience applications
High-performance networks
large data transfer

Access to Document

10.1109/PCCC.2012.6407676

Cite this

@inproceedings{bd98a77369d942b0b2595b1e703c37d8,

title = "Exploring the optimal strategy for large-scale data movement in high-performance networks",

abstract = "Advanced networking technologies and services have been rapidly developed and deployed to facilitate bulk data transfer so as to support next-generation eScience applications. However, these technologies and services have not been fully utilized due to the knowledge lack of scientific domain experts. By leveraging the functionalities of an existing data movement advising utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization. WINDMA provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. Efficacy of WINDMA has been demonstrated in several use cases based on its implementation and deployment in wide-area networks.",

keywords = "eScience applications, High-performance networks, large data transfer",

author = "Patrick Brown and Mengxia Zhu and Qishi Wu and Daqing Yun and Jason Zurawski",

year = "2012",

month = dec,

day = "1",

doi = "10.1109/PCCC.2012.6407676",

language = "English",

isbn = "9781467348812",

pages = "181--182",

booktitle = "2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012",

note = "null ; Conference date: 01-12-2012 Through 03-12-2012",

}

Brown, P, Zhu, M, Wu, Q, Yun, D & Zurawski, J 2012, Exploring the optimal strategy for large-scale data movement in high-performance networks. in 2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012., 6407676, pp. 181-182, 2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012, Austin, TX, United States, 1/12/12. https://doi.org/10.1109/PCCC.2012.6407676

TY - GEN

T1 - Exploring the optimal strategy for large-scale data movement in high-performance networks

AU - Brown, Patrick

AU - Zhu, Mengxia

AU - Wu, Qishi

AU - Yun, Daqing

AU - Zurawski, Jason

PY - 2012/12/1

Y1 - 2012/12/1

N2 - Advanced networking technologies and services have been rapidly developed and deployed to facilitate bulk data transfer so as to support next-generation eScience applications. However, these technologies and services have not been fully utilized due to the knowledge lack of scientific domain experts. By leveraging the functionalities of an existing data movement advising utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization. WINDMA provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. Efficacy of WINDMA has been demonstrated in several use cases based on its implementation and deployment in wide-area networks.

AB - Advanced networking technologies and services have been rapidly developed and deployed to facilitate bulk data transfer so as to support next-generation eScience applications. However, these technologies and services have not been fully utilized due to the knowledge lack of scientific domain experts. By leveraging the functionalities of an existing data movement advising utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization. WINDMA provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. Efficacy of WINDMA has been demonstrated in several use cases based on its implementation and deployment in wide-area networks.

KW - eScience applications

KW - High-performance networks

KW - large data transfer

UR - http://www.scopus.com/inward/record.url?scp=84874305124&partnerID=8YFLogxK

U2 - 10.1109/PCCC.2012.6407676

DO - 10.1109/PCCC.2012.6407676

M3 - Conference contribution

AN - SCOPUS:84874305124

SN - 9781467348812

SP - 181

EP - 182

BT - 2012 IEEE 31st International Performance Computing and Communications Conference, IPCCC 2012

Y2 - 1 December 2012 through 3 December 2012

ER -

Exploring the optimal strategy for large-scale data movement in high-performance networks

Abstract

Other

Keywords

Access to Document

Other files and links

Fingerprint

Cite this