Network-aware data movement advisor

Patrick Brown, Michelle Zhu, Qishi Wu, Xukang Lu

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

4 Citations (Scopus)

Abstract

Next-generation eScience applications often generate large amounts of simulation or experimental data that must be shared and managed by collaborative organizations. Advanced networking technologies and services have been rapidly developed and deployed to facilitate the massive data transport necessary for such data sharing and collaboration. However, these technologies and services have not been fully utilized by application users mainly because their use typically requires significant domain knowledge and in many cases even their existence is not made aware to the public. We design and develop a Network-aware Data Movement Advisor (NADMA) utility to enable automated discovery of network and system resources and advise the user of efficient strategies for fast and successful data transfer. NADMA is primarily a client-end program that interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP, and network resource provisioning systems such as TeraPaths and OSCARS. NADMA acts as a route planner in a typical vehicle navigation system to provide the user a set of feasible route options along with performance estimations as well as specific steps and commands to authorize and execute data transfer. We demonstrate the efficacy of NADMA in several use cases based on its implementation and deployment in wide-area networks.

Original languageEnglish
Title of host publicationNDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11
Pages31-39
Number of pages9
DOIs
StatePublished - 1 Dec 2011
EventInternational Workshop on Network-Aware Data Management, NDM'11, Held in Conjunction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC'11 - Seattle, WA, United States
Duration: 14 Nov 201114 Nov 2011

Publication series

NameNDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11

Other

OtherInternational Workshop on Network-Aware Data Management, NDM'11, Held in Conjunction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC'11
CountryUnited States
CitySeattle, WA
Period14/11/1114/11/11

Fingerprint

Data transfer
Wide area networks
Navigation systems

Keywords

  • Data transfer
  • High-performance network

Cite this

Brown, P., Zhu, M., Wu, Q., & Lu, X. (2011). Network-aware data movement advisor. In NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11 (pp. 31-39). (NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11). https://doi.org/10.1145/2110217.2110222
Brown, Patrick ; Zhu, Michelle ; Wu, Qishi ; Lu, Xukang. / Network-aware data movement advisor. NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11. 2011. pp. 31-39 (NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11).
@inproceedings{f98febc26e394e98a85e2f8fac01d6f9,
title = "Network-aware data movement advisor",
abstract = "Next-generation eScience applications often generate large amounts of simulation or experimental data that must be shared and managed by collaborative organizations. Advanced networking technologies and services have been rapidly developed and deployed to facilitate the massive data transport necessary for such data sharing and collaboration. However, these technologies and services have not been fully utilized by application users mainly because their use typically requires significant domain knowledge and in many cases even their existence is not made aware to the public. We design and develop a Network-aware Data Movement Advisor (NADMA) utility to enable automated discovery of network and system resources and advise the user of efficient strategies for fast and successful data transfer. NADMA is primarily a client-end program that interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP, and network resource provisioning systems such as TeraPaths and OSCARS. NADMA acts as a route planner in a typical vehicle navigation system to provide the user a set of feasible route options along with performance estimations as well as specific steps and commands to authorize and execute data transfer. We demonstrate the efficacy of NADMA in several use cases based on its implementation and deployment in wide-area networks.",
keywords = "Data transfer, High-performance network",
author = "Patrick Brown and Michelle Zhu and Qishi Wu and Xukang Lu",
year = "2011",
month = "12",
day = "1",
doi = "10.1145/2110217.2110222",
language = "English",
isbn = "9781450311328",
series = "NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11",
pages = "31--39",
booktitle = "NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11",

}

Brown, P, Zhu, M, Wu, Q & Lu, X 2011, Network-aware data movement advisor. in NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11. NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11, pp. 31-39, International Workshop on Network-Aware Data Management, NDM'11, Held in Conjunction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC'11, Seattle, WA, United States, 14/11/11. https://doi.org/10.1145/2110217.2110222

Network-aware data movement advisor. / Brown, Patrick; Zhu, Michelle; Wu, Qishi; Lu, Xukang.

NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11. 2011. p. 31-39 (NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11).

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

TY - GEN

T1 - Network-aware data movement advisor

AU - Brown, Patrick

AU - Zhu, Michelle

AU - Wu, Qishi

AU - Lu, Xukang

PY - 2011/12/1

Y1 - 2011/12/1

N2 - Next-generation eScience applications often generate large amounts of simulation or experimental data that must be shared and managed by collaborative organizations. Advanced networking technologies and services have been rapidly developed and deployed to facilitate the massive data transport necessary for such data sharing and collaboration. However, these technologies and services have not been fully utilized by application users mainly because their use typically requires significant domain knowledge and in many cases even their existence is not made aware to the public. We design and develop a Network-aware Data Movement Advisor (NADMA) utility to enable automated discovery of network and system resources and advise the user of efficient strategies for fast and successful data transfer. NADMA is primarily a client-end program that interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP, and network resource provisioning systems such as TeraPaths and OSCARS. NADMA acts as a route planner in a typical vehicle navigation system to provide the user a set of feasible route options along with performance estimations as well as specific steps and commands to authorize and execute data transfer. We demonstrate the efficacy of NADMA in several use cases based on its implementation and deployment in wide-area networks.

AB - Next-generation eScience applications often generate large amounts of simulation or experimental data that must be shared and managed by collaborative organizations. Advanced networking technologies and services have been rapidly developed and deployed to facilitate the massive data transport necessary for such data sharing and collaboration. However, these technologies and services have not been fully utilized by application users mainly because their use typically requires significant domain knowledge and in many cases even their existence is not made aware to the public. We design and develop a Network-aware Data Movement Advisor (NADMA) utility to enable automated discovery of network and system resources and advise the user of efficient strategies for fast and successful data transfer. NADMA is primarily a client-end program that interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP, and network resource provisioning systems such as TeraPaths and OSCARS. NADMA acts as a route planner in a typical vehicle navigation system to provide the user a set of feasible route options along with performance estimations as well as specific steps and commands to authorize and execute data transfer. We demonstrate the efficacy of NADMA in several use cases based on its implementation and deployment in wide-area networks.

KW - Data transfer

KW - High-performance network

UR - http://www.scopus.com/inward/record.url?scp=84857956440&partnerID=8YFLogxK

U2 - 10.1145/2110217.2110222

DO - 10.1145/2110217.2110222

M3 - Conference contribution

SN - 9781450311328

T3 - NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11

SP - 31

EP - 39

BT - NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11

ER -

Brown P, Zhu M, Wu Q, Lu X. Network-aware data movement advisor. In NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11. 2011. p. 31-39. (NDM'11 - Proceedings of the 2011 International Workshop on Network-Aware Data Management, Co-located with SC'11). https://doi.org/10.1145/2110217.2110222