Bandwidth Scheduling for Big Data Transfer with Deadline Constraint between Data Centers

Aiqin Hou, Chase Q. Wu, Dingyi Fang, Liudong Zuo, Michelle Zhu, Xiaoyang Zhang, Ruimin Qiao, Xiaoyan Yin

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

An increasing number of applications in scientific and other domains have moved or are in active transition to clouds, and the demand for the movement of big data between geographically distributed cloud-based data centers is rapidly growing. Many modern backbone networks leverage logically centralized controllers based on software-defined networking (SDN) to provide advance bandwidth reservation for data transfer requests. How to fully utilize the bandwidth resources of the links connecting data centers with guaranteed QoS for each user request is an important problem for cloud service providers. Most existing work focuses on bandwidth scheduling for a single request for data transfer or multiple requests using the same service model. In this work, we construct rigorous cost models to quantify user satisfaction degree and formulate a generic problem of bandwidth scheduling for multiple deadline-constrained data transfer requests of different types to maximize the request scheduling success ratio while minimizing the data transfer completion time of each request. We prove this problem to be NP-complete and design a heuristic solution. Extensive simulation results show that our scheduling scheme significantly outperforms existing methods in terms of user satisfaction degree and scheduling success ratio.

Original languageEnglish
Title of host publicationProceedings of INDIS 2018
Subtitle of host publicationInnovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages55-63
Number of pages9
ISBN (Electronic)9781728101941
DOIs
StatePublished - 21 Feb 2019
Event2018 IEEE/ACM Innovating the Network for Data-Intensive Science, INDIS 2018 - Dallas, United States
Duration: 11 Nov 2018 → …

Publication series

NameProceedings of INDIS 2018: Innovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference2018 IEEE/ACM Innovating the Network for Data-Intensive Science, INDIS 2018
CountryUnited States
CityDallas
Period11/11/18 → …

Fingerprint

Data transfer
Scheduling
Bandwidth
Telecommunication links
Quality of service
Big data
Controllers
Costs

Keywords

  • Bandwidth-scheduling
  • Big-data
  • Data-center
  • High-performance-networks
  • Software-defined-networking

Cite this

Hou, A., Wu, C. Q., Fang, D., Zuo, L., Zhu, M., Zhang, X., ... Yin, X. (2019). Bandwidth Scheduling for Big Data Transfer with Deadline Constraint between Data Centers. In Proceedings of INDIS 2018: Innovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 55-63). [8648791] (Proceedings of INDIS 2018: Innovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/INDIS.2018.00009
Hou, Aiqin ; Wu, Chase Q. ; Fang, Dingyi ; Zuo, Liudong ; Zhu, Michelle ; Zhang, Xiaoyang ; Qiao, Ruimin ; Yin, Xiaoyan. / Bandwidth Scheduling for Big Data Transfer with Deadline Constraint between Data Centers. Proceedings of INDIS 2018: Innovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 55-63 (Proceedings of INDIS 2018: Innovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis).
@inproceedings{8c63def73fb14cee912c2e1b2b8efee9,
title = "Bandwidth Scheduling for Big Data Transfer with Deadline Constraint between Data Centers",
abstract = "An increasing number of applications in scientific and other domains have moved or are in active transition to clouds, and the demand for the movement of big data between geographically distributed cloud-based data centers is rapidly growing. Many modern backbone networks leverage logically centralized controllers based on software-defined networking (SDN) to provide advance bandwidth reservation for data transfer requests. How to fully utilize the bandwidth resources of the links connecting data centers with guaranteed QoS for each user request is an important problem for cloud service providers. Most existing work focuses on bandwidth scheduling for a single request for data transfer or multiple requests using the same service model. In this work, we construct rigorous cost models to quantify user satisfaction degree and formulate a generic problem of bandwidth scheduling for multiple deadline-constrained data transfer requests of different types to maximize the request scheduling success ratio while minimizing the data transfer completion time of each request. We prove this problem to be NP-complete and design a heuristic solution. Extensive simulation results show that our scheduling scheme significantly outperforms existing methods in terms of user satisfaction degree and scheduling success ratio.",
keywords = "Bandwidth-scheduling, Big-data, Data-center, High-performance-networks, Software-defined-networking",
author = "Aiqin Hou and Wu, {Chase Q.} and Dingyi Fang and Liudong Zuo and Michelle Zhu and Xiaoyang Zhang and Ruimin Qiao and Xiaoyan Yin",
year = "2019",
month = "2",
day = "21",
doi = "10.1109/INDIS.2018.00009",
language = "English",
series = "Proceedings of INDIS 2018: Innovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "55--63",
booktitle = "Proceedings of INDIS 2018",

}

Hou, A, Wu, CQ, Fang, D, Zuo, L, Zhu, M, Zhang, X, Qiao, R & Yin, X 2019, Bandwidth Scheduling for Big Data Transfer with Deadline Constraint between Data Centers. in Proceedings of INDIS 2018: Innovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis., 8648791, Proceedings of INDIS 2018: Innovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis, Institute of Electrical and Electronics Engineers Inc., pp. 55-63, 2018 IEEE/ACM Innovating the Network for Data-Intensive Science, INDIS 2018, Dallas, United States, 11/11/18. https://doi.org/10.1109/INDIS.2018.00009

Bandwidth Scheduling for Big Data Transfer with Deadline Constraint between Data Centers. / Hou, Aiqin; Wu, Chase Q.; Fang, Dingyi; Zuo, Liudong; Zhu, Michelle; Zhang, Xiaoyang; Qiao, Ruimin; Yin, Xiaoyan.

Proceedings of INDIS 2018: Innovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2019. p. 55-63 8648791 (Proceedings of INDIS 2018: Innovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Bandwidth Scheduling for Big Data Transfer with Deadline Constraint between Data Centers

AU - Hou, Aiqin

AU - Wu, Chase Q.

AU - Fang, Dingyi

AU - Zuo, Liudong

AU - Zhu, Michelle

AU - Zhang, Xiaoyang

AU - Qiao, Ruimin

AU - Yin, Xiaoyan

PY - 2019/2/21

Y1 - 2019/2/21

N2 - An increasing number of applications in scientific and other domains have moved or are in active transition to clouds, and the demand for the movement of big data between geographically distributed cloud-based data centers is rapidly growing. Many modern backbone networks leverage logically centralized controllers based on software-defined networking (SDN) to provide advance bandwidth reservation for data transfer requests. How to fully utilize the bandwidth resources of the links connecting data centers with guaranteed QoS for each user request is an important problem for cloud service providers. Most existing work focuses on bandwidth scheduling for a single request for data transfer or multiple requests using the same service model. In this work, we construct rigorous cost models to quantify user satisfaction degree and formulate a generic problem of bandwidth scheduling for multiple deadline-constrained data transfer requests of different types to maximize the request scheduling success ratio while minimizing the data transfer completion time of each request. We prove this problem to be NP-complete and design a heuristic solution. Extensive simulation results show that our scheduling scheme significantly outperforms existing methods in terms of user satisfaction degree and scheduling success ratio.

AB - An increasing number of applications in scientific and other domains have moved or are in active transition to clouds, and the demand for the movement of big data between geographically distributed cloud-based data centers is rapidly growing. Many modern backbone networks leverage logically centralized controllers based on software-defined networking (SDN) to provide advance bandwidth reservation for data transfer requests. How to fully utilize the bandwidth resources of the links connecting data centers with guaranteed QoS for each user request is an important problem for cloud service providers. Most existing work focuses on bandwidth scheduling for a single request for data transfer or multiple requests using the same service model. In this work, we construct rigorous cost models to quantify user satisfaction degree and formulate a generic problem of bandwidth scheduling for multiple deadline-constrained data transfer requests of different types to maximize the request scheduling success ratio while minimizing the data transfer completion time of each request. We prove this problem to be NP-complete and design a heuristic solution. Extensive simulation results show that our scheduling scheme significantly outperforms existing methods in terms of user satisfaction degree and scheduling success ratio.

KW - Bandwidth-scheduling

KW - Big-data

KW - Data-center

KW - High-performance-networks

KW - Software-defined-networking

UR - http://www.scopus.com/inward/record.url?scp=85063325293&partnerID=8YFLogxK

U2 - 10.1109/INDIS.2018.00009

DO - 10.1109/INDIS.2018.00009

M3 - Conference contribution

AN - SCOPUS:85063325293

T3 - Proceedings of INDIS 2018: Innovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis

SP - 55

EP - 63

BT - Proceedings of INDIS 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Hou A, Wu CQ, Fang D, Zuo L, Zhu M, Zhang X et al. Bandwidth Scheduling for Big Data Transfer with Deadline Constraint between Data Centers. In Proceedings of INDIS 2018: Innovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc. 2019. p. 55-63. 8648791. (Proceedings of INDIS 2018: Innovating the Network for Data-Intensive Science, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis). https://doi.org/10.1109/INDIS.2018.00009