DRESS

Dynamic RESource-Reservation Scheme for Congested Data-Intensive Computing Platforms

Ying Mao, Victoria Green, Jiayin Wang, Haoyi Xiong, Zhishan Guo

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

1 Citation (Scopus)

Abstract

In the past few years, we have envisioned an increasing number of businesses start driving by big data analytics, such as Amazon recommendations and Google Advertisements. At the back-end side, the businesses are powered by big data processing platforms to quickly extract information and make decisions. Running on top of a computing cluster, those platforms utilize scheduling algorithms to allocate resources. An efficient scheduler is crucial to the system performance due to limited resources, e.g. CPU and Memory, and a large number of user demands. However, besides requests from clients and current status of the system, it has limited knowledge about execution length of the running jobs, and incoming jobs' resource demands, which make assigning resources a challenging task. If most of the resources are occupied by a long-running job, other jobs will have to keep waiting until it releases them. This paper presents a new scheduling strategy, named DRESS that particularly aims to optimize the allocation among jobs with various demands. Specifically, it classifies the jobs into two categories based on their requests, reserves a portion of resources for each of category, and dynamically adjusts the reserved ratio by monitoring the pending requests and estimating release patterns of running jobs. The results demonstrate DRESS significantly reduces the completion time for one category, up to 76.1% in our experiments, and in the meanwhile, maintains a stable overall system performance.

Original languageEnglish
Title of host publicationProceedings - 2018 IEEE International Conference on Cloud Computing, CLOUD 2018 - Part of the 2018 IEEE World Congress on Services
PublisherIEEE Computer Society
Pages694-701
Number of pages8
ISBN (Electronic)9781538672358
DOIs
StatePublished - 7 Sep 2018
Event11th IEEE International Conference on Cloud Computing, CLOUD 2018 - San Francisco, United States
Duration: 2 Jul 20187 Jul 2018

Publication series

NameIEEE International Conference on Cloud Computing, CLOUD
Volume2018-July
ISSN (Print)2159-6182
ISSN (Electronic)2159-6190

Other

Other11th IEEE International Conference on Cloud Computing, CLOUD 2018
CountryUnited States
CitySan Francisco
Period2/07/187/07/18

Fingerprint

Cluster computing
Scheduling algorithms
Program processors
Industry
Scheduling
Data storage equipment
Monitoring
Experiments
Big data

Keywords

  • Big data platforms
  • Hadoop
  • Scheduling

Cite this

Mao, Y., Green, V., Wang, J., Xiong, H., & Guo, Z. (2018). DRESS: Dynamic RESource-Reservation Scheme for Congested Data-Intensive Computing Platforms. In Proceedings - 2018 IEEE International Conference on Cloud Computing, CLOUD 2018 - Part of the 2018 IEEE World Congress on Services (pp. 694-701). [8457864] (IEEE International Conference on Cloud Computing, CLOUD; Vol. 2018-July). IEEE Computer Society. https://doi.org/10.1109/CLOUD.2018.00095
Mao, Ying ; Green, Victoria ; Wang, Jiayin ; Xiong, Haoyi ; Guo, Zhishan. / DRESS : Dynamic RESource-Reservation Scheme for Congested Data-Intensive Computing Platforms. Proceedings - 2018 IEEE International Conference on Cloud Computing, CLOUD 2018 - Part of the 2018 IEEE World Congress on Services. IEEE Computer Society, 2018. pp. 694-701 (IEEE International Conference on Cloud Computing, CLOUD).
@inproceedings{4899ac8f60344c9fb37584101c047c94,
title = "DRESS: Dynamic RESource-Reservation Scheme for Congested Data-Intensive Computing Platforms",
abstract = "In the past few years, we have envisioned an increasing number of businesses start driving by big data analytics, such as Amazon recommendations and Google Advertisements. At the back-end side, the businesses are powered by big data processing platforms to quickly extract information and make decisions. Running on top of a computing cluster, those platforms utilize scheduling algorithms to allocate resources. An efficient scheduler is crucial to the system performance due to limited resources, e.g. CPU and Memory, and a large number of user demands. However, besides requests from clients and current status of the system, it has limited knowledge about execution length of the running jobs, and incoming jobs' resource demands, which make assigning resources a challenging task. If most of the resources are occupied by a long-running job, other jobs will have to keep waiting until it releases them. This paper presents a new scheduling strategy, named DRESS that particularly aims to optimize the allocation among jobs with various demands. Specifically, it classifies the jobs into two categories based on their requests, reserves a portion of resources for each of category, and dynamically adjusts the reserved ratio by monitoring the pending requests and estimating release patterns of running jobs. The results demonstrate DRESS significantly reduces the completion time for one category, up to 76.1{\%} in our experiments, and in the meanwhile, maintains a stable overall system performance.",
keywords = "Big data platforms, Hadoop, Scheduling",
author = "Ying Mao and Victoria Green and Jiayin Wang and Haoyi Xiong and Zhishan Guo",
year = "2018",
month = "9",
day = "7",
doi = "10.1109/CLOUD.2018.00095",
language = "English",
series = "IEEE International Conference on Cloud Computing, CLOUD",
publisher = "IEEE Computer Society",
pages = "694--701",
booktitle = "Proceedings - 2018 IEEE International Conference on Cloud Computing, CLOUD 2018 - Part of the 2018 IEEE World Congress on Services",

}

Mao, Y, Green, V, Wang, J, Xiong, H & Guo, Z 2018, DRESS: Dynamic RESource-Reservation Scheme for Congested Data-Intensive Computing Platforms. in Proceedings - 2018 IEEE International Conference on Cloud Computing, CLOUD 2018 - Part of the 2018 IEEE World Congress on Services., 8457864, IEEE International Conference on Cloud Computing, CLOUD, vol. 2018-July, IEEE Computer Society, pp. 694-701, 11th IEEE International Conference on Cloud Computing, CLOUD 2018, San Francisco, United States, 2/07/18. https://doi.org/10.1109/CLOUD.2018.00095

DRESS : Dynamic RESource-Reservation Scheme for Congested Data-Intensive Computing Platforms. / Mao, Ying; Green, Victoria; Wang, Jiayin; Xiong, Haoyi; Guo, Zhishan.

Proceedings - 2018 IEEE International Conference on Cloud Computing, CLOUD 2018 - Part of the 2018 IEEE World Congress on Services. IEEE Computer Society, 2018. p. 694-701 8457864 (IEEE International Conference on Cloud Computing, CLOUD; Vol. 2018-July).

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

TY - GEN

T1 - DRESS

T2 - Dynamic RESource-Reservation Scheme for Congested Data-Intensive Computing Platforms

AU - Mao, Ying

AU - Green, Victoria

AU - Wang, Jiayin

AU - Xiong, Haoyi

AU - Guo, Zhishan

PY - 2018/9/7

Y1 - 2018/9/7

N2 - In the past few years, we have envisioned an increasing number of businesses start driving by big data analytics, such as Amazon recommendations and Google Advertisements. At the back-end side, the businesses are powered by big data processing platforms to quickly extract information and make decisions. Running on top of a computing cluster, those platforms utilize scheduling algorithms to allocate resources. An efficient scheduler is crucial to the system performance due to limited resources, e.g. CPU and Memory, and a large number of user demands. However, besides requests from clients and current status of the system, it has limited knowledge about execution length of the running jobs, and incoming jobs' resource demands, which make assigning resources a challenging task. If most of the resources are occupied by a long-running job, other jobs will have to keep waiting until it releases them. This paper presents a new scheduling strategy, named DRESS that particularly aims to optimize the allocation among jobs with various demands. Specifically, it classifies the jobs into two categories based on their requests, reserves a portion of resources for each of category, and dynamically adjusts the reserved ratio by monitoring the pending requests and estimating release patterns of running jobs. The results demonstrate DRESS significantly reduces the completion time for one category, up to 76.1% in our experiments, and in the meanwhile, maintains a stable overall system performance.

AB - In the past few years, we have envisioned an increasing number of businesses start driving by big data analytics, such as Amazon recommendations and Google Advertisements. At the back-end side, the businesses are powered by big data processing platforms to quickly extract information and make decisions. Running on top of a computing cluster, those platforms utilize scheduling algorithms to allocate resources. An efficient scheduler is crucial to the system performance due to limited resources, e.g. CPU and Memory, and a large number of user demands. However, besides requests from clients and current status of the system, it has limited knowledge about execution length of the running jobs, and incoming jobs' resource demands, which make assigning resources a challenging task. If most of the resources are occupied by a long-running job, other jobs will have to keep waiting until it releases them. This paper presents a new scheduling strategy, named DRESS that particularly aims to optimize the allocation among jobs with various demands. Specifically, it classifies the jobs into two categories based on their requests, reserves a portion of resources for each of category, and dynamically adjusts the reserved ratio by monitoring the pending requests and estimating release patterns of running jobs. The results demonstrate DRESS significantly reduces the completion time for one category, up to 76.1% in our experiments, and in the meanwhile, maintains a stable overall system performance.

KW - Big data platforms

KW - Hadoop

KW - Scheduling

UR - http://www.scopus.com/inward/record.url?scp=85057446866&partnerID=8YFLogxK

U2 - 10.1109/CLOUD.2018.00095

DO - 10.1109/CLOUD.2018.00095

M3 - Conference contribution

T3 - IEEE International Conference on Cloud Computing, CLOUD

SP - 694

EP - 701

BT - Proceedings - 2018 IEEE International Conference on Cloud Computing, CLOUD 2018 - Part of the 2018 IEEE World Congress on Services

PB - IEEE Computer Society

ER -

Mao Y, Green V, Wang J, Xiong H, Guo Z. DRESS: Dynamic RESource-Reservation Scheme for Congested Data-Intensive Computing Platforms. In Proceedings - 2018 IEEE International Conference on Cloud Computing, CLOUD 2018 - Part of the 2018 IEEE World Congress on Services. IEEE Computer Society. 2018. p. 694-701. 8457864. (IEEE International Conference on Cloud Computing, CLOUD). https://doi.org/10.1109/CLOUD.2018.00095