DRESS: Dynamic RESource-Reservation Scheme for Congested Data-Intensive Computing Platforms

Ying Mao, Victoria Green, Jiayin Wang, Haoyi Xiong, Zhishan Guo

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

In the past few years, we have envisioned an increasing number of businesses start driving by big data analytics, such as Amazon recommendations and Google Advertisements. At the back-end side, the businesses are powered by big data processing platforms to quickly extract information and make decisions. Running on top of a computing cluster, those platforms utilize scheduling algorithms to allocate resources. An efficient scheduler is crucial to the system performance due to limited resources, e.g. CPU and Memory, and a large number of user demands. However, besides requests from clients and current status of the system, it has limited knowledge about execution length of the running jobs, and incoming jobs' resource demands, which make assigning resources a challenging task. If most of the resources are occupied by a long-running job, other jobs will have to keep waiting until it releases them. This paper presents a new scheduling strategy, named DRESS that particularly aims to optimize the allocation among jobs with various demands. Specifically, it classifies the jobs into two categories based on their requests, reserves a portion of resources for each of category, and dynamically adjusts the reserved ratio by monitoring the pending requests and estimating release patterns of running jobs. The results demonstrate DRESS significantly reduces the completion time for one category, up to 76.1% in our experiments, and in the meanwhile, maintains a stable overall system performance.

Original languageEnglish
Title of host publicationProceedings - 2018 IEEE International Conference on Cloud Computing, CLOUD 2018 - Part of the 2018 IEEE World Congress on Services
PublisherIEEE Computer Society
Pages694-701
Number of pages8
ISBN (Electronic)9781538672358
DOIs
StatePublished - 7 Sep 2018
Event11th IEEE International Conference on Cloud Computing, CLOUD 2018 - San Francisco, United States
Duration: 2 Jul 20187 Jul 2018

Publication series

NameIEEE International Conference on Cloud Computing, CLOUD
Volume2018-July
ISSN (Print)2159-6182
ISSN (Electronic)2159-6190

Other

Other11th IEEE International Conference on Cloud Computing, CLOUD 2018
CountryUnited States
CitySan Francisco
Period2/07/187/07/18

Keywords

  • Big data platforms
  • Hadoop
  • Scheduling

Fingerprint Dive into the research topics of 'DRESS: Dynamic RESource-Reservation Scheme for Congested Data-Intensive Computing Platforms'. Together they form a unique fingerprint.

  • Cite this

    Mao, Y., Green, V., Wang, J., Xiong, H., & Guo, Z. (2018). DRESS: Dynamic RESource-Reservation Scheme for Congested Data-Intensive Computing Platforms. In Proceedings - 2018 IEEE International Conference on Cloud Computing, CLOUD 2018 - Part of the 2018 IEEE World Congress on Services (pp. 694-701). [8457864] (IEEE International Conference on Cloud Computing, CLOUD; Vol. 2018-July). IEEE Computer Society. https://doi.org/10.1109/CLOUD.2018.00095