Autoadmin

Automatic and dynamic resource reservation admission control in Hadoop YARN clusters

Zhengyu Yang, Janki Bhimani, Yi Yao, Cho Hsien Lin, Jiayin Wang, Ningfang Mi, Bo Sheng

Research output: Contribution to journalArticleResearchpeer-review

3 Citations (Scopus)

Abstract

Hadoop YARN is an Apache Software Foundation's open project that provides a resource management framework for large scale parallel data processing, such as MapReduce jobs. Fair scheduler is a dispatcher which has been widely used in YARN to assign resources fairly and equally to applications. However, there exists a problem of the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by Application Masters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. To solve this problem, we propose an automatic and dynamic admission control mechanism to prevent the ceasing situation happened when the requested amount of resources exceeds the cluster's resource capacity, and dynamically reserve resources for processing tasks in order to obtain good performance, e.g., reducing makespans of MapReduce jobs. After collecting resource usage information of each work node, our mechanism dynamically predicts the amount of reserved resources for processing tasks and automatically controls running jobs based on the prediction. We implement the new mechanism in Hadoop YARN and evaluate it with representative MapReduce benchmarks. The experimental results show the effectiveness and robustness of this mechanism under both homogeneous and heterogeneous workloads.

Original languageEnglish
Pages (from-to)53-67
Number of pages15
JournalScalable Computing
Volume19
Issue number1
DOIs
StatePublished - 1 Jan 2018

Fingerprint

Access control
Processing

Keywords

  • Admission control
  • Big data
  • Cloud computing
  • Cluster computing
  • Hadoop
  • MapReduce
  • Resource management
  • Scalable computing
  • Scheduling
  • YARN

Cite this

Yang, Zhengyu ; Bhimani, Janki ; Yao, Yi ; Lin, Cho Hsien ; Wang, Jiayin ; Mi, Ningfang ; Sheng, Bo. / Autoadmin : Automatic and dynamic resource reservation admission control in Hadoop YARN clusters. In: Scalable Computing. 2018 ; Vol. 19, No. 1. pp. 53-67.
@article{e186bd9f87f74706a7a48d1559bad1a1,
title = "Autoadmin: Automatic and dynamic resource reservation admission control in Hadoop YARN clusters",
abstract = "Hadoop YARN is an Apache Software Foundation's open project that provides a resource management framework for large scale parallel data processing, such as MapReduce jobs. Fair scheduler is a dispatcher which has been widely used in YARN to assign resources fairly and equally to applications. However, there exists a problem of the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by Application Masters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. To solve this problem, we propose an automatic and dynamic admission control mechanism to prevent the ceasing situation happened when the requested amount of resources exceeds the cluster's resource capacity, and dynamically reserve resources for processing tasks in order to obtain good performance, e.g., reducing makespans of MapReduce jobs. After collecting resource usage information of each work node, our mechanism dynamically predicts the amount of reserved resources for processing tasks and automatically controls running jobs based on the prediction. We implement the new mechanism in Hadoop YARN and evaluate it with representative MapReduce benchmarks. The experimental results show the effectiveness and robustness of this mechanism under both homogeneous and heterogeneous workloads.",
keywords = "Admission control, Big data, Cloud computing, Cluster computing, Hadoop, MapReduce, Resource management, Scalable computing, Scheduling, YARN",
author = "Zhengyu Yang and Janki Bhimani and Yi Yao and Lin, {Cho Hsien} and Jiayin Wang and Ningfang Mi and Bo Sheng",
year = "2018",
month = "1",
day = "1",
doi = "10.12694/scpe.v19i1.1393",
language = "English",
volume = "19",
pages = "53--67",
journal = "Scalable Computing",
issn = "1895-1767",
publisher = "universitatea de vest",
number = "1",

}

Autoadmin : Automatic and dynamic resource reservation admission control in Hadoop YARN clusters. / Yang, Zhengyu; Bhimani, Janki; Yao, Yi; Lin, Cho Hsien; Wang, Jiayin; Mi, Ningfang; Sheng, Bo.

In: Scalable Computing, Vol. 19, No. 1, 01.01.2018, p. 53-67.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Autoadmin

T2 - Automatic and dynamic resource reservation admission control in Hadoop YARN clusters

AU - Yang, Zhengyu

AU - Bhimani, Janki

AU - Yao, Yi

AU - Lin, Cho Hsien

AU - Wang, Jiayin

AU - Mi, Ningfang

AU - Sheng, Bo

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Hadoop YARN is an Apache Software Foundation's open project that provides a resource management framework for large scale parallel data processing, such as MapReduce jobs. Fair scheduler is a dispatcher which has been widely used in YARN to assign resources fairly and equally to applications. However, there exists a problem of the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by Application Masters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. To solve this problem, we propose an automatic and dynamic admission control mechanism to prevent the ceasing situation happened when the requested amount of resources exceeds the cluster's resource capacity, and dynamically reserve resources for processing tasks in order to obtain good performance, e.g., reducing makespans of MapReduce jobs. After collecting resource usage information of each work node, our mechanism dynamically predicts the amount of reserved resources for processing tasks and automatically controls running jobs based on the prediction. We implement the new mechanism in Hadoop YARN and evaluate it with representative MapReduce benchmarks. The experimental results show the effectiveness and robustness of this mechanism under both homogeneous and heterogeneous workloads.

AB - Hadoop YARN is an Apache Software Foundation's open project that provides a resource management framework for large scale parallel data processing, such as MapReduce jobs. Fair scheduler is a dispatcher which has been widely used in YARN to assign resources fairly and equally to applications. However, there exists a problem of the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by Application Masters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. To solve this problem, we propose an automatic and dynamic admission control mechanism to prevent the ceasing situation happened when the requested amount of resources exceeds the cluster's resource capacity, and dynamically reserve resources for processing tasks in order to obtain good performance, e.g., reducing makespans of MapReduce jobs. After collecting resource usage information of each work node, our mechanism dynamically predicts the amount of reserved resources for processing tasks and automatically controls running jobs based on the prediction. We implement the new mechanism in Hadoop YARN and evaluate it with representative MapReduce benchmarks. The experimental results show the effectiveness and robustness of this mechanism under both homogeneous and heterogeneous workloads.

KW - Admission control

KW - Big data

KW - Cloud computing

KW - Cluster computing

KW - Hadoop

KW - MapReduce

KW - Resource management

KW - Scalable computing

KW - Scheduling

KW - YARN

UR - http://www.scopus.com/inward/record.url?scp=85043763390&partnerID=8YFLogxK

U2 - 10.12694/scpe.v19i1.1393

DO - 10.12694/scpe.v19i1.1393

M3 - Article

VL - 19

SP - 53

EP - 67

JO - Scalable Computing

JF - Scalable Computing

SN - 1895-1767

IS - 1

ER -