New YARN Non-Exclusive Resource Management Scheme through Opportunistic Idle Resource Assignment

Zhengyu Yang, Yi Yao, Han Gao, Jiayin Wang, Ningfang Mi, Bo Sheng

Research output: Contribution to journalArticleResearchpeer-review

Abstract

Efficiently managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource manage- ment platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be re-assigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may under-utilize the cluster resources and degrade system performance. To address this issue, we propose a novel opportunistic and efficient resource allocation scheme, named O P ERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to re-assign opportunistic available resources to the pending tasks. O P ERA avoids incurring severe performance interference to active tasks by further using two approaches to efficiently balances the starvations of reserved tasks and normal queued tasks. We implement and evaluate O P ERA in Hadoop YARN v2.5.

Original languageEnglish
JournalIEEE Transactions on Cloud Computing
DOIs
StateAccepted/In press - 28 Aug 2018

Fingerprint

Electric sparks
Resource allocation
Explosions
Containers
Throughput

Keywords

  • Containers
  • Data processing
  • Hadoop YARN
  • MapReduce Scheduling
  • Opportunistic
  • Reservation
  • Resource Allocation
  • Resource management
  • Runtime
  • Spark
  • Sparks
  • Starvation
  • Task analysis
  • Yarn

Cite this

@article{71c94ceecd5648b2aec3d369ef92a06a,
title = "New YARN Non-Exclusive Resource Management Scheme through Opportunistic Idle Resource Assignment",
abstract = "Efficiently managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource manage- ment platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be re-assigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may under-utilize the cluster resources and degrade system performance. To address this issue, we propose a novel opportunistic and efficient resource allocation scheme, named O P ERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to re-assign opportunistic available resources to the pending tasks. O P ERA avoids incurring severe performance interference to active tasks by further using two approaches to efficiently balances the starvations of reserved tasks and normal queued tasks. We implement and evaluate O P ERA in Hadoop YARN v2.5.",
keywords = "Containers, Data processing, Hadoop YARN, MapReduce Scheduling, Opportunistic, Reservation, Resource Allocation, Resource management, Runtime, Spark, Sparks, Starvation, Task analysis, Yarn",
author = "Zhengyu Yang and Yi Yao and Han Gao and Jiayin Wang and Ningfang Mi and Bo Sheng",
year = "2018",
month = "8",
day = "28",
doi = "10.1109/TCC.2018.2867580",
language = "English",
journal = "IEEE Transactions on Cloud Computing",
issn = "2168-7161",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

New YARN Non-Exclusive Resource Management Scheme through Opportunistic Idle Resource Assignment. / Yang, Zhengyu; Yao, Yi; Gao, Han; Wang, Jiayin; Mi, Ningfang; Sheng, Bo.

In: IEEE Transactions on Cloud Computing, 28.08.2018.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - New YARN Non-Exclusive Resource Management Scheme through Opportunistic Idle Resource Assignment

AU - Yang, Zhengyu

AU - Yao, Yi

AU - Gao, Han

AU - Wang, Jiayin

AU - Mi, Ningfang

AU - Sheng, Bo

PY - 2018/8/28

Y1 - 2018/8/28

N2 - Efficiently managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource manage- ment platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be re-assigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may under-utilize the cluster resources and degrade system performance. To address this issue, we propose a novel opportunistic and efficient resource allocation scheme, named O P ERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to re-assign opportunistic available resources to the pending tasks. O P ERA avoids incurring severe performance interference to active tasks by further using two approaches to efficiently balances the starvations of reserved tasks and normal queued tasks. We implement and evaluate O P ERA in Hadoop YARN v2.5.

AB - Efficiently managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource manage- ment platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be re-assigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may under-utilize the cluster resources and degrade system performance. To address this issue, we propose a novel opportunistic and efficient resource allocation scheme, named O P ERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to re-assign opportunistic available resources to the pending tasks. O P ERA avoids incurring severe performance interference to active tasks by further using two approaches to efficiently balances the starvations of reserved tasks and normal queued tasks. We implement and evaluate O P ERA in Hadoop YARN v2.5.

KW - Containers

KW - Data processing

KW - Hadoop YARN

KW - MapReduce Scheduling

KW - Opportunistic

KW - Reservation

KW - Resource Allocation

KW - Resource management

KW - Runtime

KW - Spark

KW - Sparks

KW - Starvation

KW - Task analysis

KW - Yarn

UR - http://www.scopus.com/inward/record.url?scp=85052663184&partnerID=8YFLogxK

U2 - 10.1109/TCC.2018.2867580

DO - 10.1109/TCC.2018.2867580

M3 - Article

JO - IEEE Transactions on Cloud Computing

JF - IEEE Transactions on Cloud Computing

SN - 2168-7161

ER -