OpERA: Opportunistic and efficient resource allocation in hadoop YARN by harnessing idle resources

Yi Yao, Han Gao, Jiayin Wang, Ningfang Mi, Bo Sheng

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Citations (Scopus)

Abstract

Efficiently managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource management platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be re-assigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may underutilize the cluster resources and degrade system performance. To address this issue, we propose a novel opportunistic and efficient resource allocation approach, named OpERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to re-assign opportunistic available resources to the pending tasks. We implement and evaluate OpERA in Hadoop YARN v2.5. Our experimental results show that OpERA significantly reduces the average job execution time and increases the resource (CPU and memory) utilizations.

Original languageEnglish
Title of host publication2016 25th International Conference on Computer Communications and Networks, ICCCN 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781509022793
DOIs
StatePublished - 14 Sep 2016
Event25th International Conference on Computer Communications and Networks, ICCCN 2016 - Waikoloa, United States
Duration: 1 Aug 20164 Aug 2016

Publication series

Name2016 25th International Conference on Computer Communications and Networks, ICCCN 2016

Other

Other25th International Conference on Computer Communications and Networks, ICCCN 2016
CountryUnited States
CityWaikoloa
Period1/08/164/08/16

Fingerprint

Resource allocation
Electric sparks
Explosions
Program processors
Containers
Throughput
Data storage equipment

Keywords

  • Hadoop YARN
  • MapReduce scheduling
  • Resource allocation

Cite this

Yao, Y., Gao, H., Wang, J., Mi, N., & Sheng, B. (2016). OpERA: Opportunistic and efficient resource allocation in hadoop YARN by harnessing idle resources. In 2016 25th International Conference on Computer Communications and Networks, ICCCN 2016 [7568553] (2016 25th International Conference on Computer Communications and Networks, ICCCN 2016). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCCN.2016.7568553
Yao, Yi ; Gao, Han ; Wang, Jiayin ; Mi, Ningfang ; Sheng, Bo. / OpERA : Opportunistic and efficient resource allocation in hadoop YARN by harnessing idle resources. 2016 25th International Conference on Computer Communications and Networks, ICCCN 2016. Institute of Electrical and Electronics Engineers Inc., 2016. (2016 25th International Conference on Computer Communications and Networks, ICCCN 2016).
@inproceedings{ff992efc7c9746ff8fb4fb86e5c88771,
title = "OpERA: Opportunistic and efficient resource allocation in hadoop YARN by harnessing idle resources",
abstract = "Efficiently managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource management platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be re-assigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may underutilize the cluster resources and degrade system performance. To address this issue, we propose a novel opportunistic and efficient resource allocation approach, named OpERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to re-assign opportunistic available resources to the pending tasks. We implement and evaluate OpERA in Hadoop YARN v2.5. Our experimental results show that OpERA significantly reduces the average job execution time and increases the resource (CPU and memory) utilizations.",
keywords = "Hadoop YARN, MapReduce scheduling, Resource allocation",
author = "Yi Yao and Han Gao and Jiayin Wang and Ningfang Mi and Bo Sheng",
year = "2016",
month = "9",
day = "14",
doi = "10.1109/ICCCN.2016.7568553",
language = "English",
series = "2016 25th International Conference on Computer Communications and Networks, ICCCN 2016",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
booktitle = "2016 25th International Conference on Computer Communications and Networks, ICCCN 2016",

}

Yao, Y, Gao, H, Wang, J, Mi, N & Sheng, B 2016, OpERA: Opportunistic and efficient resource allocation in hadoop YARN by harnessing idle resources. in 2016 25th International Conference on Computer Communications and Networks, ICCCN 2016., 7568553, 2016 25th International Conference on Computer Communications and Networks, ICCCN 2016, Institute of Electrical and Electronics Engineers Inc., 25th International Conference on Computer Communications and Networks, ICCCN 2016, Waikoloa, United States, 1/08/16. https://doi.org/10.1109/ICCCN.2016.7568553

OpERA : Opportunistic and efficient resource allocation in hadoop YARN by harnessing idle resources. / Yao, Yi; Gao, Han; Wang, Jiayin; Mi, Ningfang; Sheng, Bo.

2016 25th International Conference on Computer Communications and Networks, ICCCN 2016. Institute of Electrical and Electronics Engineers Inc., 2016. 7568553 (2016 25th International Conference on Computer Communications and Networks, ICCCN 2016).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - OpERA

T2 - Opportunistic and efficient resource allocation in hadoop YARN by harnessing idle resources

AU - Yao, Yi

AU - Gao, Han

AU - Wang, Jiayin

AU - Mi, Ningfang

AU - Sheng, Bo

PY - 2016/9/14

Y1 - 2016/9/14

N2 - Efficiently managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource management platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be re-assigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may underutilize the cluster resources and degrade system performance. To address this issue, we propose a novel opportunistic and efficient resource allocation approach, named OpERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to re-assign opportunistic available resources to the pending tasks. We implement and evaluate OpERA in Hadoop YARN v2.5. Our experimental results show that OpERA significantly reduces the average job execution time and increases the resource (CPU and memory) utilizations.

AB - Efficiently managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource management platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be re-assigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may underutilize the cluster resources and degrade system performance. To address this issue, we propose a novel opportunistic and efficient resource allocation approach, named OpERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to re-assign opportunistic available resources to the pending tasks. We implement and evaluate OpERA in Hadoop YARN v2.5. Our experimental results show that OpERA significantly reduces the average job execution time and increases the resource (CPU and memory) utilizations.

KW - Hadoop YARN

KW - MapReduce scheduling

KW - Resource allocation

UR - http://www.scopus.com/inward/record.url?scp=84991821300&partnerID=8YFLogxK

U2 - 10.1109/ICCCN.2016.7568553

DO - 10.1109/ICCCN.2016.7568553

M3 - Conference contribution

AN - SCOPUS:84991821300

T3 - 2016 25th International Conference on Computer Communications and Networks, ICCCN 2016

BT - 2016 25th International Conference on Computer Communications and Networks, ICCCN 2016

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Yao Y, Gao H, Wang J, Mi N, Sheng B. OpERA: Opportunistic and efficient resource allocation in hadoop YARN by harnessing idle resources. In 2016 25th International Conference on Computer Communications and Networks, ICCCN 2016. Institute of Electrical and Electronics Engineers Inc. 2016. 7568553. (2016 25th International Conference on Computer Communications and Networks, ICCCN 2016). https://doi.org/10.1109/ICCCN.2016.7568553