HaSTE

Hadoop YARN scheduling based on task-dependency and resource-demand

Yi Yao, Jiayin Wang, Bo Sheng, Jason Lin, Ningfang Mi

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

30 Citations (Scopus)

Abstract

The MapReduce framework has become the de facto scheme for scalable semi-structured and un-structured data processing in recent years. The Hadoop ecosystem has evolved into its second generation, Hadoop YARN, which adopts finegrained resource management schemes for job scheduling. One of the primary performance concerns in YARN is how to minimize the total completion length, i.e., makespan, of a set of MapReduce jobs. However, the precedence constraint or fairness constraint in current widely used scheduling policies in YARN, such as FIFO and Fair, can both lead to inefficient resource allocation in the Hadoop YARN cluster. They also omit the dependency between tasks which is crucial for the efficiency of resource utilization. We thus propose a new YARN scheduler, named HaSTE, which can effectively reduce the makespan of MapReduce jobs in YARN by leveraging the information of requested resources, resource capacities, and dependency between tasks. We implemented HaSTE as a pluggable scheduler in the most recent version of Hadoop YARN, and evaluated it with classic MapReduce benchmarks. The experimental results demonstrate that our YARN scheduler effectively reduces the makespans and improves resource utilization compare to the current scheduling policies.

Original languageEnglish
Title of host publicationProceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014
EditorsCarl Kesselman
PublisherIEEE Computer Society
Pages184-191
Number of pages8
ISBN (Electronic)9781479950638
DOIs
StatePublished - 3 Dec 2014
Event7th IEEE International Conference on Cloud Computing, CLOUD 2014 - Anchorage, United States
Duration: 27 Jun 20142 Jul 2014

Publication series

NameIEEE International Conference on Cloud Computing, CLOUD
ISSN (Print)2159-6182
ISSN (Electronic)2159-6190

Other

Other7th IEEE International Conference on Cloud Computing, CLOUD 2014
CountryUnited States
CityAnchorage
Period27/06/142/07/14

Fingerprint

Scheduling
Ecosystems
Resource allocation

Cite this

Yao, Y., Wang, J., Sheng, B., Lin, J., & Mi, N. (2014). HaSTE: Hadoop YARN scheduling based on task-dependency and resource-demand. In C. Kesselman (Ed.), Proceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014 (pp. 184-191). [6973740] (IEEE International Conference on Cloud Computing, CLOUD). IEEE Computer Society. https://doi.org/10.1109/CLOUD.2014.34
Yao, Yi ; Wang, Jiayin ; Sheng, Bo ; Lin, Jason ; Mi, Ningfang. / HaSTE : Hadoop YARN scheduling based on task-dependency and resource-demand. Proceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014. editor / Carl Kesselman. IEEE Computer Society, 2014. pp. 184-191 (IEEE International Conference on Cloud Computing, CLOUD).
@inproceedings{247acaa2e7d74c0c973fdd87ed6c8a7f,
title = "HaSTE: Hadoop YARN scheduling based on task-dependency and resource-demand",
abstract = "The MapReduce framework has become the de facto scheme for scalable semi-structured and un-structured data processing in recent years. The Hadoop ecosystem has evolved into its second generation, Hadoop YARN, which adopts finegrained resource management schemes for job scheduling. One of the primary performance concerns in YARN is how to minimize the total completion length, i.e., makespan, of a set of MapReduce jobs. However, the precedence constraint or fairness constraint in current widely used scheduling policies in YARN, such as FIFO and Fair, can both lead to inefficient resource allocation in the Hadoop YARN cluster. They also omit the dependency between tasks which is crucial for the efficiency of resource utilization. We thus propose a new YARN scheduler, named HaSTE, which can effectively reduce the makespan of MapReduce jobs in YARN by leveraging the information of requested resources, resource capacities, and dependency between tasks. We implemented HaSTE as a pluggable scheduler in the most recent version of Hadoop YARN, and evaluated it with classic MapReduce benchmarks. The experimental results demonstrate that our YARN scheduler effectively reduces the makespans and improves resource utilization compare to the current scheduling policies.",
author = "Yi Yao and Jiayin Wang and Bo Sheng and Jason Lin and Ningfang Mi",
year = "2014",
month = "12",
day = "3",
doi = "10.1109/CLOUD.2014.34",
language = "English",
series = "IEEE International Conference on Cloud Computing, CLOUD",
publisher = "IEEE Computer Society",
pages = "184--191",
editor = "Carl Kesselman",
booktitle = "Proceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014",

}

Yao, Y, Wang, J, Sheng, B, Lin, J & Mi, N 2014, HaSTE: Hadoop YARN scheduling based on task-dependency and resource-demand. in C Kesselman (ed.), Proceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014., 6973740, IEEE International Conference on Cloud Computing, CLOUD, IEEE Computer Society, pp. 184-191, 7th IEEE International Conference on Cloud Computing, CLOUD 2014, Anchorage, United States, 27/06/14. https://doi.org/10.1109/CLOUD.2014.34

HaSTE : Hadoop YARN scheduling based on task-dependency and resource-demand. / Yao, Yi; Wang, Jiayin; Sheng, Bo; Lin, Jason; Mi, Ningfang.

Proceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014. ed. / Carl Kesselman. IEEE Computer Society, 2014. p. 184-191 6973740 (IEEE International Conference on Cloud Computing, CLOUD).

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

TY - GEN

T1 - HaSTE

T2 - Hadoop YARN scheduling based on task-dependency and resource-demand

AU - Yao, Yi

AU - Wang, Jiayin

AU - Sheng, Bo

AU - Lin, Jason

AU - Mi, Ningfang

PY - 2014/12/3

Y1 - 2014/12/3

N2 - The MapReduce framework has become the de facto scheme for scalable semi-structured and un-structured data processing in recent years. The Hadoop ecosystem has evolved into its second generation, Hadoop YARN, which adopts finegrained resource management schemes for job scheduling. One of the primary performance concerns in YARN is how to minimize the total completion length, i.e., makespan, of a set of MapReduce jobs. However, the precedence constraint or fairness constraint in current widely used scheduling policies in YARN, such as FIFO and Fair, can both lead to inefficient resource allocation in the Hadoop YARN cluster. They also omit the dependency between tasks which is crucial for the efficiency of resource utilization. We thus propose a new YARN scheduler, named HaSTE, which can effectively reduce the makespan of MapReduce jobs in YARN by leveraging the information of requested resources, resource capacities, and dependency between tasks. We implemented HaSTE as a pluggable scheduler in the most recent version of Hadoop YARN, and evaluated it with classic MapReduce benchmarks. The experimental results demonstrate that our YARN scheduler effectively reduces the makespans and improves resource utilization compare to the current scheduling policies.

AB - The MapReduce framework has become the de facto scheme for scalable semi-structured and un-structured data processing in recent years. The Hadoop ecosystem has evolved into its second generation, Hadoop YARN, which adopts finegrained resource management schemes for job scheduling. One of the primary performance concerns in YARN is how to minimize the total completion length, i.e., makespan, of a set of MapReduce jobs. However, the precedence constraint or fairness constraint in current widely used scheduling policies in YARN, such as FIFO and Fair, can both lead to inefficient resource allocation in the Hadoop YARN cluster. They also omit the dependency between tasks which is crucial for the efficiency of resource utilization. We thus propose a new YARN scheduler, named HaSTE, which can effectively reduce the makespan of MapReduce jobs in YARN by leveraging the information of requested resources, resource capacities, and dependency between tasks. We implemented HaSTE as a pluggable scheduler in the most recent version of Hadoop YARN, and evaluated it with classic MapReduce benchmarks. The experimental results demonstrate that our YARN scheduler effectively reduces the makespans and improves resource utilization compare to the current scheduling policies.

UR - http://www.scopus.com/inward/record.url?scp=84919784576&partnerID=8YFLogxK

U2 - 10.1109/CLOUD.2014.34

DO - 10.1109/CLOUD.2014.34

M3 - Conference contribution

T3 - IEEE International Conference on Cloud Computing, CLOUD

SP - 184

EP - 191

BT - Proceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014

A2 - Kesselman, Carl

PB - IEEE Computer Society

ER -

Yao Y, Wang J, Sheng B, Lin J, Mi N. HaSTE: Hadoop YARN scheduling based on task-dependency and resource-demand. In Kesselman C, editor, Proceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014. IEEE Computer Society. 2014. p. 184-191. 6973740. (IEEE International Conference on Cloud Computing, CLOUD). https://doi.org/10.1109/CLOUD.2014.34