Admission control in YARN clusters based on dynamic resource reservation

Yi Yao, Jason Lin, Jiayin Wang, Ningfang Mi, Bo Sheng

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

6 Citations (Scopus)

Abstract

Hadoop YARN is an open project developed by the Apache Software Foundation to provide a resource management framework for large scale parallel data processing. However, there exists a resource waiting deadlock under the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. Therefore, we develop a new admission control mechanism which dynamically reserves resources for processing tasks in order to avoid resource waiting deadlocks and meanwhile obtain good performance. We implement and evaluate our new mechanism in Hadoop YARN v2.2.0. The experimental results show the effectiveness of this mechanism under MapReduce benchmarks.

Original languageEnglish
Title of host publicationProceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015
EditorsFilip De Turck, Remi Badonnel, Carlos Raniery P. dos Santos, Jin Xiao, Shingo Ata, Voicu Groza
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages838-841
Number of pages4
ISBN (Electronic)9783901882760
DOIs
StatePublished - 1 Jan 2015
Event14th IFIP/IEEE International Symposium on Integrated Network Management, IM 2015 - Ottawa, Canada
Duration: 11 May 201515 May 2015

Other

Other14th IFIP/IEEE International Symposium on Integrated Network Management, IM 2015
CountryCanada
CityOttawa
Period11/05/1515/05/15

Fingerprint

Access control
Processing

Cite this

Yao, Y., Lin, J., Wang, J., Mi, N., & Sheng, B. (2015). Admission control in YARN clusters based on dynamic resource reservation. In F. De Turck, R. Badonnel, C. R. P. dos Santos, J. Xiao, S. Ata, & V. Groza (Eds.), Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015 (pp. 838-841). [7140389] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/INM.2015.7140389
Yao, Yi ; Lin, Jason ; Wang, Jiayin ; Mi, Ningfang ; Sheng, Bo. / Admission control in YARN clusters based on dynamic resource reservation. Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015. editor / Filip De Turck ; Remi Badonnel ; Carlos Raniery P. dos Santos ; Jin Xiao ; Shingo Ata ; Voicu Groza. Institute of Electrical and Electronics Engineers Inc., 2015. pp. 838-841
@inproceedings{eae372b1d8a94438b793eb6742263f62,
title = "Admission control in YARN clusters based on dynamic resource reservation",
abstract = "Hadoop YARN is an open project developed by the Apache Software Foundation to provide a resource management framework for large scale parallel data processing. However, there exists a resource waiting deadlock under the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. Therefore, we develop a new admission control mechanism which dynamically reserves resources for processing tasks in order to avoid resource waiting deadlocks and meanwhile obtain good performance. We implement and evaluate our new mechanism in Hadoop YARN v2.2.0. The experimental results show the effectiveness of this mechanism under MapReduce benchmarks.",
author = "Yi Yao and Jason Lin and Jiayin Wang and Ningfang Mi and Bo Sheng",
year = "2015",
month = "1",
day = "1",
doi = "10.1109/INM.2015.7140389",
language = "English",
pages = "838--841",
editor = "{De Turck}, Filip and Remi Badonnel and {dos Santos}, {Carlos Raniery P.} and Jin Xiao and Shingo Ata and Voicu Groza",
booktitle = "Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

Yao, Y, Lin, J, Wang, J, Mi, N & Sheng, B 2015, Admission control in YARN clusters based on dynamic resource reservation. in F De Turck, R Badonnel, CRP dos Santos, J Xiao, S Ata & V Groza (eds), Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015., 7140389, Institute of Electrical and Electronics Engineers Inc., pp. 838-841, 14th IFIP/IEEE International Symposium on Integrated Network Management, IM 2015, Ottawa, Canada, 11/05/15. https://doi.org/10.1109/INM.2015.7140389

Admission control in YARN clusters based on dynamic resource reservation. / Yao, Yi; Lin, Jason; Wang, Jiayin; Mi, Ningfang; Sheng, Bo.

Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015. ed. / Filip De Turck; Remi Badonnel; Carlos Raniery P. dos Santos; Jin Xiao; Shingo Ata; Voicu Groza. Institute of Electrical and Electronics Engineers Inc., 2015. p. 838-841 7140389.

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

TY - GEN

T1 - Admission control in YARN clusters based on dynamic resource reservation

AU - Yao, Yi

AU - Lin, Jason

AU - Wang, Jiayin

AU - Mi, Ningfang

AU - Sheng, Bo

PY - 2015/1/1

Y1 - 2015/1/1

N2 - Hadoop YARN is an open project developed by the Apache Software Foundation to provide a resource management framework for large scale parallel data processing. However, there exists a resource waiting deadlock under the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. Therefore, we develop a new admission control mechanism which dynamically reserves resources for processing tasks in order to avoid resource waiting deadlocks and meanwhile obtain good performance. We implement and evaluate our new mechanism in Hadoop YARN v2.2.0. The experimental results show the effectiveness of this mechanism under MapReduce benchmarks.

AB - Hadoop YARN is an open project developed by the Apache Software Foundation to provide a resource management framework for large scale parallel data processing. However, there exists a resource waiting deadlock under the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. Therefore, we develop a new admission control mechanism which dynamically reserves resources for processing tasks in order to avoid resource waiting deadlocks and meanwhile obtain good performance. We implement and evaluate our new mechanism in Hadoop YARN v2.2.0. The experimental results show the effectiveness of this mechanism under MapReduce benchmarks.

UR - http://www.scopus.com/inward/record.url?scp=84942579093&partnerID=8YFLogxK

U2 - 10.1109/INM.2015.7140389

DO - 10.1109/INM.2015.7140389

M3 - Conference contribution

SP - 838

EP - 841

BT - Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015

A2 - De Turck, Filip

A2 - Badonnel, Remi

A2 - dos Santos, Carlos Raniery P.

A2 - Xiao, Jin

A2 - Ata, Shingo

A2 - Groza, Voicu

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Yao Y, Lin J, Wang J, Mi N, Sheng B. Admission control in YARN clusters based on dynamic resource reservation. In De Turck F, Badonnel R, dos Santos CRP, Xiao J, Ata S, Groza V, editors, Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015. Institute of Electrical and Electronics Engineers Inc. 2015. p. 838-841. 7140389 https://doi.org/10.1109/INM.2015.7140389