Admission control in YARN clusters based on dynamic resource reservation

Yi Yao, Jason Lin, Jiayin Wang, Ningfang Mi, Bo Sheng

Research output: Chapter in Book/Report/Conference proceedingConference contribution

7 Scopus citations

Abstract

Hadoop YARN is an open project developed by the Apache Software Foundation to provide a resource management framework for large scale parallel data processing. However, there exists a resource waiting deadlock under the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. Therefore, we develop a new admission control mechanism which dynamically reserves resources for processing tasks in order to avoid resource waiting deadlocks and meanwhile obtain good performance. We implement and evaluate our new mechanism in Hadoop YARN v2.2.0. The experimental results show the effectiveness of this mechanism under MapReduce benchmarks.

Original languageEnglish
Title of host publicationProceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015
EditorsFilip De Turck, Remi Badonnel, Carlos Raniery P. dos Santos, Jin Xiao, Shingo Ata, Voicu Groza
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages838-841
Number of pages4
ISBN (Electronic)9783901882760
DOIs
StatePublished - 29 Jun 2015
Event14th IFIP/IEEE International Symposium on Integrated Network Management, IM 2015 - Ottawa, Canada
Duration: 11 May 201515 May 2015

Publication series

NameProceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015

Other

Other14th IFIP/IEEE International Symposium on Integrated Network Management, IM 2015
CountryCanada
CityOttawa
Period11/05/1515/05/15

Fingerprint Dive into the research topics of 'Admission control in YARN clusters based on dynamic resource reservation'. Together they form a unique fingerprint.

  • Cite this

    Yao, Y., Lin, J., Wang, J., Mi, N., & Sheng, B. (2015). Admission control in YARN clusters based on dynamic resource reservation. In F. De Turck, R. Badonnel, C. R. P. dos Santos, J. Xiao, S. Ata, & V. Groza (Eds.), Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015 (pp. 838-841). [7140389] (Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/INM.2015.7140389