TY - GEN
T1 - Admission control in YARN clusters based on dynamic resource reservation
AU - Yao, Yi
AU - Lin, Jason
AU - Wang, Jiayin
AU - Mi, Ningfang
AU - Sheng, Bo
PY - 2015/6/29
Y1 - 2015/6/29
N2 - Hadoop YARN is an open project developed by the Apache Software Foundation to provide a resource management framework for large scale parallel data processing. However, there exists a resource waiting deadlock under the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. Therefore, we develop a new admission control mechanism which dynamically reserves resources for processing tasks in order to avoid resource waiting deadlocks and meanwhile obtain good performance. We implement and evaluate our new mechanism in Hadoop YARN v2.2.0. The experimental results show the effectiveness of this mechanism under MapReduce benchmarks.
AB - Hadoop YARN is an open project developed by the Apache Software Foundation to provide a resource management framework for large scale parallel data processing. However, there exists a resource waiting deadlock under the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. Therefore, we develop a new admission control mechanism which dynamically reserves resources for processing tasks in order to avoid resource waiting deadlocks and meanwhile obtain good performance. We implement and evaluate our new mechanism in Hadoop YARN v2.2.0. The experimental results show the effectiveness of this mechanism under MapReduce benchmarks.
UR - http://www.scopus.com/inward/record.url?scp=84942579093&partnerID=8YFLogxK
U2 - 10.1109/INM.2015.7140389
DO - 10.1109/INM.2015.7140389
M3 - Conference contribution
AN - SCOPUS:84942579093
T3 - Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015
SP - 838
EP - 841
BT - Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015
A2 - De Turck, Filip
A2 - Badonnel, Remi
A2 - dos Santos, Carlos Raniery P.
A2 - Xiao, Jin
A2 - Ata, Shingo
A2 - Groza, Voicu
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 14th IFIP/IEEE International Symposium on Integrated Network Management, IM 2015
Y2 - 11 May 2015 through 15 May 2015
ER -