Abstract
Hadoop YARN is an open project developed by the Apache Software Foundation to provide a resource management framework for large scale parallel data processing. However, there exists a resource waiting deadlock under the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. Therefore, we develop a new admission control mechanism which dynamically reserves resources for processing tasks in order to avoid resource waiting deadlocks and meanwhile obtain good performance. We implement and evaluate our new mechanism in Hadoop YARN v2.2.0. The experimental results show the effectiveness of this mechanism under MapReduce benchmarks.
| Original language | English |
|---|---|
| Title of host publication | Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015 |
| Editors | Filip De Turck, Remi Badonnel, Carlos Raniery P. dos Santos, Jin Xiao, Shingo Ata, Voicu Groza |
| Publisher | Institute of Electrical and Electronics Engineers Inc. |
| Pages | 838-841 |
| Number of pages | 4 |
| ISBN (Electronic) | 9783901882760 |
| DOIs | |
| State | Published - 29 Jun 2015 |
| Event | 14th IFIP/IEEE International Symposium on Integrated Network Management, IM 2015 - Ottawa, Canada Duration: 11 May 2015 → 15 May 2015 |
Publication series
| Name | Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015 |
|---|
Other
| Other | 14th IFIP/IEEE International Symposium on Integrated Network Management, IM 2015 |
|---|---|
| Country | Canada |
| City | Ottawa |
| Period | 11/05/15 → 15/05/15 |
Fingerprint
Cite this
}
Admission control in YARN clusters based on dynamic resource reservation. / Yao, Yi; Lin, Jason; Wang, Jiayin; Mi, Ningfang; Sheng, Bo.
Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015. ed. / Filip De Turck; Remi Badonnel; Carlos Raniery P. dos Santos; Jin Xiao; Shingo Ata; Voicu Groza. Institute of Electrical and Electronics Engineers Inc., 2015. p. 838-841 7140389 (Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
TY - GEN
T1 - Admission control in YARN clusters based on dynamic resource reservation
AU - Yao, Yi
AU - Lin, Jason
AU - Wang, Jiayin
AU - Mi, Ningfang
AU - Sheng, Bo
PY - 2015/6/29
Y1 - 2015/6/29
N2 - Hadoop YARN is an open project developed by the Apache Software Foundation to provide a resource management framework for large scale parallel data processing. However, there exists a resource waiting deadlock under the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. Therefore, we develop a new admission control mechanism which dynamically reserves resources for processing tasks in order to avoid resource waiting deadlocks and meanwhile obtain good performance. We implement and evaluate our new mechanism in Hadoop YARN v2.2.0. The experimental results show the effectiveness of this mechanism under MapReduce benchmarks.
AB - Hadoop YARN is an open project developed by the Apache Software Foundation to provide a resource management framework for large scale parallel data processing. However, there exists a resource waiting deadlock under the Fair scheduler when the resource requisition of applications is beyond the amount that the cluster can provide. In such a case, the YARN system will be halted if all resources are occupied by ApplicationMasters, a special task of each job that negotiates resources for processing tasks and coordinates job execution. Therefore, we develop a new admission control mechanism which dynamically reserves resources for processing tasks in order to avoid resource waiting deadlocks and meanwhile obtain good performance. We implement and evaluate our new mechanism in Hadoop YARN v2.2.0. The experimental results show the effectiveness of this mechanism under MapReduce benchmarks.
UR - http://www.scopus.com/inward/record.url?scp=84942579093&partnerID=8YFLogxK
U2 - 10.1109/INM.2015.7140389
DO - 10.1109/INM.2015.7140389
M3 - Conference contribution
AN - SCOPUS:84942579093
T3 - Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015
SP - 838
EP - 841
BT - Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015
A2 - De Turck, Filip
A2 - Badonnel, Remi
A2 - dos Santos, Carlos Raniery P.
A2 - Xiao, Jin
A2 - Ata, Shingo
A2 - Groza, Voicu
PB - Institute of Electrical and Electronics Engineers Inc.
ER -