Abstract
Efficiently managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource management platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be re-assigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may underutilize the cluster resources and degrade system performance. To address this issue, we propose a novel opportunistic and efficient resource allocation approach, named OpERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to re-assign opportunistic available resources to the pending tasks. We implement and evaluate OpERA in Hadoop YARN v2.5. Our experimental results show that OpERA significantly reduces the average job execution time and increases the resource (CPU and memory) utilizations.
| Original language | English |
|---|---|
| Title of host publication | 2016 25th International Conference on Computer Communications and Networks, ICCCN 2016 |
| Publisher | Institute of Electrical and Electronics Engineers Inc. |
| ISBN (Electronic) | 9781509022793 |
| DOIs | |
| State | Published - 14 Sep 2016 |
| Event | 25th International Conference on Computer Communications and Networks, ICCCN 2016 - Waikoloa, United States Duration: 1 Aug 2016 → 4 Aug 2016 |
Publication series
| Name | 2016 25th International Conference on Computer Communications and Networks, ICCCN 2016 |
|---|
Other
| Other | 25th International Conference on Computer Communications and Networks, ICCCN 2016 |
|---|---|
| Country | United States |
| City | Waikoloa |
| Period | 1/08/16 → 4/08/16 |
Fingerprint
Keywords
- Hadoop YARN
- MapReduce scheduling
- Resource allocation
Cite this
}
OpERA : Opportunistic and efficient resource allocation in hadoop YARN by harnessing idle resources. / Yao, Yi; Gao, Han; Wang, Jiayin; Mi, Ningfang; Sheng, Bo.
2016 25th International Conference on Computer Communications and Networks, ICCCN 2016. Institute of Electrical and Electronics Engineers Inc., 2016. 7568553 (2016 25th International Conference on Computer Communications and Networks, ICCCN 2016).Research output: Chapter in Book/Report/Conference proceeding › Conference contribution
TY - GEN
T1 - OpERA
T2 - Opportunistic and efficient resource allocation in hadoop YARN by harnessing idle resources
AU - Yao, Yi
AU - Gao, Han
AU - Wang, Jiayin
AU - Mi, Ningfang
AU - Sheng, Bo
PY - 2016/9/14
Y1 - 2016/9/14
N2 - Efficiently managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource management platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be re-assigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may underutilize the cluster resources and degrade system performance. To address this issue, we propose a novel opportunistic and efficient resource allocation approach, named OpERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to re-assign opportunistic available resources to the pending tasks. We implement and evaluate OpERA in Hadoop YARN v2.5. Our experimental results show that OpERA significantly reduces the average job execution time and increases the resource (CPU and memory) utilizations.
AB - Efficiently managing resources and improving throughput in a large-scale cluster has become a crucial problem with the explosion of data processing applications in recent years. Hadoop YARN and Mesos, as two universal resource management platforms, have been widely adopted in the commodity cluster for co-deploying multiple data processing frameworks, such as Hadoop MapReduce and Apache Spark. However, in the existing resource management, a certain amount of resources are exclusively allocated to a running task and can only be re-assigned after that task is completed. This exclusive mode unfortunately leads to a potential problem that may underutilize the cluster resources and degrade system performance. To address this issue, we propose a novel opportunistic and efficient resource allocation approach, named OpERA, which breaks the barriers among the encapsulated resource containers by leveraging the knowledge of actual runtime resource utilizations to re-assign opportunistic available resources to the pending tasks. We implement and evaluate OpERA in Hadoop YARN v2.5. Our experimental results show that OpERA significantly reduces the average job execution time and increases the resource (CPU and memory) utilizations.
KW - Hadoop YARN
KW - MapReduce scheduling
KW - Resource allocation
UR - http://www.scopus.com/inward/record.url?scp=84991821300&partnerID=8YFLogxK
U2 - 10.1109/ICCCN.2016.7568553
DO - 10.1109/ICCCN.2016.7568553
M3 - Conference contribution
AN - SCOPUS:84991821300
T3 - 2016 25th International Conference on Computer Communications and Networks, ICCCN 2016
BT - 2016 25th International Conference on Computer Communications and Networks, ICCCN 2016
PB - Institute of Electrical and Electronics Engineers Inc.
ER -