FRESH

Fair and efficient slot configuration and scheduling for Hadoop clusters

Jiayin Wang, Yi Yao, Ying Mao, Bo Sheng, Ningfang Mi

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

20 Citations (Scopus)

Abstract

Hadoop is an emerging framework for parallel big data processing. While becoming popular, Hadoop is too complex for regular users to fully understand all the system parameters and tune them appropriately. Especially when processing a batch of jobs, default Hadoop setting may cause inefficient resource utilization and unnecessarily prolong the execution time. This paper considers an extremely important setting of slot configuration which by default is fixed and static. We proposed an enhanced Hadoop system called FRESH which can derive the best slot setting, dynamically configure slots, and appropriately assign tasks to the available slots. The experimental results show that when serving a batch of MapReduce jobs, FRESH significantly improves the makespan as well as the fairness among jobs.

Original languageEnglish
Title of host publicationProceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014
EditorsCarl Kesselman
PublisherIEEE Computer Society
Pages761-768
Number of pages8
ISBN (Electronic)9781479950638
DOIs
StatePublished - 3 Dec 2014
Event7th IEEE International Conference on Cloud Computing, CLOUD 2014 - Anchorage, United States
Duration: 27 Jun 20142 Jul 2014

Publication series

NameIEEE International Conference on Cloud Computing, CLOUD
ISSN (Print)2159-6182
ISSN (Electronic)2159-6190

Other

Other7th IEEE International Conference on Cloud Computing, CLOUD 2014
CountryUnited States
CityAnchorage
Period27/06/142/07/14

Fingerprint

Scheduling
Processing
Big data

Keywords

  • MapReduce
  • Resource Management
  • Scheduling

Cite this

Wang, J., Yao, Y., Mao, Y., Sheng, B., & Mi, N. (2014). FRESH: Fair and efficient slot configuration and scheduling for Hadoop clusters. In C. Kesselman (Ed.), Proceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014 (pp. 761-768). [6973812] (IEEE International Conference on Cloud Computing, CLOUD). IEEE Computer Society. https://doi.org/10.1109/CLOUD.2014.106
Wang, Jiayin ; Yao, Yi ; Mao, Ying ; Sheng, Bo ; Mi, Ningfang. / FRESH : Fair and efficient slot configuration and scheduling for Hadoop clusters. Proceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014. editor / Carl Kesselman. IEEE Computer Society, 2014. pp. 761-768 (IEEE International Conference on Cloud Computing, CLOUD).
@inproceedings{c5a9ed58f282472abc4a6bc9f36a430e,
title = "FRESH: Fair and efficient slot configuration and scheduling for Hadoop clusters",
abstract = "Hadoop is an emerging framework for parallel big data processing. While becoming popular, Hadoop is too complex for regular users to fully understand all the system parameters and tune them appropriately. Especially when processing a batch of jobs, default Hadoop setting may cause inefficient resource utilization and unnecessarily prolong the execution time. This paper considers an extremely important setting of slot configuration which by default is fixed and static. We proposed an enhanced Hadoop system called FRESH which can derive the best slot setting, dynamically configure slots, and appropriately assign tasks to the available slots. The experimental results show that when serving a batch of MapReduce jobs, FRESH significantly improves the makespan as well as the fairness among jobs.",
keywords = "MapReduce, Resource Management, Scheduling",
author = "Jiayin Wang and Yi Yao and Ying Mao and Bo Sheng and Ningfang Mi",
year = "2014",
month = "12",
day = "3",
doi = "10.1109/CLOUD.2014.106",
language = "English",
series = "IEEE International Conference on Cloud Computing, CLOUD",
publisher = "IEEE Computer Society",
pages = "761--768",
editor = "Carl Kesselman",
booktitle = "Proceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014",

}

Wang, J, Yao, Y, Mao, Y, Sheng, B & Mi, N 2014, FRESH: Fair and efficient slot configuration and scheduling for Hadoop clusters. in C Kesselman (ed.), Proceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014., 6973812, IEEE International Conference on Cloud Computing, CLOUD, IEEE Computer Society, pp. 761-768, 7th IEEE International Conference on Cloud Computing, CLOUD 2014, Anchorage, United States, 27/06/14. https://doi.org/10.1109/CLOUD.2014.106

FRESH : Fair and efficient slot configuration and scheduling for Hadoop clusters. / Wang, Jiayin; Yao, Yi; Mao, Ying; Sheng, Bo; Mi, Ningfang.

Proceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014. ed. / Carl Kesselman. IEEE Computer Society, 2014. p. 761-768 6973812 (IEEE International Conference on Cloud Computing, CLOUD).

Research output: Chapter in Book/Report/Conference proceedingConference contributionResearchpeer-review

TY - GEN

T1 - FRESH

T2 - Fair and efficient slot configuration and scheduling for Hadoop clusters

AU - Wang, Jiayin

AU - Yao, Yi

AU - Mao, Ying

AU - Sheng, Bo

AU - Mi, Ningfang

PY - 2014/12/3

Y1 - 2014/12/3

N2 - Hadoop is an emerging framework for parallel big data processing. While becoming popular, Hadoop is too complex for regular users to fully understand all the system parameters and tune them appropriately. Especially when processing a batch of jobs, default Hadoop setting may cause inefficient resource utilization and unnecessarily prolong the execution time. This paper considers an extremely important setting of slot configuration which by default is fixed and static. We proposed an enhanced Hadoop system called FRESH which can derive the best slot setting, dynamically configure slots, and appropriately assign tasks to the available slots. The experimental results show that when serving a batch of MapReduce jobs, FRESH significantly improves the makespan as well as the fairness among jobs.

AB - Hadoop is an emerging framework for parallel big data processing. While becoming popular, Hadoop is too complex for regular users to fully understand all the system parameters and tune them appropriately. Especially when processing a batch of jobs, default Hadoop setting may cause inefficient resource utilization and unnecessarily prolong the execution time. This paper considers an extremely important setting of slot configuration which by default is fixed and static. We proposed an enhanced Hadoop system called FRESH which can derive the best slot setting, dynamically configure slots, and appropriately assign tasks to the available slots. The experimental results show that when serving a batch of MapReduce jobs, FRESH significantly improves the makespan as well as the fairness among jobs.

KW - MapReduce

KW - Resource Management

KW - Scheduling

UR - http://www.scopus.com/inward/record.url?scp=84919828091&partnerID=8YFLogxK

U2 - 10.1109/CLOUD.2014.106

DO - 10.1109/CLOUD.2014.106

M3 - Conference contribution

T3 - IEEE International Conference on Cloud Computing, CLOUD

SP - 761

EP - 768

BT - Proceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014

A2 - Kesselman, Carl

PB - IEEE Computer Society

ER -

Wang J, Yao Y, Mao Y, Sheng B, Mi N. FRESH: Fair and efficient slot configuration and scheduling for Hadoop clusters. In Kesselman C, editor, Proceedings - 2014 IEEE 7th International Conference on Cloud Computing, CLOUD 2014. IEEE Computer Society. 2014. p. 761-768. 6973812. (IEEE International Conference on Cloud Computing, CLOUD). https://doi.org/10.1109/CLOUD.2014.106