OMO: Optimize MapReduce overlap with a good start (reduce) and a good finish (map)

Jiayin Wang, Yi Yao, Ying Mao, Bo Sheng, Ningfang Mi

Research output: Chapter in Book/Report/Conference proceedingConference contribution

8 Scopus citations

Abstract

MapReduce has become a popular data processing framework in the past few years. Scheduling algorithm is crucial to the performance of a MapReduce cluster, especially when the cluster is concurrently executing a batch of MapReduce jobs. However, the scheduling problem in MapReduce is different from the traditional job scheduling problem as the reduce phase usually starts before the map phase is finished to shuffle the intermediate data. This paper develops a new strategy, named OMO, which particularly aims to optimize the overlap between the map and reduce phases. Our solution includes two new techniques, lazy start of reduce tasks and batch finish of map tasks, which catch the characteristics of the overlap in a MapReduce process and achieve a good alignment of the two phases. We have implemented OMO on Hadoop system and evaluated the performance with extensive experiments. The results show that OMO's performance is superior in terms of total completion length (i.e., makespan) of a batch of jobs.

Original languageEnglish
Title of host publication2015 IEEE 34th International Performance Computing and Communications Conference, IPCCC 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781467385909
DOIs
StatePublished - 17 Feb 2016
Event34th IEEE International Performance Computing and Communications Conference, IPCCC 2015 - Nanjing, China
Duration: 14 Dec 201516 Dec 2015

Publication series

Name2015 IEEE 34th International Performance Computing and Communications Conference, IPCCC 2015

Other

Other34th IEEE International Performance Computing and Communications Conference, IPCCC 2015
CountryChina
CityNanjing
Period14/12/1516/12/15

Fingerprint Dive into the research topics of 'OMO: Optimize MapReduce overlap with a good start (reduce) and a good finish (map)'. Together they form a unique fingerprint.

  • Cite this

    Wang, J., Yao, Y., Mao, Y., Sheng, B., & Mi, N. (2016). OMO: Optimize MapReduce overlap with a good start (reduce) and a good finish (map). In 2015 IEEE 34th International Performance Computing and Communications Conference, IPCCC 2015 [7410279] (2015 IEEE 34th International Performance Computing and Communications Conference, IPCCC 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/PCCC.2015.7410279