Cost-efficient data acquisition on online data marketplaces for correlation analysis

Yanying Li, Haipei Sun, Boxiang Dong, Hui Wang

Research output: Contribution to journalConference articlepeer-review

Abstract

Incentivized by the enormous economic profits, the data marketplace platform has been proliferated recently. In this paper, we consider the data marketplace setting where a data shopper would like to buy data instances from the data marketplace for correlation analysis of certain attributes. We assume that the data in the marketplace is dirty and not free. The goal is to find the data instances from a large number of datasets in the marketplace whose join result not only is of high-quality and rich join informativeness, but also delivers the best correlation between the requested attributes. To achieve this goal, we design DANCE, a middleware that provides the desired data acquisition service. DANCE consists of two phases: (1) In the off-line phase, it constructs a two-layer join graph from samples. The join graph includes the information of the datasets in the marketplace at both schema and instance levels; (2) In the on- line phase, it searches for the data instances that satisfy the constraints of data quality, budget, and join informativeness, while maximizing the correlation of source and target attribute sets. We prove that the complexity of the search problem is NP-hard, and design a heuristic algorithm based on Markov chain Monte Carlo (MCMC). Experiment results on two benchmark and one real datasets demonstrate the efficiency and effectiveness of our heuristic data acquisition algorithm.

Original languageEnglish
Pages (from-to)362-375
Number of pages14
JournalProceedings of the VLDB Endowment
Volume12
Issue number4
DOIs
StatePublished - 1 Jan 2018
Event45th International Conference on Very Large Data Bases, VLDB 2019 - Los Angeles, United States
Duration: 26 Aug 201730 Aug 2017

Fingerprint Dive into the research topics of 'Cost-efficient data acquisition on online data marketplaces for correlation analysis'. Together they form a unique fingerprint.

Cite this