Revealed or concealed? transparency of procedures, decisions, and judgment calls in meta-analyses

Zeynep G. Aytug, Hannah R. Rothstein, Wencang Zhou, Mary C. Kern

Research output: Contribution to journalReview articleResearchpeer-review

47 Citations (Scopus)

Abstract

The authors examined the degree to which meta-analyses in the organizational sciences transparently report procedures, decisions, and judgment calls by systematically reviewing all (198) meta-analyses published between 1995 and 2008 in 11 top journals that publish meta-analyses in industrial and organizational psychology and organizational behavior. The authors extracted information on 54 features of each meta-analysis. On average, the meta-analyses in the sample provided 52.8% of the information needed to replicate the meta-analysis or to assess its validity and 67.6% of the information considered to be most important according to expert meta-analysts. More recently published meta-analyses exhibited somewhat more transparent reporting practices than older ones did. Overall transparency of reporting (but not reporting of the most important items) was associated with higher ranked journals; transparency was not significantly related to number of citations. The authors discuss the implications of inadequate reporting of meta-analyses for development of cumulative knowledge and effective practice and make suggestions for improving the current state of affairs.

Original languageEnglish
Pages (from-to)103-133
Number of pages31
JournalOrganizational Research Methods
Volume15
Issue number1
DOIs
StatePublished - 1 Jan 2012

Fingerprint

Transparency
Meta-analysis

Keywords

  • meta-analysis
  • missing data
  • quantitative research

Cite this

Aytug, Zeynep G. ; Rothstein, Hannah R. ; Zhou, Wencang ; Kern, Mary C. / Revealed or concealed? transparency of procedures, decisions, and judgment calls in meta-analyses. In: Organizational Research Methods. 2012 ; Vol. 15, No. 1. pp. 103-133.
@article{a56fceba13e444c99f2a32e7f79ff477,
title = "Revealed or concealed? transparency of procedures, decisions, and judgment calls in meta-analyses",
abstract = "The authors examined the degree to which meta-analyses in the organizational sciences transparently report procedures, decisions, and judgment calls by systematically reviewing all (198) meta-analyses published between 1995 and 2008 in 11 top journals that publish meta-analyses in industrial and organizational psychology and organizational behavior. The authors extracted information on 54 features of each meta-analysis. On average, the meta-analyses in the sample provided 52.8{\%} of the information needed to replicate the meta-analysis or to assess its validity and 67.6{\%} of the information considered to be most important according to expert meta-analysts. More recently published meta-analyses exhibited somewhat more transparent reporting practices than older ones did. Overall transparency of reporting (but not reporting of the most important items) was associated with higher ranked journals; transparency was not significantly related to number of citations. The authors discuss the implications of inadequate reporting of meta-analyses for development of cumulative knowledge and effective practice and make suggestions for improving the current state of affairs.",
keywords = "meta-analysis, missing data, quantitative research",
author = "Aytug, {Zeynep G.} and Rothstein, {Hannah R.} and Wencang Zhou and Kern, {Mary C.}",
year = "2012",
month = "1",
day = "1",
doi = "10.1177/1094428111403495",
language = "English",
volume = "15",
pages = "103--133",
journal = "Organizational Research Methods",
issn = "1094-4281",
publisher = "SAGE Publications Inc.",
number = "1",

}

Revealed or concealed? transparency of procedures, decisions, and judgment calls in meta-analyses. / Aytug, Zeynep G.; Rothstein, Hannah R.; Zhou, Wencang; Kern, Mary C.

In: Organizational Research Methods, Vol. 15, No. 1, 01.01.2012, p. 103-133.

Research output: Contribution to journalReview articleResearchpeer-review

TY - JOUR

T1 - Revealed or concealed? transparency of procedures, decisions, and judgment calls in meta-analyses

AU - Aytug, Zeynep G.

AU - Rothstein, Hannah R.

AU - Zhou, Wencang

AU - Kern, Mary C.

PY - 2012/1/1

Y1 - 2012/1/1

N2 - The authors examined the degree to which meta-analyses in the organizational sciences transparently report procedures, decisions, and judgment calls by systematically reviewing all (198) meta-analyses published between 1995 and 2008 in 11 top journals that publish meta-analyses in industrial and organizational psychology and organizational behavior. The authors extracted information on 54 features of each meta-analysis. On average, the meta-analyses in the sample provided 52.8% of the information needed to replicate the meta-analysis or to assess its validity and 67.6% of the information considered to be most important according to expert meta-analysts. More recently published meta-analyses exhibited somewhat more transparent reporting practices than older ones did. Overall transparency of reporting (but not reporting of the most important items) was associated with higher ranked journals; transparency was not significantly related to number of citations. The authors discuss the implications of inadequate reporting of meta-analyses for development of cumulative knowledge and effective practice and make suggestions for improving the current state of affairs.

AB - The authors examined the degree to which meta-analyses in the organizational sciences transparently report procedures, decisions, and judgment calls by systematically reviewing all (198) meta-analyses published between 1995 and 2008 in 11 top journals that publish meta-analyses in industrial and organizational psychology and organizational behavior. The authors extracted information on 54 features of each meta-analysis. On average, the meta-analyses in the sample provided 52.8% of the information needed to replicate the meta-analysis or to assess its validity and 67.6% of the information considered to be most important according to expert meta-analysts. More recently published meta-analyses exhibited somewhat more transparent reporting practices than older ones did. Overall transparency of reporting (but not reporting of the most important items) was associated with higher ranked journals; transparency was not significantly related to number of citations. The authors discuss the implications of inadequate reporting of meta-analyses for development of cumulative knowledge and effective practice and make suggestions for improving the current state of affairs.

KW - meta-analysis

KW - missing data

KW - quantitative research

UR - http://www.scopus.com/inward/record.url?scp=84055214048&partnerID=8YFLogxK

U2 - 10.1177/1094428111403495

DO - 10.1177/1094428111403495

M3 - Review article

VL - 15

SP - 103

EP - 133

JO - Organizational Research Methods

JF - Organizational Research Methods

SN - 1094-4281

IS - 1

ER -