Many Labs 3

Evaluating participant pool quality across the academic semester via replication

Charles R. Ebersole, Olivia E. Atherton, Aimee L. Belanger, Hayley M. Skulborstad, Jill M. Allen, Jonathan B. Banks, Erica Baranski, Michael J. Bernstein, Diane B.V. Bonfiglio, Leanne Boucher, Elizabeth R. Brown, Nancy I. Budiman, Athena H. Cairo, Colin A. Capaldi, Christopher R. Chartier, Joanne M. Chung, David C. Cicero, Jennifer A. Coleman, John G. Conway, William E. Davis & 44 others Thierry Devos, Melody M. Fletcher, Komi German, Jon E. Grahe, Anthony D. Hermann, Joshua A. Hicks, Nathan Honeycutt, Brandon Humphrey, Matthew Janus, David J. Johnson, Jennifer A. Joy-Gaba, Hannah Juzeler, Ashley Keres, Diana Kinney, Jacqeline Kirshenbaum, Richard A. Klein, Richard E. Lucas, Christopher J.N. Lustgraaf, Daniel Martin, Madhavi Menon, Mitchell Metzger, Jaclyn M. Moloney, Patrick J. Morse, Radmila Prislin, Timothy Razza, Daniel E. Re, Nicholas O. Rule, Donald F. Sacco, Kyle Sauerberger, Emily Shrider, Megan Shultz, Courtney Siemsen, Karin Sobocko, R. Weylin Sternglanz, Amy Summerville, Konstantin O. Tskhay, Zack van Allen, Leigh Ann Vaughn, Ryan J. Walker, Ashley Weinberg, John Paul Wilson, James H. Wirth, Jessica Wortman, Brian A. Nosek

Research output: Contribution to journalArticleResearchpeer-review

51 Citations (Scopus)

Abstract

The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences—conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects.

Original languageEnglish
Pages (from-to)68-82
Number of pages15
JournalJournal of Experimental Social Psychology
Volume67
DOIs
StatePublished - 1 Nov 2016

Fingerprint

semester
data quality
Behavioral Research
Individuality
evidence
behavioral research
mood
Data Accuracy
university
resources
time

Keywords

  • Cognitive psychology
  • Individual differences
  • Participant pool
  • Replication
  • Sampling effects
  • Situational effects
  • Social psychology

Cite this

Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., ... Nosek, B. A. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68-82. https://doi.org/10.1016/j.jesp.2015.10.012
Ebersole, Charles R. ; Atherton, Olivia E. ; Belanger, Aimee L. ; Skulborstad, Hayley M. ; Allen, Jill M. ; Banks, Jonathan B. ; Baranski, Erica ; Bernstein, Michael J. ; Bonfiglio, Diane B.V. ; Boucher, Leanne ; Brown, Elizabeth R. ; Budiman, Nancy I. ; Cairo, Athena H. ; Capaldi, Colin A. ; Chartier, Christopher R. ; Chung, Joanne M. ; Cicero, David C. ; Coleman, Jennifer A. ; Conway, John G. ; Davis, William E. ; Devos, Thierry ; Fletcher, Melody M. ; German, Komi ; Grahe, Jon E. ; Hermann, Anthony D. ; Hicks, Joshua A. ; Honeycutt, Nathan ; Humphrey, Brandon ; Janus, Matthew ; Johnson, David J. ; Joy-Gaba, Jennifer A. ; Juzeler, Hannah ; Keres, Ashley ; Kinney, Diana ; Kirshenbaum, Jacqeline ; Klein, Richard A. ; Lucas, Richard E. ; Lustgraaf, Christopher J.N. ; Martin, Daniel ; Menon, Madhavi ; Metzger, Mitchell ; Moloney, Jaclyn M. ; Morse, Patrick J. ; Prislin, Radmila ; Razza, Timothy ; Re, Daniel E. ; Rule, Nicholas O. ; Sacco, Donald F. ; Sauerberger, Kyle ; Shrider, Emily ; Shultz, Megan ; Siemsen, Courtney ; Sobocko, Karin ; Weylin Sternglanz, R. ; Summerville, Amy ; Tskhay, Konstantin O. ; van Allen, Zack ; Vaughn, Leigh Ann ; Walker, Ryan J. ; Weinberg, Ashley ; Wilson, John Paul ; Wirth, James H. ; Wortman, Jessica ; Nosek, Brian A. / Many Labs 3 : Evaluating participant pool quality across the academic semester via replication. In: Journal of Experimental Social Psychology. 2016 ; Vol. 67. pp. 68-82.
@article{3bfe5cbfeeaf4b8f9912ee0b4685a34e,
title = "Many Labs 3: Evaluating participant pool quality across the academic semester via replication",
abstract = "The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences—conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects.",
keywords = "Cognitive psychology, Individual differences, Participant pool, Replication, Sampling effects, Situational effects, Social psychology",
author = "Ebersole, {Charles R.} and Atherton, {Olivia E.} and Belanger, {Aimee L.} and Skulborstad, {Hayley M.} and Allen, {Jill M.} and Banks, {Jonathan B.} and Erica Baranski and Bernstein, {Michael J.} and Bonfiglio, {Diane B.V.} and Leanne Boucher and Brown, {Elizabeth R.} and Budiman, {Nancy I.} and Cairo, {Athena H.} and Capaldi, {Colin A.} and Chartier, {Christopher R.} and Chung, {Joanne M.} and Cicero, {David C.} and Coleman, {Jennifer A.} and Conway, {John G.} and Davis, {William E.} and Thierry Devos and Fletcher, {Melody M.} and Komi German and Grahe, {Jon E.} and Hermann, {Anthony D.} and Hicks, {Joshua A.} and Nathan Honeycutt and Brandon Humphrey and Matthew Janus and Johnson, {David J.} and Joy-Gaba, {Jennifer A.} and Hannah Juzeler and Ashley Keres and Diana Kinney and Jacqeline Kirshenbaum and Klein, {Richard A.} and Lucas, {Richard E.} and Lustgraaf, {Christopher J.N.} and Daniel Martin and Madhavi Menon and Mitchell Metzger and Moloney, {Jaclyn M.} and Morse, {Patrick J.} and Radmila Prislin and Timothy Razza and Re, {Daniel E.} and Rule, {Nicholas O.} and Sacco, {Donald F.} and Kyle Sauerberger and Emily Shrider and Megan Shultz and Courtney Siemsen and Karin Sobocko and {Weylin Sternglanz}, R. and Amy Summerville and Tskhay, {Konstantin O.} and {van Allen}, Zack and Vaughn, {Leigh Ann} and Walker, {Ryan J.} and Ashley Weinberg and Wilson, {John Paul} and Wirth, {James H.} and Jessica Wortman and Nosek, {Brian A.}",
year = "2016",
month = "11",
day = "1",
doi = "10.1016/j.jesp.2015.10.012",
language = "English",
volume = "67",
pages = "68--82",
journal = "Journal of Experimental Social Psychology",
issn = "0022-1031",
publisher = "Academic Press Inc.",

}

Ebersole, CR, Atherton, OE, Belanger, AL, Skulborstad, HM, Allen, JM, Banks, JB, Baranski, E, Bernstein, MJ, Bonfiglio, DBV, Boucher, L, Brown, ER, Budiman, NI, Cairo, AH, Capaldi, CA, Chartier, CR, Chung, JM, Cicero, DC, Coleman, JA, Conway, JG, Davis, WE, Devos, T, Fletcher, MM, German, K, Grahe, JE, Hermann, AD, Hicks, JA, Honeycutt, N, Humphrey, B, Janus, M, Johnson, DJ, Joy-Gaba, JA, Juzeler, H, Keres, A, Kinney, D, Kirshenbaum, J, Klein, RA, Lucas, RE, Lustgraaf, CJN, Martin, D, Menon, M, Metzger, M, Moloney, JM, Morse, PJ, Prislin, R, Razza, T, Re, DE, Rule, NO, Sacco, DF, Sauerberger, K, Shrider, E, Shultz, M, Siemsen, C, Sobocko, K, Weylin Sternglanz, R, Summerville, A, Tskhay, KO, van Allen, Z, Vaughn, LA, Walker, RJ, Weinberg, A, Wilson, JP, Wirth, JH, Wortman, J & Nosek, BA 2016, 'Many Labs 3: Evaluating participant pool quality across the academic semester via replication', Journal of Experimental Social Psychology, vol. 67, pp. 68-82. https://doi.org/10.1016/j.jesp.2015.10.012

Many Labs 3 : Evaluating participant pool quality across the academic semester via replication. / Ebersole, Charles R.; Atherton, Olivia E.; Belanger, Aimee L.; Skulborstad, Hayley M.; Allen, Jill M.; Banks, Jonathan B.; Baranski, Erica; Bernstein, Michael J.; Bonfiglio, Diane B.V.; Boucher, Leanne; Brown, Elizabeth R.; Budiman, Nancy I.; Cairo, Athena H.; Capaldi, Colin A.; Chartier, Christopher R.; Chung, Joanne M.; Cicero, David C.; Coleman, Jennifer A.; Conway, John G.; Davis, William E.; Devos, Thierry; Fletcher, Melody M.; German, Komi; Grahe, Jon E.; Hermann, Anthony D.; Hicks, Joshua A.; Honeycutt, Nathan; Humphrey, Brandon; Janus, Matthew; Johnson, David J.; Joy-Gaba, Jennifer A.; Juzeler, Hannah; Keres, Ashley; Kinney, Diana; Kirshenbaum, Jacqeline; Klein, Richard A.; Lucas, Richard E.; Lustgraaf, Christopher J.N.; Martin, Daniel; Menon, Madhavi; Metzger, Mitchell; Moloney, Jaclyn M.; Morse, Patrick J.; Prislin, Radmila; Razza, Timothy; Re, Daniel E.; Rule, Nicholas O.; Sacco, Donald F.; Sauerberger, Kyle; Shrider, Emily; Shultz, Megan; Siemsen, Courtney; Sobocko, Karin; Weylin Sternglanz, R.; Summerville, Amy; Tskhay, Konstantin O.; van Allen, Zack; Vaughn, Leigh Ann; Walker, Ryan J.; Weinberg, Ashley; Wilson, John Paul; Wirth, James H.; Wortman, Jessica; Nosek, Brian A.

In: Journal of Experimental Social Psychology, Vol. 67, 01.11.2016, p. 68-82.

Research output: Contribution to journalArticleResearchpeer-review

TY - JOUR

T1 - Many Labs 3

T2 - Evaluating participant pool quality across the academic semester via replication

AU - Ebersole, Charles R.

AU - Atherton, Olivia E.

AU - Belanger, Aimee L.

AU - Skulborstad, Hayley M.

AU - Allen, Jill M.

AU - Banks, Jonathan B.

AU - Baranski, Erica

AU - Bernstein, Michael J.

AU - Bonfiglio, Diane B.V.

AU - Boucher, Leanne

AU - Brown, Elizabeth R.

AU - Budiman, Nancy I.

AU - Cairo, Athena H.

AU - Capaldi, Colin A.

AU - Chartier, Christopher R.

AU - Chung, Joanne M.

AU - Cicero, David C.

AU - Coleman, Jennifer A.

AU - Conway, John G.

AU - Davis, William E.

AU - Devos, Thierry

AU - Fletcher, Melody M.

AU - German, Komi

AU - Grahe, Jon E.

AU - Hermann, Anthony D.

AU - Hicks, Joshua A.

AU - Honeycutt, Nathan

AU - Humphrey, Brandon

AU - Janus, Matthew

AU - Johnson, David J.

AU - Joy-Gaba, Jennifer A.

AU - Juzeler, Hannah

AU - Keres, Ashley

AU - Kinney, Diana

AU - Kirshenbaum, Jacqeline

AU - Klein, Richard A.

AU - Lucas, Richard E.

AU - Lustgraaf, Christopher J.N.

AU - Martin, Daniel

AU - Menon, Madhavi

AU - Metzger, Mitchell

AU - Moloney, Jaclyn M.

AU - Morse, Patrick J.

AU - Prislin, Radmila

AU - Razza, Timothy

AU - Re, Daniel E.

AU - Rule, Nicholas O.

AU - Sacco, Donald F.

AU - Sauerberger, Kyle

AU - Shrider, Emily

AU - Shultz, Megan

AU - Siemsen, Courtney

AU - Sobocko, Karin

AU - Weylin Sternglanz, R.

AU - Summerville, Amy

AU - Tskhay, Konstantin O.

AU - van Allen, Zack

AU - Vaughn, Leigh Ann

AU - Walker, Ryan J.

AU - Weinberg, Ashley

AU - Wilson, John Paul

AU - Wirth, James H.

AU - Wortman, Jessica

AU - Nosek, Brian A.

PY - 2016/11/1

Y1 - 2016/11/1

N2 - The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences—conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects.

AB - The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences—conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects.

KW - Cognitive psychology

KW - Individual differences

KW - Participant pool

KW - Replication

KW - Sampling effects

KW - Situational effects

KW - Social psychology

UR - http://www.scopus.com/inward/record.url?scp=84989936477&partnerID=8YFLogxK

U2 - 10.1016/j.jesp.2015.10.012

DO - 10.1016/j.jesp.2015.10.012

M3 - Article

VL - 67

SP - 68

EP - 82

JO - Journal of Experimental Social Psychology

JF - Journal of Experimental Social Psychology

SN - 0022-1031

ER -