Many Labs 3: Evaluating participant pool quality across the academic semester via replication

Charles R. Ebersole; Olivia E. Atherton; Aimee L. Belanger; Hayley M. Skulborstad; Jill M. Allen; Jonathan B. Banks; Erica Baranski; Michael J. Bernstein; Diane B.V. Bonfiglio; Leanne Boucher; Elizabeth R. Brown; Nancy I. Budiman; Athena H. Cairo; Colin A. Capaldi; Christopher R. Chartier; Joanne M. Chung; David C. Cicero; Jennifer A. Coleman; John G. Conway; William E. Davis; Thierry Devos; Melody M. Fletcher; Komi German; Jon E. Grahe; Anthony D. Hermann; Joshua A. Hicks; Nathan Honeycutt; Brandon Humphrey; Matthew Janus; David J. Johnson; Jennifer A. Joy-Gaba; Hannah Juzeler; Ashley Keres; Diana Kinney; Jacqeline Kirshenbaum; Richard A. Klein; Richard E. Lucas; Christopher J.N. Lustgraaf; Daniel Martin; Madhavi Menon; Mitchell Metzger; Jaclyn M. Moloney; Patrick J. Morse; Radmila Prislin; Timothy Razza; Daniel E. Re; Nicholas O. Rule; Donald F. Sacco; Kyle Sauerberger; Emily Shrider; Megan Shultz; Courtney Siemsen; Karin Sobocko; R. Weylin Sternglanz; Amy Summerville; Konstantin O. Tskhay; Zack van Allen; Leigh Ann Vaughn; Ryan J. Walker; Ashley Weinberg; John Paul Wilson; James H. Wirth; Jessica Wortman; Brian A. Nosek

doi:10.1016/j.jesp.2015.10.012

Many Labs 3: Evaluating participant pool quality across the academic semester via replication

Charles R. Ebersole, Olivia E. Atherton, Aimee L. Belanger, Hayley M. Skulborstad, Jill M. Allen, Jonathan B. Banks, Erica Baranski, Michael J. Bernstein, Diane B.V. Bonfiglio, Leanne Boucher, Elizabeth R. Brown, Nancy I. Budiman, Athena H. Cairo, Colin A. Capaldi, Christopher R. Chartier, Joanne M. Chung, David C. Cicero, Jennifer A. Coleman, John G. Conway, William E. DavisThierry Devos, Melody M. Fletcher, Komi German, Jon E. Grahe, Anthony D. Hermann, Joshua A. Hicks, Nathan Honeycutt, Brandon Humphrey, Matthew Janus, David J. Johnson, Jennifer A. Joy-Gaba, Hannah Juzeler, Ashley Keres, Diana Kinney, Jacqeline Kirshenbaum, Richard A. Klein, Richard E. Lucas, Christopher J.N. Lustgraaf, Daniel Martin, Madhavi Menon, Mitchell Metzger, Jaclyn M. Moloney, Patrick J. Morse, Radmila Prislin, Timothy Razza, Daniel E. Re, Nicholas O. Rule, Donald F. Sacco, Kyle Sauerberger, Emily Shrider, Megan Shultz, Courtney Siemsen, Karin Sobocko, R. Weylin Sternglanz, Amy Summerville, Konstantin O. Tskhay, Zack van Allen, Leigh Ann Vaughn, Ryan J. Walker, Ashley Weinberg, John Paul Wilson, James H. Wirth, Jessica Wortman, Brian A. Nosek

Psychology

Research output: Contribution to journal › Article › peer-review

250 Scopus citations

Abstract

The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences—conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects.

Original language	English
Pages (from-to)	68-82
Number of pages	15
Journal	Journal of Experimental Social Psychology
Volume	67
DOIs	https://doi.org/10.1016/j.jesp.2015.10.012
State	Published - 1 Nov 2016

Keywords

Cognitive psychology
Individual differences
Participant pool
Replication
Sampling effects
Situational effects
Social psychology

Access to Document

10.1016/j.jesp.2015.10.012

Cite this

Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., Baranski, E., Bernstein, M. J., Bonfiglio, D. B. V., Boucher, L., Brown, E. R., Budiman, N. I., Cairo, A. H., Capaldi, C. A., Chartier, C. R., Chung, J. M., Cicero, D. C., Coleman, J. A., Conway, J. G., ... Nosek, B. A. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68-82. https://doi.org/10.1016/j.jesp.2015.10.012

@article{3bfe5cbfeeaf4b8f9912ee0b4685a34e,

title = "Many Labs 3: Evaluating participant pool quality across the academic semester via replication",

abstract = "The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences—conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects.",

keywords = "Cognitive psychology, Individual differences, Participant pool, Replication, Sampling effects, Situational effects, Social psychology",

author = "Ebersole, {Charles R.} and Atherton, {Olivia E.} and Belanger, {Aimee L.} and Skulborstad, {Hayley M.} and Allen, {Jill M.} and Banks, {Jonathan B.} and Erica Baranski and Bernstein, {Michael J.} and Bonfiglio, {Diane B.V.} and Leanne Boucher and Brown, {Elizabeth R.} and Budiman, {Nancy I.} and Cairo, {Athena H.} and Capaldi, {Colin A.} and Chartier, {Christopher R.} and Chung, {Joanne M.} and Cicero, {David C.} and Coleman, {Jennifer A.} and Conway, {John G.} and Davis, {William E.} and Thierry Devos and Fletcher, {Melody M.} and Komi German and Grahe, {Jon E.} and Hermann, {Anthony D.} and Hicks, {Joshua A.} and Nathan Honeycutt and Brandon Humphrey and Matthew Janus and Johnson, {David J.} and Joy-Gaba, {Jennifer A.} and Hannah Juzeler and Ashley Keres and Diana Kinney and Jacqeline Kirshenbaum and Klein, {Richard A.} and Lucas, {Richard E.} and Lustgraaf, {Christopher J.N.} and Daniel Martin and Madhavi Menon and Mitchell Metzger and Moloney, {Jaclyn M.} and Morse, {Patrick J.} and Radmila Prislin and Timothy Razza and Re, {Daniel E.} and Rule, {Nicholas O.} and Sacco, {Donald F.} and Kyle Sauerberger and Emily Shrider and Megan Shultz and Courtney Siemsen and Karin Sobocko and {Weylin Sternglanz}, R. and Amy Summerville and Tskhay, {Konstantin O.} and {van Allen}, Zack and Vaughn, {Leigh Ann} and Walker, {Ryan J.} and Ashley Weinberg and Wilson, {John Paul} and Wirth, {James H.} and Jessica Wortman and Nosek, {Brian A.}",

note = "Publisher Copyright: {\textcopyright} 2015 Elsevier Inc.",

year = "2016",

month = nov,

day = "1",

doi = "10.1016/j.jesp.2015.10.012",

language = "English",

volume = "67",

pages = "68--82",

journal = "Journal of Experimental Social Psychology",

issn = "0022-1031",

publisher = "Academic Press Inc.",

}

Ebersole, CR, Atherton, OE, Belanger, AL, Skulborstad, HM, Allen, JM, Banks, JB, Baranski, E, Bernstein, MJ, Bonfiglio, DBV, Boucher, L, Brown, ER, Budiman, NI, Cairo, AH, Capaldi, CA, Chartier, CR, Chung, JM, Cicero, DC, Coleman, JA, Conway, JG, Davis, WE, Devos, T, Fletcher, MM, German, K, Grahe, JE, Hermann, AD, Hicks, JA, Honeycutt, N, Humphrey, B, Janus, M, Johnson, DJ, Joy-Gaba, JA, Juzeler, H, Keres, A, Kinney, D, Kirshenbaum, J, Klein, RA, Lucas, RE, Lustgraaf, CJN, Martin, D, Menon, M, Metzger, M, Moloney, JM, Morse, PJ, Prislin, R, Razza, T, Re, DE, Rule, NO, Sacco, DF, Sauerberger, K, Shrider, E, Shultz, M, Siemsen, C, Sobocko, K, Weylin Sternglanz, R, Summerville, A, Tskhay, KO, van Allen, Z, Vaughn, LA, Walker, RJ, Weinberg, A, Wilson, JP, Wirth, JH, Wortman, J & Nosek, BA 2016, 'Many Labs 3: Evaluating participant pool quality across the academic semester via replication', Journal of Experimental Social Psychology, vol. 67, pp. 68-82. https://doi.org/10.1016/j.jesp.2015.10.012

TY - JOUR

T1 - Many Labs 3

T2 - Evaluating participant pool quality across the academic semester via replication

AU - Ebersole, Charles R.

AU - Atherton, Olivia E.

AU - Belanger, Aimee L.

AU - Skulborstad, Hayley M.

AU - Allen, Jill M.

AU - Banks, Jonathan B.

AU - Baranski, Erica

AU - Bernstein, Michael J.

AU - Bonfiglio, Diane B.V.

AU - Boucher, Leanne

AU - Brown, Elizabeth R.

AU - Budiman, Nancy I.

AU - Cairo, Athena H.

AU - Capaldi, Colin A.

AU - Chartier, Christopher R.

AU - Chung, Joanne M.

AU - Cicero, David C.

AU - Coleman, Jennifer A.

AU - Conway, John G.

AU - Davis, William E.

AU - Devos, Thierry

AU - Fletcher, Melody M.

AU - German, Komi

AU - Grahe, Jon E.

AU - Hermann, Anthony D.

AU - Hicks, Joshua A.

AU - Honeycutt, Nathan

AU - Humphrey, Brandon

AU - Janus, Matthew

AU - Johnson, David J.

AU - Joy-Gaba, Jennifer A.

AU - Juzeler, Hannah

AU - Keres, Ashley

AU - Kinney, Diana

AU - Kirshenbaum, Jacqeline

AU - Klein, Richard A.

AU - Lucas, Richard E.

AU - Lustgraaf, Christopher J.N.

AU - Martin, Daniel

AU - Menon, Madhavi

AU - Metzger, Mitchell

AU - Moloney, Jaclyn M.

AU - Morse, Patrick J.

AU - Prislin, Radmila

AU - Razza, Timothy

AU - Re, Daniel E.

AU - Rule, Nicholas O.

AU - Sacco, Donald F.

AU - Sauerberger, Kyle

AU - Shrider, Emily

AU - Shultz, Megan

AU - Siemsen, Courtney

AU - Sobocko, Karin

AU - Weylin Sternglanz, R.

AU - Summerville, Amy

AU - Tskhay, Konstantin O.

AU - van Allen, Zack

AU - Vaughn, Leigh Ann

AU - Walker, Ryan J.

AU - Weinberg, Ashley

AU - Wilson, John Paul

AU - Wirth, James H.

AU - Wortman, Jessica

AU - Nosek, Brian A.

PY - 2016/11/1

Y1 - 2016/11/1

N2 - The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences—conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects.

AB - The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences—conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects.

KW - Cognitive psychology

KW - Individual differences

KW - Participant pool

KW - Replication

KW - Sampling effects

KW - Situational effects

KW - Social psychology

UR - http://www.scopus.com/inward/record.url?scp=84989936477&partnerID=8YFLogxK

U2 - 10.1016/j.jesp.2015.10.012

DO - 10.1016/j.jesp.2015.10.012

M3 - Article

AN - SCOPUS:84989936477

SN - 0022-1031

VL - 67

SP - 68

EP - 82

JO - Journal of Experimental Social Psychology

JF - Journal of Experimental Social Psychology

ER -

Many Labs 3: Evaluating participant pool quality across the academic semester via replication

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this