Korean Journal of Psychology : General
[ Regular issue ]
The Korean Journal of Psychology: General - Vol. 38, No. 4, pp.669-698
ISSN: 1229-067X (Print)
Print publication date 25 Dec 2019
Received 25 Oct 2019 Accepted 21 Dec 2019
DOI: https://doi.org/10.22257/kjp.2019.12.38.4.669

온라인 설문에서 거울 이미지를 활용한 불성실 응답 방지 방법에 대한 연구

김우영 ; 이태헌 ; 장재윤
서강대학교
중앙대학교
서강대학교
A study on the effectiveness of the use of mirror-image as a preventive measure of careless responding in online survey research
Woo Young Kim ; Tae Hun Lee ; Jae Yoon Chang
Sogang University
Chungang University
Sogang University

Correspondence to: 이태헌, 중앙대학교, 서울시 동작구 흑석로 84 중앙대학교 305관 918호, lee0267@cau.ac.kr
Correspondence to: 장재윤, 서강대학교, 서울시 마포구 백범로 35 서강대학교 다산관 337호, jych@sogang.ac.kr

초록

온라인 설문은 기존의 지필형 설문에 비해 실시의 편의성과 비용적인 측면에서 이점이 있기 때문에 일반적인 연구 장면뿐만 아니라, 고객 및 종업원 인식조사 등 실무 장면에서도 광범위하게 사용된다. 온라인 설문에서, 낮은 비율의 불성실 응답으로도 데이터의 질과 분석 결과에 미치는 부정적인 영향 및 불성실 응답의 탐지 방법에 대한 연구가 활발히 이루어지는 것에 비해, 불성실 응답을 사전에 방지하기 위한 방법에 대한 실증적인 연구는 매우 부족한 편이다. 본 연구에서는 응답자들의 불성실 응답을 방지하기 위해 사회심리학의 객관적 자기 인식이론(objective self-awareness)을 적용하여 거울이미지를 배경으로 삽입하는 온라인 설문 화면 구성하였으며, 이는 매우 간단하지만 지금까지 시도되지 않았던 방법이다. 거울이미지의 불성실 응답 방지효과를 파악하기 위해 경고지시문을 활용한 조건 및 일반적인 조사 지시문을 활용한 통제 조건과 비교하였다. 비교를 위해서 기존 불성실 응답 연구들에서 도출된 지표를 활용하였으며, 연구 결과, 경고 지시문 활용이나 일반적인 지시문을 활용한 통제조건에 비해 거울이미지를 삽입한 조건에서 불성실 응답이 감소되는 경향이 있는 것으로 나타났다. 끝으로 본 연구의 의의, 제한점 및 추후 연구방향에 관하여 논의하였다.

Abstract

Online surveys are widely used not only in general survey research but also in organizational research, such as customer and employee surveys, due to the ease of implementation and the considerations of cost-effectiveness compared to conventional paper-pencil surveys. There has been much research devoted to showing the negative impact of careless responding on subsequent data analysis and to evaluating statistical methods for detecting the careless respondents embedded in the data collected via online surveys. However, it would be fair to say that there has been a paucity of empirical research evaluating the effectiveness of a preventive measure against careless responding that would operate during the data collection phase of online survey. In this study, inspired by the objective self-awareness theory in social psychology, we proposed a simple but novel method of inserting a mirror-image in the background of each online survey page as a preventive measure of careless responding to the survey items. We collected data under three different conditions of control(standard instructions only), warning(standard instructions plus warnings against careless responding), and the mirror-image condition. Statistical properties of several existing indicators developed for detecting various types of careless respondents were compared among the three conditions. Our results suggested that the simple mirror-image method tended to reduce the careless responding more effectively than the warning and control condition. We concluded by discussing implications and limitations of our study.

Keywords:

careless responding, mirror, objective self-awareness, online survey

키워드:

객관적 자기인식 이론, 거울, 불성실 응답, 온라인 설문

References

  • 김병수, 김정현 (2013). 스트레스 척도 핸드북. 서울: 학지사.
  • 배성우, 신원식 (2005). CES-D 척도(The Center for Epidemiologic Studies-Depression Scale)의 요인구조 분석. 보건과 사회과학, 18, 165-190..
  • 이양현, 임효덕, 이종영 (1996). 한국판 20 항목 Toronto 감정표현불능증 척도(TAS-20K)의 개발과 타당도. 신경정신의학, 35(4), 888-899..
  • 최보라 (2007). 사회적 바람직성 척도 타당화. 이화여대석사학위논문..
  • Aronson, E., Fried, C., & Stone, J. (1991). Overcoming denial and increasing the intention to use condoms through the induction of hypocrisy. American Journal of Public Health, 81(12), 1636-1638.. [https://doi.org/10.2105/AJPH.81.12.1636]
  • Baer, R. A., Ballenger, J., Berry, D. T., & Wetter, M. W. (1997). Detection of random responding on the MMPI-A. Journal of Personality Assessment, 68, 139-151.. [https://doi.org/10.1207/s15327752jpa6801_11]
  • Bargh, J. A., Gollwitzer, P. M., Lee-Chai, A., Barndollar, K., & Trötschel, R. (2001). The automated will: Nonconscious activation and pursuit of behavioral goals. Journal of Personality and Social Psychology, 81, 1014-1027.. [https://doi.org/10.1037/0022-3514.81.6.1014]
  • Beaman, A. L., Klentz, B., Diener, E., & Svanum, S. (1979). Self-awareness and transgression in children: Two field studies. Journal of Personality and Social Psychology, 37, 1835-1846.. [https://doi.org/10.1037/0022-3514.37.10.1835]
  • Berkowitz, L., & LePage, A. (1967). Weapons as aggression-eliciting stimuli. Journal of Personality and Social Psychology, 7, 202-207.. [https://doi.org/10.1037/h0025008]
  • Berry, D. T., Wetter, M. W., Baer, R. A., Larsen, L., Clark, C., & Monroe, K. (1992). MMPI-2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4, 340-345.. [https://doi.org/10.1037/1040-3590.4.3.340]
  • Bowling, N. A., Huang, J. L., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111, 218-229.. [https://doi.org/10.1037/pspp0000085]
  • Breitsohl, H., & Steidelmüller, C. (2018). The impact of insufficient effort responding detection methods on substantive responses: Results from an experiment testing parameter invariance. Applied Psychology, 67(2), 284-308.. [https://doi.org/10.1111/apps.12121]
  • Bruehl, S., Lofland, K. R., Sherman, J. J., & Carlson, C. R. (1998). The Variable Responding Scale for detection of random responding on the Multidimensional Pain Inventory. Psychological Assessment, 10(1), 3-9.. [https://doi.org/10.1037/1040-3590.10.1.3]
  • Buechley, R., & Ball, H. (1952). A new test of “validity” for the group MMPI. Journal of Consulting Psychology, 16(4), 299-301.. [https://doi.org/10.1037/h0053897]
  • Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3-5.. [https://doi.org/10.1177/1745691610393980]
  • Butcher, J. N., Dahlstrom, W. G., Graham, J. R., Tellegen, A., & Kaemmer, B. (1989). Minnesota multiphasic personality inventory-2 (MMPI-2): Manual for administration and scoring. Minneapolis: University of Mennesota Press..
  • Clark, M. E., Gironda, R. J., & Young, R. W. (2003). Detection of back random responding: Effectiveness of MMPI-2 and Personality Assessment Inventory validity indices. Psychological Assessment, 15, 223-234.. [https://doi.org/10.1037/1040-3590.15.2.223]
  • Carver, C. S., & Scheier, M. S. (1981). Attention and self-regulation: A control theory approach to human behavior. New York: Springer-Verlag..
  • Costa, P. T., & McCrae, R. R. (2008). The revised neo personality inventory (neo-pi-r). The SAGE handbook of personality theory and assessment, 2, 179-198.. [https://doi.org/10.4135/9781849200479.n9]
  • Credé, M. (2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70(4), 596-612.. [https://doi.org/10.1177/0013164410366686]
  • Curran, P. G., Kotrba, L., & Denison, D. (2010). Careless responding in surveys: Applying traditional techniques to organizational settings. In 25th annual conference of Society for Industrial and Organizational Psychology, Atlanta, GA.. [https://doi.org/10.1037/e518392013-128]
  • DeSimone, J. A., DeSimone, A. J., Harms, P. D., & Wood, D. (2018). The differential impacts of two forms of insufficient effort responding. Applied Psychology, 67(2), 309-338.. [https://doi.org/10.1111/apps.12117]
  • DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36(2), 171-181.. [https://doi.org/10.1002/job.1962]
  • Devine, P. G. (1989). Stereotypes and prejudice: Their automatic and controlled components. Journal of personality and social psychology, 56(1), 5-18.. [https://doi.org/10.1037/0022-3514.56.1.5]
  • Diener, E., & Wallbom, M. (1976). Effects of self-awareness on antinormative behavior. Journal of Research in Personality, 10, 107-111.. [https://doi.org/10.1016/0092-6566(76)90088-X]
  • Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed-mode surveys: the tailored design method. Hoboken: John Wiley & Sons..
  • Dunn, A. M., Heggestad, E. D., Shanock, L. R., & Theilgard, N. (2018). Intra-individual response variability as an indicator of insufficient effort responding: Comparison to other indicators and relationships with individual differences. Journal of Business and Psychology, 33(1), 105-121.. [https://doi.org/10.1007/s10869-016-9479-0]
  • Duval, S., Duval, V. H., & Neely, R. (1979). Self-focus, felt responsibility, and helping behavior. Journal of Personality and Social Psychology, 37(10), 1769-1778.. [https://doi.org/10.1037/0022-3514.37.10.1769]
  • Duval, S., & Wicklund, R. A. (1972). A theory of objective self awareness. New York: Academic Press..
  • Ehlers, C., Greene-Shortridge, T. M., Weekley, J. A., & Zajack, M. D. (2009). The exploration of statistical methods in detecting random responding. In Annual Meeting of the Society for Industrial/Organizational Psychology, Atlanta, GA.. [https://doi.org/10.1037/e518422013-599]
  • Gibson, B., & Zielaskowski, K. (2013). Subliminal priming of winning images prompts increased betting in slot machine play. Journal of Applied Social Psychology, 43, 106-115.. [https://doi.org/10.1111/j.1559-1816.2012.00985.x]
  • Gonzales, A. L., & Hancock, J. T. (2011). Mirror, mirror on my Facebook wall: Effects of exposure to Facebook on self-esteem. Cyberpsychology, Behavior, and Social Networking, 14(1-2), 79-83.. [https://doi.org/10.1089/cyber.2009.0411]
  • Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213-224.. [https://doi.org/10.1002/bdm.1753]
  • Govern, J. M., & Marsch, L. A. (2001). Development and validation of the situational self-awareness scale. Consciousness and Cognition, 10(3), 366-378.. [https://doi.org/10.1006/ccog.2001.0506]
  • Greene, R. L. (1978). An empirically derived MMPI carelessness scale. Journal of Clinical Psychology, 34(2), 407-410.. [https://doi.org/10.1002/1097-4679(197804)34:2<407::AID-JCLP2270340231>3.0.CO;2-A]
  • Guay, R. P. Oh, I.-S. Choi, D., Mitchell, M. S., Mount, M. K., & Shin, K. (2013). The interactive effect of conscientiousness and agreeableness on job performance dimensions in South Korea. International Journal of Selection and Assessment, 21, 233-238.. [https://doi.org/10.1111/ijsa.12033]
  • Haertzen, C. A., & Hill, H. E. (1963). Assessing subjective effects of drugs: An index of carelessness and confusion for use with the Addiction Research Center Inventory (ARCI). Journal of Clinical Psychology, 19(4), 407-412.. [https://doi.org/10.1002/1097-4679(196310)19:4<407::AID-JCLP2270190410>3.0.CO;2-N]
  • Hinkin, T. R. (1998). A brief tutorial on the development of measures for use in survey questionnaires. Organizational Research Methods, 1, 104-121.. [https://doi.org/10.1177/109442819800100106]
  • Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99-114.. [https://doi.org/10.1007/s10869-011-9231-8]
  • Huang, J. L., Liu, M., & Bowling, N. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100(3), 828-845.. [https://doi.org/10.1037/a0038510]
  • Ickes, W. J., Wicklund, R. A., & Ferris, C. B. (1973). Objective self awareness and self esteem. Journal of Experimental Social Psychology, 9(3), 202-219.. [https://doi.org/10.1016/0022-1031(73)90010-3]
  • Johnson, J. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39(1), 103-129.. [https://doi.org/10.1016/j.jrp.2004.09.009]
  • Kanfer, F. H. (1970). Self-regulation: Research, issues, and speculations. Behavior Modification in Clinical Psychology, 74, 178-220..
  • Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254-284.. [https://doi.org/10.1037/0033-2909.119.2.254]
  • Kreuter, F., Yan, T., & Tourangeau, R. (2008). Good item or bad-can latent class analysis tell?: the utility of latent class analysis for the evaluation of survey questions. Journal of the Royal Statistical Society: Series A (Statistics in Society), 171, 723-738.. [https://doi.org/10.1111/j.1467-985X.2007.00530.x]
  • Lawson, G., Stedmon, A. W., Zhang, K., Eubanks, D. L., & Frumkin, L. A. (2013). The effects of self-awareness on body movement indicators of the intention to deceive. Applied Ergonomics, 44(5), 687-693.. [https://doi.org/10.1016/j.apergo.2012.04.018]
  • Liu, M., Bowling, N. A., Huang, J. L., & Kent, T. A. (2013). Insufficient effort responding to surveys as a threat to validity: The perceptions and practices of SIOP members. The Industrial-Organizational Psychologist, 51, 32-38..
  • Maniaci, M. R., & Rogge, R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61-83.. [https://doi.org/10.1016/j.jrp.2013.09.008]
  • McGrath, R. E., Mitchell, M., Kim, B. H., & Hough, L. (2010). Evidence for response bias as a source of error variance in applied assessment. Psychological Bulletin, 136(3), 450-470.. [https://doi.org/10.1037/a0019216]
  • Mead, G. H. (1934). Mind, self and society (Vol. 111). Chicago: University of Chicago Press..
  • Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437-455.. [https://doi.org/10.1037/a0028085]
  • Morey, L. C. (1991). Personality assessment inventory. Odessa, FL: Psychological Assessment Resources.. [https://doi.org/10.1037/t03903-000]
  • Mueller-Hanson, R., Heggestad, E. D., & Thornton III, G. C. (2003). Faking and selection: Considering the use of personality from select-in and select-out perspectives. Journal of Applied Psychology, 88, 348-355.. [https://doi.org/10.1037/0021-9010.88.2.348]
  • Niessen, A. S. M., Meijer, R. R., & Tendeiro, J. N. (2016). Detecting careless respondents in web-based questionnaires: Which method to use?. Journal of Research in Personality, 63, 1-11.. [https://doi.org/10.1016/j.jrp.2016.04.010]
  • Northcraft, G. B., Schmidt, A. M., & Ashford, S. J. (2011). Feedback and the rationing of time and effort among competing tasks. Journal of Applied Psychology, 96(5), 1076-1086.. [https://doi.org/10.1037/a0023221]
  • Ones, D. S., Viswesvaran, C., & Reiss, A. D. (1996). Role of social desirability in personality testing for personnel selection: The red herring. Journal of Applied Psychology, 81, 660-679.. [https://doi.org/10.1037/0021-9010.81.6.660]
  • Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867-872.. [https://doi.org/10.1016/j.jesp.2009.03.009]
  • Payne, B. K., Brown-Iannuzzi, J. L., & Loersch, C. (2016). Replicable effects of primes on human behavior. Journal of Experimental Psychology: General, 145, 1269-1279.. [https://doi.org/10.1037/xge0000201]
  • Perdue, C. W., & Gurtman, M. B. (1990). Evidence for the automaticity of ageism. Journal of Experimental Social Psychology, 26(3), 199-216.. [https://doi.org/10.1016/0022-1031(90)90035-K]
  • Pinsoneault, T. B. (2005). Detecting random, partially random, and nonrandom Minnesota Multiphasic Personality Inventory-Adolescent Protocols. Psychological Assessment, 17(4), 476-480.. [https://doi.org/10.1037/1040-3590.17.4.476]
  • Pratto, F., & Bargh, J. A. (1991). Stereotyping based on apparently individuating information: Trait and global components of sex stereotypes under attention overload. Journal of Experimental Social Psychology, 27(1), 26-47.. [https://doi.org/10.1016/0022-1031(91)90009-U]
  • R Development Core Team. (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL http://www.R-project.org, ..
  • Schultz, D. P. (1969). The human subject in psychological research. Psychological Bulletin, 72, 214-228.. [https://doi.org/10.1037/h0027880]
  • Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93-105.. [https://doi.org/10.1037/0003-066X.54.2.93]
  • Silvia, P. J., & Duval, T. S. (2001). Objective self-awareness theory: Recent progress and enduring problems. Personality and Social Psychology Review, 5, 230-241.. [https://doi.org/10.1207/S15327957PSPR0503_4]
  • Silvia, P. J., & Eichstaedt, J. (2004). A self-novelty manipulation of self-focused attention for Internet and laboratory experiments. Behavior Research Methods, Instruments, & Computers, 36(2), 325-330.. [https://doi.org/10.3758/BF03195578]
  • Silvia, P. J., & Phillips, A. G. (2013). Self-awareness without awareness? Implicit self-focused attention and behavioral self-regulation. Self and Identity, 12, 114-127.. [https://doi.org/10.1080/15298868.2011.639550]
  • Spelke, E., Hirst, W., & Neisser, U. (1976). Skills of divided attention. Cognition, 4, 215-230.. [https://doi.org/10.1016/0010-0277(76)90018-4]
  • Stone, J., Aronson, E., Crain, A. L., Winslow, M. P., & Fried, C. B. (1994). Inducing hypocrisy as a means of encouraging young adults to use condoms. Personality and Social Psychology Bulletin, 20(1), 116-128.. [https://doi.org/10.1177/0146167294201012]
  • Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. New York: Cambridge University Press.. [https://doi.org/10.1017/CBO9780511819322]
  • Uleman, J. S., & Bargh, J. A. (Eds.). (1989). Unintended thought. New York: Guilford Press..
  • Ward, M. K., & Meade, A. W. (2018). Applying social psychology to prevent careless responding during online surveys. Applied Psychology, 67, 231-263.. [https://doi.org/10.1111/apps.12118]
  • Ward, M. K., & Pond III, S. B. (2015). Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys. Computers in Human Behavior, 48, 554-568.. [https://doi.org/10.1016/j.chb.2015.01.070]
  • Weingarten, E., Chen, Q., McAdams, M., Yi, J., Hepler, J., & Albarracín, D. (2016). From primed concepts to action: A meta-analysis of the behavioral effects of incidentally presented words. Psychological Bulletin, 142, 472-497.. [https://doi.org/10.1037/bul0000030]
  • Woods, C. M. (2006). Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28, 186-191.. [https://doi.org/10.1007/s10862-005-9004-7]