Audit Studies: Behind the Scenes with Theory, Method, and Nuance Table of Contents Introduction I. The Theory Behind and History of Audit Studies Chapter 1 Making it Count: Action Research and the Practice of Auditing for Discrimination Frances Cherry and Marc Bendick, Jr. Chapter 2 Correspondence Experiments Between 2006 and 2015: What Did They Tell Us About Success Factors in the Labour Market? Stijn Baert II. The Method of Audit Studies: Planning, Implementation, and Analysis Chapter 3 The Ethics of Audit Studies: Best Practices and Academic Cooperation Chapter 4 Technical Aspects of Correspondence Studies Joanna Lahey and Ryan Beasley Chapter 5 Which Jobs, Which Employers, Which Labor Markets?: Challenges in Designing and Conducting a Labor Market Resume Study William Carbonaro and Jon Schwarz Chapter 6 Tentative Chapter on Analysis TBD III. Nuance in Audit Studies: Context, Signals, and Technology Chapter 7 Geography and Social Perception: Experimental Evaluation Methods of Place-Based Stigma Max Besbris, Jacob Faber, Peter Rich, Patrick Sharkey Chapter 8 By Any Other Name? Social Class Perceptions of Names
Chapter 9 Social Science by Email: Adapting and Improving Email Audit Studies Charles Crabtree Chapter 10 Technology and Audit Studies: A Return to Phone Audits and Using Mechanical Turk for Pre-Tests Heather Kugelmass IV. The Future of Audit Studies: Challenges and Limitations Chapter 11 Emerging Frontiers in Audit Study Research: Mechanisms, Variation, and Generalizability David S. Pedulla Chapter 12 Summary/Conclusion Chapter David Neumark
Audit Studies: Behind the Scenes with Theory, Method, and Nuance List of Contributors Stijn Baert Tenure Track Professor of Empirical Labour Economics Ghent University Ryan Beasley Computer Scientist SimQuest Solutions, Inc. Marc Bendick, Jr. Bendick and Egan Economic Consultants, Inc. Max Besbris Ph.D. candidate in Sociology New York University William Carbonaro Associate Professor of Sociology University of Notre Dame Frances Cherry Professor of Psychology Carleton University Charles Crabtree Ph.D. candidate in Political Science University of Michigan Jacob Faber Assistant Professor of Public Service New York University Assistant Professor of Sociology and Demography The Pennsylvania State University Heather Kugelmass Ph.D. candidate in Sociology Princeton University Joanna Lahey Associate Professor of Government and Public Service Texas A&M University
David Neumark Professor of Economics University of California at Irvine David S. Pedulla Assistant Professor of Sociology Stanford University Peter Rich Assistant Professor of Policy Analysis and Management Cornell University Jon Schwarz Ph.D. candidate in Sociology University of Notre Dame Patrick Sharkey Associate Professor of Sociology New York University
Audit Studies: Behind the Scenes with Theory, Method, and Nuance Abstracts Introduction I. The Theory Behind and History of Audit Studies Chapter 1 Making it Count: Action Research and the Practice of Auditing for Discrimination Frances Cherry and Marc Bendick, Jr. Community self-surveys and related social action research from the target s perspective has been an important component of civil rights activism since the 1940s. The current generation of employment discrimination auditing (testing), which started around 1990, can in part be understood as a more rigorous version of this tradition. Paralleling the expanding scope of the civil rights movement beyond its initial roots in race and ethnicity, the subjects of auditing have grown to include a diverse range of personal characteristics such as gender, age, disability, sexual orientation/gender identity, and personal appearance, as well as life circumstances such as employment credentials, place of residence, and criminal records. Auditing has also creatively expanded the range of mechanism of social change where it has been mobilized, including shaping individual attitudes, influencing law and regulations, providing practical job-seeking advice, supporting legal enforcement, and advancing basic social science research. This evolution suggests the untapped potential for activist scholars of this powerful and flexible tool. Chapter 2 Correspondence Experiments Between 2006 and 2015: What Did They Tell Us About Success Factors in the Labour Market? Stijn Baert In this chapter, we synthesize the main conclusions of the correspondence experiments testing employers hiring behaviour conducted between 2006 and 2015. On the one hand, we review the impact on positive call-back of discrimination grounds based on which unequal treatment is prohibited by law in most OECD countries, with a focus on ethnicity, gender, sexual orientation, age, and disability. On the other hand, we review the main findings of the more recent literature using correspondence experiments to measure the impact of aspects of educational credentials and former labour market experiences on employment opportunities. II. The Basics of Audit Studies: Planning, Implementation, and Analysis Chapter 3 The Ethics of Audit Studies: Best Practices and Academic Cooperation The process of submitting research to an institutional review board for approval is often a private and personal one for researchers involving few people making important distinctions and
determinations regarding the ethics and impact of research. Arguably, the process has not adapted to provide institutional agents with clear guidelines on dealing with audit studies, particularly those conducted online. In this chapter, I review the critical language and definitions of human subjects research as it relates to audit studies. I provide a list of best practices to help researchers new to field experiments get started and make sure they follow all ethical guidelines. Furthermore, I discuss the ways in which researchers generally might work together to reduce the burden of audit studies and ensure that institutional review boards continue to approve these studies. Chapter 4 Technical Aspects of Correspondence Studies Joanna Lahey and Ryan Beasley Using the need for external validity as a framework, this chapter discusses technical concerns and choices that arise when crafting a correspondence or audit study. The chapter will go into the details of resume creation including power analysis, choice of inputs, pros and cons of matching pairs, solutions to the limited template problem, and making sure that instruments indicate what the experimenters want them to indicate. Further topics about implementation include when and for how long to field a study, deciding on a participant pool, and whether or not to use replacement from the participant pool. More technical topics include matching outcomes to inputs, data storage, and analysis issues such as when to use clustering, when not to use fixed effects, and how to measure heterogeneous and interactive effects. The chapter ends with a technical checklist that experimenters can utilize prior to fielding a correspondence study. Chapter 5 Which Jobs, Which Employers, Which Labor Markets?: Challenges in Designing and Conducting a Labor Market Resume Study William Carbonaro and Jon Schwarz In this chapter, we will reflect on our experiences conducting an audit study that used resumes to examine how race, high school credentials, and academic grades were related to callbacks for jobs. The paper will focus on three main issues. First, we will examine whether resume studies may be better suited to studying large urban labor markets rather than smaller urban labor markets. Second, we will review the literature on employers behaviors in hiring, and consider whether recent technological changes make prior research in this area outdated. Finally, we consider whether resume studies generate a sample of jobs that reasonably approximates the full array of jobs that job applicants typically seek. Our main goal is to help researchers who hope to conduct similar studies in the future by highlighting and reflecting upon challenges, obstacles, and unexplored opportunities in our work. III. Nuance in Audit Studies: Context, Signals, and Technology Chapter 7 Geography and Social Perception: Experimental Evaluation Methods of Place-Based Stigma
Max Besbris, Jacob Faber, Peter Rich, Patrick Sharkey The United States remains a spatially segregated nation by many measures including race, income, wealth, political views, education, and immigration status. Scholars have, for many years, grappled with questions stemming from demographic spatial patterns and have come to recognize the neighborhood in which an individual lives as predictive of many individual-level outcomes. However, the mechanisms that underlie this relationship remain relatively underexplored. In this chapter, we show how the use of experimental audits can help uncover interaction-based mechanisms such as place-based social stigma. Specifically, we describe the methodology of a previous study (Besbris et al. 2015) that revealed how neighborhood of residence matters for individual desirability in economic transactions. We focus on the study s operationalization of neighborhoods. In doing so, we show how future research can operationalize non-individual-level treatment characteristics like units of space. Doing so helps future research to 1) better understand the causal relationship between space and individual-level outcomes, and 2) better parse the effects of individual-level variables versus non-individual-level variables, which are often conflated in non-experimental research. We close by suggesting the implementation of experimental audit studies in testing for effects at other geographic scales, such as metropolitan area, state, region, country, or continent. Chapter 8 By Any Other Name? Social Class Perceptions of Names Prior research using names to convey race in audit studies has been concerned about the effects of social class in choosing names. However, minimal scientific study of this issue has been conducted. In this chapter, I present the results of a survey experiment on individual perceptions of social class from names. I conduct a survey experiment using participants from Amazon s Mechanical Turk. Each participants is randomly assigned a set of twenty names with varying combinations of including first only or first and last names and asked to either identify the race or social class they associate with each name. The results suggest that social class is less salient than race: respondents asked to identify the social class of names take longer to complete the survey than respondents asked to identify the race of names. Additionally, respondents are less likely to correctly recognize social class among lower rather than middle or upper class names. I find a number of variations in effects by the respondent s own demographic characteristics. These results suggest that the effect sizes of discrimination are likely to significantly vary based on the researcher s choice of names as a signal. Chapter 9 Social Science by Email: Adapting and Improving Email Audit Studies Charles Crabtree In recent years, there has been a dramatic increase in the number of audit studies conducted throughout the social sciences. This growth has been fueled, in part, by the invention and diffusion of electronic mail, which enables experimentalists to correspond with study participants at low cost and over great distances. In this chapter, I suggest ways that email audit studies can be adapted and improved. First, I review the history of email audit studies. I then
examine several possible limitations of using email as a study medium and propose some design features that address these shortcomings. I then discuss how audit studies can be adapted to investigate a broader range of social phenomena than discrimination. Finally, I provide a set of recommendations about how to improve audit study implementation and data collection. Chapter 10 Technology and Audit Studies: A Return to Phone Audits and Using Mechanical Turk for Pre-Tests Heather Kugelmass In recent years, emphasis has shifted from phone-based audit studies to web-based audit studies. The prevalence of online marketplaces and ubiquity of email communication has enabled a rapid increase in the use of online audit studies, which can be conducted inexpensively and quickly. Yet, online services have great potential to increase the popularity of phone audits as well, by reducing logistical hurdles and cost. For example, freelancer websites serve as good resources for locating low-cost voice actors to serve as testers. Mechanical Turk s pool of survey-takers provides a cheap, quick, and effective way to improve experimental stimuli and pre-validate manipulations. Google offers a free voice service through which calls can be made and voicemails can be collected and transcribed. Phone audits continue to occupy an important space among field experiments. Discrimination in access to goods, services, and opportunities persists via subtle phone-based communication, particularly within some target professions such as health care. Furthermore, voice is not only a powerful racial cue it can also be leveraged as strong indicator of social class, which can be difficult to communicate through written stimuli. In this chapter, I describe the progression and potential of phone audit studies, with particular attention to their applications in the health care sector. IV. The Future of Audit Studies: Challenges and Limitations Chapter 11 Emerging Frontiers in Audit Study Research: Mechanisms, Variation, and Generalizability David S. Pedulla Audit studies have gained popularity in sociology, particularly among researchers examining discrimination and bias. While the audit study method has clear strengths combining high levels of internal validity with the examination of real world behavior it can also suffer from important limitations. After briefly highlighting the benefits of the method, this chapter moves on to discuss three of the central challenges faced by audit study research: 1) mechanisms, 2) variation, and 3) generalizability. First, audit studies often have difficulty identifying the mechanisms that drive the empirical patterns that are observed, leaving a black box of speculation between the independent and dependent variables. Second, while justifiable given their design, audit studies generally emphasize the importance of average treatment effects, leaving additional theoretically important variation unexamined. Finally, the generalizability of the findings from audit studies is often uncertain, making it difficult to provide clear scope conditions on the results that are generated. I then summarize and present recent audit study research that utilizes innovative approaches to address these concerns and discuss potential paths
forward to continue to advance the design, implementation, and contribution of this method to sociological research. Chapter 12 Summary/Conclusion Chapter David Neumark