Over the last 40 years, a rich body of literature has emerged about the design, implementation, analysis, and interpretation of experiments in field settings. This course then has three purposes. The first is to introduce students to recent methodological advances in the design and analysis of field experiments, particularly in school settings. The second is for students to read and discuss well-known field experiments that have had important implications for policy, and/or our understanding of science. The readings will also help students observe challenges that researchers have encountered in the field, and how these problems were (or were not) addressed. The third is to demonstrate that although the course is about field experiments, many of the issues that are addressed here are either directly applicable – or extends easily – to the design and analysis of observational studies, including concerns about attrition, treatment non-compliance, and violations to SUTVA. 

To achieve these purposes, we begin by discussing methodological issues related to the planning and designing of field experiments. We review basic concepts from experimental design, discuss the two most common experimental designs used in field evaluations of school interventions (the cluster and randomized block designs), and address appropriate methods for analysis. We examine special issues related to the hierarchal structure of data often found in school settings, as well as discuss approaches for calculating statistical power, and ensuring that the planned randomized experiment addresses the research question of interest. Because randomized experiments in field settings are rarely (if ever) perfect, the second half of the course focuses on methods for detecting and addressing implementation problems in field experiments. This includes measuring treatment implementation fidelity, addressing non-compliance of treatment assignment, examining and addressing attrition and unhappy random assignment, and detecting violations to the Stable Unit Treatment Value Assumption (SUTVA). Finally, we discuss tips for writing and presenting of experimental designs and results in proposals and in journal articles.

At the end of the course, students should be able to design and plan their own randomized control trial in a realistic field setting. As such, throughout the semester, students will work in small groups to develop and write their own Institute for Education Sciences (IES) Goal 3 Efficacy proposals. Proposals should use random assignment to evaluate a school intervention on an IES program area of interest. Proposals will focus on the “Significance” and “Research Plan” components of the Project Narrative. At the end of the semester, students will present their proposals to a “reviewer panel” (that will consist of the class).