Randomized experiments have been recently gaining traction in decision sciences and operations management. However, several internal and statistical validity threats can undermine the conclusions drawn from experimental studies. In this tutorial, we will briefly introduce some of these threats, as well as several potential solutions. After having introduced the endogeneity problem typical of observational studies (i.e., the correlation between a regressor and the disturbance term in a regression model), we will focus on major sources of endogeneity in experimental research. For instance, we will illustrate the perils of demand effects, the risks of running analyses conditional on non-randomized mediators, moderators, or covariates, as well as the problems related to running manipulation checks before measuring the main dependent variable of interest. We will finally present several experimental design strategies, as well as some statistical techniques (e.g., instrumental variable estimation) to cope with the abovementioned problems.
Brief bio: Sirio is a management scholar interested in quantitative research methods, focusing on causal analysis, experimental methods, and structural equation modeling. He also conducts multidisciplinary research on organizational leadership, relying on econometric analysis of observational data and behavioral experiments. His work has been recently published in the Journal of Operations Management and in The Leadership Quarterly.
Ketokivi, M., McIntosh, C.N., 2017. Addressing the endogeneity dilemma in operations management research: theoretical, empirical, and pragmatic considerations. Journal of Operations Management, 52, 1–14.
Lonati, S., Quiroga, B.F., Zehnder, C. and Antonakis, J., 2018. On doing relevant and rigorous experiments: Review and recommendations. Journal of Operations Management, 64, 19-40.
Montgomery, J.M., Nyhan, B., Torres, M., 2016. How conditioning on posttreatment variables can ruin your experiment and what to do about it. American Journal of Political Science, 62 (3), 760–775.
Sajons, G. B. (2020). Estimating the causal effect of measured endogenous variables: A tutorial on experimentally randomized instrumental variables. The Leadership Quarterly, 101348.
Zizzo, D. J. (2010). Experimenter demand effects in economic experiments. Experimental Economics, 13(1), 75-98.