Lecture 19 Treatment Effect Methods and Review

Some Pointers

• Last time we talked about heterogeneous treatment effects and how our methods produce different averages of those effects
• But we don’t need to be limited to that!
• There are plenty of methods - many of them new - that let us estimate a distribution of treatment effects
• We won’t be going super far into detail with them, but I’ll mostly just be letting you know they exist and some pointers for looking further
• I’ll favor pointers to packages over papers, but if you look in the help files you’ll generally find paper citations

Sorted Effects

• The sorted effects method uses covariates to look at variation in the treatment effect, and produces a distribution of treatment effects
• It also lets you see who is at each part of the distribution

library(SortedEffects)
# Data on being denied for a mortgage.
data(mortgage)
# Save the formula to reuse later
fm <- deny ~ black + p_irat + hse_inc + ccred + mcred + pubrec + ltv_med + 
           ltv_high + denpmi + selfemp + single + hischl
# spe() for "Sorted Partial Effects"
m <- spe(fm, data = mortgage,
         var = 'black', # black is the treatment variable
         method = 'logit',
         us = c(2:98)/100, # Get the distribution from the 2nd to 98th percentile
         b = 500, bc = TRUE) # Use bootstrapped SEs and bias-correction 

Sorted Effects

• See the distribution in the effect of black on being denied

Sorted Effects

• Who is most and least affected by the “treatment” of being black?

t <- c("deny", "p_irat", "black", "hse_inc", "ccred", "mcred", "pubrec",
       "denpmi", "selfemp", "single", "hischl", "ltv_med", "ltv_high")
classify <- ca(fm, t = t, data = mortgage, var = 'black', method = 'logit',
               cl = 'both', # Get WHO the most and least are, not how different they are
               b = 500, bc = TRUE)

## Using 1 CPUs now.

results <- summary(classify) %>%
  as_tibble() %>%
  mutate(Group = row.names(summary(classify)),
         Ratio = Most/Least) %>%
  select(Group, Most, Least, Ratio) %>%
  arrange(Ratio)

Sorted Effects

• Those who were denied for insurance (denpmi) had smallest effects of black, those who were single had the biggest

Bayesian Hierarchical Modeling

• A very old method! But it works. An extension of random effects
• Instead of just letting the constant vary, let any coefficient vary, and give each its own function to vary over controls! Those controls can let the effect vary

\[ Y = \beta_0 + \beta_1X + \varepsilon \] \[ \beta_0 = \gamma_{00} + \nu_{00} \] \[ \beta_1 = \gamma_{10} + \gamma_{11}W + \nu_{01} \]

• Terminology difference: “fixed effects” means “coefficients that don’t vary”

Bayesian Hierarchical Modeling

library(lme4)
# The whole thing would be super slow so for now let's just do a few effects
m <- lmer(deny ~ p_irat + hse_inc + ccred + mcred + pubrec + ltv_med + 
           ltv_high + denpmi + selfemp + single + hischl +
       (single + hischl | black),
     data = mortgage)

Bayesian Hierarchical Modeling (cut off)

## Linear mixed model fit by REML ['lmerMod']
## Formula: deny ~ p_irat + hse_inc + ccred + mcred + pubrec + ltv_med +  
##     ltv_high + denpmi + selfemp + single + hischl + (single +  
##     hischl | black)
##    Data: mortgage
## 
## REML criterion at convergence: 722.8
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -3.4138 -0.4242 -0.1776 -0.0076  3.9268 
## 
## Random effects:
##  Groups   Name        Variance  Std.Dev.  Corr       
##  black    (Intercept) 3.803e-03 0.0616722            
##           single      6.488e-07 0.0008055 -1.00      
##           hischl      6.975e-05 0.0083517 -1.00  1.00
##  Residual             7.726e-02 0.2779482            
## Number of obs: 2380, groups:  black, 2
## 
## Fixed effects:
##              Estimate Std. Error t value
## (Intercept) -0.011466   0.069643  -0.165
## p_irat       0.449568   0.086586   5.192
## hse_inc     -0.072378   0.095866  -0.755
## ccred        0.031008   0.003673   8.441
## mcred        0.015003   0.011154   1.345
## pubrec       0.198644   0.023095   8.601
## ltv_med      0.032981   0.012356   2.669
## ltv_high     0.191246   0.033116   5.775
## denpmi       0.704414   0.041051  17.160
## selfemp      0.060240   0.017941   3.358
## single       0.034120   0.011944   2.857
## hischl      -0.134258   0.045705  -2.937
## 
## Correlation of Fixed Effects:
##          (Intr) p_irat hse_nc ccred  mcred  pubrec ltv_md ltv_hg denpmi selfmp
## p_irat   -0.155                                                               
## hse_inc  -0.042 -0.781                                                        
## ccred    -0.090 -0.073  0.077                                                 
## mcred    -0.267  0.073 -0.106 -0.127                                          
## pubrec    0.019 -0.054  0.012 -0.252  0.003                                   
## ltv_med  -0.015 -0.063  0.020 -0.020 -0.150 -0.043                            
## ltv_high  0.031 -0.048  0.003 -0.019 -0.074 -0.044  0.180                     
## denpmi    0.017 -0.030  0.020 -0.006 -0.018 -0.058 -0.076 -0.073              
## selfemp  -0.032 -0.078  0.064  0.012  0.040 -0.039  0.075  0.001  0.009       
## single   -0.018  0.018 -0.043  0.022 -0.154  0.016  0.025 -0.018 -0.005  0.009
## hischl   -0.746  0.017  0.051  0.013  0.045 -0.001 -0.007 -0.033 -0.023 -0.002
##          single
## p_irat         
## hse_inc        
## ccred          
## mcred          
## pubrec         
## ltv_med        
## ltv_high       
## denpmi         
## selfemp        
## single         
## hischl   -0.058
## optimizer (nloptwrap) convergence code: 0 (OK)
## boundary (singular) fit: see ?isSingular

Machine Learning

• The biggest contributions of machine learning to causal inference thus far have been in heterogeneous treatment effects
• (there are other things too, like matrix completion, which is way too complex to get into here)
• Allowing a zillion different things to vary is easy in machine learning!
• Note: machine learning tends to use “training” and “holdout” data. So estimate the model using a training subset, and then estimate your treatment effect distribution by sending your holdout data through that model

LASSO and interactions

• LASSO is a “regularized regression” that does regression but doesn’t JUST minimize sum of squared errors, it also has a second goal of shrinking coefficients
• There are several forms of regularized regression. LASSO tends to set coefficeints to 0, i.e. chuck them out
• So… just interact treatment with everything and see what interactions are worth keeping!

LASSO and interactions

• This is commonly applied in instrumental variables settings, where these interactions can improve first-stage power and ease the weak-instrument problem
• But can be applied elsewhere too to find variation in a treatment effect
• I won’t do a walkthrough here, but the package glmnet is typically used to estimate LASSO. See this walkthrough.

Causal Forest

• A random forest is a prediction method. Take your data, go through every covariate you have, and every of each covariate, and split the data based on the split that minimizes sum of squared error
• Then do that again for each of the splits, and again and again. Each time only use a random subset of the variables
• Stop once the splits get too small
• Causal forest does the exact same thing except instead of minimizing the SSE, it maximizes the difference in causal effect between the splits
• i.e. it hunts for causal effect differences! You end up with an effect prediction for each individual

Causal Forest

• grf does causal forest, and even has an IV version if you need an IV to identify

library(grf)
mortgage <- mortgage %>% mutate(holdout = runif(n()) > .5)
holdout <- mortgage %>% filter(holdout)
training <- mortgage %>% filter(!holdout)
W = training %>% pull(black) %>% as.matrix()
X = training %>% select(p_irat, hse_inc, ccred, mcred, pubrec,
       denpmi, selfemp, single, hischl, ltv_med, ltv_high) %>% as.matrix()
Y = training %>% pull(deny) %>% as.matrix()
m <- causal_forest(X, Y, W, tune.parameters = 'alpha')

Causal Forest

X.holdout <- holdout %>% select(p_irat, hse_inc, ccred, mcred, pubrec,
       denpmi, selfemp, single, hischl, ltv_med, ltv_high) %>% as.matrix()
indiv_effects <- predict(m, X.holdout)
holdout <- holdout %>% mutate(effect = indiv_effects$predictions)

Causal Forest

Causal Forest

• Who is affected? Let’s do a similar test to what SortedEffects did (although we could look at it plenty of other ways)

Treatment Effect Methods

• Anyway, there’s some stuff for you to check out!
• Obviously there are zillions of causal-inference methods we don’t have time to cover
• Bartik instruments, matrix completion, causal discovery, and so on and so on and so on
• Consider this a good starting place

Exam Review

• Just a reminder of some stuff we’ve covered

Fixed Effects

• If we have data where we observe the same people over and over, we can implement fixed effects by controlling for individual
• This accounts for everything that’s constant within individual. If, for example, “individual” was city, that would include geography, state, founding year, etc.
• Doesn’t account for things that vary within individual over time, like Laws

Difference-in-Difference

• Difference-in-Difference applies when you have a group that you can observe both before and after the policy
• You worry that time is a confounder, but you can’t control for it
• Unless you add a control group that DIDN’T get the policy
• We must be careful to check that parallel trends holds

Difference-in-Difference

Difference-in-Difference

• Get the before-after difference for both groups
• Then subtract out the difference for the control

diddata <- tibble(Group=c(rep("C",2500),rep("T",2500)),
                  Time=rep(c(rep("Before",1250),rep("After",1250)),2)) %>%
  mutate(Treated = (Group == "T") & Time == "After") %>%
  mutate(Y = 2*(Group == "T") + 1.5*(Time == "After") + 3*Treated + rnorm(5000))
did <- diddata %>% group_by(Group,Time) %>% summarize(Y = mean(Y))
before.after.control <- did$Y[1] - did$Y[2]
before.after.treated <- did$Y[3] - did$Y[4]
did.effect <- before.after.treated - before.after.control
did.effect

## [1] 2.936178

m <- feols(Y ~ Treated | Group + Time, data = diddata)
coef(m)

## TreatedTRUE 
##    2.936178

Regression Discontinuity

• If we have a treatment D that is assigned based on a cutoff in a running variable, we can use regression discontinuity
• Focus right around the cutoff and compare above-cutoff to below-cutoff
• We’ve isolated a great set of treatment/control groups because in this area it’s basically random whether you’re above or below the cutoff

Regression Discontinuity

Regression Discontinuity

• Estimate by fitting a line that jumps at the cutoff and estimating the jump
• Use local regression and bandwidths to avoid being affected by far-away observations
• “Fuzzy” designs where treatment only jumps partially scale the effect using IV

Regression Discontinuity

• Expressed well in graphs! Treatment should jump at cutoff. If not perfectly from 0% to 100%, use IV too

Regression Discontinuity

• Variables other than Y and treatment shouldn’t jump at cutoff - they should be balanced

Instrumental Variables

• An instrumental variable affects treatment (relevant) but has no back doors itself or paths to \(Y\) except through \(X\) (valid)
• We move the no-open-back-doors assumption to the IV rather than the treatment
• We isolate JUST the variation that comes from Z. No back doors in that variation! We have a causal effect
• Can conceptually think of it as (or literally apply it to) an experiment where randomization doesn’t work perfectly

Instrumental Variables

Treatment Effects

• There isn’t a treatment effect. They vary across time, space, individual
• Our methods give us averages - ATE (experiment), ATT (DID), LATE (IV, RDD), variance-weighted (regression w/ controls), etc.
• We must pay close attention to what our design and estimator gives us

That’s it!

• In a very condensed way, that’s the material we covered!
• I recommend looking back over slides, notes, homeworks