Dataloopr

Here, each $w_g$ is the weight assigned to a particular 2x2 DiD comparison (e.g., early-treated vs. later-treated, treated vs. never-treated), and $\text{DiD}_g$ is the estimated effect from that comparison. The overall TWFE estimate is a weighted average of all these DiDs.

Example: Suppose you have three groups: never treated, early treated, and late treated. Bacon decomposition will tell you how much of your TWFE estimate comes from comparing early- to late-treated units, and how much from comparing treated to never-treated.

Real-world application: Evaluating the effect of minimum wage hikes rolled out at different times across states.

Assumptions / Limitations: Shows you where bias might come from; doesn’t solve the problem itself.

3.8. Advanced Staggered DiD: Callaway & Sant’Anna, Sun & Abraham, Borusyak et al.

As panel data and staggered policy rollouts became the norm, researchers developed robust new estimators for DiD with staggered adoption and heterogeneous treatment effects.

3.8.1. Callaway & Sant’Anna (2021)

Problem it solves: TWFE estimates are biased when treatment effects vary over time or across groups.

Intuition: Estimate group-time average treatment effects (ATTs) for each cohort (units treated in the same period) and event time (relative to treatment), then aggregate.

Equation:

• $G=g$: group treated in period $g$
• $t$: calendar time

Aggregate these to get overall ATT.

Example: For each campaign launch wave, compare outcomes to appropriate controls at each event time.

Real-world application: Multi-city marketing campaigns with staggered launches.

Assumptions / Limitations: Requires at least one group never treated or not-yet-treated units for control; more complex to estimate and report.

3.8.2. Sun & Abraham (2021)

Problem it solves: Event-study coefficients from TWFE are biased in the presence of treatment effect heterogeneity.

Intuition: Estimate event-study coefficients separately for each cohort and then aggregate, avoiding contamination from already-treated units.

Equation:

• $D_{i,t}^{(k)}$: indicator for event time $k$ (relative to treatment)

Example: For each city, estimate the effect at each event time (e.g., 2 weeks before, 1 week before, etc.) using only proper controls.

Real-world application: Policy rollouts in education, marketing, or healthcare.

Assumptions / Limitations: Requires sufficient untreated or not-yet-treated units at each event time.

3.8.3. Borusyak, Jaravel & Spiess (2021)

Problem it solves: Offers a simple and robust estimator for staggered adoption settings.

Intuition: Impute untreated potential outcomes for treated units, then average the gaps.

Equation:

• $t(i)$: time unit $i$ is treated

Example: For each treated city, predict what would have happened without treatment, using pre-treatment data and contemporary controls.

Real-world application: Tech rollouts, phased implementations in business analytics.

Assumptions / Limitations: Requires correct model for imputation; implementation can be computationally intensive.

3.9. Event Studies: Visualizing Dynamic Effects & Testing Parallel Trends

Problem it solves: Treatment effects may vary over time; we need to check for pre-trends (parallel trends assumption).

Intuition: Estimate treatment effects at each “event time” (e.g., weeks before and after treatment) to visualize dynamics and diagnose bias.

Equation:

Where $\beta_k$ is the effect at event time $k$ (e.g., $k = -3$ is three weeks before treatment).

Example: Plot sales in treated cities relative to control cities from 4 weeks before to 6 weeks after campaign launch.

Real-world application: Retail promotions, policy impacts, A/B test diagnostics.

Assumptions / Limitations: Relies on enough pre- and post-periods; cannot fully verify parallel trends, but can detect clear violations.

3.10. de Chaisemartin & D’Haultfoeuille: Robust DiD for Treatment Reversals and Switching

Problem it solves: Some units receive, lose, and regain treatment (e.g., on-off campaign exposure).

Intuition: Provide DiD estimators that remain valid when treatment reverses, using only appropriate control periods and units.

Equation:

• Weights $w_{g,t}$ ensure each comparison uses valid controls.

Example: A city receives the campaign, then the campaign is paused, then restarted. This method compares only periods where treatment status changes.

Real-world application: Subscription price changes, repeated marketing blasts, toggling interventions.

Assumptions / Limitations: Requires detailed treatment timing and enough switching for identification.

4. Deep Dive: The Pitfalls of Two-Way Fixed Effects (TWFE)

TWFE is popular because it’s simple and easy to implement. But when treatment is staggered and effects are heterogeneous, it can go wrong. Let’s see why, with a timeline:

At $t = 4$, Group A is treated, Group B and C are controls. But at $t = 6$, Group B is just treated, and Group A is already treated—yet TWFE may use Group A as a control for Group B, even though both are post-treatment!

Bacon decomposition shows that TWFE combines many such comparisons, some of which are not valid counterfactuals. If treatment effects change over time, or differ between groups, this leads to bias.

Modern DiD estimators (Callaway & Sant’Anna, Sun & Abraham, Borusyak et al.) provide valid alternatives by comparing only to never-treated or not-yet-treated units.

5. Event Studies: Diagnosing Pre-Trends and Dynamic Effects

Event studies plot the estimated treatment effect at each time relative to treatment:

• $k = -3$: Three periods before treatment
• $k = 0$: Period of treatment
• $k = +2$: Two periods after treatment

If pre-treatment coefficients ($k<0$) are flat and close to zero, this supports the parallel trends assumption. If not, it signals problems!

An event study plot could look like this:

import matplotlib.pyplot as plt
plt.axhline(0, color='gray', linestyle='--')
plt.plot(event_times, betas, marker='o')
plt.xlabel('Event Time')
plt.ylabel('Estimated Effect')
plt.title('Event Study: Campaign Effect on Sales')
plt.show()

6. Case Study: E-Commerce Campaign with Staggered Rollout and Changing Customer Composition

Let’s bring it all together with a realistic example.

The Business Question

Your company launches a personalized email campaign to boost conversion and revenue. The campaign is rolled out gradually:

• Wave 1: Large cities in Week 2
• Wave 2: Medium cities in Week 4
• Wave 3: Small cities in Week 6

Your KPIs: conversion rate, revenue per user, and ROAS (return on ad spend).

Challenges

• Not all cities treated at the same time (staggered rollout)
• Customer demographics shift over time (city growth, new markets open)
• The effect of the campaign may fade or grow over time
• Some cities drop out (panel imbalance)

Step 1: Naive Before-After

Compare average revenue before and after for treated cities. Result: +$200 per city.

Flaw: Doesn’t control for broader trends; overstates effect.

Step 2: Simple DiD

Compute DiD using cities not yet treated as controls. Still problematic—parallel trends may not hold due to city differences.

Step 3: DiD with Covariates

Add city size and average customer age as controls. Estimate is now +$120.

Step 4: TWFE

Fit a TWFE model:

library(fixest)
feols(revenue ~ treated*post | city + week, data = ecommerce_panel)

Estimate: +$110 per city per week.

But Bacon decomposition reveals that some comparisons are between already-treated cities and later-treated cities—not valid counterfactuals!

Step 5: Callaway & Sant’Anna

Estimate group-time ATTs using never-treated and not-yet-treated cities as controls. Now you find:

• Wave 1 (large cities): +$100
• Wave 2 (medium): +$80
• Wave 3 (small): +$50

Event study shows no pre-trend for large and medium cities, but a slight positive trend for small cities—suggesting some bias remains for the smallest markets.

Step 6: Outcome Regression & Doubly Robust

Impute counterfactual outcomes using both outcome regression and IPW. The doubly robust estimator gives +$95, confident that you’re less vulnerable to misspecification.

Interpretation

The true effect of the campaign is likely closer to +$80 to +$100 per city per week, with larger effects in bigger markets.

7. Core Equations Explained in Plain English

• Difference-in-Differences:
  $$ \text{DiD} = (Y_{1,1} - Y_{1,0}) - (Y_{0,1} - Y_{0,0}) $$

  “Take the change in outcome for the treated group, subtract the change for the control group. The difference is your estimated effect.”

• Average Treatment Effect on the Treated (ATT):
  $$ ATT = E[Y(1) - Y(0) | D=1] $$

  “The average difference between treated units’ outcomes with and without treatment.”

• Parallel Trends Assumption:
  $$ E[Y_{1,1}^{0} - Y_{1,0}^{0} | D=1] = E[Y_{0,1}^{0} - Y_{0,0}^{0} | D=0] $$

  “If untreated, the treated group would have changed like the control group.”

• Two-Way Fixed Effects:
  $$ Y_{it} = \alpha_i + \lambda_t + \beta D_{it} + \varepsilon_{it} $$

  “Control for average differences across units and time, estimate the effect of treatment.”

• Event Study:
  $$ Y_{it} = \alpha_i + \lambda_t + \sum_{k \neq -1} \beta_k D_{i,t}^{(k)} + \varepsilon_{it} $$

  “Estimate the effect at each time relative to treatment; use pre-treatment coefficients to check for pre-trends.”

• Inverse Probability Weighting:
  $$ w_i = \frac{D_i}{e(X_i)} + \frac{1 - D_i}{1 - e(X_i)} $$

  “Weight each observation by how likely it was to be treated, to balance covariates.”

8. Practical Guide: Which Method to Use for Which Problem?

9. Common Mistakes and How to Avoid Them

• Ignoring parallel trends: Always check for pre-treatment trends using event studies. If trends diverge before treatment, DiD estimates are likely biased.
• Using TWFE with staggered treatment and heterogeneous effects: This can produce misleading results. Use modern estimators instead.
• Not adjusting for observable differences: If treated and control groups differ on key covariates, adjust or reweight to reduce bias.
• Over-interpreting post-treatment dynamics: If effects fade or grow, simple DiD may not capture the full story—use event studies or dynamic DiD estimators.
• Neglecting robustness checks: Always check sensitivity to model specification, sample changes, and weighting.

10. Five Key Takeaways for Practitioners

1. Start simple, but diagnose rigorously: Begin with basic DiD, but always check assumptions—especially parallel trends—before trusting results.
2. Modern problems require modern solutions: With staggered rollout, panel imbalances, and dynamic effects, use advanced methods like Callaway & Sant’Anna or Sun & Abraham.
3. Visualize, don’t just summarize: Event study plots reveal pre-trend violations and dynamic effects much better than a single number.
4. Balance is everything: Use covariate adjustment or weighting to ensure treated and control groups are comparable.
5. Method choice depends on your data and context: There’s no one-size-fits-all—always match your method to your problem and test robustness.

Conclusion

Causal inference is as much an art as a science. In the fast-moving world of analytics, understanding not just the tools but when and how to use them is crucial. Modern Difference-in-Differences methods let us evaluate the true impact of policies, campaigns, or business interventions—even in complex, staggered, and dynamic real-world settings. But honest answers require careful diagnostics, transparent reporting, and a healthy skepticism for easy fixes. Next time you see a sales spike after a campaign, you’ll know how to ask—and answer—the right causal question.

For further reading, check out Scott Cunningham’s Causal Inference: The Mixtape—a hands-on, story-driven resource that inspired much of this tutorial.