# Measuring Incrementality in Social: Uplift, Holdouts, and Bandits

Attribution dashboards are descriptive, not causal. To answer 'what would have happened otherwise?' you need disciplined experimentation. Here is a practitioner’s guide that your marketing, RevOps, and data teams can run together.

---

## 1) Design Principles
- Start with a narrow, testable question (e.g., 'Does evidence-first structure increase qualified demos?')
- Define exposure rules (who sees content) and outcome windows (e.g., 7/14 days)
- Pre-register success metrics and guardrails before posting
- Prefer simpler designs with strong logging over fragile, complex setups

---

## 2) Experiment Frameworks
- Time-split holdout: alternate test/control posting windows; measure downstream conversions
- Geography/segment holdout: prioritize by region or persona lists where spillover is minimal
- CUPED: variance reduction using pre-period outcomes to increase power
- Uplift modeling: causal forests or two-model approach to target segments with higher treatment effect
- Bandits (e.g., Thompson sampling): adaptively allocate impressions across structures as evidence accumulates

---

## 3) Instrumentation and Data
- Canonical post IDs and variant IDs; persistent topic/structure tags
- UTM hygiene: campaign/medium/content reflect topic and structure
- Session stitching: resolve social session to site session and CRM event
- Outcome definitions: MQL-qualified meeting, self-serve signup, assisted pipeline within 14 days

Warehouse tables to confirm readiness: fct_post, fct_interaction, fct_session, fct_conversion.

---

## 4) Bandit Loop (Pseudo-Code)
    initialize priors for K structures (Beta/Normal as appropriate)
    for each publish opportunity t:
      sample theta_k ~ posterior_k for all k
      choose structure = argmax(theta_k)
      publish post with chosen structure
      observe reward r (e.g., qualified clicks or assisted conversions)
      update posterior for chosen structure with r
    weekly: freeze posterior snapshot → report → retire low performers

Reward choices: proxy (qualified click), intermediate (profile clicks with dwell), hard (assisted conversions).

---

## 5) Analysis and Reporting
- Primary: difference-in-differences between test/control windows with CUPED
- Secondary: uplift segments (ICP vs. non-ICP, seniority tiers)
- Robustness: placebo tests, permutation checks; flag Simpson’s paradox risks
- Decision: promote structures with positive lift and stable risk metrics

---

## 6) Guardrails and Ethics
- Frequency caps and rate limits to avoid user fatigue
- No dark patterns; keep summaries in-post to reduce bounce risk
- Respect privacy; redact customer data in visuals

---

## 7) Weekly Operating Plan
- Mon: choose 2–3 structures to pit via bandit; define thresholds
- Tue–Thu: run posts; maintain comment/expansion ladders
- Fri: analyze CUPED results; update priors; promote winners
- Always-on: log learnings to the snippet and asset library

Outcome: lift becomes measurable, repeatable, and explainable. The organization learns quickly without sacrificing rigor.

How to quantify causal lift from social programs using holdouts, CUPED, causal forests, and multi-armed bandits—plus a weekly operating plan that balances rigor with speed.

Having trouble with engagement?