Skip to main content

Experiments, bandits, and holdouts

Experiments measure whether flag variations cause a meaningful outcome. Open Project → Experiments → Create experiment, choose an environment and flag, then define the decision before looking at results.

Experiment design​

Configure:

  • a name and falsifiable hypothesis;
  • the flag rule that assigns treatments;
  • randomization context kind and bucket attribute;
  • control and treatment variations;
  • audience allocation and treatment split;
  • primary metric, secondary metrics, and guardrails;
  • Bayesian or frequentist analysis settings, thresholds, and measurement windows;
  • optional mutual-exclusion layer, holdout, attributes, and reshuffling behavior.

Statuses are draft, ready, running, stopped, completed, and archived. Iterations preserve treatment, allocation, metric, and analysis configuration at the time an experiment ran. Start a new iteration instead of rewriting a completed result.

The Results view reports exposures, sample sizes, credible/confidence intervals, lift, probability of being best, and health warnings. The Exposures view helps diagnose missing evaluation events or sample-ratio mismatch. Do not declare a winner when the sample, event instrumentation, or guardrail health is incomplete.

Multi-armed bandits​

Bandits dynamically shift traffic toward better-performing variations while retaining exploration traffic. Use them for continuous optimization when opportunity cost matters more than a fixed-horizon causal estimate. Store the reward metric, minimum exploration rate, update cadence, eligibility rule, and winning-treatment behavior in Project → Experiments → Multi-armed bandits.

Holdouts​

A holdout reserves a stable group that does not receive one or more treatments, allowing long-term cumulative impact measurement. Define the context kind, bucket attribute, percentage, included flags/experiments, and active window. Mutual-exclusion layers prevent one context from entering competing experiments; a holdout measures the broader combined effect.

Migration behavior​

LaunchDarkly experiment metadata, iterations, treatments, allocations, metrics, secondary metrics, analysis configuration, status, and raw provider payload are synced. Rerunning sync updates the source record by provider ID without creating a second experiment.