Instructor: Kyle Carlson

Experiments are commonly used in software and internet businesses to evaluate product changes, marketing strategies, and other decisions. Expertise in applying experiments in this context is valuable skill that this course will help students develop. We will cover fundamentals of causality, experimental design, and statistical analysis. In addition we will devote special attention to the particular challenges of experiments in a software business environment. We will analyze how these challenges relate to the theory of causality and assumptions of statistical approaches. Students will learn to build intuition about these problems using simulation.

Course Goals

Students should learn the following by completing the course:


The course will be graded based on the following components:


You will lose 10% of this grade for every class period you miss. And yes, that can turn negative (e.g. if you miss 11 class periods, your attendance score is -10%).

Lecture plan

Class schedule

The course roughly splits in half. The first half focuses on establishing frameworks for thinking about experiments rigorously. In the second half we apply those frameworks to understand the complications of experimentation in practice.

Date Topics
8/19 Monday Lecture 1 - Causality fundamentals
8/26 (Canceled)  
8/30 Friday Lecture 2 - Uncertainty
9/2 (Canceled: Labor Day)  
9/9 Monday Lecture 3 - Estimation
9/16 Monday Lecture 4 - Complications in practice: Bias
9/23 Monday Lecture 5 - Complications in practice: Inference
9/30 Monday Lecture 6 - Decision-making
10/7 Monday Lecture 7 - Review
10/14 Monday Lecture 8 - TBD

Class outlines

Lecture 1 - Causality fundamentals

Lecture 2 - Uncertainty

Lecture 3 - Estimation

Lecture 4 - Complications in practice: Biased estimates

Lecture 5 - Complications in practice: Inference

Lecture 6 - Experiments and economic decision-making

Lecture 7 - Review

Lecture 8 - TBD

Additional materials

Reference books

Edifying readings