http://blog.boyangyue.com/tags/causal-inference/Recent content in Causal Inference on Boyang YueHugoen-usA Good Year AheadSun, 06 Aug 2023 15:30:00 +0800http://blog.boyangyue.com/2023/08/why-experiment-wins-underdeliver/Sun, 06 Aug 2023 15:30:00 +0800http://blog.boyangyue.com/2023/08/why-experiment-wins-underdeliver/<p>A product team ships ten experiment wins in a quarter. Each one cleared the decision rule, each showed a lift on its primary metric, and no single analysis looked obviously broken. Then the quarter closes and the top-line number barely moves. That disappointment is common enough that it deserves a name.</p> <p>An A/B test is one of the few instruments a product team has that can yield genuinely causal evidence. Random assignment, not the <span class="katex"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>p</mi></mrow><annotation encoding="application/x-tex">p</annotation></semantics></math></span>-value, gives the comparison its causal interpretation: this change, on average, moved this metric for the users represented by the analysis. Statistical significance says that, if the null model and test assumptions held, a difference this large or larger would be rare. That is why experiments are so valuable, and why the quarter&rsquo;s disappointment usually traces to their limits rather than to the method itself.</p>