Trustworthy Online Controlled Experiments 1st edition by Ron Kohavi, Diane Tang, Ya Xu – Ebook PDF Instant Download/Delivery: 1108724264 , 978-1108724265
Full download Trustworthy Online Controlled Experiments 1st edition after payment
Product details:
ISBN 10: 1108724264
ISBN 13: 978-1108724265
Author: Ron Kohavi, Diane Tang, Ya Xu
Getting numbers is easy; getting numbers you can trust is hard. This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests.
“A/B testing is the gold standard of creating verifiable and repeatable experiments, and this book is its definitive text” — Steve Blank, father of modern entrepreneurship, author of The Startup Owner’s Manual and The Four Steps to the Epiphany
“This book is a great resource for executives, leaders, researchers or engineers looking to use online controlled experiments” — Harry Shum, Executive Vice President, Microsoft Artificial Intelligence and Research Group
“A great book that is both rigorous and accessible. Readers will learn how to bring trustworthy controlled experiments, which have revolutionized internet product development, to their organizations” — Adam D’Angelo, Co-founder and CEO of Quora and prior CTO of Facebook
“Kohavi, Tang and Xu have a wealth of experience and excellent advice to convey, so the book has lots of practical real world examples and lessons learned over many years of the application of these techniques at scale.” — Jeff Dean, Google Senior Fellow, and SVP, Google Research
“The secret sauce for a successful online business is experimentation. But it is a secret no longer. Here three masters of the art describe the ABCs of A/B testing so that you too can continuously improve your online services.” — Hal Varian, Chief Economist, Google, and author of Intermediate Microeconomics: A Modern Approach
“This is the new bible of how to get from data to decisions in the digital age.” — Scott Cook, Intuit Co-founder & Chairman of the executive committee
Based on practical experiences at companies that each run more than 20,000 controlled experiments a year, the authors share examples, pitfalls, and advice for students and industry professionals getting started with experiments, plus deeper dives into advanced topics for practitioners who want to improve the way they make data-driven decisions. Learn how to
- Use the scientific method to evaluate hypotheses using controlled experiments.
- Define key metrics and ideally an Overall Evaluation Criterion.
- Test for trustworthiness of the results and alert experimentersto violated assumptions.
- Build a scalable platform that lowers the marginal cost of experiments close to zero.
- Avoid pitfalls like carryover effects and Twyman’s law * Understand how statistical issues play out in practice.
Trustworthy Online Controlled Experiments 1st Table of contents:
Acknowledgments
Part I Introductory Topics for Everyone
1 Introduction and Motivation
Online Controlled Experiments Terminology
Why Experiment? Correlations, Causality, and Trustworthiness
Necessary Ingredients for Running Useful Controlled Experiments
Tenets
Tenet 1: The Organization Wants to Make Data-Driven Decisions and Has Formalized an OEC
Tenet 2: The Organization Is Willing to Invest in the Infrastructure and Tests to Run Controlled Experiments and Ensure That Their Results Are Trustworthy
Tenet 3: The Organization Recognizes That It Is Poor at Assessing the Value of Ideas
Improvements over Time
Google Ads Example
Bing Relevance Example
Bing Ads Example
Examples of Interesting Online Controlled Experiments
UI Example: 41 Shades of Blue
Making an Offer at the Right Time
Personalized Recommendations
Speed Matters a LOT
Malware Reduction
Backend Changes
Strategy, Tactics, and Their Relationship to Experiments
Scenario 1: You Have a Business Strategy and You Have a Product with Enough Users to Experiment
Scenario 2: You Have a Product, You Have a Strategy, but the Results Suggest That You Need to Consider a Pivot
Additional Reading
2 Running and Analyzing Experiments
Setting up the Example
Hypothesis Testing: Establishing Statistical Significance
Designing the Experiment
Running the Experiment and Getting Data
Interpreting the Results
From Results to Decisions
3 Twyman’s Law and Experimentation Trustworthiness
Misinterpretation of the Statistical Results
Lack of Statistical Power
Misinterpreting p-values
Peeking at p-values
Multiple Hypothesis Tests
Confidence Intervals
Threats to Internal Validity
Violations of SUTVA
Survivorship Bias
Intention-to-Treat
Sample Ratio Mismatch (SRM)
Threats to External Validity
Primacy Effects
Novelty Effects
Detecting Primacy and Novelty Effects
Segment Differences
Segmented View of a Metric
Segmented View of the Treatment Effect (Heterogeneous Treatment Effect)
Analysis by Segments Impacted by Treatment Can Mislead
Simpson’s Paradox
Encourage Healthy Skepticism
4 Experimentation Platform and Culture
Experimentation Maturity Models
Leadership
Process
Build vs. Buy
Can an External Platform Provide the Functionality You Need?
What Would the Cost Be to Build Your Own?
What’s the Trajectory of Your Experimentation Needs?
Do You Need to Integrate into Your System’s Configuration and Deployment Methods?
Infrastructure and Tools
Experiment Definition, Set-up, and Management
Experiment Deployment
Experiment Instrumentation
Scaling Experimentation: Digging into Variant Assignment
Single-Layer Method
Concurrent Experiments
Experimentation Analytics
Part II Selected Topics for Everyone
5 Speed Matters
Key Assumption: Local Linear Approximation
How to Measure Website Performance
The Slowdown Experiment Design
Impact of Different Page Elements Differs
Extreme Results
6 Organizational Metrics
Metrics Taxonomy
Formulating Metrics: Principles and Techniques
Evaluating Metrics
Evolving Metrics
Additional Resources
SIDEBAR: Guardrail Metrics
SIDEBAR: Gameability
7 Metrics for Experimentation and the Overall Evaluation Criterion
From Business Metrics to Metrics Appropriate for Experimentation
Combining Key Metrics into an OEC
Example: OEC for E-mail at Amazon
Example: OEC for Bing’s Search Engine
Goodhart’s Law, Campbell’s Law, and the Lucas Critique
8 Institutional Memory and Meta-Analysis
What Is Institutional Memory?
Why Is Institutional Memory Useful?
9 Ethics in Controlled Experiments
Background
Risk
Benefits
Provide Choices
Data Collection
Culture and Processes
SIDEBAR: User Identifiers
Part III Complementary and Alternative Techniques to Controlled Experiments
10 Complementary Techniques
The Space of Complementary Techniques
Logs-based Analysis
Human Evaluation
User Experience Research (UER)
Focus Groups
Surveys
External Data
Putting It All Together
11 Observational Causal Studies
When Controlled Experiments Are Not Possible
Designs for Observational Causal Studies
Interrupted Time Series
Interleaved Experiments
Regression Discontinuity Design
Instrumented Variables (IV) and Natural Experiments
Propensity Score Matching
Difference in Differences
Pitfalls
SIDEBAR: Refuted Observational Causal Studies
Part IV Advanced Topics for Building an Experimentation Platform
12 Client-Side Experiments
Differences between Server and Client Side
Difference #1: Release Process
Difference #2: Data Communication between Client and Server
Implications for Experiments
Implication #1: Anticipate Changes Early and Parameterize
Implication #2: Expect a Delayed Logging and Effective Starting Time
Implication #3: Create a Failsafe to Handle Offline or Startup Cases
Implication #4: Triggered Analysis May Need Client-Side Experiment Assignment Tracking
Implication #5: Track Important Guardrails on Device and App Level Health
Implication #6: Monitor Overall App Release through Quasi-experimental Methods
Implication #7: Watch Out for Multiple Devices/Platforms and Interactions between Them
Conclusions
13 Instrumentation
Client-Side vs. Server-Side Instrumentation
Processing Logs from Multiple Sources
Culture of Instrumentation
14 Choosing a Randomization Unit
Randomization Unit and Analysis Unit
User-level Randomization
15 Ramping Experiment Exposure: Trading Off Speed, Quality, and Risk
What Is Ramping?
SQR Ramping Framework
Four Ramp Phases
Ramp Phase One: Pre-MPR
Ramp Phase Two: MPR
Ramp Phase Three: Post-MPR
Ramp Phase Four: Long-Term Holdout or Replication
Post Final Ramp
16 Scaling Experiment Analyses
Data Processing
Data Computation
Results Summary and Visualization
Part V Advanced Topics for Analyzing Experiments
17 The Statistics behind Online Controlled Experiments
Two-Sample t-Test
p-Value and Confidence Interval
Normality Assumption
Type I/II Errors and Power
Bias
Multiple Testing
Fisher’s Meta-analysis
18 Variance Estimation and Improved Sensitivity: Pitfalls and Solutions
Common Pitfalls
Delta vs. Delta %
Ratio Metrics. When Analysis Unit Is Different from Experiment Unit
Outliers
Improving Sensitivity
Variance of Other Statistics
19 The A/A Test
Why A/A Tests?
Example 1: Analysis Unit Differs from Randomization Unit
Example 2: Optimizely Encouraged Stopping When Results Were Statistically Significant
Example 3: Browser Redirects
Example 4: Unequal Percentages
Example 5: Hardware Differences
How to Run A/A Tests
When the A/A Test Fails
20 Triggering for Improved Sensitivity
Examples of Triggering
Example 1: Intentional Partial Exposure
Example 2: Conditional Exposure
Example 3: Coverage Increase
Example 4: Coverage Change
Example 5: Counterfactual Triggering for Machine Learning Models
A Numerical Example (Kohavi, Longbotham et al. )
Optimal and Conservative Triggering
Overall Treatment Effect
Example 1
Example 2
Trustworthy Triggering
Common Pitfalls
Pitfall 1: Experimenting on Tiny Segments That Are Hard to Generalize
Pitfall 2: A Triggered User Is Not Properly Triggered for the Remaining Experiment Duration
Pitfall 3: Performance Impact of Counterfactual Logging
Open Questions
Question 1: Triggering Unit
Question 2: Plotting Metrics over Time
21 Sample Ratio Mismatch and Other Trust-Related Guardrail Metrics
Sample Ratio Mismatch
Scenario 1
Scenario 2
SRM Causes
Debugging SRMs
Other Trust-Related Guardrail Metrics
22 Leakage and Interference between Variants
Examples
Direct Connections
Indirect Connections
Some Practical Solutions
Rule-of-Thumb: Ecosystem Value of an Action
Isolation
Edge-Level Analysis
Detecting and Monitoring Interference
23 Measuring Long-Term Treatment Effects
What Are Long-Term Effects?
Reasons the Treatment Effect May Differ between Short-Term and Long-Term
Why Measure Long-Term Effects?
Long-Running Experiments
Alternative Methods for Long-Running Experiments
Method #1: Cohort Analysis
Method #2: Post-Period Analysis
Method #3: Time-Staggered Treatments
Method #4: Holdback and Reverse Experiment
References
Index
People also search for Trustworthy Online Controlled Experiments 1st :
trustworthy online controlled experiments free pdf
trustworthy online controlled experiments 1st edition
trustworthy online controlled experiments summary
trustworthy online controlled experiments pdf free download
trustworthy online controlled experiments five puzzling outcomes explained
Tags: Ron Kohavi, Diane Tang, Ya Xu, Trustworthy Online, Controlled Experiments