论文标题
测量在线实验中的电子商务指标变更
Measuring e-Commerce Metric Changes in Online Experiments
论文作者
论文摘要
数字技术组织通常使用在线实验（例如A/B测试）指导其产品和业务决策。在电子商务中，我们经常衡量对基于交易或项目的业务指标的变化，例如平均篮子价值（ABV），平均篮子尺寸（ABS）和平均销售价格（ASP）；然而，在实验设计和分析过程中，忽略交易/项目的价值/大小之间的依赖性仍然是一个普遍的陷阱。我们提供了有关这种依赖性的经验证据，其对测量不确定性的影响以及对A/B测试结果的实际含义，如果不明显。通过提供证据，我们希望提高电子商务实验者之间对陷阱的认识，从而鼓励采用已建立的缓解方法。当将选定的缓解方法纳入当前生产中的实验分析平台中时，我们还分享了经验教训。
Digital technology organizations routinely use online experiments (e.g. A/B tests) to guide their product and business decisions. In e-commerce, we often measure changes to transaction- or item-based business metrics such as Average Basket Value (ABV), Average Basket Size (ABS), and Average Selling Price (ASP); yet it remains a common pitfall to ignore the dependency between the value/size of transactions/items during experiment design and analysis. We present empirical evidence on such dependency, its impact on measurement uncertainty, and practical implications on A/B test outcomes if left unmitigated. By making the evidence available, we hope to drive awareness of the pitfall among experimenters in e-commerce and hence encourage the adoption of established mitigation approaches. We also share lessons learned when incorporating selected mitigation approaches into our experimentation analysis platform currently in production.
