https://news.ycombinator.com/item?id=28821498 hn_throwaway_99 10 hours ago [–] I've also found very similar issues with extremely "data driven" organizations that live and die by A/B test performance. It's not that there is anything fundamentally wrong with the science behind A/B testing, it's just that individuals are incentivized to run tests on things that are easily measured. Things where results will take a long time to show, or things that may be initially disruptive but then beneficial, are discounted. I see this theme in many, many areas: business, sports, politics, academia, etc. When we have tons of data and a desire to make things as "objective" as possible, it's easy to get stuck in homogeneous "local maxima" because we just grade by the things that are easiest to measure. reply
Comments