Note: I did not A/B test the experiment covered in the Mixpanel article. I did just release an A/B testing framework (A/Bingo) today. You can find it here:
1) I run tests until I get statistically significant results. How long that takes is outside the scope of this comment, except to say that if you're testing something with a high conversion rate and testing two very different alternatives you can get significant results in ~250 users or so. This lets me do very rapid iterating. Changes in my core interaction can be A/B tested in under a week, which works well for me (I do most development on weekends).
2) I run lots of tests in parallel. For why that is not as bad an idea as you might think it is, see the A/Bingo FAQ.
3) That is a core design consideration for A/Bingo. I obsessively track users to make sure they always get the same alternative. A/Bingo is the only Rails A/B testing framework that does this, to my knowledge.
http://news.ycombinator.com/item?id=764508
1) I run tests until I get statistically significant results. How long that takes is outside the scope of this comment, except to say that if you're testing something with a high conversion rate and testing two very different alternatives you can get significant results in ~250 users or so. This lets me do very rapid iterating. Changes in my core interaction can be A/B tested in under a week, which works well for me (I do most development on weekends).
2) I run lots of tests in parallel. For why that is not as bad an idea as you might think it is, see the A/Bingo FAQ.
3) That is a core design consideration for A/Bingo. I obsessively track users to make sure they always get the same alternative. A/Bingo is the only Rails A/B testing framework that does this, to my knowledge.