Thanks for this great article and thanks for sharing the code! Methinks I'll be installing it sometime in the next week.
Quick questions about your A/B testing...
* What kind of user volumes are you looking to do A/B testing on? How does that relate to how long each of your tests runs/will run?
* Are you running several A/B tests in parallel?
* Do you make each user an A or B, or do you keep the A/B testing page-based? I'm assuming the former, since otherwise the same user might see both sides of the test, which is not desirable.
Note: I did not A/B test the experiment covered in the Mixpanel article. I did just release an A/B testing framework (A/Bingo) today. You can find it here:
1) I run tests until I get statistically significant results. How long that takes is outside the scope of this comment, except to say that if you're testing something with a high conversion rate and testing two very different alternatives you can get significant results in ~250 users or so. This lets me do very rapid iterating. Changes in my core interaction can be A/B tested in under a week, which works well for me (I do most development on weekends).
2) I run lots of tests in parallel. For why that is not as bad an idea as you might think it is, see the A/Bingo FAQ.
3) That is a core design consideration for A/Bingo. I obsessively track users to make sure they always get the same alternative. A/Bingo is the only Rails A/B testing framework that does this, to my knowledge.
Quick questions about your A/B testing...
* What kind of user volumes are you looking to do A/B testing on? How does that relate to how long each of your tests runs/will run?
* Are you running several A/B tests in parallel?
* Do you make each user an A or B, or do you keep the A/B testing page-based? I'm assuming the former, since otherwise the same user might see both sides of the test, which is not desirable.