Very interesting. Thanks for sharing the insights!
Would've it made more sense to separate this testing out to a different instance of your product? This would've probably helped distinguish between real and bounty users.
From this perspective, that makes total sense. We thought about it, but spinning up a dedicated replica would mean duplicating most of our infra, and I’ve always been hesitant about that.. it’s hard to guarantee it behaves exactly like production. Bugs sometimes hide in those edge differences.
But yes, I agree even a simple flag (“tester”) on accounts would have gone a long way. At least then we could separate metrics and clean up the noise afterwards without burning so much time. Definitely something I’d do differently if we ever tried again.
Very interesting. Thanks for sharing the insights!
Would've it made more sense to separate this testing out to a different instance of your product? This would've probably helped distinguish between real and bounty users.
From this perspective, that makes total sense. We thought about it, but spinning up a dedicated replica would mean duplicating most of our infra, and I’ve always been hesitant about that.. it’s hard to guarantee it behaves exactly like production. Bugs sometimes hide in those edge differences.
But yes, I agree even a simple flag (“tester”) on accounts would have gone a long way. At least then we could separate metrics and clean up the noise afterwards without burning so much time. Definitely something I’d do differently if we ever tried again.