Mozilla Marketing
Multivariate testing to improve Firefox conversion
Here at Mozilla, we’re always trying to tweak messaging, graphics, layouts, etc. that are found throughout Mozilla.com and Mozilla’s other web properties (add-ons, spreadfirefox, etc.). Rarely are these changes done just because we think it’ll make things look pretty. Usually we have a good hypothesis that we want to test to improve a user’s experience or increase conversion (usually for Mozilla.com this is equivalent to a download of Firefox) and we go about setting up a test, looking at the effect these changes have on conversion over a certain period of time.
In the past we haven’t had a very good mechanism for doing this type of testing. So today we’re trying a new service that allows us to roll-out tests without placing a large additional workload on the webdev or IT groups. Our first order of business is to rerun a test that we tried earlier this summer with very inconclusive results (screenshots are shown below). The goal is to see what effect (if any) these alternate layouts have on conversion. In industry analytic language, this is called an A/B test.
The new capabilities of the service we’re taking for a test-drive actually go a step further than simple A/B tests; it’s called multivariate testing. In a nutshell, multivariate testing tests the effect that different elements on a page may have on conversion. So instead of page A vs. page B, you can test the effect that different elements on a page may contribute to conversion (e.g. moving a graphic, changing the size of a button, modifying the copy, etc.). Cool stuff.
I’ll share the results of these tests in future posts.
The Current Firefox download page (a.k.a. control page)
Creative Test Page #2 (this is a long version — we are also testing a shorter version which looks very similar)
Discussion
4 comments for “Multivariate testing to improve Firefox conversion”
Post a comment
Categories
Hi,
I love the simpleness of the first Test Page but the second Test Page looks more serious/respectable and informative.
[...] we’ve been testing some variations of the US Firefox download page. You can read about the details in-depth here, but our goal was pretty basic — increase the download conversion rate (individuals that [...]
The 3rd page is very good. But the original Download Firefox button is the best. The page could have included more awards received by Firefox.
[...] we’ll be doing fairly straight-forward A/B testing. I’ve written about this type of website optimization before, but the basic mechanics of this test are to run different versions of the download page [...]