General marketing

Recap: A More Dynamic Experience

By DR ⋅ April 23, 2009 ⋅  Print This Post ⋅ Post a comment

In a previous post, I wrote about the details of a test we were going to conduct on the Firefox download page.  To recap, the goals of the test were two-fold: improve the visit-to-download conversion rate, and provide a better user experience for site visitors by displaying relevant information about Firefox based on their browser type.  We’ve done A/B testing before to try and improve metrics like conversion rate, but this is the first time we’ve incorporated browser detection into the mix in order to display more relevant information to Firefox.com visitors.

Making Converts

Most users that visit the Firefox download page that don’t have Firefox arrive with Internet Explorer.  Since IE makes up the majority of visits, we created a page specifically addressing IE users, with information about Firefox vs. IE features, reasons to switch, and of course a prominent download button.  The end result was that over the week-long test period, the test page improved visit-to-download conversion by 0.3% over the incumbent page at a statistically significant level (for you stats geeks out there, that’s at a 99% Confidence Level where z=~4).  To put this in perspective, for the en-US page specifically, that type of improvement would net about a quarter of a million new downloads per year.  More testing is on the way to try and continue to improve this result.  In the meantime, if you visit the Firefox download page and you have Internet Explorer in the US, you’ll get this new “control” page.

Content Just For You, Awesome Firefox User

One of the key insights prior to this test was the discovery that a huge percentage of users that already have Firefox (about 30%!)  end up on the Firefox download page.  From looking at some anonymous click-stream data in conjunction with funnelcake experiments, a large number of these users seem to be confused about how to update their Firefox.  Once again, browser detection to the rescue (or so the theory of the test goes)!

So we presented two more test pages for users that visited with Firefox–one for those with the latest flavor of version 3, and a page for those that visited with a version older than 3.  The first page (we’ll refer to it as the “Upgrade” page), focused on encouraging users to upgrade and prominently featuring tips and tricks.  The second page (we’ll refer to it as the “Get Personal” page), encouraged users with the latest and greatest version of Firefox (good job!) to explore the world of browser customization and get involved with the Mozilla community.

To measure how well these two pages performed against our current page (our well known Firefox download page, a.k.a. the “control”), we used bounce-rate, defined as single page views/entry pages as our main metric.  In other words, compared to the control page, did users see one of these new test pages and indicate an interest in learning more by clicking on links for customization,  tips and tricks, or by upgrading their browser?  Or did they see the page, and with an unsatisfied feeling, “bounce” away somewhere else on the web?

It turns out that presenting visitors with relevant content actually seems to work! The bounce rate for the Upgrade Page bested the control by 4% (the lower the number for bounce rate, the better) and for the personalize page by 18%. In other words, people were more likely to find the information they were looking for with one of the test pages vs. our old friend, the control page.

Since the test concluded, we’ve implemented these test pages (“Upgrade” and “Get Personal”) for the en-US locale.  And we’re already incorporating these pages into the plans for Firefox 3.5 launch based on the findings of this test.  So look for even more relevant content on Mozilla.com in the future.

Tags: ab testing, bounce rate, browser detection, download conversion, dynamic content, firefox marketing, Internet Explorer, personalization, web analytics, Website Optimization