Mozilla Marketing
The What’s New Page test results are in
In a previous post, I described a program that was designed to test a new approach to the page that users see when they update to a new version of Firefox — we call it the “What’s New Page”. Well, the results are in, and they reveal some interesting things about what’s interesting to Firefox users and the potential education value of the page.
I’ve included a slide that illustrates how the test was setup and the corresponding resulting click-thru paths. In brief, test page 1 was the clear winner, encouraging nearly 3.5 times more users to learn more about add-ons.
The Test Results
In addition, the test revealed some interesting things about the likelihood to download from one of the click-thru pages (bottom row in the graphic); over 14% of the time, a user that landed there downloaded an add-on on-the-spot, right from the page.
So why do these numbers really matter?
Based on traffic for this test, these results would translate into an additional increase of over 200,000 incremental add-on downloads over the current page (control) over a one week period after the Firefox update. In other words, implementing a page similar to “test page 1″ could greatly increase awareness and ultimately, drive a huge number of add-on trials.
Some other interesting results and observations, in brief:
- User perception of updates to Firefox may generally be neutral, or even negative, but we can make it a more positive experience by offering relevant product suggestions, news, and tips about how using Firefox can help lead to a more productive and enjoyable online experience
- Clear organization and messaging about the value of add-ons encourages trial. This test helps validate the great work that the add-ons team has done in thinking about how to re-organize AMO to be more consumer friendly, searchable, and helpful to those that may not be as familiar with customizing their Firefox browser. Check out the new mockups that will be rolling out soon.
There are more tests like this one on the horizon. And there are more interesting issues to tackle — how do we make the offer on the What’s New page fresh and relevant to our users, how do we scale this internationally, and so on.
Discussion
12 comments for “The What’s New Page test results are in”
Post a comment
Categories
[...] check the full story here [...]
The current dutch update page tells you can install the dutch dictionary. This confused my dad so much that he thought it was required to reinstall the dutch dictionary each time firefox wend trough a minor update.
Confusions like that might be something you want to keep in mind when designing the update page.
Interesting! Please keep sharing your research findings, this is great stuff.
One question, however… I noticed that the winning test page had 2 options: 1. Customize it now; and 2) No thanks, I’m just here for the internet.
Does the 8.4% click through rate include those who said “No Thanks?” or did 8.4% say “Customize it Now?”
sorry,8.2% I mean…
Nice page, but I think you’ve chosen an odd set of add-ons to promote so highly.
I would suggest choosing some general categories, giving the top one or two addons from those categories, and inviting the user to look at more from those categories. For example:
Adblock Plus, perhaps repagination, and “See more add-ons that make the web less annoying”
Foxmarks, perhaps stumbleupon, “see more add-ons that work with online services”.
It’s All Text, FireGPG, “see more add-ons that work with programs on your system”.
Customize Google, perhaps Firefox companion for eBay, “see more add-ons that customize a particular website”.
Web Developer, perhaps Firebug, and “See more add-ons for web developers”.
Download Statusbar, Fission, “see more add-ons that change the look of the browser”.
It’s great to see the work being done in optimizing the various different parts of the Mozilla website. It will all help to increase Firefox’s growth rate, and IMNSHO we can’t be rid of IE6 too soon!
Actually, FoxyTunes would probably make another good entry in the “add-ons that work with programs on your system” category, consider its popularity.
Re: Jeremy’s question above — the 8.2% are folks that clicked “Customize it Now” and arrived at the secondary page.
I have serious doubt about your result.
What you did is answering a question about how successful the page can be to *push* users to go to addons page.
The question that remains is – is that a real goal of the update note page?
Is the goal to push them to addons? Do you really believe that in general it’s a good thing to push so many people to addons?
I really doubt that. I didn’t do deep studies on the subject, but from my experience in working on Mozilla communities it seems that addons are designed for minority. Trying to force it for majority reminds me testing Linux configuration that better promotes the usage of the shell for users.
Addons are for hi-tech Internet users who’ll always should stay minority. Firefox should reach majority with default setup and allow those who want to go further. What you can achieve by exposing addons more is that people who are not the target will investigate addons use and they may even fall into a rabbit hole that’s way deeper than they want the browser to be. (“I did something, the browser is different now, and I don’t know how to do this, and now I’m afraid of touching anything”)
I’d like to second Staszek Malolepszy here. This page should ENSURE users that they are SAFE now. That should be the loudest, clearest message on the screen. They risked clicking “yes” on the update screen. They may be worried about what’s happening now (yes, that’s the common picture of my friends seeing Update alert and asking me what to do). People who see this page are people who literally *risked* clicking “yes”. We should calm them down. Tell them that what happened made them safe, secure. Congratulate them on the decision and braveness and make them feel confident that they made a good choice.
THEN we may propose them to do anything more with their browser, but I’d like to make sure we don’t get addons to the wrong target group so I’d probably say sth like “If you want to make your internet experience richer, you can customize and empower your browser with addons” – so that those tiny, little, Joe Average kind users can safely run away without having to risk more.
From your screen 1 the message is *totally* different, not to say *opposite*. You ignore the message about update. It’s complete, nothing to talk about, move on.
Now you scream to them that “it’s time to do more” – I can see with my soul eyes my friends, my mum, my girlfriend seeing this call to “do more” and running away.
Sorry, for me you’re going to make them scared. I don’t know if they will run away from Firefox, probably no, I don’t know if it will influence they memory connection map with the word Firefox and relations to Safe, Scary, Difficult, Easy etc. But I’d guess it may. And even if you’ll get more addons downloaded, the overall result will be wrong.
This message is great for experienced Internet users, who feel “safe” while browsing, who know what they do, who are reading the messages instead of panicking on each new screen – to minority of users.
Please, think twice before you do the swap. Test elder users, less experienced. Not the ones who installs Firefox because they find it cool, but the ones who got Firefox installed by their son or friend. Or even made it on their own and believe that installing Firefox was the most important, difficult and terrifying thing they did on the Internet.
I’d prefer not to make them not want to click the “Update” button next time…
Compare this with the ‘What’s new’ release notes for the latest OpenOffice linked to from their download page:
http://development.openoffice.org/releases/2.3.0.html
where exactly do these tests take place?
Oh, and here is staszek’s blog post about the Update page: http://smalolepszy.wordpress.com/2007/09/20/redesigning-the-firefox-update-page/