Why We Don’t Test on Windows Phones

Recently there have been some really good comments on my Responsive Mobile UI Split-Test: Icons or Text? post. Over the weekend a commenter named Yorke said:

Also might be worthwhile doing a quick broad device test to see how it renders across the ever changing continuum of device specs. On a quick test on a windows phone it doesn’t render well.

I replied:

We have not been testing on Windows phone to be honest. It represents 0.25% of our visitors in 2014. We monitor our user base and try to test only on the devices/browsers that a substantial portion of our users use.

(I wrote specifically about which devices we test on and why in my post Browser & Usability Testing…Without Going Insane.)

If the goal was to build a perfect website that functions beautifully on all devices, then yes, we would absolutely have a Windows Phone and test on it.

However, that’s not the goal. The goal is to build a great business. It would be silly, maybe even insane, to spend time testing and refining on a device that represents 0.25% of our market when there are countless other things we could be doing to improve user experience for the majority of our customers, drive more traffic, increase conversion rates, increase efficiencies, and/or increase profitability.

It’s nothing personal. I actually think Windows Phones are pretty cool and innovative. I also think Ubuntu is cool, but I stopped testing on it when I realized none of our customers use it. Curiosity did get the best of me and I spent a few minutes using the Windows Phone emulation in Internet Explorer, and everything looked/functioned fine. That’s as far as I’m willing to go for now.

If our Windows Phone user base grows over time, then we will start testing on actual devices, although I wouldn’t bet on it, especially now that Nokia is releasing Android phones.

3 comments on Why We Don’t Test on Windows Phones

  1. JD says:

    Maybe Windows Phone users don’t go to your site because it doesn’t work well for them. Also, if you think Microsoft is going to advance the FORKED Android project beyond low level devices, then your assessment of the entire mobile market is deserving of scrutiny. You probably would have exclusively catered to Blackberry several years ago. Look at growth trends to see what’s “next” and be a leader. Leadership is where greatness lies. Following the herd as a “yet another” will give you some short term gains if you can beat all the rest of the competing crumb snatchers. What do I know? IBM, Blackberry and GM are still on top because they were once too dominant to fall, right?

    • Adam McFarland says:

      Thanks for reading JD.

      Maybe Windows Phone users don’t go to your site because it doesn’t work well for them.

      It’s possible, although I doubt it. We literally have never heard one question/comment/complaint from a Windows user, when I tested with the emulator everything worked fine, and even given our small sample size the pages/visit and average visit duration are within normal ranges. If you have a screenshot or specific UI/UX functionality that does not work properly, please let me know.

      Also, if you think Microsoft is going to advance the FORKED Android project beyond low level devices, then your assessment of the entire mobile market is deserving of scrutiny.

      It’s a pretty popular topic in the tech community these past few weeks. When Nokia is fully absorbed into Microsoft, they may kill it or may continue it. We don’t know. The fact that Nokia saw the need to fork Android at the low end should raise some red flags. They have low end Windows devices that would work in emerging markets.

      Look at growth trends to see what’s “next” and be a leader.

      If you read my “Browser & Usability Testing…Without Going Insane” article I linked to above, you’ll see that we do that. Windows Phone is likely not what’s “next”.

      IBM, Blackberry and GM are still on top because they were once too dominant to fall, right?

      We watch trends within the industry and look at our stats. If Windows Phone or Firefox OS or Ubuntu Phone or something else gains significant market share then we’ll start testing with it. Until then, we’ll prioritize projects that will benefit the majority of our user base.

  2. Tim says:

    I think another thing worth exploring is AOV based on platform, OS and browser. No business, or at least any I’ve ever seen or heard of, have unlimited resources (time and qualified personnel) to have a perfect world for all devices, you focus on the most profitable first and spread the rest of your resources thin to accommodate as many as possible – any other strategy is pure madness and will not lead to success.

Leave a Reply

Your email address will not be published. Required fields are marked *

Commenting Rules

I'm honored that you found this post interesting enough to leave a comment. Before posting, I have a few ground rules:

  • Please keep your comments as relevant to the post as possible.
  • No personal attacks or any other nastiness.
  • Your first comment is subject to my approval.

Thanks!