Where BBC News developers blog about responsive design.

Opinions expressed on this blog are those of the individual contributors, and are not necessarily those of the BBC as a whole.

Responsive News Testing

A year ago I wrote a post about the mobile browsers, operating systems and devices that I used to test the responsive news codebase. After updating the original post 5 times, I thought a retrospective on the past year would show what I have learned, what new hardware and software make up our test suite and some tools to help in the process.

We launched the News site a year ago, adding features and making our way to a fully responsive desktop site. We have been focusing on mobile, but recently on tablet features. This has prompted a lot more tablet testing. This post will talk about everything mobile and lead right up to the point before desktop browsers.

I have decided to keep desktop testing separate as there is a lot more automation testing involved in that area for News. For that reason, I will follow this post up with another post about News’s desktop browser automation and testing.

My go-to devices

If testing responsive sites all day, you’ll probably have a group of devices, your favourites, your special ones, the ones that you wouldn’t go on without. These are based on several factors, News stats, general market share and browsers that non UK users are using. This ensures the coverage is always up to date, but also that we don’t miss a key user group. I approach the day by picking up a collection of feature phones, smartphones, tablets and lots of chargers. For me, these devices include the usual suspects, Nokia feature phones, Nokia feature phones masquerading as smartphones, Galaxy Android phones, Blackberries, Windows Phone and an iPhone. Originally I had a list of just a few devices, the minimum I called it, but actually, the minimum is bigger now. This is mainly due to more choice in the market, but also lingering browsers that just won’t seem to go away. We will keep the same principals as before, browser requirements and hardware requirements, this should cover everything from IE’s to Chrome Android, but also 240px phones to phones with a pixel ratio of 3.

This is achieved with the following devices: Nokia N96 (Symbian 9.4), Nokia C3 (Nokia Xpress browser), Galaxy S2 (Android 2.3), Galaxy S4 (Android 4.2 & Chrome), iPhone 5 (Safari 6), Blackberry 9900 (BB7 Webkit browser), HTC 8X (IE10). I don’t really have retail GS4, but have used prototypes with this high pixel density display.

With these browsers and hardware, you can test pixel ratio of 1, 2 and 3, resolutions from 240x320px to 1080x1920px, all major mobile browsers, but first party and third and have different input methods, such as trackpad and 5 way input. This currently represents my go-to devices, except the S4, but that is on the way. These 7 devices are the ones most likely to show any issues with your site given the browsers and hardware differences.

Browsers

In general, we should concern ourselves with browsers rather than hardware, so although we talk about Android and iOS, it is better to talk about Chrome, IE, Firefox, Safari, Opera and AOSP(Android Open Source Project) browser. One of the more interesting developments is the separation of browser and operating system. Anything with Android 4 can have Chrome, which means a world of terrible, underpowered phones are using Chrome, that opens up a lot of interesting testing use cases for performance. At the moment, we have high powered phones running browsers with lacklustre support, but also low powered phones running great browsers. This should be taken into account when testing performance. Right now, our browser support list is small compared the number of devices, we test on Chrome, Safari, ASOP 2-4, Opera Mobile & Mini, Nokia Ovi, Symbian, Dolphin, Blackberry 6-10, and Firefox.

Here is a break down of just browsers used on Android, the majority of users stick the AOSP browser, but more and more are switching to Chrome. Android has the most variety of third party browsers, but it is also important to note that some OEM’s maybe alter the AOSP browser slighty.

Year to Year

 we look at the same list last year, it is relatively similar, though we did cut some devices. We found that there were big trends last year, Blackberry 4 browsers dropped off the map, so we dropped testing for them altogether. Blackberry 5 and AOSP 2.1 have also seen steep declines, we actually see more android 1.5 traffic than AOSP 2.1, though this is probably due to our global audience. Windows Phone 7 with IE9, it came, it went, we let it die. Our stats have shown WP8 with IE10 to be the browser that is worth investing into, having better support and features for new browsers standards. So, what have we seen dying a death, Blackberry 4 & 5 browsers, AOSP 2.1 & 2.2, IE9 and S60 9.4 browsers.

Although these browsers are going away, we have seen new ones hit the market, with better support and better performance. A quick run down of browsers launched in the last 12 months: Chrome for Android and iOS, IE 10, Blackberry 10, Opera Webkit, iOS 6, Firefox and Android 4. I included Android 4 at the end as over the last 12 months, many Android 3 tablets got the 4.0 update, bringing the AOSP(stock) browser to many users.

Big phones and tablets

That’s right, phablets, 5 inch phones, whatever you call them, they are here to stay. With lots of the newer phones set to come with 1080p displays, this type of device should be a household item in the next few months. I would expect most of you to buy one of these, simply because the Galaxy S4 will probably be the most purchase phone of all time.

This leads me on to tablets, a year ago was a different story completely. The iPad 3 had just come out, the playbook was a flop, Android tablets were just starting to get Android 4, but there was one other good tablet, the Kindle Fire. This tablet was cheap, came with Android 2.3, was cheap, was 7 inches, had a new browser called Silk, and did I mention is was cheap? With only 2 tablets having market share, it was actually quite easy to test tablets a year ago, but tablets is where we have seen the biggest change in test resources. The Nexus 7, Galaxy tab 2 7, Kindle Fire HD, Kindle Fire 8.9, iPad Mini, Nexus 10 and Microsoft Surface have all come out, with 7 inch tablets being the more interesting devices.

There isn’t a set size, or resolution, in fact, the Kindle Fire, Fire HD, Nexus 7 and iPad mini all have different report sizes, resolutions and different physical sizes. This is where having devices really pays off, especially given that Amazon uses Silk, Nexus uses Chrome and iOS uses Safari. Apple were also forced to maintain the aspect and resolution of the iPad, but also keeps the UA string. This makes separating the smaller version from the larger impossible. My go to tablets, Nexus 7, iPad & MS Surface.

Further testing

In this section last time, I talked about what I felt would offer more in-depth browser testing. When added to our go-to devices, we have the following:

  • Nokia N96 (Symbian 9.4)
  • Nokia C3 (Nokia Xpress browser)
  • Galaxy S (AOSP 2.2)
  • Galaxy S2 (AOSP 2.3)
  • Galaxy S3 (AOSP 4.0 & Chrome)
  • Galaxy S4 (AOSP 4.2 & Chrome)
  • iPhone 5 (Safari 6)
  • Blackberry 9700 (BB6 Webkit browser)
  • Blackberry 9900 (BB7 Webkit browser)
  • Blackberry Z10 (BB 10)
  • HTC 8X (IE10)
  • Nexus 7 (Chrome)
  • Nexus 10 (Chrome)
  • Motorola Xoom – (AOSP 3.2)
  • Kindle Fire (Silk)
  • iPad Mini (Safari)
  • iPad (Safari 6)
  • Microsoft Surface (IE10)

You can see the inclusion of some extra AOSP versions and Blackberry 10. The reason BB10 wasn’t included before is because it has limited release so far, hasn’t show much growth and has yet to prove it will become and established browser. Having used it, the BB10 browser is one of the better browsers available, so don’t feel the need to go buy one, your local phone store should give you enough hands on time. In total, my go-to devices number 10, 7 phones, 3 tablets, going up to 18 for a more broad testing suite.

Tools

Each of the following are used for testing, some tools are BBC specific, so I will link to components to make up the tools.

  • PhantomJS
  • Imagemagick
  • Image diffing
  • WEINRE
  • Remote Preview
  • Chrome & iOS USB debugging
  • Skia debugger
  • Telemetry
  • YSLOW
  • IE VM’s

For the full background to all the browsers and what the differences are, check out my original post

Our Projects Our Events Our Jobs

Ashley theme by Jxnblk