We’ve finally crossed the line.

After almost 4 years, 5000 pull requests from 50 contributors, we’ve migrated the BBC News site to a totally new stack. We now have a single code base for all devices, in 30 different languages thanks to RWD.

How can it possibly take 4 years? Let’s start from the beginning …

The BBC News site is big. Not just in terms of the number of pages and micro sites but the sheer number of stakeholders involved. We started with a small team overseen by Matt Chadburn in 2011, and thanks to his guidance we moved fast. The old mobile site which we replaced, was not much better than WAP and had minimal interest from the stakeholders. We were free to innovate and release with little interference from outside the team. It was like a startup without the bean bags.

We were lucky to have people who were comfortable taking risks. People like Julian Kirby and Tom Maslen broke the mould and defined new approaches. One example being “Cuts The Mustard” which at the time was totally new and is now common knowledge in our industry.

Based on a User Agent lookup from a DB we were able to redirect device types to different domains. We used this to incrementally move device types to the new site. Starting from old school feature phones we built the basic story and index pages.

Looking back, using Mobile First was genius. We were able to strip everything back to the core content, the stuff that mattered to users. No JavaScript. No cruft. Just the good stuff. Journalism at its best.

Feature phones were also a small part of our traffic, just around 2%. This allowed us to innovate with low risk. We simply couldn’t take the same risks with BBC News as we can with other products, it’s an institution.

Progressive Enhancement

As we built up features and confidence we redirected all smartphone users to the new site. At this point we started to layer on JavaScript aka progressive enhancement. Performance was always a key driver for us. We made hard decisions to ensure we kept page weight low and JS enhancement quick. This is where “cuts the mustard” came in. We were able to check the ability of the JS engine and only load the enhanced experience if it was a modern browser. Older browsers with poor rendering got a simple but content complete experience.

CTM freed us from jQuery and other large frameworks, we just used HTML and native JS. We were also hostile to other teams front end components, we built stuff from scratch where ever we could unless they met our high standard. Another reason was backend API calls in the PHP stack. We built the site on the existing BBC PHP framework that allowed you to pull in modules from other teams at run time - they could do anything. Such as making the page uncacheable and slowing the request by making a barmy number of API calls.

This led us to another key decision. Based on our limited servers and the load on News, each PHP request should only make 1 API call, even with Memcache. This however makes it difficult to build rich page with data from multiple sources. However resilience was for obvious reasons, pretty important. We got round this by loading components client side. Core components that had core content were always baked into the page, but other stuff like “Stories from this section” were pulled in with JS and were cached by Varnish separately.

So onto Varnish. Thank goodness for that. We’ll eventually be moving to cloud compute, but for now its traditional in-house stuff. These servers also serve up iPlayer, Sport, homepage and everything from CBBC to Gardening. With News traffic, every page needs to be cached at the edge to protect our front-end servers. This affected how we architected pages, all personalisation was client side, everyone got the same core page from cache. If you CTM we then enhance the experience on top, pulling in further components with XHR. For the core experience we’d simply link to another page with the same content on.

The slow down

As the Responsive News team became integrated with the desktop team and world service, things started to slow. We gained more people and far more stakeholders. Scaling up was painful and for the few of us who started it all, it felt like things were changing for the worse. We lost a few of of our core team and we were dealt some rough tasks to add features to the old desktop site (still serving 90% of users). For example, we were asked to build a new video experience, oh and by the way please make it work in the old static published site with velocity and SSI templates! In retrospect we should have made a harsh decision to stop development on the old site. It sucked resources and morale and that cost us dearly by delaying our strategic move to “Responsive News”. We should have carefully considered expensive projects that were not aligned with our strategic technical direction.

If you haven’t heard about the BBC World Service, go Google it. It’s the jewel in our crown and delivers the only native unbiased journalism available in some parts of the world. Previously every one of the 30 language sites had a separate site. As WS became integrated, we used the Responsive News code base to power them instead. This was a massive undertaking, WS use a different CMS and there’s plenty of front-end challenges including right to left and custom script fonts such as Burmese. The World Service team did a great job of updating the code base to support multi languages from a completely different CMS. I’m still amazed at how well it has worked.

One downside of Mobile First is you can end up waiting until the end to solve some of the most difficult problems. How do you support a wide navigation and mobile style nav on the same page? How do you load a wide experience while maintaing a perfomant mobile experience? What do you do about IE<=8? The lesson we leant here was to start from mobile but get to desktop quickly, even if its for one page type. You can always iterate! One area that slowed our progress was advertising. The adverts were integrated far too late and parts of the UX had to change to fit them in for users outside of the UK. It’s actually quite a balance getting a page looking good with and without adverts, especially when the adverts are not responsive.

Getting over the line

A fresh product owner team prompted us to take stock and inject some much needed direction and fresh commitment to the Responsive News project - we didn’t touch the old site again, and guess what? No one complained.

We didn’t want to just lift and shift the old desktop site. Our product owner worked hard to only migrate the components that added value. Unfortunately our stats system didn’t give us great data. Many of the decisions became subjective and a lot of negotiation with journalists whose reputation was affected by their position on the front page. When it comes down to it, all of the hard decisions were about that front page. The position of a story on the front page makes a huge difference to it’s click rate. The page is longer than I’d like, but we’ve made hard decisions to remove as much cruft as we can.

Last November we added an opt-in banner for Tablets and finally opt-in for desktops in January. We used a survey and focus groups to get as much information from our users as we could. The feedback made a big difference to our end product and helped set users’ expectations of what was coming, something we leant from our previous update.

Whats next? We’ve built a solid foundation and the next few months will see us integrate linked data powered topic pages and further personalisation, to create a product that provides news that users want, when they want it.

We got a lot right but there’s a whole host of things we’d do differently - I’ll leave that for another post.

So hopefully the new site is on www.bbc.co.uk/news and now that the old site is gone we can move forward and iterate further.

This has been a massive investment from a lot of people. I’m especially tired, unfortunately no time to rest - it’s Election time. News never stops.

-cleve