How to prioritise 3,000 pages? Start with data

In my current role as Digital Strategy Lead at Citizens Advice, prioritisation is one of the most complex issues I face on a near daily basis. I was inspired and driven to find the best, most efficient way of dealing with it.

Citizens Advice is for everyone. We offer advice on a huge range of issues, to a vast number of people. Our digital service includes over 3,000 pages of advice content that are visited by millions of people every year, often in very difficult circumstances.

Our job as a digital team is to help many more people get the advice they need online — and help our advisers help people in person. In practice, this means making sure our digital advice is as practical and actionable as possible. But with so many thousands of pages, it is a big challenge knowing what to prioritise. We want to make sure we are focusing our limited resources where people most need our help.

Initially, we based our priorities on page visits and searches as a pretty good indicator of the advice people were looking for. But this meant manually crunching lots of data, and it didn’t give us any idea of whether people were getting what they need from our advice. We wanted to see if we could find a more user-centred, automated way of prioritising our work to help us make decisions quickly, accurately and based on performance.

That’s when I came up with the idea to automate it. I took my idea to our wonderful Data Scientist, who worked his inimitable magic on my concept, and with that “Backlogger” was born!

How we identified pages that needed work

Our first priority is solving people’s problems. So we added the simple question — “Did this advice help?” — to every page. There’s also a feedback box for people to let us know if anything on the page needs fixing or changing.

This helped us work out what advice people were finding useful, and where we needed to improve.

We also want to help as many people as possible. So we combined the results of this satisfaction measure with the number of visits to pages. That’s visits from both the public and our advisers.

The results told us what our priorities should be, and we built a tool to visualise them.

Using data to decide priorities:

Here’s what Backlogger looks like:

The colour and position of the dots help us decide what to do with each page:

Need to improve: high number of visits and lower satisfaction — these pages are the next to be improved.

Need to rethink: lower number of visits and lower satisfaction — maybe we don’t need these pages any more, or we need a different approach.

Keep going: high satisfaction and lower visits — we can leave these pages alone or do a lot less work on them or make small improvements.

Doing great: high number of visits and high satisfaction — looking good!

We never want to stop improving our advice so the red quarter will never be empty. As improved pages move into the green half, the tool reassesses all the pages. The tool gets stricter with its criteria for a green page, so the quality improves over time.

How backlogger helps us make better decisions:

We called the tool Backlogger because we’re using it to inform our ‘backlog’ of priorities — essentially it helps us decide what to do and in what order.

Quick testing with content and data teams improved the tool — for example, by adding a search function to look up specific pages.

Already, the tool has transformed the way we make decisions about content. We know which areas of advice we need to work on next, and how big redesigns need to be. Some areas need a complete re-think, some just need a tidy up.

By using the tool we’ve learnt that not every area of content needs a full rethink. This will save us a lot of time. And we’re making decisions more confidently now we’ve got real time data to guide us.

Rebecca’s working on a section of our website about problems where you live. She said:

“I used the tool to see how the pages within the section were performing. The page called ‘Problems in your local environment’ was getting low satisfaction scores, which had made the whole section red even though the other pages were doing better. I focused my research on the local environment content and realised that people were going there from Google hoping for help with nature and green issues. So I’m completely rethinking that page — especially the title — but spending less time on the others in the section”.

What's next?

We’re training the whole team to use the tool and get more feedback. And we’re also looking at adding extra data, for example how complex a page is.

When we tested the tool we realised we’d find it useful to know which pages aren’t included — it only shows pages that people have visited and given feedback on. It’s useful to have a list of these so we can look into why we don’t have a score for them — perhaps they’re so long that people aren’t making it to part of the page with the survey on. So we’ll look at collecting data directly from our content management system so we can see any pages that have no score.

We think Backlogger could be helpful for other teams grappling with complex website design or content projects. If you’d like to hear more or talk to us about it, or if you have any feedback or ideas for how we could improve it, please get in touch.

Feature image credit: Alex Anderson