Success in a digital product launch: why ongoing customer testing is key

OPINION / 14th March 2019

Guerrilla usability testing is one of a wider toolkit of customer testing methods that Arriva UK Bus and ORM are working together on to deliver a best-in-class website and app. It is a rapid and effective technique used by CX experts to improve the customer experience. It typically involves testing a prototype website or app with the public or end users in situ. Using live consumer feedback, CX designers can then iterate the prototype in real time, refining the user experience.

This form of user testing can help designers to uncover unknown or unexpected issues with an app / portal / website and fix the problems early in the process.

The public who take part give their objective opinions and highlight unnecessary features, or steps in the engagement process. This type of research helps to validate (and invalidate) critical assumptions, as well as uncover other unknowns, such as the time it takes a user to complete a ‘journey’.

Guerrilla user testing is relatively inexpensive, which means it’s possible to conduct multiple tests, at a variety of locations, taking feedback from a variety of demographics.  The pros of doing guerrilla testing is that results can be fed back into the design process almost instantly.

ORM uses guerrilla testing as part of their wider customer testing programme with Arriva UK Bus app

ORM recently conducted guerrilla user testing with our client partner Arriva UK Bus’ journey planning app. We spoke with Arriva UK Bus commercial and marketing graduate, April Moores, to find out more about the test, and to get the low-down on how it helped us to improve the customer experience within the app.

What were we testing?

We were conducting guerrilla tests on our prototype journey planning app.

Where did the test take place?

We spent a day at the bus station and travel centre in Luton.

How did the test work?

We wanted to see how we could make the user journey easier for our customers within our journey-planning app. Customers were shown a variety of options, such as buttons with / without text to see which they found easier to use.

When users got stuck in the process, we’d flag it up to the designer, who would amend the app in real time, ready for the next customer test.

Did we go in with a set hypothesis or take a more open approach to testing?

Customer testing always works better in an environment when you’re not expecting any answers. This means you’re letting the customer speak, and the results reflect the customer’s voice, rather than your own.

It’s important not to premeditate what the customer is going to experience. If you have a hypothesis, say “if a customer likes this button, or this button” then you’re excluding the fact that they might not have even seen that button on the screen.

When you go out and test you must ask the customer what they like, and if they were to do a journey, how would they do it? The customer will then give you a more natural answer.

Why is it important to guerrilla test on actual customers?

In the past, researchers tested prototypes on those who were involved in the project or just those in the business. These people tended to have a biased view. So, actually going out and testing the prototype in public, with our own customers, gives us more reliable, unbiased feedback.

Can you give an example of how we incorporated the real-time feedback into the prototype?

We were testing the journey flow and whether the customer could logically navigate the journey planning function of the new app.

We tested the positioning of the buttons around planning a journey such as “location” or “stops near me”, and whether having buttons in different places made the search easier or more complicated for the customer.

Guerrilla testing 3Guerrilla testing 3

How important is it to consult with customers when developing a product that they’ll ultimately use?

Your product is ultimately there to make your customer’s life easier. So, if you haven’t tested the product, then you don’t know if your customers can use it. If they can’t or don’t use it, then ultimately the whole purpose of that product is redundant.

How many tests are we planning on conducting?

There will be four live guerilla tests of the prototype in total. Our next live customer test will be at the Travel shop in Leicester.  We wanted to go out to our Midlands-based customers to see if there are any differences between their expectations and our Luton customers’, who may be influenced by the “London effect”.

The next test after Leicester will be with our stakeholders, in an office environment. Our final test will be lab-based. We will pre-select the participants and invite them in and record their feedback.

Alongside these tests, we’ve sent a detailed survey to our customers, asking them what they’re looking for in a new app, and what they’d use our website for. Ultimately, the testing is about finding out about how customers can use our app, and the survey is being used to find out whether our customers will actually go and use the product.

We’ll combine the feedback from both forms of testing to iterate the app before it goes live later this year.

Jason Bell Jason Bell Solution Consulting Director Jason Bell