Written by on . Pegasus.

Measuring impact and a defence of ‘vanity’ metrics

Sam Hughes outlines how we use best practice to measure the impact of our campaigns

‘Impact Week’ is an agency-wide initiative to reinforce good practice for impact measurement across our teams, and to ensure we continue proving to our clients, and ourselves, that the work we do is making a genuine difference.

Here are some key insights from this year’s Impact Week:

  1. Showing a change happened is not the same as proving impact

Behaviour change is at the core of Pegasus’ mission to inspire healthy decisions. We build our communications to encourage specific behaviours and overcome barriers to them. For us, having  ‘impact’ means changing behaviour.

But we must ask some questions to work out not only whether there has been a change, but whether our communications caused it. For instance, if sales increased, was it definitely our target audience buying? Did our communications specifically influence these purchases, and if so which channels and messages were most effective?

Essentially, measuring impact is about being honest in assessing the role that our communications played in effecting a positive change.

  1. Decide how and what to measure before a campaign goes live

It’s critical to understand how we will measure impact before a campaign goes live because sometimes it’s only possible to see impact by comparing pre- and post-campaign measurements.

For instance, the key behavioural objective for our  ‘Small Talk Saves Lives’ campaign with the Samaritans, Network Rail and the British Transport Police was to encourage members of the public taking the train to use small talk (such as conversations about the weather) to speak to any fellow passengers in distress. This behaviour fed into the campaign objective to reduce suicides on the rail network.

To prove our communications had the desired impact, we conducted pre-campaign research with National Rail to determine public attitudes and intent towards approaching someone who might be in distress. This enabled us to then take the same measurements after the campaign to show that public intent had increased and, using Network Rail data, that the number of public interventions had increased by 20 per cent. Coupled with recall metrics, proof of impact becomes much clearer.

Importantly, without first identifying a specific behaviour to change, and without measuring a benchmark before the campaign went live, we wouldn’t have been able to prove that it was our campaign that made the difference.

  1. ‘Vanity’ metrics might simply be metrics used incorrectly

Proving campaign impact is like telling a story. A key behavioural measure can demonstrate an impact on behaviour, but other metrics show how and why this happened.

Metrics perceived as needlessly big and showy, such as page views, email opens, or traffic volume are increasingly criticised as ‘vanity metrics’. But this is only true if they’re presented as singular proof of a campaign’s success. When measured and analysed alongside other metrics they become useful in telling the story of an impact.

A case in point is the British Government claiming success by hitting a target of 100,000 daily coronavirus tests. It has been criticised because the 100,000 target tell the story of what impact testing may have. It doesn’t take into account who is being tested or why.

Testing aims to identify people who should self-isolate, as well as track the geographic spread of the virus. This will enable the Government to focus resources more effectively to combat the outbreak. But by concentrating just on the volume of tests, regardless of region or occupation, the government has lost sight of the impact tests could have in the fight against coronavirus, turning a valuable metric into a vanity metric.

  1. At Pegasus we take impact seriously

We have a dedicated Insights team to develop and execute measurement projects across the qualitative and quantitative spectrum. We can build bespoke surveys, utilise existing feedback options on platforms such as Facebook and YouTube, and gain a deeper understanding of online behaviour via Brandwatch, Google Analytics and YouGov Profiles.

These tools, alongside close collaboration with our clients, are vital to demonstrating that we are inspiring healthy decisions and proving that our campaigns deliver genuine impact.