Last year I was prompted to write a suitably festive Christmas blog. Nearly a year on the ideas are still relevant? I often get asked to look at campaign reviews and results and more often than I’d like I see people making decisions based on scant evidence. I also find that the comparison of apples and pears is quite common.
Now, I’m not a purist. I don’t put much stock in statistical significance (not for the type of reviewing I do), I find the application of common sense much more useful in these situations. But comparing two campaigns that are totally different, come on. Really?
This is what the Christmas blog was about. Comparing Christmas against anything is an exercise in futility. Yet I see it all the time.
“Why oh why did these two appeals perform so badly?” It’s another issue that comes into play as well. Why do you assume that two appeals were bad? Why not assume that one was good? Then to compound matters, often there’s only two points of comparison; say Christmas against September, or Christmas this year against Christmas last year.
Comparing this year’s sales or donations against last year’s is pointless. All it will tell you is that one figure is here and one figure is there. Usually the wheels of the retail PR train always put it in the context of this year: being up or down against last year, but don’t fall for it. This is spin.
Look at the chart below. It has the sales figures for 2016 and 2015. 2016 is less than 2015. Oh dear, we are doomed.
But let’s see what this looks like against the previous five years.
Adding in extra values and expanding the series it all that is needed to provide some context and get a much clearer view of performance.
You can see that 2016 is a clear increase on previous years, displaying a continuation of an upward trend in sales values. 2015 was an outlier – a response to a complex set of variables during that specific period, a set of variables that we can neither fully identify or replicate.
There are two valuable lessons to be learnt here.
The first, and most obvious, is that you should never compare just two figures or outputs. Invariably it will paint an incorrect picture. All data, stats, facts and figures need context and this comes from widening out the view and setting your information within its wider landscape.
My second point is more subtle. Think of Christmas like Las Vegas: What happens in Christmas, stays in Christmas. This is really important. On the one hand I encourage you to set your data into its landscape to provide context but you need to ensure the landscape is the correct one. By this I mean that the figures that you are trying to evaluate need to be compared to, and given context from similar data. You wouldn’t dream of comparing the sales data of breakfast cereals and Mercedes Benz, likewise don’t compare Christmas with Easter or Summer, all of which represent specific times of the year each with its own unique outlook and reasons for giving. It’s not just about Christmas either, think about how a campaign might differ from those it’s being compared to.
I fear many a sales or fundraising strategy has been founded on both or either of these two fundamental mistakes. Inevitably the strategy will fail and so more data is pulled together and presented back, and will include the same mistakes. And so we go on.