Last night, I published the results of my eagerly awaited jazz-sushi survey, where I attempted to find whether or not there was a correlation between liking jazz and liking sushi.
If you have not already done so you can read it here. (Feel free to skip the numbers bit if that stuff bores you.)
So I proved that a correlation existed and made it into a law and all is well. Well not quite. I should come clean about something. It didn’t really prove anything and I’ll explain why.
Firstly, (as many people pointed out), my questions asked for Yes/No answers to complex questions. There are lots of different types of jazz and usually someone doesn’t like or dislike all of them. My survey forced them to interpret the question as they saw fit. Worse, it caused people to give me long-winded answers which I had to interpret.
Why’s that worse? Well I knew what I wanted the outcome of my survey to be and while I didn’t consciously seek to influence the results in this manner, I am hardly in the best position to be a neutral judge.
Also as @mapsadaisical quickly pointed out, I had a self-selecting sample. This means people were free to choose whether to take part or not. Why is that bad? Well people knew that I was trying to find a correlation between people who liked jazz and people who liked sushi. When people know what is trying to be proven it influences whether they respond or not.
On Friday night I did my sums and found that there was a correlation but it was not significant enough to prove anything one way or the other. I explained this on Twitter and asked for some more responses. Of the next 12 responses 11 were either likes both or likes neither. This wasn’t coincidence, it was simply people wanting to help me show a correlation. Those who did like both or neither kindly though “I’ll help you out.”
Another example of this came when I was watching a morning day time TV show a few years ago. It was GMTV, or Anne & Nick or Richard & Judy or some bollocks, and they had a phone in poll. A phone in poll is even worse for this problem than Twitter because the effort of making the call is greater and they charge you money for doing so. You aren’t going to bother voting unless you have some compelling reason to do so.
The poll asked people to vote on whether or not they were currently in an abusive relationship. About 50% said yes. At no point did the programme mention that the surprisingly high result could be influenced by the fact that this poll was much more important to someone in an abusive relationship and they were therefore more compelled to vote than someone who wasn’t. In fairness to the programme they didn’t try to conclude that 50% of all relationships were abusive.
There is another problem with the way in which I gathered the stats. Even if everyone who saw the question had responded, I didn’t survey a proper cross-section of the public. Supposing I did a poll on Twitter to find out whether people thought Social Networking sites were a good thing. I would certainly get a higher proportion saying Yes than if I stopped people in the street and asked. Although there is no obvious reason for people who use Twitter to have different views on jazz/sushi to the public at large, the whole experiment was to find a correlation between two seemingly unrelated things so really I should have excluded any other similarities between the respondents.
A good example of this is in the polls which newspapers do on their online websites. If the Daily Mail asks a question about immigration on its website is the response going to reflect the views of the country at large? Probably not, because people who read the Daily Mail website are likely to have different views on immigration than the average person on the street.
You should treat with skepticism any survey that can’t show clearly how it gathered and interpreted its data to avoid external factors like this affecting the results. Companies like Ipsos MORI go to huge lengths to try to minimise these problems. I didn’t and as such you should just interpret my survey as a bit of fun.
RedEaredRabbit
