The real problem with F1 surveys

Another F1 season, another F1 fan survey – this time from the GPDA. Since survey design is my day job, Todd asked me to take a look at the survey and report my thoughts back to FBC. Consider this like watching a race with Charsley, only slightly slower moving and about collecting information. At least surveys don’t have DRS, so we have that going for us.

Before we get start, a bit of disclaimer: I don’t know who did this survey for GPDA but I’m guessing it is a marketing research firm, as opposed to a survey research firm which in and of itself, isn’t inherently good or bad, but the points I mention below are often afterthoughts in the world of market research. Market researchers aren’t necessarily concerned with collecting data in a scientific manner, but quick turnaround and pulse point surveys. In other words, this survey will tell the GPDA something but that something is antidotal and is not generalizable to the larger population of Formula 1 fans and we as data consumers need to know that because they are neither concerned nor will they disclose it.

How do you plan to use the data collected from this survey to make an informed decision?

This should always be your first question, because knowing why the data is collected informs every other component in survey design. If you take a look at the GPDA’s survey it is all over the place. It asks questions about favorite teams, preferred method for watching races, and how much you think General Admission tickets should cost. What is the plan for this data when it comes in? What are they comparing? What decisions are they interested in making at the end of the day?

Unfortunately, people often ask questions first and then figure out what they want when they see the data or only see gathering information as a checkbox required to move forward with a project. This is getting worse as we become a more data driven society. Don’t get me wrong – I heart data – but having your mechanic give you a survey and then asking you to mark all ‘excellent’ so he or she can get a bigger bonus, is not the scientific use of a survey. Or what does a respondent’s dedication to Ferrari or a love of Alex Wurz, have to do with their preference for watching races online or randomly introduced sprinklers? It is important to have a plan for how the data will be used before it is collected, instead of just throwing the kitchen sink at it and waiting to see what sticks. That is not how science works and is probably an inefficient use of one’s money.

How are you collecting the data?

A web survey on a commercial website that allows for multiple entries by a single individual is not a scientific sample. Frankly, I’m not sure it even counts as informative but it is important to keep this point in mind. The GPDA survey measures the opinions of individuals who found out about the survey, went to motorsport.com website, and completed the survey – that does not include all F1 fans by a long shot. But that in and of itself isn’t what’s important – what’s important is how those individuals captured differ from the individuals that your data does not capture. So even if one does generalize to a larger population, what does that population represent: British F1 fans? Hard core F1 fans? What about people who go to motorsport.com vs Autosport, or ESPN? Are these different kinds of fans? What about people without Internet access? (yes, these people do exist) Or if they don’t speak English? Yes, a scientific random sample of Formula 1 fans would be ideal, but probably cost prohibitive, so the question is really about can one live with this compromise. Does the mode of data collection and the people you are collecting the information from meet your original goal?

This also brings up the important points of people who start the survey but don’t finish it. This survey is long, the questions are unfocused, and all the questions require answers. People just don’t care enough or have enough energy to answer this many questions, even about a sport they love and the survey provided a progress indicator to boot so people know exactly how much longer this survey is going to take. The question format is also very tedious and cognitively burdensome. A better answer is to narrow the survey down to a few key constructs and measure those constructs well and instead, what they’ve done is forced survey participants into giving easy answers or not attending to items in order to get the damn thing over with. Or worse yet, they just quit the survey. Bad data IS NOT better than any data and I’d be hard pressed to think the GPDA has gotten much use out of this exercise. Of course, that wouldn’t stop people from reporting out their findings but when that happens, I’ll be here for you.

 

 

 

0 0 votes
Article Rating
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

10 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments

Paul KieferJr

Given that I really have no knowledge of the science of surveys, how would you design this survey? What questions would you ask, how would you go about collecting the data, and how would you interpret it?

Bacon Wrapped Sushi

Interesting article and very true about this survey. I actually went and filled it out. I think it took me nearly 20-30 minutes to do, and was all over the place. As a passionate fan, I felt it was my responsibility to finish it no matter what, but I can definitely see many people bailing. You bring up excellent points about the type of audience who fills this out as well as “Ask questions first, figure it out later” mantra, as this survey definitely was that.

Alianora La Canta

Tracking favourite team/driver data allows patterns to be seen that help indicate what affects a fan’s viewpoint. It’s like tracking demographic data in a general survey – there are things that can’t and/or shouldn’t be done with it, but it allows the rest of the data to make more sense. If fans of big teams want customer cars and fans of small teams don’t, that shows where the likely upsides and downsides are going to be – or at least which opinion has the best marketing… I’m not entirely convinced it’s “ask questions first, decide the purpose later” – if… Read more »

MIE

Grace, have you looked at the Haymarket magazines (AUTOSPORT, F1 Racing, etc) survey? It’s yes / no / not applicable tick boxes, rather than a sliding scale. There is one free text box for technical regulation changes you would like to see.
I worry about yes/no answers, as the last survey asked if we wanted more overtaking, and that was interpreted as DRS.

runnah

You can have the best data in the world but if its interpreted by knuckleheads you won’t get good conclusions.

jakobusvdl

Thanks Grace, its good to get a bit of insight from an expert.
Taking a cynical view, if the survey is unfocused, does that give the GPDA’s more of a free rein to pick out the bits they like and use them to promote whatever causes they prefer?

Are Buntz

Thanks Grace.

“Bad data IS NOT better than any data”

@_canuck_

I quit the gpda survey less the 50% done, It was the same type of survey the Fia or F1 did a few years ago that gave us drs. The Magazine survey had better questions so i completed that one.

Fred Talmadge

Asking me prepared questions that have anticipated answers and possibly
leading questions to the result you want are not in my interest. If you want to sit down and have an open discussion then I’m open to that.

PM

I think the first point hits the nail on the head. What is the objective of the survey? Are the changes that fans propose actually going to be implemented? Highly unlikely. At best, this survey is going to lead to some sort of push in how f1 CONTENT is delivered since I feel that was the most relevant question to ask the fans. At worst, its going to be used as by the f1 politicians to push their own agenda when the survey happens to align with their wishes for the sport.