Ad Watchers: What Does it Take To Get Consumer Perception Surveys Right?
May 25, 2022
Consumer perception surveys are notoriously difficult to get right – from what you ask, to who you ask, to how you ask it, there is a lot of room for things to go wrong.
In this episode of Ad Watchers, hosts provide an overview of some of the biggest points to consider when creating consumer perception surveys, such as whether it has an appropriate universe and a representative sample, how to determine if you have a well-designed questionnaire, and how to use these surveys effectively at NAD. Hosts were also joined by Joel Steckel, marketing professor and the vice dean for doctoral education at NYU’s Stern School of Business, who spoke about his philosophy on survey design, the importance of how respondents interpret questions, and the most common mistakes companies make when it comes to designing surveys.
___________________________________
Related Resources
National Advertising Division's Frequently Asked Questions
How Should You Present Scientific Evidence to Support Your Ad Claim?
___________________________________
Show Notes
In this second episode of the season, hosts La Toya and Eric are joined by a special guest, Professor Joel Steckel. Professor Steckel is a Marketing Professor at the Stern School of Business at New York University and the Vice Dean for Doctoral Education. He joins in on a dynamic conversation centered on consumer perception surveys. In season one’s episode, “What Does it Mean to Have a “Reasonable Basis” Standard,” your hosts promised loyal listeners a conversation addressing reliable consumer surveys.
La Toya and Eric fulfill this promise by beginning their discussions with a definition and intended purpose of consumer perception surveys. La Toya mentions that companies often decide to conduct this particular survey for ad development purposes. Advertisers seek to identify the appropriate messages conveyed by proposed ads via consumer perception survey results. A word of advice shared by La Toya, “It’s a good thing to do even before you get caught up in a NAD challenge or litigation.”
Eric then moves the conversation into survey design basics. He warns that the National Advertising Divison (NAD) examines the survey design with a critical eye as it is considered evidence. Flaws found in the design deem the survey unreliable. Therefore, NAD won’t credit it.
Moving on from design, the hosts provide a rundown of other items NAD frequently looks for when reviewing cases involving consumer perception surveys. They begin with whether or not the survey represents an appropriate universe sample. The questionnaire should also be well designed to ensure instructions and questions are not leading. NAD also reviews for the proper control cell and data analysis. The control is used to mimic a test ad that excludes the hypothesized message conveyed. Also, how the survey data is analyzed matters, and responses must be coded appropriately.
Professor Steckel adds to the intricate conversation with three vital criteria that constitutes a consumer perception survey—construct validity, internal validity, and external validity. Professor Steckel notes that construct validity refers to whether or not the survey measures what it is intended to measure. The internal validity looks to understand if the survey tests what it was developed to test. The external validity asks, does the behavior exhibited in the survey reflect marketplace conditions?
Professor Steckel adds that if one of these criteria could be sacrificed, it is external validity. “If a measure doesn’t measure what it’s supposed to measure or a test doesn’t test what it’s supposed to test, at least in the context of a survey, then the researcher has nothing,” Professor Steckel defines.
He also provides further insights on La Toya and Eric’s points shared at the beginning of the episode. Professor Steckel contributes the following points on each:
A sample used in a survey must represent a target universe. It doesn’t have to mirror the target universe, but it must represent it.
The benefits of internet-based surveys outweigh the challenges compared to the more traditional intercept survey.
Designing a survey control is both an art and a science.
When designing a control, the fewer elements you can change, the better, but the elements you change have to be consistent with the allegations of deception.
When reviewing the verbatim responses of a survey, remove the experts from the process.
The most common mistake advertisers make in designing a consumer perception survey is eliminating pretests to cut costs. Not pretesting the survey produces a significant unknown. Advertisers won’t learn how respondents will interpret the questions, whether they can guess the survey’s purpose, who the client is, et cetera.
La Toya leaves listeners with this final note, “Every part of your survey needs to be backed up by objective evidence. This is the correct way to test the hypothesis. It’s not just what one expert thinks is correct. It’s what the field would believe is appropriate.”
Latest Podcasts
Ad Watchers: Endorsements & Testimonials - So What’s New?
In this episode of Ad Watchers, our hosts discuss recent cases before the National Advertising Division (NAD) that demonstrate issues outlined in the FTC’s updated Endorsement Guides and outline the best practices you should follow to stay off the radar of the FTC and NAD.
The COPPA Rule: Proposed Changes, the Impact, & the Magic 8-Ball
Privacy for Start-Ups
With tens of thousands of entrepreneurs in the United States, how do these business leaders ensure privacy is part of any pivots or growth plans? What are the data wants vs the must haves? Priv hosts are joined by the Tech Diplomacy Network’s Katharina Koerner and Santa Clara University’s Professor Linsey Krolik to discuss the privacy questions entrepreneurs face when getting their business started.
The Government Purchase of Private Data
In this episode of Privacy Abbreviated, professor Matthew Tokson joins our hosts to discuss how the collection and sale of private data may help government agencies circumvent legal requirements.