Ad Watchers: What Does it Take To Get Consumer Perception Surveys Right?
May 25, 2022
Consumer perception surveys are notoriously difficult to get right – from what you ask, to who you ask, to how you ask it, there is a lot of room for things to go wrong.
In this episode of Ad Watchers, hosts provide an overview of some of the biggest points to consider when creating consumer perception surveys, such as whether it has an appropriate universe and a representative sample, how to determine if you have a well-designed questionnaire, and how to use these surveys effectively at NAD. Hosts were also joined by Joel Steckel, marketing professor and the vice dean for doctoral education at NYU’s Stern School of Business, who spoke about his philosophy on survey design, the importance of how respondents interpret questions, and the most common mistakes companies make when it comes to designing surveys.
___________________________________
Related Resources
National Advertising Division's Frequently Asked Questions
How Should You Present Scientific Evidence to Support Your Ad Claim?
___________________________________
Show Notes
In this second episode of the season, hosts La Toya and Eric are joined by a special guest, Professor Joel Steckel. Professor Steckel is a Marketing Professor at the Stern School of Business at New York University and the Vice Dean for Doctoral Education. He joins in on a dynamic conversation centered on consumer perception surveys. In season one’s episode, “What Does it Mean to Have a “Reasonable Basis” Standard,” your hosts promised loyal listeners a conversation addressing reliable consumer surveys.
La Toya and Eric fulfill this promise by beginning their discussions with a definition and intended purpose of consumer perception surveys. La Toya mentions that companies often decide to conduct this particular survey for ad development purposes. Advertisers seek to identify the appropriate messages conveyed by proposed ads via consumer perception survey results. A word of advice shared by La Toya, “It’s a good thing to do even before you get caught up in a NAD challenge or litigation.”
Eric then moves the conversation into survey design basics. He warns that the National Advertising Divison (NAD) examines the survey design with a critical eye as it is considered evidence. Flaws found in the design deem the survey unreliable. Therefore, NAD won’t credit it.
Moving on from design, the hosts provide a rundown of other items NAD frequently looks for when reviewing cases involving consumer perception surveys. They begin with whether or not the survey represents an appropriate universe sample. The questionnaire should also be well designed to ensure instructions and questions are not leading. NAD also reviews for the proper control cell and data analysis. The control is used to mimic a test ad that excludes the hypothesized message conveyed. Also, how the survey data is analyzed matters, and responses must be coded appropriately.
Professor Steckel adds to the intricate conversation with three vital criteria that constitutes a consumer perception survey—construct validity, internal validity, and external validity. Professor Steckel notes that construct validity refers to whether or not the survey measures what it is intended to measure. The internal validity looks to understand if the survey tests what it was developed to test. The external validity asks, does the behavior exhibited in the survey reflect marketplace conditions?
Professor Steckel adds that if one of these criteria could be sacrificed, it is external validity. “If a measure doesn’t measure what it’s supposed to measure or a test doesn’t test what it’s supposed to test, at least in the context of a survey, then the researcher has nothing,” Professor Steckel defines.
He also provides further insights on La Toya and Eric’s points shared at the beginning of the episode. Professor Steckel contributes the following points on each:
A sample used in a survey must represent a target universe. It doesn’t have to mirror the target universe, but it must represent it.
The benefits of internet-based surveys outweigh the challenges compared to the more traditional intercept survey.
Designing a survey control is both an art and a science.
When designing a control, the fewer elements you can change, the better, but the elements you change have to be consistent with the allegations of deception.
When reviewing the verbatim responses of a survey, remove the experts from the process.
The most common mistake advertisers make in designing a consumer perception survey is eliminating pretests to cut costs. Not pretesting the survey produces a significant unknown. Advertisers won’t learn how respondents will interpret the questions, whether they can guess the survey’s purpose, who the client is, et cetera.
La Toya leaves listeners with this final note, “Every part of your survey needs to be backed up by objective evidence. This is the correct way to test the hypothesis. It’s not just what one expert thinks is correct. It’s what the field would believe is appropriate.”
Latest Podcasts
Likely to be Accessed: Do You Know Who Your Users Are?
Join us for this episode of Priv, where Dona Fraser is joined by Phyllis Marcus to discuss the broad operational, financial, and logistical impacts and challenges of trying to protect both children and teens online under the same laws and regulations.
Ad Watchers: Clear and Conspicuous Disclosures: Can You Read the Fine Print?
From small fonts to fast talking and distracting music, our hosts revisit this common issue area in advertising law, discuss what it takes for a disclosure to be considered clear and conspicuous, and share some lessons learned from a series of advertisements, both print and in TV commercials, that didn’t quite meet the clear and conspicuous standard.
Breaking Down AdTech: Cookies and Pixels and SDKs, Oh My!
This episode of Priv breaks down the most talked about issues in the adtech space, including the impact of the death of the cookie, the focus of regulators on the newest kid on the block - the pixel, lessons learned from recent SDK legal cases, what all of this looks like for children and teens, and what the legislative and regulatory road ahead looks like.
Ad Watchers: The best subject in advertising law: Is it puffery?
For this episode of Ad Watchers, join us for Eric’s favorite ad law topic: puffery, an exaggerated, blustering, or boastful statement or general claim that could only be understood to be an expression of opinion, not a statement of fact. But where is the line between puffery and a claim that needs a reasonable basis?