Ad Watchers: What Does it Take To Get Consumer Perception Surveys Right?

May 25, 2022

Consumer perception surveys are notoriously difficult to get right – from what you ask, to who you ask, to how you ask it, there is a lot of room for things to go wrong.

In this episode of Ad Watchers, hosts provide an overview of some of the biggest points to consider when creating consumer perception surveys, such as whether it has an appropriate universe and a representative sample, how to determine if you have a well-designed questionnaire, and how to use these surveys effectively at NAD. Hosts were also joined by Joel Steckel, marketing professor and the vice dean for doctoral education at NYU’s Stern School of Business, who spoke about his philosophy on survey design, the importance of how respondents interpret questions, and the most common mistakes companies make when it comes to designing surveys. 

 

___________________________________

Related Resources

National Advertising Division's Frequently Asked Questions

How Should You Present Scientific Evidence to Support Your Ad Claim?

Truth In Advertising 101

___________________________________

 

Show Notes

In this second episode of the season, hosts La Toya and Eric are joined by a special guest, Professor Joel Steckel. Professor Steckel is a Marketing Professor at the Stern School of Business at New York University and the Vice Dean for Doctoral Education. He joins in on a dynamic conversation centered on consumer perception surveys. In season one’s episode, “What Does it Mean to Have a “Reasonable Basis” Standard,” your hosts promised loyal listeners a conversation addressing reliable consumer surveys. 

La Toya and Eric fulfill this promise by beginning their discussions with a definition and intended purpose of consumer perception surveys. La Toya mentions that companies often decide to conduct this particular survey for ad development purposes. Advertisers seek to identify the appropriate messages conveyed by proposed ads via consumer perception survey results. A word of advice shared by La Toya, “It’s a good thing to do even before you get caught up in a NAD challenge or litigation.” 

Eric then moves the conversation into survey design basics. He warns that the National Advertising Divison (NAD) examines the survey design with a critical eye as it is considered evidence. Flaws found in the design deem the survey unreliable. Therefore, NAD won’t credit it.  

Moving on from design, the hosts provide a rundown of other items NAD frequently looks for when reviewing cases involving consumer perception surveys. They begin with whether or not the survey represents an appropriate universe sample. The questionnaire should also be well designed to ensure instructions and questions are not leading. NAD also reviews for the proper control cell and data analysis. The control is used to mimic a test ad that excludes the hypothesized message conveyed. Also, how the survey data is analyzed matters, and responses must be coded appropriately.  

Professor Steckel adds to the intricate conversation with three vital criteria that constitutes a consumer perception survey—construct validity, internal validity, and external validity. Professor Steckel notes that construct validity refers to whether or not the survey measures what it is intended to measure. The internal validity looks to understand if the survey tests what it was developed to test. The external validity asks, does the behavior exhibited in the survey reflect marketplace conditions?  

Professor Steckel adds that if one of these criteria could be sacrificed, it is external validity. “If a measure doesn’t measure what it’s supposed to measure or a test doesn’t test what it’s supposed to test, at least in the context of a survey, then the researcher has nothing,” Professor Steckel defines. 

He also provides further insights on La Toya and Eric’s points shared at the beginning of the episode. Professor Steckel contributes the following points on each:  

  • A sample used in a survey must represent a target universe. It doesn’t have to mirror the target universe, but it must represent it.  

  • The benefits of internet-based surveys outweigh the challenges compared to the more traditional intercept survey. 

  • Designing a survey control is both an art and a science.  

  • When designing a control, the fewer elements you can change, the better, but the elements you change have to be consistent with the allegations of deception. 

  • When reviewing the verbatim responses of a survey, remove the experts from the process. 

  • The most common mistake advertisers make in designing a consumer perception survey is eliminating pretests to cut costs. Not pretesting the survey produces a significant unknown. Advertisers won’t learn how respondents will interpret the questions, whether they can guess the survey’s purpose, who the client is, et cetera. 

La Toya leaves listeners with this final note, “Every part of your survey needs to be backed up by objective evidence. This is the correct way to test the hypothesis. It’s not just what one expert thinks is correct. It’s what the field would believe is appropriate.”  

Go to the next episode of The Ad Watchers.

Latest Podcasts

Podcast

The Future of EU-U.S. Data Transfers

On October 7, the negotiations between the U.S. and European Commission regarding the future of the data privacy frameworks behind the Privacy Shield program were completed with the release of a Presidential Executive...

Listen to the Podcast
Podcast

Ad Watchers: Where is the Line Between Ethical Design and Dark Patterns?

Think about the times you felt tricked or frustrated by a membership or subscription that had a seamless signup process but was later difficult to cancel. Something that should be simple and transparent can be complicated, intentionally or unintentionally, in ways that impair consumer choice. These are examples of dark patterns. Unfortunately,...

Listen to the Podcast
Podcast

The Metaverse Will Blur the Lines Between Physical and Online Privacy

On this episode of Privacy Abbreviated, hosts Dona and Catherine are joined by Tracy Shapiro, a privacy expert, and partner at Wilson, Sonsini, Goodrich, & Rosati. Together, they discuss the many questions related to how virtual reality will force privacy standards to evolve in the coming years. Though no one has concrete answers yet, Tracy offers her predictions on...

Listen to the Podcast
Podcast

Ad Watchers: How is Direct Selling Advertising Different?

Ad Watchers hosts are joined by Direct Selling Self-Regulatory Council Vice President, Peter Marinello, who shares with listeners how DSSRC has worked to support the direct selling industry. Listen now to understand self-regulation's impact on the market and what to expect during a direct selling self-regulatory case.

Listen to the Podcast