Ad Watchers: What Does it Take To Get Consumer Perception Surveys Right?

May 25, 2022

Consumer perception surveys are notoriously difficult to get right – from what you ask, to who you ask, to how you ask it, there is a lot of room for things to go wrong.

In this episode of Ad Watchers, hosts provide an overview of some of the biggest points to consider when creating consumer perception surveys, such as whether it has an appropriate universe and a representative sample, how to determine if you have a well-designed questionnaire, and how to use these surveys effectively at NAD. Hosts were also joined by Joel Steckel, marketing professor and the vice dean for doctoral education at NYU’s Stern School of Business, who spoke about his philosophy on survey design, the importance of how respondents interpret questions, and the most common mistakes companies make when it comes to designing surveys. 



Related Resources

National Advertising Division's Frequently Asked Questions

How Should You Present Scientific Evidence to Support Your Ad Claim?

Truth In Advertising 101



Show Notes

In this second episode of the season, hosts La Toya and Eric are joined by a special guest, Professor Joel Steckel. Professor Steckel is a Marketing Professor at the Stern School of Business at New York University and the Vice Dean for Doctoral Education. He joins in on a dynamic conversation centered on consumer perception surveys. In season one’s episode, “What Does it Mean to Have a “Reasonable Basis” Standard,” your hosts promised loyal listeners a conversation addressing reliable consumer surveys. 

La Toya and Eric fulfill this promise by beginning their discussions with a definition and intended purpose of consumer perception surveys. La Toya mentions that companies often decide to conduct this particular survey for ad development purposes. Advertisers seek to identify the appropriate messages conveyed by proposed ads via consumer perception survey results. A word of advice shared by La Toya, “It’s a good thing to do even before you get caught up in a NAD challenge or litigation.” 

Eric then moves the conversation into survey design basics. He warns that the National Advertising Divison (NAD) examines the survey design with a critical eye as it is considered evidence. Flaws found in the design deem the survey unreliable. Therefore, NAD won’t credit it.  

Moving on from design, the hosts provide a rundown of other items NAD frequently looks for when reviewing cases involving consumer perception surveys. They begin with whether or not the survey represents an appropriate universe sample. The questionnaire should also be well designed to ensure instructions and questions are not leading. NAD also reviews for the proper control cell and data analysis. The control is used to mimic a test ad that excludes the hypothesized message conveyed. Also, how the survey data is analyzed matters, and responses must be coded appropriately.  

Professor Steckel adds to the intricate conversation with three vital criteria that constitutes a consumer perception survey—construct validity, internal validity, and external validity. Professor Steckel notes that construct validity refers to whether or not the survey measures what it is intended to measure. The internal validity looks to understand if the survey tests what it was developed to test. The external validity asks, does the behavior exhibited in the survey reflect marketplace conditions?  

Professor Steckel adds that if one of these criteria could be sacrificed, it is external validity. “If a measure doesn’t measure what it’s supposed to measure or a test doesn’t test what it’s supposed to test, at least in the context of a survey, then the researcher has nothing,” Professor Steckel defines. 

He also provides further insights on La Toya and Eric’s points shared at the beginning of the episode. Professor Steckel contributes the following points on each:  

  • A sample used in a survey must represent a target universe. It doesn’t have to mirror the target universe, but it must represent it.  

  • The benefits of internet-based surveys outweigh the challenges compared to the more traditional intercept survey. 

  • Designing a survey control is both an art and a science.  

  • When designing a control, the fewer elements you can change, the better, but the elements you change have to be consistent with the allegations of deception. 

  • When reviewing the verbatim responses of a survey, remove the experts from the process. 

  • The most common mistake advertisers make in designing a consumer perception survey is eliminating pretests to cut costs. Not pretesting the survey produces a significant unknown. Advertisers won’t learn how respondents will interpret the questions, whether they can guess the survey’s purpose, who the client is, et cetera. 

La Toya leaves listeners with this final note, “Every part of your survey needs to be backed up by objective evidence. This is the correct way to test the hypothesis. It’s not just what one expert thinks is correct. It’s what the field would believe is appropriate.”  

Go to the next episode of The Ad Watchers.

Latest Podcasts


Consumer Privacy in Telehealth: An Interview with the ATA

In this episode of Priv, Dona Fraser is joined by Kyle Zebley from the American Telemedicine Association (ATA) to get a check-up on consumer health data privacy in the telehealth industry. From HIPAA to the pandemic to Dobbs to a hodge podge of new state-level privacy laws, Dona and Kyle discuss the companies navigating this complex terrain, how the world of telehealth has changed,...

Listen to the Podcast

Ad Watchers: Endorsements & Testimonials - So What’s New?

In this episode of Ad Watchers, our hosts discuss recent cases before the National Advertising Division (NAD) that demonstrate issues outlined in the FTC’s updated Endorsement Guides and outline the best practices you should follow to stay off the radar of the FTC and NAD.

Listen to the Podcast

The COPPA Rule: Proposed Changes, the Impact, & the Magic 8-Ball

Priv host Dona Fraser is joined by SIIA Vice President, Education & Children’s Policy, Sara Kloek, and CARU Director, Rukiya Bonner, to discuss the COPPA Rule review and how we got here, what the proposed changes mean, the potential impacts of these changes for businesses and Safe Harbors, as well as some predictions on the road ahead.
Listen to the Podcast

Privacy for Start-Ups

With tens of thousands of entrepreneurs in the United States, how do these business leaders ensure privacy is part of any pivots or growth plans? What are the data wants vs the must haves? Priv hosts are joined by the Tech Diplomacy Network’s Katharina Koerner and Santa Clara University’s Professor Linsey Krolik to discuss the privacy questions entrepreneurs face when getting their business started.

    Listen to the Podcast