Case study
SAN
tests winning advertisements with
AI-powered creative testing from MetrixLab
Business challenge
The Advertiser Jury Foundation (Stichting Adverteerdersjury Nederland, or SAN), based in the Netherlands, is dedicated to sharing knowledge to advance the advertising profession. Their goals are to celebrate the best creative work by awarding excellent campaigns, and to provide a platform for professionals to connect. Every year, advertisers and agencies submit their best campaigns to be judged by a jury of marketing professionals. Since 2017, SAN has partnered with MetrixLab to research and understand the elements that create successful advertising campaigns.
This spirit of helping advertising professionals optimize their campaigns to the fullest is what inspired SAN and MetrixLab to conduct a meta-study where campaigns submitted to compete for a SAN Accent Award were subjected to an innovative, AI-driven creative evaluation.
This spirit of helping advertising professionals optimize their campaigns to the fullest is what inspired SAN and MetrixLab to conduct a meta-study where campaigns submitted to compete for a SAN Accent Award were subjected to an innovative, AI-driven creative evaluation. The central question was: To what extent can AI predict the effectiveness of advertising and distinguish winning ads?
Our approach
As an AI-powered solution, MetrixLab’s ACT Instant is unique in that it combines algorithmic analysis with human expertise: 70% of the analysis is AI-driven, but 30% is human. And it’s 100% grounded in proven methodologies. The predictive model benchmarks results against a database of MetrixLab’s strong history of copy testing via traditional methods, comparing current ads with years of survey-based research across platforms and formats.
Every ad run through the model is classified across 130+ creative dimensions. The model assesses two popular dimensions of advertising effectiveness, breakthrough and response:
Every ad run through the model is classified across 130+ creative dimensions. The model assesses two popular dimensions of advertising effectiveness, breakthrough and response: Is an ad able to attract attention, realize brand and message memory and then promote brand interest or even buying intention among consumers?
Machine learning and adaptive regression modeling develop performance results and predictive scoring of an ad’s performance. 54 commercials (22 from winning SAN cases and 32 from non-winning SAN cases) from the past three years were coded by the ACT Instant algorithm. The more interpretative variables, such as the use of humor and emotion or number of messages, were coded by advertising experts.
Results
On average, all submitted SAN cases scored significantly better than the benchmark. But there was a clear difference between the ads that had won SAN’s Accent Award and the other entries. The winning ads scored 24% better against the benchmark, compared to 19% for the non-winning ads. In terms of breakthrough specifically, the winning ads scored 9% higher than the benchmark (non-winning ads 4%).
ACT Instant also determined specific elements that influenced winning ads’ performance. Ads from winning campaigns more often use storylines with brand and product intertwined and with more emotional elements. Music (often popular music) also featured more prominently. Although AI is capable of coding the majority of creative dimensions, the most distinguishing variables here stemmed from a mix of coding by algorithms and humans.
Although AI is capable of coding the majority of creative dimensions, the most distinguishing variables here stemmed from a mix of coding by algorithms and humans.
“AI is able to distinguish and organize dozens of characteristics of campaigns in a split second. But as we’ve seen, human coding is an indispensable complement to AI,” says Ron Schneider, SAN Chairman of the Jury. “You have to think of it as a cook who gets all kinds of ingredients for a successful dinner thanks to AI, but he or she still has to give it the finishing touch.”
Implications for brands are clear: They can test ads that were previously unable to be tested due to time and/or budget. “Teams can now make data-based decisions much faster, more affordably and earlier in the process of creative development. You can have an algorithm and a predictive model do much of the work,” says Andy Santegoeds, Senior Solution Consultant at MetrixLab. “ACT Instant is increasingly being used by advertisers to evaluate online video or banner ads more quickly for go-or-no-go decisions, or to analyze large batches of creatives at the same time. That is the need we see in today’s market, because there is more and more content that brands want to distribute on all kinds of platforms.”
SAN and MetrixLab made the results of this research public at the Marketing Insights Event in the Netherlands on 8 June 2021. Get a broader view on this research in the interview with Andy and Ron that was published in MarketingTribune (in Dutch).
Download our journal article to read more about AI as co-pilot for human decision making.
