One, Ten, One Hundred: A Flawed Experiment
How Wistia’s report, “Does Production Quality Matter in Video Advertising?” is flawed and why you shouldn’t act on its findings.
If you haven’t read the study published last week, here’s a link to the story.
Wistia, a software company, hired Sandwich Video, a production company, to create three versions of the same product video — for three different budgets: $1k, $10k, and $100k. Wistia then used analytics to determine which spot performed the best per dollar spent. The result was that the video produced for $10k performed twice as well as the other videos.
After taking a closer look at the study’s methodology in their in-depth report, it’s clear that this experiment is flawed in its basic premise, and it is nothing more than an ad campaign for Wistia, their product Soapbox, and Sandwich Video. Here’s why:
Flaw number 1: The “control” in this experiment is the concept, which, all things considered, is about a $10k concept. Ballpark at least, when factoring in locations, talent, camerawork, and post-production needed for this type of product. This concept as written does not require a significantly higher or lower production budget. Videos produced with factor-of-ten smaller or larger budgets will feel far less appropriate than the $10k ad.
The authors of the report admit that they were not surprised by the $10k ad performing the best, which is a telling fact that further demonstrates the lack of integrity in the study. It was never a fair fight by design.
Flaw number 2: Egregious overspending. I’ll be the first to tell you that good creative does not necessarily require big budgets. But having a bigger production budget will give your brand more options in terms of what you can do, if you‘ve partnered with the right team. Where this study fails is that rather than putting the money into the creative, they are needlessly wasteful with the production budget — spending lots on camera and crew where it doesn’t serve to enhance the story or central idea. The authors then concluded, “Good storytelling matters far more than production quality.”
Equating “bells and whistles” and “gloss” to production quality is overly simplistic, and an insult to the thousands of real directors, DPs, animators, and designers who take pride in the quality of their work.
On the other end of the spectrum: less expensive does not have to mean elementary. It likely took more time and money creating practical paper cutouts than they would if they just designed an equivalent digital art card or pulled out the real clapperboard.
Flaw number 3: Soft costs. I have a hard time believing the true cost of the $100k ad was actually $100k. Sure they’re using an Arri Amira camera ($1,300/day rental) on a Fisher dolly, artificial lights, hair and makeup, and paid actors, but a lot of cost was sunk into motion graphics and animation. Much of these expenses are in-house at Sandwich and can be arbitrarily inflated to expand into the budget.
Find me the animator who is proud of the tasteless over-the-top graphics in this ad, and I’ll raise you two thoughtful creative animators from my team who understand the value in their work — any day of the week. But Brian, you say, they’re just being over-the-top to make a point. Well if their point is that bigger budgets equal gaudy and needless spending, then how is this study a fair process?
Flaw number 4: Ad rollout. Wistia’s methodology for deploying the ads concerns me as well. The viewer either watched them in sequence from smallest budget to biggest or they got to see all three next to each other to compare them. This approach inherently forces a comparison between them rather than seeing how the ads performed on their own. And because it’s a $10k concept, the cards were always in that video’s favor.
Conclusion: I co-own a video agency, so obviously I’m going to be a bit upset by a report that ostensibly claims that production quality matters less than I believe it should, as a creative. But what bothers me more is that this purportedly research-based report could be misleading brands in a way that could have real impact on the ever-shrinking budgets that my brothers and sisters in the creative community have to contend with.
It doesn’t just affect our bottom line, but it affects the relationships with our go-to vendors and contractors that depend on our work. And it dilutes the already watered-down landscape of mediocre video, something we’re constantly striving to improve, unlike, it seems, Wistia. It’s not easy finding talented directors and artists working in this field, and undercutting their value based on a misleading study is harmful to our existence.
It saddens me to see a fellow production shop participate in a stunt like this and although the authors admit to the study’s limitations in the appendix, it’s worrisome that brands will be using it as fodder to slash already-dwindling budgets. Buried in that appendix, they say: “A great video ad requires great creative talent, and this inevitably comes with a cost — the average small business cannot commission an ad from Sandwich Video for $10,000.”
A good quality pair of pants is expensive because of the materials and skilled labor that went into their design and creation, not because someone slapped on 50 zippered pockets. Likewise, what makes good quality video is a lot more nuanced than just special effects, a fancy camera, or a bloated crew. Finding a production partner you can trust, who understands your brand and your vision, and who’ll deliver the goods when it counts is what you‘re looking for. Once you’ve found them, hold on tight!
Good creative is intangible. Good video is nuanced. Good partnerships are triumphant.
