Build a creative testing plan

Short answer: Write every test as a hypothesis that changes one variable, choose one primary metric and a decision rule before launch, run it for a fixed period, and log the result under a test ID that matches your file names. Then put each finding into the next brief as a rule to keep, a variant to try again, or an idea to retire.

Who this guide is for

This is for anyone who runs paid or organic short-form video and has had the experience of looking at last quarter’s ads without being able to say what was learned. It assumes you can run A/B tests in an ad platform such as Meta Ads Manager. The organic section applies if you post the same assets to TikTok, Instagram, Facebook or YouTube.

SGNL runs creative testing for clients, so this is close to our own work. Everything below can be done with a spreadsheet and the tools inside the ad platforms; you do not need an agency to do it.

Why tests get lost

Tests get lost for ordinary reasons. Two ads differ in their hook, their length and their caption, so a winner cannot tell you which change mattered. Files are renamed “final” and “final-2”. Results live in an ad account that someone else later reorganizes. Nobody wrote down, before launch, what result would count as a win.

The cost is that you pay for the same lesson twice. Budget goes into variants that cannot teach you anything, and winners get dropped because nobody recorded why they won. A plan that logs each test turns ad spend into knowledge you keep.

The idea that ads should be tested and judged by measured response is old. Claude Hopkins argued for it in Scientific Advertising in 1923, using keyed coupons to compare one ad with another (a later edition at the Internet Archive; Library of Congress record). The same discipline applies to platform tests today: record what you changed, and record what happened.

Step 1: Write a hypothesis for each test

Meta’s own guidance is to start from a measurable hypothesis (Meta, Best practices for A/B testing). A template that works:

Because [what we observed], we believe that [one change] for [audience] will [raise or lower] [primary metric], measured over [period].

For example: “Because our best organic posts open with a direct question, we believe that opening with a question instead of a product claim, for cold audiences in the US, will lower cost per registration, measured over 14 days.” Cold audiences are people who have not interacted with you before; warm audiences have.

A hypothesis should come from something you saw: a past result, a customer comment, a sales call, a competitor’s ad. If you cannot fill in the first blank, you are guessing, which is allowed, but label it as an exploratory test.

Step 2: Choose one variable per test

Meta recommends keeping test versions identical except for the variable you are testing (Meta, Best practices for A/B testing). Not all variables are equal. I order them from the largest likely effect to the smallest, and test from the top down:

• Concept or angle. The core argument: price, outcome, fear of missing out, social proof, a demonstration.

• Hook. The first two to three seconds: the opening line, image or on-screen text.

• Talent. Who is on camera: founder, expert, customer, creator.

• Format. Talking head, screen recording, demonstration, interview, text-on-screen.

• Offer. What you ask for and at what price.

• Length and pacing. For example a 15-second cut against a 30-second cut of the same footage.

• Call to action and caption.

The order is my working view, not a published finding. The practical point is to settle the large questions before you spend weeks on small ones.

Step 3: Set measurement rules before launch

Write these rules down once and apply them to every test. Rules set after the results are in tend to confirm what the team already wanted to believe, which is how weak ads keep their budget.

• One primary metric. Pick the metric closest to revenue that gets enough volume to read, such as cost per purchase, cost per lead or cost per registration. Secondary metrics (hook rate, the share of viewers who watch past the first few seconds; hold rate, the share who keep watching longer; click-through rate) explain why a result happened; they do not decide the test.

• A fixed period. Meta recommends at least 7 days and allows a maximum of 30; if your customers usually take more than a week to convert, run longer, such as 10 days (Meta, Best practices for A/B testing).

• No edits during the test. Significant edits send an ad set back into the learning phase, which Meta says usually ends after about 50 results in the week after the last significant edit (Meta, About the learning phase).

• Use the platform’s test tool. Meta advises against testing by turning ads on and off manually, because it can lead to overlapping audiences and unreliable results (Meta, About A/B testing). It also advises against using the test audience in other campaigns at the same time.

Enough volume to tell variants apart

Small differences need large samples. Using the standard two-proportion formula at 95% confidence and 80% power, detecting a change in conversion rate from 2.0% to 2.5% needs about 12,748 people per variant; detecting 2.0% to 3.0% needs about 3,292 (my calculation; you can check it with Evan Miller’s sample size calculator). If your budget cannot reach that volume, test bigger differences, such as concepts, before small ones.

• A decision rule. Write what result means “keep”, “try again” and “retire” before the test starts, so you are not deciding after you have seen the numbers.

• A record of what else changed. Budget changes, price changes, a promotion, press coverage, a holiday. Any of these can move results during a test.

For organic posts, platform A/B tools are not available in the same way, so the rules are simpler and the conclusions weaker. At SGNL we judge a post at a fixed age, 72 hours after it goes live, so every post is read on the same basis, and we leave unknown values blank rather than entering zero. Organic results are useful for choosing what to put into paid tests.

On their own they rarely show cause, because budget, timing and the algorithm all change from one post to the next; the Vaunt example below shows how ordinary posts did not separate from the baseline.

Step 4: Keep a testing matrix

The matrix is the single record of what you are testing. Each row is one test, and the IDs match the file names.

The matrix below is hypothetical. It does not describe an SGNL client or real results.

Add three columns once a test ends: Result (the numbers, with dates), Verdict and Next action.

Naming. The asset IDs in the matrix follow the convention SGNL uses internally: a client code, the year and month the asset was created, and a running number, such as ABC-2611-014. File names add a short description and a two-digit version, such as ABC-2611-014_question-hook_v03. Revisions are new versions of the same file, never new names, and IDs are never reused. Put the asset ID in the ad name inside the platform as well, so the ad account and the matrix always match.

Step 5: Read and log the results

Use four verdicts and nothing else. These are the labels SGNL uses in its weekly review:

• Winner: the variant (the new version) beat the control (the current best version you compare against) on the primary metric under the decision rule. It becomes the new control.

• Neutral: no clear difference. Keep the cheaper or simpler version and move on.

• Test again: the result was promising but the volume was too low, or something else changed during the test.

• Retire: the variant lost. Record why you think it lost, briefly, so nobody makes it again next quarter.

Write one sentence of learning per test, in plain language, with dates: “Question hooks lowered cost per registration for cold US audiences in the November 2026 test (T-2611-01); not yet tested on warm audiences.”

Step 6: Bring findings into production

A finding only matters if it changes what you make next. This is the loop we use:

• Weekly: read finished tests, set verdicts, and record one-line learnings. At SGNL this happens in a Thursday review that feeds a Friday team call.

• Monthly: roll the weekly learnings into one page: what became the new control, what to test again, what is retired.

• Before the next shoot or creator brief: add a section to the brief called “What we learned last cycle,” listing the current controls and the next hypotheses. The brief asks for the footage those hypotheses need, for example three alternative openings for each script.

• Decide the split in advance: how much of next month’s production goes to variations on current winners and how much to new concepts. There is no correct ratio. Write yours down and review it each month.

• Retire on purpose: remove retired concepts from the brief so they do not come back under a new name.

Vaunt example: what the record shows so far

Vaunt, a private aviation membership app, is an SGNL client, and its 2026 data shows why the rules above matter. These are organic and paid observations, not controlled tests.

Ordinary posts did not separate from the baseline

From June to September 2026, days with a post logged in our content audit averaged 480 new users, and days without one averaged 504 (SGNL analysis of Vaunt’s daily new-user reports, October 4, 2026). Only outlier posts lined up with spikes: after two TikToks on June 14 and 15 (291,100 and 597,500 views), new users averaged 553 a day from June 15 to 22 against 365 from June 7 to 14.

Timing only; it does not show which post did what.

The primary metric changes the winner

In partial data visible on October 4, 2026 (about 93 of 237 rows of the dashboard table), the ad creative “phil10” had 2,059 downloads and 330 connected calls, 16.0% calls per download. The creative “phil1” had more volume, 7,846 downloads, and 745 calls, 9.5% (Finding Valence, Vaunt Meta Ads Post Mortem, October 9, 2026). Judged on downloads, phil1 wins; judged on the step closer to revenue, phil10 does.

• Tests need new creative coming in. Finding Valence’s post mortem says no new creative entered Vaunt’s ad account in September, while new users fell from about 479 a day in the week of September 7 to 387 in the week of September 20.

The change that follows: from the October 3, 2026 shoot, each Vaunt post is planned to carry its own keyword or link, so organic results can be read per post instead of by timing. Limits: the daily new-user count includes organic and paid users and cannot be split by source; the ads are managed by Finding Valence, not SGNL.

Tradeoffs and limitations

• Platform tests measure what the platform can see. Meta’s A/B tests compare versions on the platform’s own results, which depend on its attribution. They do not account for effects on other channels.

• Small budgets cannot detect small differences. If you cannot reach the volumes in Step 3, test concepts and formats, which in my judgment tend to differ more (see the variable order in Step 2), and treat close results as neutral.

• Winners fade. An ad that wins in November may not win in March. This is a judgment, not a measured rate: audiences see the same ad repeatedly and competitors change. Re-test controls periodically against new challengers.

• Results do not transfer automatically. A hook that wins for cold audiences in the US may not win for warm audiences or in another market. Record the conditions with the result.

• Organic reads are weak evidence. Use them to choose paid tests, not to declare winners.

• The variable order is a judgment. It reflects how I prioritize, not a controlled study.

Next step

Copy the matrix, write your next three hypotheses using the template, and set your measurement rules before you brief the next shoot. If you are budgeting for the assets those tests need, see How Much Does Creator-Led Content Cost?. If you want help running the plan, see SGNL’s performance creative service.

Sources

All accessed October 9, 2026.

• Meta Business Help Center, “Best practices for A/B testing.” https://www.facebook.com/business/help/290009911394576

• Meta Business Help Center, “About A/B testing.” https://www.facebook.com/business/help/1738164643098669

• Meta Business Help Center, “About the learning phase.” https://www.facebook.com/business/help/112167992830700

• Evan Miller, “Sample Size Calculator.” https://www.evanmiller.org/ab-testing/sample-size.html

• Claude C. Hopkins, Scientific Advertising (1923). Later edition at the Internet Archive: https://archive.org/details/scientificadvert0000hopk ; Library of Congress catalog record: https://www.loc.gov/item/23009362/

• SGNL Media internal standard operating procedures. asset naming and IDs; weekly stats review (drafts, September to October 2026). Not public.

• SGNL Media internal data for Vaunt (not public). daily new-user reports from Vaunt’s Looker/Data Studio dashboard, prepared by David Dickson of Button Up Management (January 14 to October 4, 2026), stitched into a daily series in SGNL’s case-study files (October 4, 2026).

• SGNL Media, flyvaunt content audit (logged-out capture, September 30, 2026) and content-library accuracy audit (October 4, 2026). Not public.

• Finding Valence, “Vaunt Meta Ads Post Mortem. September 2026,” October 9, 2026. Prepared for Vaunt; not public.

About the author

Andrew Moullin is co-founder and Creative Director of SGNL Media, where he leads creative and strategy for a four-person team that builds media systems for founders, experts and brands. Before marketing, he played professional soccer for San Antonio FC and Atlanta United 2, then coached.