Compare Short-Form Posts Without Fooling Yourself

Write the decision first

“Which post won?” is too vague. Decide what choice the comparison will inform: keep a series, revise an opening, change runtime, or route viewers to a different offer. Name one primary outcome and two guardrails. For a tutorial series, the outcome might be qualified clicks; guardrails might be engaged-view rate and production hours. A post with more starts but worse lead quality may not win.

Match the comparison set

Group posts by platform, format, audience, objective, length band, and distribution condition. YouTube’s Shorts analytics guidance explicitly recommends comparing videos of the same type because audience behavior differs across formats (YouTube Help). Exclude paid boosts from an organic comparison or label them separately. Do not combine a local offer, an entertainment loop, and an educational explainer into one average.

Freeze the observation window

Take each reading at the same post age—such as 48 hours and 14 days—rather than comparing a new post with a six-month-old one. Note outages, holidays, collaborations, media spend, major news, and account changes. Use complete windows. If a platform backfills or revises data, record the extraction date and avoid pretending the snapshot is immutable.

Use medians and show the spread

For a group of at least several comparable posts, report the median primary metric and the range or individual values. The median resists one exceptional hit better than the mean, but it does not fix a biased sample. Keep sample size visible. A hypothetical group with qualified-click rates of 0.2%, 0.3%, 0.3%, 0.4%, and 2.8% has a 0.3% median; the outlier deserves investigation, not automatic cloning.

Run the causality check

Before writing “the new hook caused the lift,” ask: Was only the hook changed? Was the audience exposure similar? Did topic demand shift? Was there a collaborator, trend, paid push, or better offer? YouTube Advanced Mode supports groups, filters, date-range comparisons, and content comparisons (YouTube Help), but the interface cannot remove confounding on your behalf. Use “associated with” unless the design supports stronger language.

Add a short sensitivity check before deciding. Recalculate the comparison without the highest and lowest post, then inspect whether the direction changes. Repeat using a later maturity window when available. If the recommendation flips, the evidence is fragile: collect more matched posts or make only a reversible change. Preserve the calculation and the excluded items rather than reporting only the preferred version.

End each review with one of four decisions: repeat unchanged, repeat with one variable changed, collect more observations, or stop. Preserve the losing examples; they explain the boundary of the finding. Do not set a universal benchmark from a small internal sample, and do not move the success metric after seeing the result.