The Shortlist Report

Methodology

Every number on this site comes from the same four steps. If you want to argue with a ranking, argue with these.

1. The questions come from buyers, not keywords

Each report starts with a question a buyer would actually type — “what are the best agencies for TikTok Shop live selling?” — not a search term reverse-engineered from a volume tool. One question, one report.

2. We ask repeatedly, because the answer moves

AI assistants are not deterministic. Ask the same question five times and you get five overlapping but different answers. A single response tells you almost nothing, so we run each question multiple times and count across the whole set. The sample size behind every table is printed above it.

3. We count answers, not mentions

A firm named four times in one response scores that response once. Otherwise a verbose paragraph outweighs a genuine recommendation. Extraction is done by reading each response and listing the firms it names — we do not search for firms we already knew about, because the point is to find the ones we didn’t.

Names get normalised where the same firm is written several ways. Where a firm operates more than one domain, we merge them and say so on the page.

4. Share is a plain fraction

Share is the number of responses naming a firm divided by the total responses in that sample. Nothing is weighted, scored, or blended into an index. A firm at 36% appeared in roughly one of every three answers.

5. Tiers, because a ranking would be lying

We group firms into tiers instead of ranking them one through fifteen. Here is why, using our own first attempt at this.

The opening sample had four firms on 25, 24, 24 and 22 of 52 responses. Ordered as a list, one of them is “number one.” So we split those 52 responses in half at random, three separate times, and ranked each half on its own. A different firm came first in all three splits. The top group was stable — the same four firms every time — but the order inside it was noise. One extra mention flipped the lead.

So a tier holds every firm the sample cannot separate from that tier’s leader. The test is McNemar’s test at 95%, which is the right one here because every firm is scored against the same set of responses. It looks only at the responses where two firms disagree — one named, the other not — since responses naming both, or neither, say nothing about which is preferred.

Order within a tier is arbitrary. Read the counts, not the position. As samples grow, tiers will split — and a firm moving between them is a real change, not a reshuffle.

What these numbers do not mean

Money

No firm can pay to appear, to move up, or to be removed. There are no affiliate links and no sponsored placements. Outbound links to firms carry nofollow. If that ever changes, it will be disclosed on every affected page before it changes, not after.

Corrections

If a firm is misnamed, misdescribed, or merged wrongly, we will fix it and note the change. What we will not do is add a firm to a ranking because it asked — the count is the count.