Supplier Intel

Field notes

We Fed 100 "Manufacturer" Names Into a Real Validation Pipeline. 56 Weren't What They Claimed to Be.

If you're doing supplier sourcing right now, there's a decent chance an AI tool generated at least part of your candidate list. It's fast, it's cheap, and it sounds authoritative. The problem is nobody seems to have actually measured how often those names hold up once you check them.

So we checked.

The setup

We were working a packaging machinery sourcing task — a real project, not a demo. The starting point was a list of 100 candidate suppliers, names generated by ChatGPT in response to a sourcing brief.

Instead of taking the list at face value, we ran every name through an actual validation pipeline: confirming the domain resolves and belongs to the company it claims to be, cross-checking the company against UK Companies House records, and screening out anything that turned out to be a marketplace listing, a trading intermediary, or a company that doesn't actually match what the name implied.

This isn't a "does this look plausible" check. It's the same kind of due diligence a careful sourcing consultant would do by hand — we just did it systematically, on all 100, and kept a record of what broke and why.

The result

44 out of 100 were real, correctly-matched manufacturing companies. 56 were not.

Breaking down the 56:

  • Dead or wrong domains — the company either doesn't have the web presence the list implied, or the domain now belongs to someone else entirely.
  • Wrong-company matches — a name that's real, but not the company the brief was actually asking for. Same or similar name, different business, sometimes a different country.
  • Marketplace and trader listings dressed as manufacturers — pages that read like a factory profile but are actually a listing on a B2B marketplace or a trading company sitting between you and whoever actually makes the part.

None of these are exotic failure modes. They're exactly the kind of thing a generative model has no way to check for itself — it's pattern-matching plausible-sounding company names and descriptions against its training data, not verifying that a business exists, is who it says it is, and does what you need.

A 44% real-and-matched rate isn't a knock on AI-assisted sourcing as an idea. It's a data point about what happens when you skip the verification step. If you're currently taking AI-generated supplier lists and sending outreach without checking each one, roughly half your effort may be going toward companies that can't fulfil the brief at all.

A finding we didn't expect: how you phrase the search matters more than you'd think

While building the validation pipeline, we ran a separate test on search-query framing for a different sourcing task, in UK warehouse racking. Searching with "manufacturer"-framed queries returned 0% real, correctly-matched UK manufacturers at scale. Reframing the same search around "dealer" and "supplier" language returned roughly 25%.

Same underlying market, same country, same target companies — the only variable was the word choice in the query. Our read on why: a lot of legitimate businesses in this space describe themselves as dealers or suppliers rather than manufacturers, even when they're the right point of contact, so a manufacturer-only search structurally misses them. It's a narrow test — one sector, one query pattern — so we're not claiming it generalizes to every category. But it's a concrete illustration of how much search-framing bias can silently shape a sourcing result set before any AI model even gets involved.

What "wrong" actually looked like

One specific bug worth mentioning, because it's the kind of thing that's easy to miss if you're not looking for it: during validation, we caught cases where two genuinely different companies were being treated as the same supplier because they shared a common word tied to their industry. Nothing exotic — just enough shared vocabulary in the company name and description for a naive match to conflate them. Once caught, it's an obvious problem. Before you've specifically tested for it, it's invisible, and it can quietly corrupt a "verified" list without anyone noticing.

We mention this not to sell a fix, but because it's a useful thing for anyone doing this work — manually or with tooling — to test for directly.

Why this matters if you do sourcing for clients

If you're a freelance sourcing consultant or run a small trading company, none of this is news to you in the abstract — you already know AI-generated lists need checking, because you're the one who ends up checking them. What might be useful is having an actual number to put on it: on this run, the false-positive rate was 56%, and it broke down into three specific, checkable failure types.

That's a number worth knowing whether or not you ever use a tool to help you check it — it's a reasonable planning assumption for how much validation effort an AI-sourced list actually needs.

A note on what we built, and what it doesn't do

We built the pipeline that ran this validation — a supplier sourcing and verification tool, working name Supplier Intel — because we kept doing this checking by hand and wanted it to be faster and more consistent. It does discovery, domain and company matching, Companies House verification, reverse-image checks on factory photos, and certification-claim extraction with links back to the actual registries.

It does not decide whether a supplier passes or fails. That call stays with a person, every time, because judgment calls about supplier fit involve context — pricing, capacity, relationship risk — that the evidence alone doesn't settle. What the tool does is gather and surface evidence; a human still makes the qualification call. We're being specific about that distinction on purpose: "evidence-gathered, human-reviewed" is an accurate description of what happens, "verified supplier" is not.

If you do this kind of validation work manually for clients and any of this matches what you've run into, I'd be glad to compare notes — what breaks most often in your process, what a tool would actually need to get right to be worth using alongside your own judgment, that kind of thing.

Doing supplier verification for clients? Supplier Intel is built for exactly this workflow — get in touch if you'd like to compare notes.