A Quarter of the Skills on a Job Posting Were Never in the Posting

Every job site shows you a list of skills on a posting. Almost none of them tell you where that list came from. We checked ours, and the answer is uncomfortable enough to publish.

Our pipeline records the provenance of every skill tag it assigns: whether the skill was found in the posting description, inherited from the source's own tag list, read off the title, or added by enrichment. That makes a question answerable that is usually invisible: does the posting actually say this?

Across 883,827 tag instances, 25.7% are not backed by the posting text.

The failure is not uniform, and that is the useful part

If a quarter of tags were wrong at random, the lists would just be noisy. They are not random. Concrete nouns hold up almost perfectly and abstract categories collapse.

That table is also its own positive control. If our provenance recording were broken, sql and python would look as bad as platform engineering. They do not. The instrument works; the tags for abstract categories genuinely are not in the text.

Why this happens

A named tool is hard to write by accident. PostgreSQL appears in a posting because somebody typed PostgreSQL. An abstract category like "cloud computing" is a bucket that something else gets filed into: a source's own taxonomy assigns it, an alias expands into it, an enrichment step infers it. Every one of those is a reasonable engineering decision, and stacked together they produce a skill list that reads like the employer wrote it when a quarter of it was written by us.

What it invalidates

Anything that ranks skills by tag frequency. That includes work we have published. A "most in-demand skills" ranking built on raw tag counts will put abstract categories near the top, because those are exactly the tags that get assigned without appearing in the text. Two of our own blog posts were headlined on precisely such tags and are being retracted and redirected.

We also audited our published skill pages against provenance. Across the 17 pages that publish skill demand percentages, 31 individual figures are under 50% text-backed. The worst example: one page publishes "Diversity Inclusion, 28.2%" for a skill that is 0.0% text-backed across 4,273 tag instances. Not low. Zero.

The damage is concentrated rather than universal. Our engineering-focused pages are clean, with medians between 92% and 97%. The sales and product-management pages are where it hurts.

What we are doing about it

Three things, in order:

1. Not silently deleting anything. The affected figures get corrected in public with the old numbers preserved, the same way we handled our seniority miscount.
2. Reporting text-backing alongside demand wherever we publish a skill percentage, so a reader can tell an extracted skill from an inherited one.
3. Refusing tag-count rankings. A frequency table over a tag space we partly authored is a measurement of us.

The part worth stealing

If you build anything that extracts structure from text, record where each field came from at write time. We can answer "is this skill actually in the posting?" only because a provenance map was written alongside the tags. Without it, this entire investigation would have been impossible after the fact, and the wrong numbers would have stayed up because nothing would have contradicted them.

Provenance is cheap to write and impossible to reconstruct later.