TL;DR — Deleting the name column does not de-identify a dataset. What identifies a person is usually the combination of innocuous fields — site, cohort, age, category — and in the eight-row export below, removing every name still left 8 of 8 rows uniquely identifiable on four such fields. The measurable property is k-anonymity: the size of the smallest group sharing a quasi-identifier combination, where k=1 means "this row is one person." Generalising age into bands raised k from 1 to 2, and also dropping the site raised it to 4 — while cutting the number of analysable groups from 8 to 2, which is the real trade and the reason this cannot be automated away. Three failures sit outside the maths entirely: the identifier printed inside a scanned image rather than in a column, the mapping file that reconstructs everything, and the fact that deleting a committed file does not remove it from version-control history. This stops being sufficient the moment an adversary holds an auxiliary dataset you have not seen, because k-anonymity is defined against the fields you thought of.
1. Simple explanation
De-identification is the step between "we hold data about people" and "we can share it with a colleague, a contractor, or a model provider." The intuitive version — find the names, delete them — fails, and it fails in a way that produces confident-looking safe data.
It fails because identity is rarely carried by one field. A dataset with no names, no addresses and no ID numbers can still contain exactly one person who is 17, at the Northgate site, in the 2025 electrical cohort. Anyone holding a list of who was in that cohort — the site itself, a colleague, a parent, anyone with the enrolment sheet — can point at that row and name them. The fields that do this are called quasi-identifiers: individually unremarkable, jointly unique.
There is also a category of failure that has nothing to do with columns. If the underlying record is a scan — a photographed or scanned document — the name is a set of pixels in the top-right corner of the page. Renaming the file from jane_smith.pdf to cand-01.pdf changes nothing at all about the image inside it, and it is a common and expensive mistake because the file listing afterwards looks perfectly clean.
Analogy — a hospital publishing "anonymous" discharge statistics. They strip names and addresses, and publish age, postcode, admission date and diagnosis, because that is the useful part. Then a journalist takes the local electoral roll, filters to the one 84-year-old on that postcode, matches the admission date to a story that ran in the paper, and names the patient. No field in the published data was an identifier. The combination was. The hospital did not leak a name; it leaked a coordinate precise enough to find one, and the person who completed the identification did so with a public list the hospital never considered.
2. Diagram
WHAT PEOPLE THINK THEY REMOVED WHAT ACTUALLY IDENTIFIES SOMEONE
direct identifiers quasi-identifiers
+----------------------+ +----------------------------+
| name | <- deleted | site Northgate |
| contact details | <- deleted | trade electrical | none of these
| national ID | <- deleted | cohort 2025 | is an identifier
+----------------------+ | age 17 | on its own
+----------------------------+
"it's anonymous now" |
v
+-------------------------------------+
| THE JOIN |
| adversary holds the enrolment list |
| exactly one 17-year-old, Northgate, |
| 2025, electrical -> named |
+-------------------------------------+
THE MEASURABLE PROPERTY — k = size of the smallest quasi-identifier group
(measured, §5)
raw export k = 1 8 of 8 rows unique 8 analysable groups
age -> bands k = 2 0 unique 4 groups
+ drop site k = 4 0 unique 2 groups
^^^^^ ^^^^^^^^
safer less useful
\____________ this is the trade ________/
THE THREE FAILURES THAT ARE NOT ABOUT COLUMNS AT ALL
scan.pdf mapping.csv git history
+-----------+ cand-01 -> J. Smith commit A: added names
| J. SMITH | <- the cand-02 -> R. Patel commit B: "removed names"
| ratings | name => still in A, forever
+-----------+ is pixels
renaming the file one file that undoes `git rm` is not deletion
changes nothing every other control
3. How it works
3.1 Direct identifiers versus quasi-identifiers
A direct identifier names a person on its own: full name, phone number, national ID, email. Removing these is necessary and takes five minutes.
A quasi-identifier does not name anyone alone but narrows the population when combined with others: site, cohort, age, job role, date of assessment, first three digits of a postcode. These are exactly the fields people keep, because they are the fields that make the data worth analysing. That tension is the entire problem — the analytical value and the re-identification risk live in the same columns.
The practical consequence is that "which fields are quasi-identifiers?" is not a property of your dataset. It depends on what other data exists in the world. A cohort year is harmless if nobody has the enrolment list and identifying if someone does, and you generally cannot enumerate who has what.
3.2 The linkage attack, concretely
Re-identification is a join. The adversary has an auxiliary table containing both the quasi-identifiers and the names — an enrolment sheet, a public register, a social media profile, an internal roster. They join it against your "anonymous" export on the quasi-identifiers. Every row that matches exactly one auxiliary record is re-identified.
Nothing sophisticated happens here. There is no cryptography to break and no model to train. It is a spreadsheet operation, which is why it is worth taking seriously: the barrier is not skill or resources, it is whether anyone bothers.
3.3 k-anonymity, and what it does not promise
A dataset is k-anonymous if every row shares its quasi-identifier combination with at least k−1 others. k=1 means at least one row is unique and therefore re-identifiable by the join above. Raising k means generalising (exact age becomes a band), suppressing (drop the site), or aggregating until the small groups merge.
Two limits matter more than the definition. First, k is computed against the quasi-identifiers you listed. A field you did not think of — an unusual score, a free-text comment mentioning a location, a timestamp — can be identifying, and k will not notice. Second, k-anonymity says nothing about the sensitive value. If all four members of a k=4 group have the same outcome, learning which group someone is in reveals their outcome exactly; the group hid who, not what. Refinements exist for this, but the honest summary is that k is a floor, not a guarantee.
3.4 Redact the artifact, not the filename
When the record is a scan or photograph, the identifier is inside the image. This defeats every text-based control: a grep for names finds nothing, a schema review finds nothing, and the file listing looks clean. It also survives most format conversions, and if the pipeline runs OCR downstream, the name is dutifully extracted back into text.
Redaction here means altering the pixels — drawing an opaque box and re-encoding the image — before the artifact leaves the holder. Two traps: covering a region in a PDF viewer's annotation layer usually leaves the original content beneath it, recoverable by anyone who selects the text; and rotating or cropping does not remove metadata, which frequently carries the device, the timestamp and sometimes the location.
The strongest arrangement is that the holder redacts before sending. If de-identification happens at the source, the receiving party never possesses identifiable data, which usually simplifies the governance question rather than answering it after the fact.
3.5 The mapping file is the whole system
Pseudonymisation — replacing names with cand-01 — is reversible by design, because someone has to be able to answer "which candidate was that?" That reversal lives in a mapping file, and that file is now the single artifact that undoes every other control.
It belongs with the data holder, not in the analysis environment, and never in the repository. The failure mode is mundane: someone commits it "temporarily" to unblock a colleague.
3.6 Deleting a committed file does not delete it
Version control keeps history. git rm secrets.csv followed by a commit removes the file from the current tree while leaving it fully intact in every prior commit, in every clone anyone has pulled, and in any fork or backup. The repository still contains the data; it is merely not in the latest snapshot.
Actually removing it requires rewriting history and force-pushing, invalidating every existing clone, plus rotating anything the exposure compromised. This is disruptive enough that the practical control is upstream: never let identifiable data reach a commit. An ignore rule and a pre-commit check cost minutes; a history rewrite costs a day and an apology.
4. The math
4.1 k-anonymity
For a dataset D and a chosen set of quasi-identifiers Q, partition the rows into equivalence classes by their Q values:
groups(D, Q) = { g : all rows in g share the same Q values }
k(D, Q) = min over g of |g|
re-identifiable rows = count of g where |g| == 1
Generalising a field merges classes, so k is non-decreasing under generalisation and the number of distinct groups is non-increasing. That is the trade in one line: safety rises and analytical resolution falls, together.
4.2 Worked example
Using the eight-row export from §5 with Q = (site, trade, cohort, age):
raw:
8 rows, 8 distinct Q combinations
k = min group size = 1 -> 8 of 8 rows unique, every row re-identifiable
analysable groups = 8
generalise age -> two bands (16-18, 19-21):
(Northgate, electrical, 2025, 16-18) <- absorbs ages 17 and 18 |g| = 2
(Northgate, welding, 2025, 19-21) <- absorbs ages 19 and 20 |g| = 2
(Southbank, electrical, 2025, 16-18) |g| = 2
(Southbank, welding, 2025, 19-21) |g| = 2
k = 2, unique rows = 0, analysable groups = 4
also suppress site:
(-, electrical, 2025, 16-18) <- both sites merge |g| = 4
(-, welding, 2025, 19-21) |g| = 4
k = 4, unique rows = 0, analysable groups = 2
Going from k=1 to k=4 cost 75% of the analysable groups (8 → 2). Any claim that de-identification is free is a claim that the analysis did not need those distinctions.
5. Real code
from collections import Counter
# A "de-identified" export: names replaced by IDs. Everything else kept
# because it "seemed useful for analysis".
ROWS = [
# id, centre, trade, cohort, age, score
("cand-01", "Northgate", "electrical", 2025, 17, 72),
("cand-02", "Northgate", "electrical", 2025, 18, 65),
("cand-03", "Northgate", "welding", 2025, 19, 81),
("cand-04", "Northgate", "welding", 2025, 20, 48),
("cand-05", "Southbank", "electrical", 2025, 17, 55),
("cand-06", "Southbank", "electrical", 2025, 18, 77),
("cand-07", "Southbank", "welding", 2025, 19, 90),
("cand-08", "Southbank", "welding", 2025, 20, 60),
]
FIELDS = ("id", "centre", "trade", "cohort", "age", "score")
QUASI = ("centre", "trade", "cohort", "age") # not identifiers alone; identifying together
def k_anonymity(rows, quasi_idx):
"""Smallest group size over the quasi-identifier combination. k=1 => unique => re-identifiable."""
groups = Counter(tuple(r[i] for i in quasi_idx) for r in rows)
return min(groups.values()), groups
qi = [FIELDS.index(f) for f in QUASI]
k, groups = k_anonymity(ROWS, qi)
print(f"quasi-identifiers: {QUASI}")
print(f"k-anonymity of the 'de-identified' export: k = {k}")
uniques = [g for g, n in groups.items() if n == 1]
print(f"rows that are UNIQUE on those four fields: {len(uniques)} of {len(ROWS)}")
for g in uniques:
print(f" {g}")
# Generalise the one field doing the most damage: exact age -> band.
def band(a):
return "16-18" if a <= 18 else "19-21"
GEN = [(r[0], r[1], r[2], r[3], band(r[4]), r[5]) for r in ROWS]
k2, groups2 = k_anonymity(GEN, qi)
print(f"\nafter banding age: k = {k2}, unique rows = {sum(1 for n in groups2.values() if n == 1)}")
# Drop the centre as well.
GEN2 = [(r[0], "-", r[2], r[3], r[4], r[5]) for r in GEN]
k3, groups3 = k_anonymity(GEN2, qi)
print(f"after also dropping centre: k = {k3}, unique rows = {sum(1 for n in groups3.values() if n == 1)}")
print("\nutility cost of each step (distinct analysable groups):")
for label, g in (("raw", groups), ("age banded", groups2), ("age banded + no centre", groups3)):
print(f" {label:<24} {len(g)} groups")
assert k == 1, "the naive export should contain unique rows"
assert k3 > k, "generalisation must raise k"
print("\nasserts passed")
# Output:
# quasi-identifiers: ('centre', 'trade', 'cohort', 'age')
# k-anonymity of the 'de-identified' export: k = 1
# rows that are UNIQUE on those four fields: 8 of 8
# ('Northgate', 'electrical', 2025, 17)
# ('Northgate', 'electrical', 2025, 18)
# ('Northgate', 'welding', 2025, 19)
# ('Northgate', 'welding', 2025, 20)
# ('Southbank', 'electrical', 2025, 17)
# ('Southbank', 'electrical', 2025, 18)
# ('Southbank', 'welding', 2025, 19)
# ('Southbank', 'welding', 2025, 20)
#
# after banding age: k = 2, unique rows = 0
# after also dropping centre: k = 4, unique rows = 0
#
# utility cost of each step (distinct analysable groups):
# raw 8 groups
# age banded 4 groups
# age banded + no centre 2 groups
#
# asserts passed
Every name was already gone before this script ran. The export was still, by the only measure that matters, a list of eight identifiable people.
6. Real-world example
A team needed real marked assessments to test an automated scoring pipeline. A vocational training centre agreed to supply them. The centre did what everyone does: renamed the scanned files from candidate names to cand-01.pdf through cand-40.pdf, exported the marks to CSV with the same IDs, and sent the lot over.
Three problems arrived with it. The scans still had the candidate's handwritten name in the header block of page one, because renaming a file does not touch its contents — and since the pipeline ran OCR, the names were extracted straight back into text on the first ingest. The CSV carried site, trade, cohort and exact age "for analysis", which as computed above left most rows unique. And a mapping.csv was included in the same archive, helpfully, so the team could report results back per candidate — which meant one file in the delivery reversed the pseudonymisation of every other.
The costly part was the fix. Someone had committed the archive before anyone looked closely. Removing it meant rewriting the repository's history and force-pushing, which invalidated every clone on the team, and re-requesting the data from the centre with an explicit redaction specification. The engineering work was perhaps two hours; the coordination was a fortnight.
The lesson the team drew was not "be more careful with data." It was that the redaction must happen at the holder, before transfer, because every control after that point is a compensating control for a mistake that has already been made.
7. Interview questions companies actually ask
Q1. Is removing names sufficient to anonymise a dataset? No, and it is the most common misconception in this area. Identity is usually carried by the combination of quasi-identifiers — site, age, cohort, role, dates — none of which is an identifier alone. In the worked example above, an export with every name removed still had all eight rows uniquely identifiable on four such fields. The attack is a join against any auxiliary table containing those fields plus names, which requires a spreadsheet rather than any technical sophistication.
Q2. What is k-anonymity and what does it fail to protect? A dataset is k-anonymous if every row shares its quasi-identifier values with at least k−1 other rows, so k=1 means some row is unique. It has two well-known gaps: it is computed only over the fields you nominated as quasi-identifiers, so an unconsidered field silently breaks it; and it protects which individual a row belongs to without protecting the sensitive value, since a group whose members all share the same outcome reveals that outcome for anyone known to be in it.
Q3. Your source data is scanned handwritten documents. What changes? The identifier is pixels inside the image rather than a value in a column, so every text-level control misses it — grep finds nothing, schema review finds nothing, and the file listing looks clean after renaming. Redaction has to modify and re-encode the image itself. Annotation-layer boxes in a PDF viewer are not redaction, since the underlying content usually remains selectable, and image metadata can independently carry device, timestamp and location.
Q4. Where should the pseudonym mapping live? With the data holder, never in the analysis environment and never in a repository. It is the one artifact that reverses every other control, so its blast radius is the whole dataset. If results must be reported per subject, the holder performs the re-association on their side rather than the analyst holding the key.
Q5. Someone committed a file with personal data and then deleted it. Is the problem solved? No. The deletion removes it from the current tree while every prior commit still contains it, in the repository and in every clone and fork already pulled. Fixing it properly means rewriting history and force-pushing, which invalidates existing clones, plus rotating anything the exposure compromised. Because remediation is that disruptive, the effective control is preventing the commit in the first place.
Q6. How do you decide how much to generalise? By measuring both sides. Raising k costs analytical resolution — in the worked example, going from k=1 to k=4 reduced the number of distinct analysable groups from eight to two, a 75% loss. The decision requires knowing which distinctions the analysis actually depends on, so it is a question for whoever will use the data, not one the person performing de-identification can answer alone.
8. When to use / tradeoffs
Reach for structured de-identification when:
- Data about people leaves the system that collected it — to a colleague, a contractor, a model provider, or a repository.
- The records are documents or images rather than rows, where identifiers hide inside the artifact.
- You intend to publish aggregate results and need to state what was protected and how.
- Small subgroups exist, since these are where k collapses to 1 first.
| Situation | Why it breaks | Use instead |
|---|---|---|
| Small population, many quasi-identifiers | Nearly every row is unique; reaching a useful k destroys the data | Do not export row-level data — release aggregates, or keep analysis inside the holder's environment |
| Adversary may hold auxiliary data you cannot enumerate | k is defined against the fields you listed; an unknown auxiliary table can defeat any k | Contractual and access controls alongside the technical ones; assume k is a floor |
| Free-text fields present | A comment can name a place, an incident or a person; k-anonymity does not read prose | Strip or manually review free text — it is the field that most often leaks after everything else is handled |
| Data must be re-associated with individuals later | Irreversible anonymisation forecloses it | Pseudonymise, and keep the mapping with the holder — but note the data is now personal data, not anonymous |
Honest limits. k-anonymity is a floor and a rough one: it protects against identity disclosure and not attribute disclosure, and it is defined entirely over the quasi-identifiers someone chose to list, so its guarantee is only as good as that judgement. The eight-row example makes the arithmetic legible and is far too small to be representative — real datasets have long tails of rare combinations where k collapses no matter how you generalise, and the honest response there is usually not to release row-level data at all. Stronger frameworks exist, including formal differential privacy, which offers a guarantee that does not depend on enumerating auxiliary data; they carry their own costs and are out of scope here. Nothing in this article is legal advice, and the jurisdiction you operate in may impose specific obligations that are stricter than anything measured above. And the whole apparatus addresses disclosure risk only — it says nothing about whether you were permitted to collect the data in the first place, which is a separate question with a separate answer.
9. Summary + related articles
- Deleting names is not de-identification. In the measured example, 8 of 8 rows stayed uniquely identifiable on four ordinary fields after every name was removed.
- Identity lives in the combination of quasi-identifiers, and re-identification is a join against auxiliary data — a spreadsheet operation, not an attack requiring skill.
- k-anonymity is the measurable property: k=1 means re-identifiable. Banding age raised k to 2; also dropping the site raised it to 4.
- Safety and resolution move together. That change cost 75% of the analysable groups, and no tool can decide that trade for you.
- When records are scans, redact the pixels before transfer. Renaming files, annotation-layer boxes, and cropping are all insufficient.
- The pseudonym mapping reverses every other control. It stays with the holder.
- Deleting a committed file leaves it in history, in every clone. Prevent the commit; remediation is a history rewrite.
- Boundary condition: k-anonymity is defined only over the quasi-identifiers you nominated and protects identity rather than attributes — an unlisted field or a homogeneous group defeats it.
Related:
- Guardrails & Output Validation — preventing a model from emitting personal data at inference time, the runtime counterpart to this build-time control
- Hallucination Detection & Grounding — checking what a model asserts, once the data it was given is safe to use
- Fail-Closed Safety Gates: Enforcing Hard Constraints in Safety-Critical AI — refusing by default when a check cannot be completed
- Vector Databases: Dimensions, Footprint, and the Migration Nobody Plans For — what happens to personal data once it has been embedded, where deletion is materially harder
Resources
- Sweeney, L., "k-Anonymity: A Model for Protecting Privacy," International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 10(5), 2002 — the original definition and the linkage attacks that motivated it.
- Machanavajjhala et al., "l-Diversity: Privacy Beyond k-Anonymity," ACM TKDD 1(1), 2007 — the attribute-disclosure gap described in §3.3, and one response to it.
- Narayanan & Shmatikov, "Robust De-anonymization of Large Sparse Datasets," IEEE S&P 2008 — re-identification of a public "anonymised" release using auxiliary data.
- Dwork & Roth, The Algorithmic Foundations of Differential Privacy, 2014 — Chapters 1–3 for a guarantee that does not depend on enumerating auxiliary datasets.
- git-filter-repo — the currently recommended tool for removing data from repository history, and its documentation on why remediation is disruptive.