Identity-blind where it matters — and why it isn’t enough

Agents at Work — CC BY 4.0

Here’s a genuinely good design instinct: if an agent is judging people, don’t let it see who they are. Strip the name, the photo, the age, the address — feed it only what the person can actually do. Judge the work, not the identity.

It’s the right instinct, it’s grounded in the law, and this lesson is going to spend most of its length telling you why it isn’t enough on its own. Both halves matter. Skip the first and you’ve no defence at all; believe the first is the whole answer and you’ll ship bias while feeling safe — the worst of both.

Why identity-blind is right (and lawful)

New Zealand’s Privacy Act has a principle — the first one, IPP1 — that you should collect only the personal information you actually need for the job, and shouldn’t require identifying detail the purpose doesn’t call for. If the task is “match skills to a role,” the name, the photo, and the date of birth aren’t inputs — they’re liabilities. Feeding them in isn’t just risky; it cuts against the principle.

So the design move is real: redact identity and protected-attribute signals before the agent sees anything, and give it only job-relevant content. For the Recruiter that means skills and experience, not name, age, gender, ethnicity, or photo. Necessary, and a good habit. Now the hard part.

Why it isn’t enough — bias travels by proxy

Stripping the obvious signals doesn’t strip the information. It leaks back in through proxies the redaction never touched:

• A school or university name carries class, ethnicity, and age.
• An address or postcode carries socioeconomic and ethnic signal — the old “redlining” path, back by another door.
• Employment gaps hint at caregiving, health, age.
• Even writing style and word choice correlate with background.

Modern models are very good at picking these up. Redact the name and a capable system can reconstruct much of what you hid from the pattern of everything else — and then score on it, invisibly. You’ve removed the evidence of bias, not the bias.

And it can be worse than merely ineffective. A large, peer-reviewed field experiment run through a national public employment service found that anonymising résumés actually made firms less likely to interview and hire the very groups it was meant to help. Stripped of context that let evaluators account for disadvantage, the process turned against the intended beneficiaries. Redaction isn’t a dial that always points the safe way. Applied blindly, it can backfire.

The limit, stated plainly

Identity-blind is necessary but not sufficient. It’s a floor — do it — not a defence you can lean your weight on. Anyone selling you “we anonymise the CVs, so our AI screening is fair” is standing on that floor as if it were a roof. It isn’t. The evidence says the bias is still in there, riding the proxies, and possibly pointing the wrong way.

This is why the course keeps circling back to the same, less comfortable conclusion. You cannot design your way to a safe people-ranking agent with redaction alone. What actually protects you is the combination the next tier builds:

• Narrow, non-ranking scope — the agent surfaces evidence against criteria; it doesn’t rank or reject.
• Adverse-impact testing — you measure the agent’s outcomes across groups instead of trusting that redaction worked (Tier 3). This is the only way you catch proxy leakage, because you can’t see it by looking.
• A genuine human decision, backed by that testing.

And sometimes, when the stakes are high enough and the fixes all leak, the honest design is the one the Recruiter teaches: don’t automate the decision.

The design move

If your agent touches people:

• Redact identity and protected attributes before it sees the data — because IPP1, and because it’s the floor.
• Assume proxies survive the redaction — because they do.
• Plan the test now — decide, at design time, that you’ll measure outcomes across groups before you trust the thing. Don’t let “we redacted it” be the end of the sentence.

Next

You’ve decided what’s fair game and designed the agent to be narrow, checkable, and blind where it should be. Tier 3 makes it real: the guardrails you can actually write, the tests that catch what design can’t — and building one, with Claude Code, including the one you build to watch fail.