High-volume hiring has a brutal arithmetic problem. A single popular role can draw hundreds of applications, and a recruiting team can spend days on first-pass screening before a hiring manager sees a single candidate. By then the strongest candidates, the ones with options, are often already gone. Screening is the obvious place to point AI, and it’s also one of the most dangerous places to point it carelessly, because a hiring decision affects a person’s livelihood and is, rightly, heavily regulated.
This is a case study about doing it the careful way. The headline result is specific: 191 CVs screened in 75 minutes, at 88.2% accuracy against expert judgment. But the most important design decision in the build isn’t in that number. It’s that the system ranks and screens in, and never auto-rejects. That single choice is what separates an evidence-backed, defensible screening stack from the “AI rejected your resume” systems that generate lawsuits and bad press. Here’s what was built, why it’s designed this way, and how the architecture scales from 191 CVs to a full pipeline.
Results at a glance
| Metric | Result |
| Volume / speed | 191 CVs screened in 75 minutes |
| Accuracy | 88.2% agreement with expert screening judgment |
| What it does | Parses, scores, and ranks/shortlists candidates against role criteria |
| What it deliberately does NOT do | Autonomously reject candidates |
| Human role | Owns the decisions: reviews the shortlist, owns all rejections |
| Designed for | Fairness testing, human oversight, audit trail (high-risk-aware) |
The 191 / 75-minute / 88.2% figures are from a real build. “Pipeline at scale” describes how that proven architecture extends, not a larger claimed client result.
The problem: speed versus responsibility, and you need both
The naive version of this project is easy and wrong: point an LLM at the resume pile, have it accept or reject, done. It would be fast. It would also be a liability, because hiring AI sits in the most regulated, highest-stakes corner of business automation.
- The EU AI Act classifies AI used in recruitment and CV screening as high-risk, with core obligations (risk assessment, data governance, bias testing, human oversight, transparency, logging). Under the 2026 Omnibus agreement these high-risk obligations apply from December 2027, while the Act’s broader framework is already taking effect from August 2026.
- NYC Local Law 144 already requires an independent bias audit of automated employment decision tools, within a year of use, with published impact ratios by sex, race, and ethnicity. DCWP has been enforcing it since July 2023.
- Under GDPR Article 22, candidates have the right to contest a purely automated rejection.
Best practice across the field is explicit: AI should be used to screen in or rank top candidates, not to autonomously execute rejections. So the real engineering brief wasn’t “screen CVs fast.” It was “screen CVs fast and defensibly,” at a speed a hiring team trusts and a regulator (or a rejected candidate’s lawyer) can’t fault. Those two goals feel opposed. The architecture is how you get both.
What was built: an evidence-backed, fairness-aware screening stack
1. Parse and structure (the unglamorous foundation)
CVs are messy: every format, every layout. The first stage reliably extracts structured information (experience, skills, qualifications) from unstructured documents. Get this wrong and everything downstream is built on noise, which is why most of the real engineering effort lives here, not in the “AI” everyone pictures.
2. Score against role-specific, job-relevant criteria
The system scores each candidate against criteria defined for the role, the skills and experience relevant to the job, and deliberately not proxies that correlate with protected characteristics. That’s both a fairness requirement (the EU AI Act and non-discrimination law demand job-relevant, bias-tested criteria) and an accuracy requirement: scoring on what actually matters for the role is what produced the 88.2% agreement with expert judgment.
3. Rank and shortlist, never auto-reject
This is the load-bearing design decision. The output is a ranked shortlist, not accept/reject verdicts. The accuracy figure is exactly why this matters. Around 88% agreement with experts is excellent for prioritizing a pile so humans see the strongest candidates first, and nowhere near grounds to let the remaining ~12% disagreement become automated rejections of real people. The AI compresses days of triage into the time it takes to review a ranked list, and the human makes every consequential call.
4. Human-in-the-loop, by design
A recruiter reviews the shortlist and owns all decisions, especially rejections. This isn’t a grudging safety step. It’s the design that makes the system both better (human plus AI screening beats either alone) and compliant (human oversight is a hard requirement for high-risk hiring AI). It’s our human-in-the-loop philosophy applied exactly where the stakes demand it.
5. Bias testing and an audit trail
Every AI-supported decision is logged, and the scoring is bias-tested and monitorable. This is what turns “we think it’s fair” into “we can demonstrate it’s fair,” which is precisely what NYC LL 144’s bias-audit requirement and the EU AI Act’s monitoring obligations ask for. The audit trail isn’t bureaucracy. It’s the difference between a defensible tool and an indefensible one, and it’s the same discipline behind AI governance without bureaucracy.
From 191 CVs to a pipeline at scale
The 191-in-75-minutes result is a proof point, not a ceiling. The architecture scales because the expensive, hard part, reliable parsing, job-relevant scoring, the human-in-the-loop and audit scaffolding, is built once and then runs against any volume. Going from one role’s 191 applicants to a continuous, multi-role pipeline is mostly a matter of throughput and integration (connecting to the ATS, handling concurrent roles), not re-solving the problem. The same build-once discipline shows up in our production-readiness checklist.
What doesn’t change as you scale, and must not, is the discipline: ranking rather than rejecting, human oversight on decisions, bias testing, and logging at every volume. A stack that’s fair and defensible at 191 CVs stays fair and defensible at 10,000 only if those controls scale with it. That’s the honest version of “pipeline at scale”: the speed compounds, and so must the safeguards.
What to look for when buying AI screening
If you’re evaluating AI screening vendors, this build argues for a specific buying lens.
- “Screen in,” not “auto-reject.” Any system that autonomously rejects candidates is selling you legal and reputational risk. Rank-and-shortlist is the defensible pattern. Ask which one they do.
- Ask for the accuracy number and what it’s measured against. “88.2% against expert judgment” is meaningful; “highly accurate” is marketing. And ask what they do with the disagreement. If it’s “auto-reject,” walk away.
- Demand human-in-the-loop and an audit trail as architecture, not add-ons. With EU AI Act high-risk obligations arriving (December 2027) and NYC LL 144 already in force, this is becoming non-negotiable for anyone hiring in or from those jurisdictions.
- Insist on bias testing. NYC LL 144 already requires it for automated employment decision tools. It’s table stakes, not a premium feature.
A screening stack built this way doesn’t just save days of first-pass triage. It does so in a way you can defend to a hiring manager, a compliance officer, and a rejected candidate. That’s the difference between AI that accelerates your pipeline and AI that becomes your next legal problem. For another evidence-backed build, see how a structured audit caught 23 critical issues before launch.
The bottom line
The temptation in high-volume hiring is to let AI make the reject decisions, because that’s where the time goes. The evidence-backed stack does the opposite. It puts AI on the speed (parsing 191 CVs in 75 minutes, ranking them at 88.2% agreement with experts) and keeps humans on the decisions, because hiring is high-stakes, heavily regulated, and about people’s livelihoods.
That’s not the slow, cautious version of AI screening. It’s the fast and defensible one, the only kind worth building when the EU AI Act, NYC LL 144, and a candidate’s right to contest are all in the room. Speed where it’s safe, humans where it counts, evidence and an audit trail throughout. That’s how you get from 191 CVs in 75 minutes to a pipeline you can run at scale and stand behind.
Build your screening stack
Drowning in applications, but wary of the legal and fairness risk of “AI that rejects resumes”? You can have the speed without the liability, but only if the stack is built to rank, not reject, with oversight and bias testing baked in.
Build Your Screening Stack → We’ll build an evidence-backed, fairness-aware screening pipeline for your roles: reliable parsing, job-relevant scoring, ranked shortlists (never auto-rejection), human-in-the-loop on decisions, and the bias testing and audit trail that high-risk hiring AI now requires. Fast first-pass screening you can defend to a regulator. Book a scoping call.
FAQs
In a real build, an AI screening stack reached 88.2% agreement with expert screening judgment while processing 191 CVs in 75 minutes. Accuracy only means something relative to a baseline: “88.2% against expert judgment” is meaningful, while vague claims like “highly accurate” are not. Even strong accuracy is grounds for ranking candidates, not for autonomous rejection. The disagreement rate is exactly why a human must own reject decisions.
Yes, with safeguards. Hiring AI is classified high-risk under the EU AI Act (its high-risk obligations apply from December 2027 under the 2026 Omnibus agreement), and NYC’s Local Law 144 already requires annual independent bias audits of automated employment decision tools. Best practice, and increasingly law, requires job-relevant criteria, bias testing, human oversight, transparency, logging, and the candidate’s right to contest. AI should rank and screen in, not autonomously reject.
No. The consistent best practice and regulatory direction is that AI should rank or shortlist candidates while humans make the consequential decisions, especially rejections. Under GDPR Article 22, candidates have the right to contest a purely automated rejection. Beyond compliance, human-in-the-loop screening is also more accurate than either AI or humans alone, so keeping people on the decisions is both safer and better.
By scoring candidates on job-relevant skills and experience rather than proxies that correlate with protected characteristics, testing the scoring for disparate impact, keeping an audit trail of AI-supported decisions, and maintaining human oversight. NYC Local Law 144 requires publishing impact ratios by sex, race, and ethnicity for automated tools, and the EU AI Act requires bias mitigation and continuous monitoring. Fairness has to be designed in and demonstrable, not assumed.
Yes. The hard, expensive parts (reliable parsing, job-relevant scoring, human-in-the-loop, and audit scaffolding) are built once and then run at any volume, so scaling is mostly throughput and ATS integration. The critical caveat is that the safeguards must scale with the speed: ranking-not-rejecting, human oversight, bias testing, and logging have to hold at 10,000 CVs as firmly as at 191, or the system becomes fast and indefensible.
Four things: that it ranks and screens in rather than auto-rejecting; an accuracy figure stated against a clear baseline (such as expert judgment) plus what it does with disagreement; human-in-the-loop and an audit trail as architecture, not add-ons; and bias testing. With EU AI Act high-risk obligations arriving and NYC LL 144 already in force, any vendor that can’t speak to these is selling you risk, not just software.
