Privacy officers are increasingly concerned about automated decision-making, not just in theory but in practice. These systems, already in use, flag suspicious transactions, score vendor risk, or terminate employee access based on behavioral triggers.
This concern grew after the Dutch Data Protection Authority fined Uber €825 million ($959.2 million) for using an automated system to deactivate driver accounts based on customer review scores. Nearly a billion dollars for letting an algorithm make employment decisions without adequate human oversight.
Here's what privacy officers are discussing in Slack channels and hallway conversations.
Do We Need Human Review for Every Automated Decision?
Not every decision needs human review, but you must identify which ones have legal or significant effects. The GDPR distinguishes between routine automation and decisions that impact individuals significantly, like employment termination, credit denial, or restricting essential services. If your algorithm makes these calls without human context review, you're at risk.
The issue with Uber wasn't the use of automation itself. It was that drivers lost their income source through a purely algorithmic process without meaningful human intervention.
A practical guideline: If the decision would require documentation in an employee file or trigger a right to appeal under your policies, it needs human review before execution.
How Much Transparency Do We Owe About the Algorithm?
Enough for individuals to challenge the decision. Article 22 of the GDPR gives people the right not to be subject to decisions based solely on automated processing. This includes understanding the logic involved and the significance of the processing.
Explaining "the logic involved" doesn't mean sharing your source code. It means clarifying the factors the system weighs, how they combine to produce an outcome, and what triggers action. If your algorithm deactivates accounts after three poor ratings, people need to know that's the rule.
Privacy officers often mistake algorithm transparency for trade secret protection. Your competitive edge isn't the scoring formula but how you act on the scores. Document decision logic in plain language that anyone can understand and contest.
You'll know your transparency is lacking if someone can't understand why they received a specific outcome.
Can We Use Automated Scoring If We Add a Human Review Step?
Yes, but the human review must be substantive. Adding a manager who approves 99.8% of algorithmic recommendations isn't oversight; it's liability theater.
Real oversight means the reviewer can access the underlying data, understand what triggered the flag, apply contextual judgment, and override the recommendation without special justification. If your process assumes the human will almost always agree with the algorithm, you haven't added real judgment.
Document what you expect the reviewer to evaluate. "Manager reviews account activity and determines whether circumstances warrant exception to standard policy" is substantive. "Manager confirms system recommendation" is not.
What Counts as a "Solely Automated" Decision?
This question often confuses privacy officers. If a human clicks a button to execute what the algorithm recommended, is that still solely automated?
Regulators say if the human review is pro forma or lacks practical ability to override the algorithm, it's still solely automated. The test isn't whether a person touched the process but whether meaningful human judgment influenced the outcome.
Consider your system design. Can the reviewer access information beyond what the algorithm surfaces? Do they have time for actual review, or are they processing 200 cases per hour? Can they override without escalating to a supervisor? If the answers show procedural rather than substantive involvement, you're still running an automated system under GDPR.
Do These Rules Apply to Our Vendor Risk Scoring?
Probably not with the same intensity, but don't assume exemption. GDPR's Article 22 protections apply to natural persons, not companies. Automated vendor risk scoring affecting business relationships generally falls outside these requirements.
Complications arise if your system blocks individual contractors or sole proprietors from doing business with you. The person operating as a business entity still has GDPR rights as an individual.
Also, watch for indirect effects. If your system flags certain regions or business categories as high-risk, preventing individuals from those areas from accessing your platform, regulators may see that as significantly affecting natural persons even if you're technically scoring business entities.
How Should We Document Our Automated Decision-Making Systems?
Start with a register of all systems that make or significantly influence decisions about individuals. For each system, document:
- The decision the system makes or influences
- Data inputs feeding the decision logic
- Whether the decision produces legal or significant effects
- Human oversight and at what stage
- How individuals can contest decisions
- Testing done to identify bias or errors
Article 35 GDPR requires a Data Protection Impact Assessment for processing involving systematic evaluation of personal aspects based on automated processing. If your system scores individuals and those scores drive consequential decisions, you need a DPIA addressing the automated decision-making risks.
Your documentation should let you quickly answer: "Show me every system that could terminate, suspend, or significantly restrict someone's access without a human making the final call."
Where Do We Go From Here?
Review your highest-risk automated systems first. Employment decisions, account suspensions affecting income, credit or lending decisions, and access to essential services should top your list.
For each system, trace the decision path from data input to action. Identify where human judgment enters the process and whether it's real or cosmetic. If you can't articulate what the human reviewer evaluates beyond "confirm the system recommendation," redesign the process.
The €825 million fine isn't just about Uber. It signals that data protection authorities prioritize automated decision-making enforcement. Your automated systems need documented human oversight, clear decision logic, and a way for affected individuals to understand and contest outcomes.
Start documenting now. The next enforcement action won't wait for you to figure this out.
