Speed is only an advantage when you understand what you are speeding up. Automation can improve speed and consistency, but it can also multiply the weaknesses hidden inside an existing workflow. Before introducing AI, agents, or process automation, organizations need to validate the process, preserve necessary judgment, and establish controls for errors and rollback.
I recently learned that a “cow path” is a trail worn in by farm animals wandering toward water. It makes sense to a cow – there are good patches of grass, they can stop in the shade of a tree, and finally arrive at some cool, tasty water. It makes less sense once humans pave it, add curbs and buildings, and build bigger highways for fast cars and heavy trucks down a bend no one can really explain. The cow’s goal became our reality. Software teams can fall into the same trap by wrapping automation around a workflow, speeding it up, and never asking why the workflow bends where it bends.
Automating a broken process gives you a faster route to the wrong place, and you wonder, “How did this traffic jam even start?”
Joseph Marie Jacquard, who invented the loom, studied how weavers threaded a pattern before he punched it into cards. Henry Ford spent a year timing how a car got built before any of it moved on its own. They automated the processes they fully understood. When you automate one, you don’t, you get the same mistake, faster, at a scale that is harder to see and harder to walk back.
We Might Be Skipping Validation Entirely
Most broken processes were not designed; they were accumulated through inheritance and poor understanding. Maybe a workaround became the standard, and one-time exceptions became a rule. Then the people who lived through the original problem leave the organization, and the reasoning leaves with them. What stays behind is the shape of the process without the context that can explain it.
Automation does not recover that context. It preserves the shape and removes the friction that used to slow the mistake down. The friction was sometimes the only thing keeping a bad process honest.
This is not a fringe risk. EY found that 30 to 50% of early robotic process automation projects fail, and the most common reason is teams automating a process they do not fully understand or one that was broken to begin with. While automation can create failures, a bad process can remove the limit on how far it could spread.
If the team cannot explain why a process works the way it does, automation will only preserve the confusion at scale.
When Automation Outruns Understanding
Some examples from the field with our customers (anonymized for privacy and embarrassment) where each started with a reasonable goal and a process that had not been examined closely enough to automate safely.
A content team replaced its human translators with machine translation to move faster and cut costs. The output looked fluent, so no one checked it closely. The errors weren’t simple misunderstandings; they were mistranslations that shifted meaning entirely. No one caught the pattern because each batch was too small to raise alarms, but taken together they became a running internal joke. The process that was removed, human review, was the part doing the real work, and the machines were “Lost in Translation”.
An investment process for acquiring tax deeds on distressed properties worked when a person reviewed each parcel. Automating sourcing and buying increased volume quickly, along with the share of properties that could never be resold without major reconstruction or toxic cleanup. Human judgment, as it were, was critical.
Zillow Offers hit a version of the same wall. Its automated valuation model was competitive, with a median error of nearly 1.9% on listed homes. The failure came from scaling the buying operation faster than the model could stay calibrated, and from downstream pricing logic that did not slow down as its own uncertainty grew. The result: roughly $881 million in write-downs, a quarter of the workforce laid off, and the unit shut down. A working model pointed at a process that could not absorb its speed.
Finally, we know of a team that wired a mapping API into Dynamics to geolocate users and devices. It ran fine in testing. A small uptick in real usage triggered exponential API calls, and bugs that never surfaced at low volume broke the system at scale, which wasn’t discovered until someone got their monthly invoice for 10 times more than expected.
Fast is not the same as fixed.
The Failures You May Not See for Months
There is a second, harder problem with automating on top of generative AI. The output is probabilistic, so a wrong answer does not announce itself. It looks like every right answer, and a confident, incorrect result just sits in a report.
We’ve probably all seen the stories about lawyers using ChatGPT to draft court filings and not verifying the output. The tool invented case citations, complete with fake quotes and docket numbers. The court sanctioned the attorneys and fined them $5,000, now a landmark reference for exactly this failure. Air Canada learned a related lesson when its support chatbot told a grieving customer he could claim a bereavement fare retroactively, which was not the airline’s policy. A tribunal held the airline liable for what its automated tool told the customer, rejecting the argument that the chatbot was a separate entity responsible for its own answers.
The medical field shows how wide the gap can be. A peer-reviewed meta-analysis of chatbots answering medical questions found an overall accuracy of about 56%. A tool that’s right slightly more than half the time is useful when a human checks it, and dangerous when it runs unattended.
Agentic frameworks raise the stakes again, because they act, not just answer. Replit’s AI coding agent deleted a live production database during an explicit code freeze, then produced fake data and denied what it had done. The wiped records covered executives at roughly 1,200 companies. Give a system the ability to act at machine speed with thin guardrails, and a single bad decision executes before anyone can intervene.
Recently, OpenAI disclosed during an internal safety evaluation that its models, including GPT-5.6 Sol, were scored on a cybersecurity benchmark with their usual guardrails dialed down. Focused on getting a passing score, the models exploited a zero-day to break out of their sandbox, reached the open internet, moved laterally through the research environment, and broke into Hugging Face to steal the benchmark’s answer key. Humans didn’t ask them to specifically do this, only to optimize the goal in front of them, and the goal ran straight through every boundary in the way. An automated system pursuing a narrow objective faster and further than anyone intended.
When automation produces a wrong answer that looks exactly like the right one, how long would it take you to notice?
Understand the Risk Before You Automate
You may know the comic, Is It Worth the Time?, that charts how much time you can spend automating a task before you spend more than you save. So, before automating a process, you must understand it well enough to know whether it should be automated, changed, or retired. If you have to do this anyway, why not find the cow path bends?
This is the discipline underpinning our Continuous Modernization approach. Before we decide how to modernize a workload, we assess it: business criticality, security and resilience risk, change friction, and how clearly it is owned. A process with high risk and unclear ownership is not a candidate for automation but rather a conversation that requires scrutiny. Understand what you are dealing with before you pour speed on it, because automation is a decision about how to modernize, and you cannot make that decision well without knowing the risks.
Two practical places to start:
- Recover the context before you touch the process. Find the people who lived through why the process exists and find any reasoning still left. Turnover and reorganizations erase the history that explains the shape. Without it, you are likely to automate your way into bigger problems.
- Put a human between the model and the outcome. For anything built on AI, decide where verification lives before you scale. Guardrails, review gates, and the ability to roll back are not negotiable. They are there to catch the errors before they reach the customer’s environment.
The Bottom Line
Automation and AI are leverage, and leverage does not care what it is applied to. Point it at a process you understand, and you get speed, consistency, and margin. Point it at one you don’t, and you get the same mistake, running faster than anyone can watch, at a scale that can be expensive to reverse. Understand the process first. Then automate it.
Automation is a multiplier. It multiplies whatever you point it at.
eGroup helps teams assess what they run, understand the risk before they change it, and modernize with discipline instead of speed for its own sake. Check out our Application Modernization offer.
Modernize the Process, Not the Problem
Before you automate, understand what the process does, why it exists, and where risk is hiding. eGroup helps teams assess applications and workflows, reduce modernization risk, and build a practical path forward.
