By Frank Palermo COO of NewRocket
Image: Sasun Bughdaryan - Unsplash
After spending enough time in conference rooms, connecting with clients or strategizing with board members, a common question eventually gets asked: what happens if, or even when, AI does something that it was not intended to do?
This question, which may be in everyone’s minds, is all too quickly brushed aside and shoved under the rug. Then, the conversation quickly moves on to other topics. That reaction, our tendency to dismiss the uncomfortable questions before we answer them, may be the bigger risk, rather than the technology itself. As AI becomes embedded in more consequential decisions and business processes, those questions will have to be answered sooner rather than later. Organizations and their customers deserve answers to these questions that are clearly defined, accountability and intentional governance.
This is one of many examples of rogue AI, and while it has garnered a lot of media attention, it is not the only one nor is it the only example of what rogue AI can look like.
For organizations, issues begin to arise when their recommendation engine quietly starts favoring one vendor over another because of a data pattern nobody flagged. Across industries, this can look different, but the patterns are there.
This could be a customer service bot that promises a refund policy that does not exist, or a hiring tool that filters out qualified candidates for reasons buried three layers deep in a model nobody on staff can explain. In software it could look like when a coding agent fixes a vulnerability by changing an authentication control that locks out customers or introduces a different vulnerability elsewhere. Or in human resources it could be when a workforce agent learns from historical promotion patterns and begins recommending people who resemble previous successful leaders, perpetuating historical biases.
These issues by themselves seem small and easily resolvable. But they can compound over time to become a large organizational crisis. Small, boring failures are the kinds that bankrupt companies: nobody sees them coming until it is too late.
There is also the money you cannot count as easily. When a customer’s trust erodes, it does not show up on a balance sheet until they decide to look elsewhere for what they need. By that point organizations are paying to win it back at a much higher price than it would have cost to keep them as a customer.
A single, poor automated decision can undo years of brand goodwill. You cannot buy that back with an increased marketing budget. Companies have to earn it back slowly, except now people are closely watching and monitoring for the next mistake.
Third, and this one gets talked about the least, is the internal cost. The internal cost is when employees don’t use or trust the systems that an organization has implemented for them to use.
This could look like when teams start double-checking everything, which defeats the purpose of automating in the first place. Or worse, they stop questioning the output altogether because it is easier to defer to the machine than to push back.
Companies also need to test how agents fail before they go live, rather than discovering those failures when a customer, employee, or regulator finds them. So many organizations rush to deploy without doing the proper work ahead of time.
And if and when something does go wrong, organizations need to have the ability to act immediately. Every autonomous system requires continuous monitoring, defined escalation paths, trained teams to identify and monitor problems, and a kill switch that can be activated without waiting for three committees and a change-management form.
When it comes to AI, the cost can quickly exceed whatever you hoped to gain from the technology in the first place. AI does not stop being your responsibility once you deploy it. I would argue that is when the responsibility actually begins. The companies that will come out ahead are the ones who can honestly say that they understand what their systems are doing and why.
Reviewed by Irfan Ahmad.
Read next:
• Workplace AI Adoption Soars as Risky Practices and Poor Oversight Undermine Organizational Safety
• Apple, Samsung, Xiaomi and Google Phones Come Loaded With Apps, Here’s Which You Can Remove
After spending enough time in conference rooms, connecting with clients or strategizing with board members, a common question eventually gets asked: what happens if, or even when, AI does something that it was not intended to do?
This question, which may be in everyone’s minds, is all too quickly brushed aside and shoved under the rug. Then, the conversation quickly moves on to other topics. That reaction, our tendency to dismiss the uncomfortable questions before we answer them, may be the bigger risk, rather than the technology itself. As AI becomes embedded in more consequential decisions and business processes, those questions will have to be answered sooner rather than later. Organizations and their customers deserve answers to these questions that are clearly defined, accountability and intentional governance.
Examining rogue AI and what it means for organizations
The past few months have seen a surge in headlines about rogue AI. This is defined as autonomous AI agents acting outside their intended instructions or safety parameters. The Hugging Face incident, for example, occurred after autonomous agents were able to circumvent their digital sandbox and execute an unauthorized cyberattack.This is one of many examples of rogue AI, and while it has garnered a lot of media attention, it is not the only one nor is it the only example of what rogue AI can look like.
For organizations, issues begin to arise when their recommendation engine quietly starts favoring one vendor over another because of a data pattern nobody flagged. Across industries, this can look different, but the patterns are there.
This could be a customer service bot that promises a refund policy that does not exist, or a hiring tool that filters out qualified candidates for reasons buried three layers deep in a model nobody on staff can explain. In software it could look like when a coding agent fixes a vulnerability by changing an authentication control that locks out customers or introduces a different vulnerability elsewhere. Or in human resources it could be when a workforce agent learns from historical promotion patterns and begins recommending people who resemble previous successful leaders, perpetuating historical biases.
These issues by themselves seem small and easily resolvable. But they can compound over time to become a large organizational crisis. Small, boring failures are the kinds that bankrupt companies: nobody sees them coming until it is too late.
What the cost of rogue AI actually looks like
There are the obvious costs associated with rogue AI. Fines, settlements, and the legal bills after regulators start asking questions. These costs can be more easily built into budgets and reported and tracked.There is also the money you cannot count as easily. When a customer’s trust erodes, it does not show up on a balance sheet until they decide to look elsewhere for what they need. By that point organizations are paying to win it back at a much higher price than it would have cost to keep them as a customer.
A single, poor automated decision can undo years of brand goodwill. You cannot buy that back with an increased marketing budget. Companies have to earn it back slowly, except now people are closely watching and monitoring for the next mistake.
Third, and this one gets talked about the least, is the internal cost. The internal cost is when employees don’t use or trust the systems that an organization has implemented for them to use.
This could look like when teams start double-checking everything, which defeats the purpose of automating in the first place. Or worse, they stop questioning the output altogether because it is easier to defer to the machine than to push back.
What to do about rogue AI
Organizations need to start treating AI oversight as a leadership responsibility. Senior leaders need to own the outcomes these systems produce. It is not as simple as being in the room and green lighting the decision to buy or deploy them then wiping your hands of the impact.Companies also need to test how agents fail before they go live, rather than discovering those failures when a customer, employee, or regulator finds them. So many organizations rush to deploy without doing the proper work ahead of time.
And if and when something does go wrong, organizations need to have the ability to act immediately. Every autonomous system requires continuous monitoring, defined escalation paths, trained teams to identify and monitor problems, and a kill switch that can be activated without waiting for three committees and a change-management form.
When it comes to AI, the cost can quickly exceed whatever you hoped to gain from the technology in the first place. AI does not stop being your responsibility once you deploy it. I would argue that is when the responsibility actually begins. The companies that will come out ahead are the ones who can honestly say that they understand what their systems are doing and why.
Reviewed by Irfan Ahmad.
Read next:
• Workplace AI Adoption Soars as Risky Practices and Poor Oversight Undermine Organizational Safety
• Apple, Samsung, Xiaomi and Google Phones Come Loaded With Apps, Here’s Which You Can Remove
