AI model security has become a critical concern as financial institutions increasingly rely on machine learning to automate lending, fraud detection, trading, and risk management. While these systems can improve decision-making and operational efficiency, they must also be protected against manipulated data, adversarial attacks, and other forms of model poisoning that could compromise their performance.
As AI adoption accelerates across financial services, organizations are placing greater emphasis on validating training data, testing models before deployment, and implementing safeguards that detect suspicious inputs. Building resilient AI systems requires not only strong cybersecurity practices but also ongoing human oversight to ensure models continue to operate as intended.
The experts have proposed a few approaches that could cripple the processes AI manages, but mitigation approaches seem relatively clear. The experts argue that misleading inputs, such as fake trading data, would trip up the AI. But if fake data can be easily injected without notice, it would likely also fool a human. More importantly, assuming the model was trained in a controlled environment and the model tested before deployment, then it can be tested for how it would behave if it is fed fake data. Better yet, a model could be trained to detect fake data and shut the system down.
A specific example given is that the lending model might be flooded with phony loan applications that would alter the model in a negative way. In the 2017 Report Bringing AI Into the Enterprise: A Machine Learning Primer, Mercator identified the importance of thoroughly vetting training data. A machine learning professional should never train a model on data that hasn’t been carefully evaluated to assure it matches reality and is also not including biased data. Even after validating the training data the model needs to be tested. Perhaps testing should include ingestion of malicious data, but that shouldn’t be difficult to implement.
To inflict systemic damage to the bank or the entire financial system would require the system be operating without human observation and executing either extremely high value transactions or many relatively low value transactions extremely quickly. Consider for example card authorizations. Large banks process millions of transactions every day, but the input data is highly secure and is forced to adhere to a strict ISO format. Altering the input on a large scale isn’t likely. Perhaps more likely is that an AI-based fraud detection company is infiltrated and a trojan model distributed to the endpoints. This could be catastrophic but it isn’t an AI-based attack, it’s a traditional cyber-attack that in my opinion is far more likely to be pursued because it uses existing capabilities and create maximum damage downstream:
Machine-learning models vary in their levels of sophistication, from those that use relatively simple algorithms to complex black-box AI systems, so named because, like human brains, they can’t be simply opened up to see exactly how decisions are being made. And like human brains, AI platforms can be susceptible to being fed faulty information, including by attackers seeking to manipulate them.
Russian expertise in using the Internet and social media to disseminate disinformation could easily be turned against machine-learning models that, like other investors, turn to the Internet to try to gauge market sentiment.
‘Misinformation about a takeover being imminent, or a public-relations debacle unfolding, could easily fool a financial institution’s trading systems, Mr. Gupta said’”
I expect this would also fool human traders?
Protecting AI model security requires more than developing accurate machine learning models—it demands rigorous testing, continuous monitoring, and strong governance throughout the model lifecycle. Financial institutions must ensure that training data is trustworthy, models are resilient against manipulation, and automated systems can identify abnormal behavior before it creates broader operational or financial risks.
While AI introduces new attack surfaces, many of the greatest threats still originate from traditional cyberattacks targeting the systems that support machine learning. Combining robust cybersecurity with responsible AI development will be essential to maintaining trust in AI-driven financial services.
Overview by Tim Sloane, VP, Payments Innovation at Mercator Advisory Group
