Why is continuous monitoring essential?
Remember the panic just a few months ago when Facebook was down for almost six hours? Their customers around the globe could not engage and the damages were apparent on different levels.
Now imagine if it had been a major financial institution.
That would be very problematic for the business on different fronts. In today’s digital era, where consumers are expecting to be served immediately and efficiently, banks and insurance companies cannot afford to be offline at all, let alone for six hours.
For financial institutions, dynamic business monitoring systems that continuously check for Key Performance Indicators (KPIs) and for model performance are now a necessity. In the UK, for example, there are some insurance companies with quote volumes for an auto insurance policy in the millions per day – imagine what could go wrong in a day without proper monitoring.
Every company monitors its KPIs in one way or another. However deep analysis is typically only done on an ad-hoc or infrequent basis. As such detection of KPIs “drifts” might be uncovered either late or potentially not at all. This is a major concern for business executives; the fear and anxiety of learning too late that they have a significant problem. In addition, the focus of some monitoring reports is on the KPIs themselves and not on the underlying factors comprising them. Monitoring only KPIs might miss changes to the underlying factors; for example, there could be opposite trends with respect to changes in the mix of business and model(s) performance such that KPIs are not affected but, if one of the trends intensifies, there could be an issue.
A key concept of the system is that it continuously monitors KPIs concurrently checking for changes in the data, and tracking performance of the various models behind the KPIs. In many cases, changes in the mix of business or errors in the data pipelines are not revealed until quite late in the process which can cause a lot of issues. An intelligent monitoring system prevents this from happening by continuously checking for such errors or changes. It could, for example, alert the business that its internal rating system generates only price discounts due to some error or that KPIs will not be achieved due to incorrect imputation of missing data.
The system should generate alerts specific to organizational roles, with a focus on business executives. The alerts need to be meaningful with a call to action, based on a “laser sharp” comprehensive root cause analysis as to the sources of the drift.
Checking for KPIs, model and data drifts, and identifying the root cause behind the drifts involves computation and testing of numerous statistical metrics behind the scenes. For more technical users, the system should include detailed reports which feature various charts and statistical tests behind all the alerts that enable them to take a deeper look and investigate. However, since time is of the essence, it is critical to have meaningful and timely alerts so the business can act immediately and not wait for data scientists to analyze the situation. That takes some time, and the business often needs to respond with more urgency.
To properly monitor KPIs on a continuous basis, there needs to be a constant feed of outcomes into the system; unfortunately, the outcome of a transaction often is not known or cannot be verified until only after a certain period of time. As such, the main tool which is used to anticipate changes in KPIs is through monitoring changes to the data process and the mix of business. If there is a certain development of KPIs that can be modeled based on historical data, this can be factored into the monitoring system.
An advanced feature of the system is its ability to not only alert and analyze, but also provide continuous updated forecasts and early warning signs of poor performance. In addition, it can update automatically the various predictive models via advanced Machine Learning algorithms making sure the business can move quickly. Once a drift is discovered, alerts are sent, and predictive models are updated, action is needed. The specific action will depend on the business environment and goals it needs or wants to achieve. In some situations, the system could be configured on an “auto” pilot mode and automatically deploy alternative pricing strategies. In most cases, human intervention is needed before moving forward with a revised pricing strategy.
When Facebook goes down, users are upset – but a financial institution? That is potentially catastrophic to the business. With the reliability and analytics capabilities of today’s AI and automation technology, now is the time to upgrade and invest in an intelligent monitoring solution. This will not only ensure that the business is continuously watched for but also provide the peace of mind to business executives knowing that the business they manage is protected.