Companies are aware of the virtually unlimited potential of artificial intelligence. Credit goes in part to generative AI, which has extended awareness of AI capabilities beyond industry insiders, but also to the results many companies have already achieved in terms of efficiency, productivity, automation, and support for strategic and operational decisions.
Expectations are very high. Artificial intelligence can influence competitiveness, but it is crucial that each organization can measure its actual performance (AI performance), namely the benefits the company has obtained and is obtaining from its investments. Unfortunately, this is not straightforward.
AI performance, asking the right questions
Companies betting on AI must account for a considerable investment in skills, infrastructure, technologies and consulting. This creates an immediate need to demonstrate its value as accurately and quickly as possible. Only then can investments be justified, new investments secured, stakeholder trust gained, and alignment of ongoing initiatives with business objectives demonstrated.
From this perspective, it might seem that measuring AI performance means merely calculating the ROI of investments. However, this is not entirely correct, as ROI is a central but not exhaustive indicator. Instead, the company must clearly define the objectives of individual initiatives and adopt indicators that allow them to monitor the solution's effectiveness and progress made so far. In other words, the company must first ask the right questions about how to evaluate the effectiveness of the solutions being implemented, which naturally depends on their purpose.
- How has the CX improved since we introduced artificial intelligence?
- Has employee turnover decreased?
- Have production costs decreased?
- Has downtime decreased?
- How is the solution helping us achieve our ESG goals?
- What is the impact on employee engagement?
- Is AI actually helping the organization make better decisions?
Success metrics: from adoption to model accuracy
In the data-driven era, measuring success means defining and tracking specific performance indicators (KPIs), which once again depend on the initiative's goal and the questions posed. Starting from this premise, it is possible to highlight the most common ones.
Adoption rate
If AI is integrated into an internal solution, measuring the adoption rate is crucial to understanding its actual or perceived usefulness. We talk about perceived usefulness because an unsatisfactory adoption rate does not only depend on the solution's ineffectiveness, but also on the company's inability to manage the complex issue of change and transform/evolve corporate culture.
Customer experience metrics
If AI is introduced to enhance customer relations, the primary indicators are those of customer experience (CX). It starts with the traditional NPS (Net Promoter Score), and then delves into more specialized indicators such as CSAT (Customer Satisfaction Score), churn rate, and many others. Rather than comparing with market benchmarks, the key aspect here is monitoring one's progress over time.
Employee Experience Assessment
Maintaining high morale among people is crucial in a productivity-oriented company. Here, artificial intelligence can act as a personal assistant to employees, help them with many tasks (e.g., searching for complex information), and relieve them from routine operations (e.g., chatbots). In this case, success metrics include turnover rate, various engagement KPIs, satisfaction index, and absenteeism rate.
AI model performance
The term AI performance does not only refer to the business effects of the solution but also the effectiveness, or rather the accuracy, of the AI technology or model implemented. Here, various indicators used by data scientists come to the rescue, which are essential for building trust in artificial intelligence: F1 Score, Confusion Matrix, Mean Squared Error (MSE), and many others. There are also benchmarks like Glue and SuperGlue to evaluate specific features such as natural language understanding capabilities.
ROI of AI investments, the key indicator
We mentioned that ROI is not the only indicator to monitor, but it remains central because it measures the profitability of the investment, calculated as the ratio between the invested capital and the realized profit. The ROI of AI investments is an indicator of the effectiveness of artificial intelligence, regardless of its purpose.
Calculating AI ROI presents significant challenges, primarily for two reasons:
- The involvement of multiple intangible elements;
- The highly evolutionary nature of AI. For example, the accuracy of machine learning model results changes over time; this affects the beneficial business effects and, consequently, the ROI of initiatives.
Additionally, it appears that companies make several errors precisely when measuring the return on investment in AI. A few years ago, PwC identified the top three.
- Not accounting for the uncertainty due to AI models not having 100% accuracy. Therefore, the cost of errors must be considered, which few do;
- Evaluating ROI in a static way. As seen, AI performance changes over time, and ROI measurement must take this into account;
- Evaluating projects as single units without seeing the big picture of the entire portfolio and its effects on the overall business.
A path to calculating AI ROI
There are several approaches to calculating return on investment. The problem is not the calculation methodology, for which a simple cost-benefit ratio or more complex methods such as payback period calculation can be used, but rather identifying all direct, indirect, and somewhat hidden entries that fall into the macro areas of benefits and costs.
1. Quantification of Tangible and Intangible Benefits
Quantifying tangible and intangible benefits is a key element in calculating ROI. Tangible benefits are easily quantifiable and measurable in economic terms, such as operational cost savings and increased direct revenues.
Intangible benefits are harder to quantify but equally significant. Consider, for example, a reduction in product defects, an enhancement of brand reputation, or organizational improvement in terms of agility. These are elements that can accelerate (or not) the success of any organization but are not always translatable into financial terms.
2. Quantification of investment costs
Identifying all costs is (obviously) fundamental, but again this is a highly complex factor due to the number of non-apparent or easily quantifiable items. For example, the following should be considered:
- Development and implementation costs of the solution;
- Consultancy costs;
- Recurring operational costs, such as maintenance and system updates;
- Change management initiative costs;
- Training costs necessary to fully exploit the solution;
- Integration costs with existing systems;
- Data management costs.
3. Definition of an adequate timeframe
As mentioned, it is very important to establish the right moment to measure ROI, as models and technologies are continually evolving and changing. The timeframe should be selected considering that solutions require a period of adaptation and optimization before producing significant results. This approach reduces the unrealistic expectations that unfortunately often accompany the entire topic of artificial intelligence.
The importance of measuring AI performance
The promises of AI are fascinating, and there is a risk that companies might adopt the technology without having defined realistic expectations and adequate budgets.
Integrating AI into a company entails significant costs and requires not only skills and technologies but also careful planning. It is therefore essential to set objectives, define methods to monitor progress, and predict and evaluate actual benefits. Relying on an expert consultant can make a difference.
The topic is constantly evolving, and the above represents only an overview of a much broader and more complex subject that will become increasingly relevant over time. However, companies should immediately equip themselves not only on the technical front but also on the performance evaluation front to get the most out of it.
-1.png "002-Kirey Group - una riga - (color)-1")
