How to measure and communicate the ROI of AI tools 

Editor’s Note: This article is an extension of Mr. Rebmann’s presentation at HousingWire’s 2025 AI Summit held in mid-August. 

Lenders are under unrelenting pressure to cut costs, accelerate processes and enhance borrower experiences while navigating a less-than-certain originations environment. Artificial Intelligence (AI) has emerged as a tool to meet these demands, yet quantifying its business value can be elusive. 

In a 2024 Gartner survey of more than 600 organizations, nearly half said their number-one barrier to adopting AI was the difficulty of estimating and demonstrating its business value. This outweighed concerns over a lack of talent, data or even trust in AI. 

In the lending world, where margins are thin and every investment must show a clear return, this challenge can delay or derail initiatives that might otherwise deliver transformative results. 

Why the dollars don’t add up easily 

One reason it’s hard to pin down AI’s ROI is the way AI costs are structured. Lenders are used to working with Software as a Service (SaaS) offerings, which tend to have predictable pricing, often aligned to a price per loan or, like in the case of Dark Matter’s Empower loan origination system (LOS), a price per closed loan. Loan-based pricing is easy to estimate and align to Gain-on-Sale calculations. 

Generative AI costs are priced quite differently. Gen AI charges per use, using tokens to measure use. Gen AI charges per use, using tokens to measure use. Roughly translating to four characters of English text, token use can vary wildly based on use case, and even within a use case. 

Using a chat use case as an example, even a short input from a user can generate thousands of tokens once you add in all the extra components of the process needed to ensure a reliable and relevant response — like system prompts, context, guardrails and the response itself. In addition, the uncertainty introduced by the end user determines when and how often to use AI, making costs unpredictable.  

Finally, the choice of model used in the use case can dramatically impact cost. For example, ChatGPT 5.0 can be twenty times more expensive than ChatGPT 5.0 Nano for the same token use. If the use case doesn’t require the heavier reasoning capabilities of ChatGPT 5.0, a lender can save significant money and get faster responses using ChatGPT 5.0 Nano. 

Another reason business value is difficult to quantify is that AI is inherently iterative. Similar to traditional software projects, AI requires continuous refinement. The difference is the pace of iteration and the need to continuously test to ensure models are not drifting or changing, even if inputs haven’t changed.  

Models often begin with promising results on a small proof of concept or pilot, then need significant tuning, retraining and prompt adjustments before they perform at production standards across the various scenarios in a full pipeline. AI’s fluid nature means that costs and benefits are constantly moving targets, making estimating value a matter of educated guesses.

This makes predicting the measurable business improvements that justify the investment a challenge. AI might save time on a particular task, improve loan quality, provide better compliance accuracy or a more seamless borrower experience. Additionally, many of the high-level business metrics we want to influence, like gain on sale, have significant latency in impact. Today’s changes may not be seen in those numbers for months or quarters. This leads to a breadcrumb trail of KPIs rooted in assumptions about effects, which may or may not have any basis.

Shifting the focus from tech specs to business impact

When evaluating AI’s return, lenders must focus on business outcomes, not just technical metrics. Accuracy percentages and processing speeds have their place, but the real test is whether AI improves the lender’s business.

That means defining the key performance indicators that the AI-enabled solution should move. Ideally, these are no more than one degree away from that primary business objective. If the primary aim is to gain sales, KPIs such as reducing cycle times, lowering defect rates or reducing investor findings can be good measures. Using these types of measures reinforces that AI is a component in a broader operational solution, with its performance measured both by technical metrics of the model and the business objective and key results they are intended to improve.  

Ultimately, if high-level business metrics are not moving favorably even when model metrics are impressive, the investment is of questionable value. 

Proxy metrics can be helpful for truly intangible gains reported by teams, like borrower satisfaction or customer stickiness. For example, if AI improves borrower satisfaction, customer satisfaction measures should improve. If customers are more sticky, retention rates should improve. The key is to connect intangible benefits to measurable business outcomes.

Why the “hours saved” formula falls short

Now that we’re measuring the impact, we must resist the urge to translate that impact to dollars for use in a business case. For example, to calculate ROI on a time reduction benefit, many lenders use a traditional formula of determining the time saved on a task, multiplying by the number of times that task is performed, and multiplying by the fully loaded hourly rate of the staff performing it. This process will result in awe-inspiring savings numbers. However, this method has a critical flaw in that the value of an hour saved is not automatically the staff member’s hourly rate. It’s an increase in capacity, which represents latent opportunity.

Unstructured capacity results in leakage. In their Q2 2025 CFO Report, Gartner reported that only 30% of the capacity gained is put toward work that improves business outcomes. Capturing AI’s full value means intentionally redeploying the capacity it frees toward activities that matter, such as resolving complex exceptions or handling a higher volume of loans without adding headcount. Driving this work through structured capacity management and task management capabilities allows lenders to capitalize on the capacity gains made by implementing AI or other efficiency practices. 

A practical ROI framework for lenders begins with defining the desired business outcomes (such as shortening cycle times, reducing errors or raising borrower satisfaction scores) and establishing a baseline before implementation. During proofs of concept and pilots, measure the costs incurred closely along with the business metrics, as this is the best opportunity to get a true sense of what token consumption will be. 

To the extent possible, run in parallel with current processes, to help reduce the impact of other variables like market movement, measuring gains both in tangible results and through proxy metrics for intangibles. Feedback loops should be built into the process, from end users and more automated feedback forms, such as corrections to responses. Use this feedback to help identify opportunities to improve the capabilities or areas where a human’s judgment will be required.

Results should be communicated in the context of strategic goals. For example, “AI reduced post-close review time by 20%, enabling us to process an additional $50 million in loan volume annually without increasing headcount” is far more compelling than citing hours saved alone. 

Turning potential into proven results

We generally believe that AI can be transformative in mortgage lending, but only data from measuring business impact will prove this out. Expectations should be realistic and reflect the iterative nature of AI, especially early on when models are still being tuned. The focus must remain on business outcomes, with every hour of freed capacity intentionally used to drive measurable lift.

Craig Rebmann is the Product Evangelist at Dark Matter Technology.
This column does not necessarily reflect the opinion of HousingWire’s editorial department and its owners. To contact the editor responsible for this piece: zeb@hwmedia.com.