Who’s Responsible When the AI Gets It Wrong?

Introduction

At NAGGL’s Annual Conference in San Diego, lenders were asked whether they had tried building their own AI agents. Plenty of hands went up. Asked about their biggest challenges, the answers came quickly: consistency and accuracy.

A follow-up question drew laughter: Who had dedicated resources to manage those agents? For many, it had become another responsibility alongside their existing jobs.

The exchange captured where AI adoption stands across much of SBA lending. Interest is high, experimentation is widespread, and lenders are discovering what it takes to make the technology dependable.

During “State of AI: How Are You Using AI in Your Shop?”, Lama AI’s CEO Omri Yacubovich joined fellow panelists and lenders in the audience to discuss their experiences. Four lessons emerged:

1) Think about how AI will work across your institution

A tool that helps one experienced lender presents a different set of considerations when an entire team begins using it.

An attendee described a concern: two underwriters could ask essentially the same question using different prompts and receive different answers. The institution’s standards, however, need to remain consistent.

That makes the surrounding process essential. AI needs the right context, defined boundaries, and validation. Employees need clear guidance on which tools they can use and what information they can share. Changes need testing before they reach the wider team.

Institutions evaluating their options should consider those responsibilities from the outset. Who maintains the technology? How are results checked? What happens when the underlying model changes? These questions determine whether an experiment can become a reliable part of daily operations.

“AI is not our strategy. It is supplementing and supporting our strategy, but it can’t be the strategy.”

2) Use AI where it creates measurable business value

The session included examples of AI preparing loan-review checklists, supporting equity-injection analysis, and helping lenders examine portfolio data. Each addressed work that consumes time and attention.

But the discussion also exposed a complication: an answer generated quickly can still take considerable time to verify.

One lender described an AI-generated equity analysis that included a deposit unsupported by the source documents. She caught the error during review. In another instance, the tool recommended contacting the customer about a $2.50 duplicate debit, a detail that offered little value to the equity-injection review. Together, the examples illustrated why useful output needs both accuracy and relevance. Producing more findings can create more work when the system cannot distinguish what matters.

Evaluate the full task, including preparation, review, corrections, and moving the result into the loan file. Consider whether AI is the appropriate tool for each step. Some activities benefit from interpreting unstructured information; others call for established rules and straightforward automation.

The business objective should guide those choices: faster processing, greater capacity, better borrower service, or more time for credit analysis.

“I need to know that they can call the AI out if it’s wrong.”

3) Make human oversight practical

Keeping humans in the loop was a recurring theme. The discussion gave that familiar phrase a more concrete meaning.

Reviewers need to understand where information came from, what the technology did with it, and what someone subsequently changed. They also need the ability to intervene when an exception requires judgment.

As Yacubovich described, a useful audit history can follow information from the arrival of a document through extraction and subsequent adjustments. That visibility helps employees review the work and explain how a result was reached.

Expertise remains just as important. One lender shared that before new reviewers can start using AI they are required to do things “the good old way”, and understand the underlying process, so they develop the knowledge needed to challenge an incorrect answer.

The same principle applies to credit narratives. AI can organize information and improve clarity. The lender still needs to understand the business and explain why the credit makes sense.

4) See how the entire process works. Not a demo.

For lenders evaluating technology, the panel offered direct advice: ask to see it live, using your own data, from start to finish.

A demonstration becomes more useful when it reflects the documents, policies, and exceptions employees encounter every day. Can the team verify an answer? Does the information stay connected to the deal? What happens when a person needs to review or change something?

Look at the handoffs, too. If employees must repeatedly copy information between tools, reconcile outputs, and update records manually, that work remains part of the lending process.

The broader opportunity is to connect the steps: information arrives, analysis is prepared, exceptions reach the appropriate person, and the deal moves forward with its context intact.

Building for what comes next

The discussion in San Diego showed an industry beginning to work through the practical consequences of AI adoption. Lenders are finding opportunities throughout the loan lifecycle, while confronting questions about consistency, accountability, and how to develop the next generation of lending expertise. Their experiences suggest that lasting progress will depend on how well institutions connect AI to their policies, people, data, and everyday workflows.

The session also offered a glimpse further ahead: business owners whose AI assistants could communicate their financing needs directly to lenders, reducing the need to complete forms and repeatedly supply information. That possibility brings today’s decisions about lending infrastructure into sharper focus. Institutions need a foundation that can evolve as technology and borrower expectations change. For SBA lenders, the opportunity is to expand how much business they can serve while giving their people more time to understand borrowers, exercise judgment, and help sound businesses access capital.