“Best LLM for customer support” is usually answered with a list of models that sound impressive. Real data on how they perform at actual customer support tasks is often missing. Research from the Qiscus AI team, published at ICADEIS 2026 (IEEE), tested two popular models directly, GPT-4.1 and Gemini 2.5 Flash. The test focused on the single most critical task in AI-powered customer support, deciding when a bot should hand a conversation over to a human.
This article covers five LLMs relevant to customer support, with the deepest coverage going to the two models that were actually tested. The answer to “which LLM is best” turns out to be more complicated than “model A is smarter than model B”. The result depends heavily on the AI Agent architecture in use, not just the raw capability of the model itself.
What an LLM for Customer Support Actually Does
An LLM for customer support is the large language model behind an AI Agent. It understands customer questions, searches for answers, and decides when a conversation needs to go to a human. The quality of this LLM determines how well the AI Agent understands intent and how accurate its handover decisions are.
Not every LLM is built with the same priorities. Some models are optimized for speed and low cost. Others are optimized for complex reasoning. Some are optimized specifically for following long, detailed business instructions consistently. This difference in priorities is why “best LLM” is not a single answer. The right answer depends on the AI Agent architecture in use and the type of questions your CS team handles most.
How the Data Behind This Comparison Was Collected
The credibility of the comparison below depends on how representative the underlying test data actually is. This section covers how the ICADEIS 2026 research built its test data.
1. The QiscusCS Dataset Was Built from Real Conversations, Not Simulations
The research built a dataset called QiscusCS, made up of 100 real customer service dialogues with 4,265 utterances total. Of those, 3,612 utterances were labeled “transferable”, meaning they needed to go to a human. The remaining 653 were labeled “normal”. Each dialogue averaged about 44 utterances, long enough to reflect real customer support conversations rather than a single simple question.
2. Labeling Was Done Manually by an Actual Customer Service Team
Every utterance in the dataset was manually labeled by three domain experts from the Qiscus customer service team, not by an automated system. This approach means the “transferable” or “normal” label reflects real human judgment from people who actually handle escalations, not a rule someone guessed at. All personal customer data was replaced with dummy data before the research was conducted.
Why Indonesian and Regional Businesses Need the Right LLM for Customer Support
Picking the wrong LLM has a direct impact on customer experience, not just a technical problem hidden behind the scenes.
1. Your LLM Choice Determines Handover Accuracy
A weak LLM can hold onto a customer too long before escalating, or hand off a simple conversation it could have solved itself. The ICADEIS 2026 research shows this performance gap between LLMs is real and measurable, not just an assumption.
2. Operating Costs Vary a Lot Between Models
Models with more reasoning capacity usually cost more per API call. A business that picks a model without considering conversation volume can end up paying far more than necessary, without a matching improvement in results.
3. Architecture Can Change How Much Your LLM Choice Even Matters
This is the finding most businesses overlook. The same research found that LLM choice is highly decisive in a single-agent architecture, but its influence shrinks dramatically once a multi-agent architecture is used. Businesses that understand this can save on cost without sacrificing performance.
4. Representative Test Data Makes the Decision More Trustworthy
LLM decisions are often based on vendor demos using ideal scenarios, not messy real conversation data. The research above avoids that trap by building a dataset from real customer service conversations, manually labeled by an actual CS team. Businesses that follow a similar approach when evaluating LLMs for their own needs get a far more realistic picture than trusting marketing claims alone. This matters especially for teams evaluating multiple vendors at once. Every vendor’s demo tends to look polished under controlled conditions that rarely match day-to-day support volume.
5 Best LLMs for Customer Support
The five models below are ordered by depth of available data. The first two were tested directly for the handover task. The remaining three have not been tested for this specific task, so they are covered based on publicly known characteristics instead.
1. GPT-4.1 (OpenAI), Strong Performance in Single-Agent Architecture
GPT-4.1 scored a Macro F1-Score of 0.5356 in a single-agent architecture for the handover task, well above Gemini 2.5 Flash on the same architecture. Its precision on the majority class reached 0.9608, and its AUC was 0.7017, the highest of the four model-and-architecture combinations tested in this research. In a multi-agent architecture, GPT-4.1 scored a Macro F1-Score of 0.6159, nearly matching Gemini 2.5 Flash on the same architecture. GPT-4.1 is a solid choice for both single-agent and multi-agent setups, with more consistent performance across both architectures than Gemini.
2. Gemini 2.5 Flash (Google), Best Performance Only in Multi-Agent Architecture
Gemini 2.5 Flash produced a sharp contrast between architectures. In single-agent, its Macro F1-Score was only 0.3333, far below GPT-4.1. Recall on the minority class was actually high at 0.9219, but precision was very low at 0.1259. This is a sign the model held onto conversations too often when it should have escalated. The picture flips completely in multi-agent architecture. Gemini 2.5 Flash actually posted the highest Macro F1-Score in the entire experiment, 0.6260, slightly ahead of GPT-4.1. Gemini 2.5 Flash is a strong choice, but only when paired with multi-agent architecture, not used on its own.
3. Claude (Anthropic), Known for Following Complex Instructions
Claude was not tested specifically for the handover task in this research, so performance data for this scenario is not yet available. The model is broadly known for consistently following long, complex instructions, a trait relevant to customer support with intricate business rules. Businesses considering Claude for customer support should run internal testing first, ideally following a similar methodology to the research above, before rolling it out fully.
4. Llama (Meta), an Open-Weight Option for Full Control
Llama offers a different advantage from closed-source models like GPT and Gemini, the ability to self-host. This matters for businesses with strict data compliance needs, such as banking or healthcare. These businesses may not want customer conversation data passing through a third-party server. No performance data exists for the handover task in this research. Performance claims for this specific scenario still need internal verification, especially since open-weight model performance can vary a lot depending on how it’s fine-tuned.
5. Mistral, an Efficient Option for High Volume with Controlled Cost
Mistral is known for being relatively lightweight and cost-efficient compared to larger models. This trait can appeal to businesses with high conversation volume but relatively low question complexity, like an online store handling order-status questions. As with Claude and Llama, no specific handover-task testing exists for this research. Performance claims remain general rather than directly tested, so they still need validation against your own business’s conversation data before wide adoption.
LLM Comparison Table for Customer Support
The table below summarizes the available data from the ICADEIS 2026 research for the two directly tested models.
| Model and Architecture | Macro F1-Score | AUC | Majority Precision | Minority Recall |
|---|---|---|---|---|
| GPT-4.1, Single-Agent | 0.5356 | 0.7017 | 0.9608 | 0.7656 |
| GPT-4.1, Multi-Agent | 0.6159 | 0.6881 | 0.9430 | 0.5521 |
| Gemini 2.5 Flash, Single-Agent | 0.3333 | 0.6053 | 0.9708 | 0.9219 |
| Gemini 2.5 Flash, Multi-Agent | 0.6260 | 0.6646 | 0.9354 | 0.4583 |
One insight stands out from this table. GPT-4.1 stays competitive across both architectures, while Gemini 2.5 Flash only pulls ahead once paired with multi-agent architecture. Research from the Qiscus AI team published at IEEE concluded that LLM capability is highly dominant in single-agent systems. That influence drops off significantly once the system is multi-agent.
Criteria for Choosing the Right LLM for Customer Support
The three criteria below help determine which LLM best fits your business needs.
1. Decide Your AI Agent Architecture First
The architecture decision, single-agent or multi-agent, should come before the LLM decision, not after. The data above shows that the right LLM for single-agent can be suboptimal for multi-agent, and the other way around.
2. Consider Volume and Question Complexity
Businesses with high volume and relatively simple questions can consider a more cost-efficient model. Businesses with complex, higher-risk questions should prioritize accuracy over cost efficiency.
3. Test with Real Conversation Data Before Deciding
The ICADEIS 2026 methodology used a real conversation dataset, not artificial scenarios, to produce reliable numbers. Businesses that want to compare LLMs for their own needs should follow a similar approach, using real conversation data rather than just vendor demos.
4. Weigh the Precision and Recall Trade-off
The table above shows that a model with high recall on hard cases doesn’t always have good precision, and vice versa. Businesses need to decide upfront which matters more to them. The choice is between catching every hard case, even at the risk of false alarms, or keeping precision high even if cases slip through. This decision should be based on the real cost of each type of mistake for your business. Don’t just pick whichever number looks highest in the comparison table.
5. Watch for the Same Trade-off During Your Own Testing
When you run your own internal comparison, the same precision-recall pattern will likely show up again. A model that looks strong on one metric can look weak on another, depending on which type of mistake costs your business more. Deciding this trade-off before testing, rather than after seeing the results, keeps the evaluation honest. It also stops the temptation to favor whichever model happens to score best on the metric you didn’t originally prioritize.
Why AI Agent Architecture Matters More Than Picking an LLM
Focusing too much on “which LLM is best” can distract from the decision that actually matters more, the AI Agent architecture itself.
1. Multi-Agent Architecture Reduces the Risk of Picking the Wrong LLM
The data above shows the performance gap between LLMs is much smaller in multi-agent architecture. Businesses using this architecture have more room to switch LLMs later without rebuilding their system from scratch.
2. Single-Agent Architecture Demands More Care When Choosing an LLM
Businesses using single-agent architecture need to be more careful with LLM choice. A wrong model choice shows up directly in handover decision quality, as seen with Gemini 2.5 Flash in the test above. Teams running single-agent setups should budget extra time for model evaluation upfront. The cost of getting this one decision wrong is higher than it would be in a multi-agent setup.
How Qiscus AgentLabs Helps Businesses Avoid Getting Locked into One LLM
Customer service AI solutions from Qiscus are built to support multi-agent AI architecture. This lines up with the research finding above that this architecture reduces how dependent results are on any single LLM. Businesses don’t have to commit to one model and can adjust their LLM choice as needs change. They avoid rebuilding their AI Agent architecture from scratch every time they want to experiment with a new model.
For a broader look at tool options, the most complete comparison of AI agent tools for customer service covers wider platform choices. This goes beyond just the underlying LLM. AI customer support software options reviews specific platforms in more depth. Talk to Qiscus about your AI Agent needs to see which LLM and architecture configuration fits your business.
Best LLM vs Best Architecture, What’s the Difference
Many businesses assume “which LLM is best” and “which architecture is best” are the same question. They influence each other, but they are not identical.
| Aspect | Best LLM | Best Architecture |
|---|---|---|
| What it measures | The model’s own capability | How that model is used and how roles are divided |
| Impact if chosen wrong | Varies, small in multi-agent, large in single-agent | Consistently affects the quality of every handover decision |
| Example from the research | Gemini 2.5 Flash weak in single-agent, strong in multi-agent | Multi-agent handles hard cases more consistently |
The same logic behind AI Agent vs chatbot applies here too. Just as an AI Agent isn’t simply a chatbot with a smarter script, the best LLM isn’t simply the model with the most parameters. It’s the model that fits the architecture and use case you’ve chosen.
How to Start Choosing an LLM for Customer Support
The five steps below help your team start the LLM selection process in a structured way.
1. Map Your Customer Question Volume and Type
Gather historical CS conversation data to understand the split between simple and complex questions before considering which model fits.
2. Decide Your AI Agent Architecture First
Determine whether your business fits better with single-agent or multi-agent AI. This decision affects how much your LLM choice will actually influence the final result.
3. Test at Least Two Models on Real Conversation Data
Run a small test with two or more models on a sample of real conversations. Measure metrics like precision and recall for the cases your CS team runs into most often.
4. Compare Cost per Conversation, Not Just Cost per Token
Calculate total cost based on actual conversation volume. A model that looks cheap per token can end up more expensive if it needs more back-and-forth turns to resolve a single case.
5. Build In a Path to Switch Models Later
Make sure your AI Agent architecture isn’t too tightly coupled to one specific LLM. This gives your business the flexibility to adapt once a better model shows up on the market.
The Best LLM Choice Always Depends on the Architecture Around It
No single LLM is automatically the best choice for every customer support scenario. GPT-4.1 performed consistently across both tested architectures, while Gemini 2.5 Flash only pulled ahead when paired with multi-agent architecture. Other models like Claude, Llama, and Mistral remain worth considering, but they need internal testing since no specific handover-task data exists for them yet.
The more important question isn’t just “which LLM is best”, it’s how that LLM will actually be used within your AI Agent architecture. Explore Qiscus customer engagement solutions to see how the right architecture can help your business choose an LLM with more confidence.
Frequently Asked Questions About the Best LLM for Customer Support
Based on the ICADEIS 2026 research, GPT-4.1 performed most consistently across both tested architectures. Gemini 2.5 Flash only pulled significantly ahead when used in a multi-agent architecture.
Not always. The research shows that AI Agent architecture, single-agent or multi-agent, can affect the final result more. This effect is often bigger than the raw size or capability of the LLM itself.
It depends on the architecture. In single-agent setups, LLM choice has a major impact on the result. In multi-agent setups, the impact of LLM choice is much smaller, since specialized roles already help balance out a weaker model’s shortcomings.
Gather real conversation samples from your CS team, then test a few LLMs on the same data. Measure metrics like precision and recall for the cases that come up most often.
Yes, especially if the architecture is designed to be flexible, like multi-agent AI. Businesses with a rigid single-agent architecture usually need more effort to switch models later.
