IT teams planning to build their own chatbot usually get stuck on one question, where does a usable dataset chatbot actually come from. A dataset chatbot is a collection of conversation data, question and answer pairs, or labeled dialogue used to train and test conversational AI models. Building a dataset that genuinely represents real customer conversation patterns turns out to be far harder than just downloading a CSV file from the internet.
This article covers the types of dataset chatbot available, and how to build one from scratch. It also covers a real case study from the Qiscus AI team, who built a dataset from the ground up for their own research. At the end, there is a build versus buy comparison so the decision rests on data, not assumptions.
What Is a Dataset Chatbot and Its Main Types
A dataset chatbot is a collection of text messages, dialogues, or question and answer pairs used to train and test conversational AI models. The quality of this dataset directly determines how well the model understands context and user intent.
1. Open Domain Dataset
An open domain dataset contains free-form, casual conversation, such as movie subtitles or social media threads, used to train general purpose conversational bots. This type suits a chatbot that needs to sound natural, but is less suited to business cases that demand high precision.
2. Task Oriented Dataset
A task oriented dataset contains structured dialogue designed for a specific goal, such as booking a ticket or making a reservation. Each conversation turn is usually tied directly to a step toward completing that task.
3. Customer Support Dataset
A customer support dataset contains question and answer pairs focused on intent detection and issue resolution. This is the type most relevant for businesses that want to build their own AI Agent for customer service.
4. Multilingual Dataset
Businesses serving customers across multiple languages need a dataset that covers the language variations relevant to their market. This type is usually the hardest to find publicly, since dialect or code-switched language representation rarely exists in large volumes. A multilingual AI chatbot depends heavily on this kind of coverage to perform consistently across every language it serves.
How to Build a Dataset Chatbot From Scratch
Building a dataset chatbot fit for production requires a much longer process than simply collecting conversation text. The four stages below are the standard workflow most data science teams follow.
1. Collect Conversation Data From Relevant Sources
Collect real conversation logs from channels the business already uses, such as WhatsApp, live chat, or email. Synthetic data can be a supplement, but real conversation data still represents actual customer language patterns more faithfully.
2. Clean and Anonymize Sensitive Data
Replace all personal identifying information, such as names, emails, and phone numbers, with dummy data before the dataset is used for training. This step is mandatory to protect customer privacy and meet data protection regulations.
3. Manually Annotate Every Conversation
Every message in the dataset needs a relevant label, such as whether it requires escalation to a human or can be handled automatically. This process is ideally done by a domain expert who understands the business context, not a generic annotator. The QiscusCS case study below shows this approach directly, with three domain experts from the Customer Service team brought in specifically to label every utterance. Generic annotators unfamiliar with Qiscus’s customer service context were not involved in this process.
4. Validate Label Quality Before Using It for Training
Run cross checks between annotators to confirm the resulting labels are consistent. A dataset with inconsistent labels will lower model accuracy, even if the data volume is large. A binary label scheme like the one used in QiscusCS, transferable or not transferable, tends to be easier to validate. A scheme with many categories usually takes more effort to keep consistent.
This four stage process is a general outline. Once the dataset is ready, businesses need to go further, starting with how to train an AI Agent properly.
Case Study, How the Qiscus AI Team Built Their Own Dataset Chatbot
The challenge of building a dataset is not an abstract story. The Qiscus AI team went through it directly while researching the handover process from chatbot to human agent.
1. Why the Qiscus Team Built Their Own Dataset
Research published by the Qiscus AI team at IEEE notes one important gap. A public dataset for the chatbot to human handover task that reflects real customer service conversation patterns simply did not exist. The team ended up building their own dataset, named QiscusCS, drawn from conversation data on the Qiscus customer engagement platform.
2. Scale and Annotation Process of the QiscusCS Dataset
The QiscusCS dataset consists of 100 dialogues covering more than 4,000 utterances, precisely 4,265 utterances, with an average of 44.43 utterances per dialogue. Every utterance was manually annotated by three domain experts from the Qiscus Customer Service team, using a binary label of transferable or not transferable. All customer identity data was replaced with dummy data before use, to protect privacy.
3. The Data Structure the Qiscus Team Used
Every row in the QiscusCS dataset stores six columns, namely user name, user email, conversation room id, message content, sender role, and the transferable label. A structure this simple turned out to be enough to train a model that detects when a conversation needs escalation to a human.
4. A Good Dataset Alone Does Not Guarantee Model Performance
The same QiscusCS dataset was used to compare two architectures, Single-Agent and Multi-Agent. The Multi-Agent architecture with Gemini 2.5 Flash recorded a Macro F1-Score of 0.6260. That number is far above the Single-Agent architecture using the same model, which recorded only 0.3333. The dataset was identical, but performance could differ by nearly double depending on the architecture used to process it.
This finding matters for teams planning to build their own dataset. Time invested in careful annotation can go to waste if the architecture sitting on top of that dataset is not designed correctly.
What This Architecture Comparison Teaches Us
The performance gap between Single-Agent and Multi-Agent on the same dataset offers a concrete lesson. It matters for teams considering an agentic AI approach for their own needs.
1. The Base Model Matters Less on a Multi-Agent System
On the Multi-Agent architecture, Gemini 2.5 Flash actually scored slightly higher than GPT 4.1, even though it is generally considered a lighter model. This result shows that architectural design can matter more than simply picking the largest available base model.
2. The Base Model Matters a Lot on a Single-Agent System
On the Single-Agent architecture, GPT 4.1 recorded a Macro F1-Score of 0.5356, far above Gemini 2.5 Flash at only 0.3333. On a simpler system, the capability of the base model becomes a much bigger factor.
3. Escalation Timing Affects Accuracy on Hard Cases
The Single-Agent Gemini model tended to hold onto conversations too long before escalating, sometimes never escalating at all. This pattern lowered precision on the minority class cases that genuinely needed expert handling.
4. No Single Metric Is Enough to Judge Model Quality
Interestingly, the Single-Agent OpenAI architecture recorded the highest AUC among all four setups, at 0.7017. This happened even though its Macro F1-Score trailed far behind the Multi-Agent setups. AUC measures a model’s ability to distinguish between classes across every threshold. Macro F1-Score measures the balance of precision and recall at one specific threshold instead. Teams that look at only one metric risk drawing the wrong conclusion about which model is actually production ready.
The Hidden Cost of Building a Dataset Chatbot From Scratch
The cost of building a dataset chatbot rarely stops at annotation hours alone. The following components are often missing from a team’s initial estimate.
1. Domain Expert Time Diverted Into Annotation
The QiscusCS case study above involved three domain experts from the Customer Service team, not general annotators hired on the side. Domain expert time that would normally go toward daily operational work instead got allocated to labeling thousands of utterances.
2. Architecture Experiments That Require a Trial and Error Cycle
As the Single-Agent versus Multi-Agent comparison above shows, a good dataset alone is not enough to guarantee good results. Teams need to test multiple architectures and base models before finding a combination that genuinely works for their business case. This often means testing different AI Agent template configurations before landing on a setup that fits.
3. Ongoing Dataset Maintenance
Customer question patterns shift over time, especially when a new product or policy launches. A dataset that is not updated regularly will become steadily less relevant to the real conversations coming in every day.
Data Compliance Considerations When Building a Dataset Chatbot
A dataset chatbot built from real customer conversation logs automatically contains personal data. Teams that treat compliance as an afterthought risk facing a legal problem, not just a technical one.
1. Anonymization Is Not Just Deleting Names
The QiscusCS case study above shows the right approach, with all customer identity data replaced by dummy data before it was used for research. Proper anonymization also needs to consider combinations of data that could indirectly identify someone, not just names and emails treated in isolation.
2. Consent for AI Training Use Should Be Explicit
Businesses collecting customer conversation data for model training purposes still fall under data protection regulation in most markets. Consent to use data for AI training should be stated explicitly in the privacy policy. It should never be assumed to be automatically covered by standard customer service consent.
3. Restrict Access to the Raw Dataset
Limit access to the raw dataset to only the team members genuinely involved in annotation and model training. The more people who can access raw data containing customer conversations, the greater the risk of a data leak.
Dataset Chatbot From Scratch vs Ready to Use AI Agent, Which Makes More Sense
This is not a question of which option is more technically advanced. It is a question of time allocation and the risk a team is willing to carry. Two examples below show how this calculation differs depending on business scale.
An e-commerce business with thousands of conversations a day usually already has enough raw data to build its own dataset. The problem is, a business like this rarely has an internal data science team to process that data. A mid-sized business with lower volume usually faces the opposite problem. Its raw data is not yet large enough to train a reliable model, which makes the ready to use option more sensible from the start.
| Aspect | Build Your Own Dataset | Ready to Use AI Agent |
|---|---|---|
| Time to production | Months, including annotation and architecture testing | Weeks, mostly knowledge base configuration |
| Team required | Domain experts for annotation, data engineers for model testing | A CS or IT team managing the knowledge base |
| Performance risk | Depends on dataset quality and architecture choice | Architecture already tested, just needs business context |
| Data control | Full, data and model stay entirely internal | Customer data still managed inside the platform in use |
The practical takeaway from the table above is simple. Building your own dataset makes sense for a research team that genuinely needs full control over the model. For a business chasing fast time to production, the ready to use option is usually more realistic.
How a Ready to Use AI Agent Avoids the Dataset Burden
Businesses without a dedicated data science team can still get an AI Agent that performs well. This is possible without going through the process of building a dataset from scratch.
1. A Knowledge Base Replaces the Need for Raw Conversation Data
AI Agent that skips building a training dataset from scratch runs on a knowledge base filled from existing business documents and FAQs. The source is not thousands of utterances that need manual annotation. Teams simply feed in material that already exists, rather than building a conversation dataset from zero.
2. The Architecture Is Already Tested, So Teams Skip the Experimentation
The QiscusCS case study above shows just how large the performance gap between architectures can be. Using an AI Agent whose architecture has already gone through internal testing changes the equation. A business team does not need to repeat that experimentation cycle from the start.
3. Team Focus Shifts to Business Context, Not Model Engineering
Without the burden of building and maintaining a dataset, IT and CS teams can shift their focus. They can refine the knowledge base and escalation flow to fit business needs instead. This territory sits much closer to their actual expertise than machine learning model engineering. Personalizing an AI Agent to match brand voice becomes the team’s main work, not rebuilding a dataset every time performance dips.
PCS cut repetitive work by 30 percent with Qiscus AI after moving from a basic bot setup to a more capable AI Agent. That shift happened without needing to build an internal data science team to get there.
This approach is also relevant for teams weighing the risk of AI Agent hallucination when building conversational AI in house. An unproven architecture trained on a small internal dataset carries a real risk of generating confident but incorrect answers.
How to Start, Whether Building Your Own or Evaluating a Ready to Use AI Agent
The steps below help a team decide the most realistic direction based on the capacity actually available.
1. Audit Internal Data Science Capacity
Map out whether the internal team has the capacity for manual annotation, architecture experimentation, and long term dataset maintenance. A capacity gap in any of these areas is usually a strong signal to consider the ready to use option. Understanding when to escalate from AI to human support becomes easier once a team stops treating escalation logic as something built entirely from scratch.
2. Map the Volume and Type of Conversations to Handle
Estimate daily conversation volume and the types of questions that come in most often. This data is useful both for designing a custom dataset and for populating a ready to use AI Agent’s knowledge base.
3. Compare Estimated Time to Production
Compare the estimated time to build and test a dataset and architecture in house against the implementation time of a ready to use AI Agent. The difference in time to production is often the deciding factor, especially for a business already feeling the operational impact of an overwhelmed CS team. As a rough reference, manual annotation for just one hundred dialogues in the QiscusCS case study involved three people working in parallel. A business that needs a dataset covering a wider range of topics should plan for more annotation time. Time needed tends to scale up proportionally with dialogue count and topic variety.
4. Start Small Before Committing Fully
Whether building a dataset in house or adopting a ready to use AI Agent, start with a limited topic scope first. Expand to the full conversation flow only after that scope proves out. Starting small makes results easier to evaluate before committing significant resources.
5. Revisit the Decision Every Few Months
Business needs and conversation volume can change, so a build or buy decision made early on does not have to be permanent. Periodic review helps a team adjust strategy as soon as scale or complexity shifts. This also applies to teams rolling out AI Agent benefits gradually across channels, rather than all at once.
Dataset Is Not Everything, Execution Determines the Result
The QiscusCS case study proves one important point. A well annotated dataset is only a starting point, not a guarantee of results. The architecture processing that dataset, and how consistently it gets maintained over time, matter just as much as the quality of the data itself. Businesses that understand all three factors early are better equipped to make a realistic build or buy decision.
Teams with real internal research capacity can get significant value from building their own dataset. Teams chasing fast time to production usually benefit more from an AI Agent whose architecture has already been proven.
Explore Qiscus to see how AI Agent capabilities can run without burdening a team with a dataset project from scratch. The performance is already tested, not the result of an unproven internal experiment.
Frequently Asked Questions About Dataset Chatbot
A dataset chatbot is a collection of conversation data, question and answer pairs, or labeled dialogue used to train and test conversational AI models. The quality and relevance of this dataset directly affects how well the model understands user context.
A dataset chatbot can come from public repositories like Kaggle or GitHub for general use cases, or be built from internal conversation logs for a specific business case. Public datasets usually represent a particular business context less accurately than a dataset built directly from a company’s own real conversation data.
There is no fixed number that applies universally, since the required data volume depends on the complexity of the case being handled. The QiscusCS case study, for example, used 100 dialogues with more than 4,000 utterances for architecture comparison research, not for training a production model from scratch.
Not always. The QiscusCS case study shows that two architectures using an identical dataset can produce very different performance, with a gap of nearly double on Macro F1-Score. The architecture processing the data matters just as much as the dataset’s volume.
A business without an internal data science team, or one that needs fast implementation, usually benefits more from a ready to use AI Agent. Building a dataset from scratch makes more sense for a research team that genuinely needs full control over the data and the model.
