Blog / Build with AI /
Building a chatbot on your own documents
Wouldn’t it be handy if you could ask a question and get an answer from your own documents, not from the whole internet? A shipping policy, a product manual, a set of client notes, a help centre. That’s what a chatbot on your own documents does.
The technique has a name: retrieval-augmented generation, usually shortened to RAG. In plain words, the chatbot first finds the most relevant passages in your documents, then gives them to an AI model together with the question and says “answer using only this”.
That’s the whole trick. The model isn’t retrained on your files. It’s handed the right pages at question time, a bit like an open-book exam.
How it works, step by step
Imagine a library with a very fast assistant.
- Prepare the documents. Your files are cut into chunks, maybe a few paragraphs each. Whole documents are too big and too vague to search well.
- Turn chunks into numbers. Each chunk is converted into an embedding, a list of numbers that captures its meaning, so similar ideas sit close together. I explained that in embeddings explained simply.
- Store them. The numbers go into a searchable store, often called a vector database. See vector databases explained without maths.
- Search at question time. When someone asks something, their question is turned into numbers too, and the closest chunks are found.
- Answer from the evidence. The best few chunks go to the AI model along with the question and an instruction to answer from them only, ideally saying where the answer came from.
If you’ve understood the library analogy, you’ve understood the system.
Do you actually need to build one?
Before you build anything, check if a simpler option works:
- Small set of documents? You may be able to paste the lot into a modern AI chat that accepts long inputs or file uploads. No code needed. Context windows explained for builders explains the limits.
- Hosted tools. Several services let you upload documents and get a chatbot you can embed on a site. They’re quicker to set up, but check pricing, data policy and how well they cite sources.
- Custom build. Makes sense when you need control over data, design, or cost, or want to learn how it works.
For a first project, I’d say start with the simplest option and only build when you hit a real limit.
A minimal build plan
Here’s the shape of a small version you could build with AI help. It’s a plan, not a finished product.
Goal: a private page where you ask questions about a folder of text files and get answers with the source file named.
Parts:
- A script that reads your files and splits them into chunks of a few hundred words, with a little overlap so ideas aren’t cut mid-thought.
- A call to an embeddings service to convert each chunk to numbers, saved to a file or small database.
- A search step: embed the question, compare it with the stored chunks, pick the top few.
- A prompt that sends those chunks and the question to a chat model.
- A simple page with a text box and an answer area.
A prompt to start with:
I want to build a small question-answering tool over a folder of .txt files, using Python. Plan it in steps before writing code. Use an embeddings API to embed chunks of about 300 words with 50 words of overlap, store them in a local file, and for each question retrieve the top 4 chunks by similarity. Then ask a chat model to answer using only those chunks and to name the source files. If the chunks don’t contain the answer, it must say so. Keep every API key in environment variables.
Build it in small pieces. Test the chunking first. Then test search alone, before adding the chat step. If search returns the wrong chunks, the answer will be wrong no matter how clever the model is.
Where it goes wrong
- Bad retrieval. The most common failure. The right passage exists but isn’t found, so the model answers from nothing or from the wrong text. Try different chunk sizes and check what is retrieved.
- Confident wrong answers. Even with your documents, models can blend in guesses. The instruction “say you don’t know” helps but isn’t perfect. Evaluating AI output: simple tests for quality gives you a way to check.
- Out-of-date documents. If your policy changed and the old PDF is still in the store, the bot will quote the old one.
- Messy sources. Scanned pages, tables and PDFs with odd layouts often come out as garbage text.
- Privacy. Documents sent to an online service leave your machine. Don’t upload anything confidential without permission, and if you want the data to stay local, look at running an LLM on your own computer.
- Prompt injection. A document can contain text that tries to instruct the model (“ignore your rules…”). For a public bot reading untrusted content, this is a real risk.
A test set beats a feeling
Write 15 to 20 real questions, with the answers you expect. Run them after every change. Mark each as right, partly right or wrong, and note where the answer came from. This will show you whether chunk size, wording or model choice made things better or worse. It also stops you fooling yourself after seeing three good demos.
What it means for a website owner
A documents chatbot is most useful where people ask the same questions again and again: shipping, returns, sizing, set-up guides. It can cut repetitive support. It also puts a wrong answer in front of a customer in your name. Keep a clear route to a human, label it as an automated assistant, and don’t let it promise things your policy doesn’t say.
Key takeaways
- A documents chatbot finds relevant passages first, then asks a model to answer from them.
- Retrieval quality decides answer quality.
- Try a no-code or hosted option before building.
- Test with a fixed list of real questions.
- Keep sensitive documents and keys under control.
Related reading
Which collection of documents do you wish you could simply ask a question?