Skip to main content

Training overview

"Training" in SiteMesh means giving your chatbot content to answer from. Every piece of content you add is a data source, and a chatbot can have as many as you like.

The four source types​

TypeBest forGuide
Website URLYour public pages: services, pricing, about, blog, FAQWebsites
File uploadBrochures, manuals, price lists, policy documentsFiles
Pasted textAnything not published anywhere — internal policies, seasonal notesText and YouTube
YouTubeProduct demos, tutorials, recorded webinarsText and YouTube

Mix them freely. A typical setup is one recursive website crawl plus two or three uploaded PDFs.

Training statuses​

Each data source moves through a lifecycle you can watch on the Training page:

StatusMeaning
PendingQueued, not started yet
ProcessingBeing fetched, read and indexed
DoneIndexed successfully and available to the chatbot
ErrorSomething went wrong — an error message explains what

Your chatbot has its own status alongside these:

StatusMeaning
IdleNo content yet — it cannot answer anything
TrainingAt least one source is still processing
ReadyIndexed and answering
ErrorTraining failed; check the individual sources
tip

A chatbot in training still answers using whatever has already finished. You do not have to wait for a large crawl to complete before testing.

Pages and chunks​

After training you will see two numbers.

Pages is how many documents were read — web pages crawled, or one per uploaded file.

Chunks is how many searchable pieces those pages were broken into. Each chunk is roughly a thousand characters, with deliberate overlap between neighbours so no sentence gets orphaned by a split.

Chunks are what the chatbot actually searches, so the chunk count is the better measure of how much your bot knows. A 40-page site producing 600 chunks is healthy; 40 pages producing 45 chunks means most of those pages were nearly empty.

How much content is enough?​

More is not automatically better. A focused knowledge base beats a sprawling one, because every irrelevant chunk is one more thing competing to be retrieved.

Good practice:

  • Include the pages that answer real customer questions — services, pricing, policies, contact details, FAQ, how-to content
  • Skip thin pages with no substance: tag archives, pagination, image galleries, login pages
  • Split very large documents into focused ones rather than uploading a single 400-page manual

Removing content​

Deleting a data source removes its indexed content too — the chatbot stops being able to answer from it immediately. There is no separate "unlearn" step.

This is the right move when you have taken a product off sale or a policy has been withdrawn. Leaving stale content indexed is how a chatbot ends up quoting a discount you no longer offer.

Keeping it current​

Content is indexed when you crawl it, not read live from your site. When your site changes, the index needs to catch up — either manually or on a schedule.

→ Keeping content fresh

When answers are not good enough​

There is a diagnostic playbook for this — start with the crawl summary rather than rewriting your system prompt.

→ Improving answers