Keeping content fresh
Your chatbot answers from content indexed at crawl time, not read live from your site. Change your prices on Monday and the chatbot keeps quoting Friday's until the index catches up.
There are three ways to catch it up.
Refresh a single source
Re-reads one data source and replaces its indexed content. Everything else is left alone.
Use this when one specific thing changed — you updated the pricing page, you replaced a PDF, you fixed the captions on a video.
It is fast, because it only touches what you pointed at.
Recrawl the whole chatbot
Re-ingests every data source attached to the chatbot.
Use this after a broad change: a site redesign, a migration, a batch of new pages, or any time you are not sure what moved.
It takes as long as the original training did, and the chatbot stays answerable from existing content while it runs.
Automatic refresh
Set a schedule in your chatbot settings and SiteMesh re-indexes on its own.
| Setting | Behaviour |
|---|---|
| None | No automatic refresh. You refresh manually. |
| Daily | Re-indexed once a day |
| Weekly | Re-indexed once a week |
| Monthly | Re-indexed once a month |
A blog or a shop with rotating stock benefits from daily. A brochure site whose content changed twice last year does not — monthly is plenty, and avoids re-crawling the same unchanged pages every night.
Automatic refresh applies to sources that have a live original to re-read — websites and YouTube videos. Uploaded files and pasted text have no external source to check, so they stay exactly as you left them until you change them yourself.
What to do when content is removed
Refreshing a crawl picks up pages that changed. It does not always resolve content you have deleted from the site, and it certainly does not remove an uploaded PDF you no longer stand behind.
When you withdraw something — a discontinued product, an expired promotion, an old policy — delete the data source that carries it. Deleting a source removes its indexed content immediately.
This matters more than it sounds. A chatbot cheerfully offering last year's 20% discount code is a live commitment to a customer.
A sensible maintenance routine
| When | Do this |
|---|---|
| You edited one page or document | Refresh that source |
| You launched a redesign or migrated the site | Recrawl the whole chatbot |
| You discontinued something | Delete the source that describes it |
| Ongoing | Set auto-refresh to match how often your site really changes |
| Quarterly | Read the live inbox for questions the bot could not answer, and fill the gaps with pasted text |
That last one is the highest-value habit in this whole page. Real conversations tell you exactly what your knowledge base is missing.