Training on a website

The fastest way to train a chatbot is to point it at your own site.

Adding a website source

1. Go to Training and add a new source
2. Choose Website URL
3. Paste the URL
4. Decide whether to crawl recursively
5. Start the crawl

Single page vs recursive

Recursive off indexes exactly the URL you gave and nothing else. Use it when you want one specific page — a detailed pricing page, a single policy — without dragging in the rest of the site.

Recursive on follows the internal links it finds and indexes the pages behind them. This is what you want for a normal setup: paste your homepage, turn recursion on, and let it map the site.

Recursive crawling only follows links within your own site. It will not wander off to external sites you link to.

Sitemaps

If your site publishes a sitemap, SiteMesh uses it to discover pages rather than relying purely on link-following. That means better coverage of pages that are not linked prominently from the homepage.

You do not need to do anything to enable this — if a sitemap is found, it is used.

JavaScript-rendered sites

Some sites send the browser an empty shell and build the page with JavaScript. React, Vue, Next.js, Nuxt and most modern app frameworks work this way. A basic crawler pointed at such a site sees almost nothing.

SiteMesh detects this and switches approach automatically: it loads the page in a real browser, waits for the content to render, and only then extracts the text.

You will see the detected site type on the source — static, spa, nextjs, nuxt, or unknown. No action is needed either way, but it is useful when diagnosing a thin crawl.

Rendered crawls take longer

Loading pages in a real browser is slower than fetching plain HTML. A large JavaScript site can take several minutes. The chatbot answers from whatever has finished so far while the rest continues.

Reading the crawl summary

When a crawl finishes, the source shows a summary. It is the single most useful diagnostic in the product.

Figure	What it tells you
Attempted	How many URLs the crawler tried
Succeeded	How many were read and indexed
Failed	How many errored — usually server errors or timeouts
Skipped	How many were deliberately passed over, with the URLs listed

If a customer question is not being answered, check here first: the page you expected may be sitting in skipped or failed.

Pages get skipped for good reasons — duplicates, non-HTML files, pages with essentially no text. Pages fail for infrastructure reasons: the page timed out, returned an error, or was blocked.

Sparse content

A source can be flagged as having sparse content, meaning very little usable text was extracted relative to the pages visited.

Common causes:

• The site is JavaScript-rendered and rendering did not complete
• The content is inside images or embedded PDFs rather than as text
• The pages genuinely are mostly navigation and imagery

If you see this flag, look at the chunk count. A near-zero chunk count means your chatbot has almost nothing to answer from — add the content as uploaded files or pasted text instead.

Pages that will not crawl well

Situation	What to do instead
Content behind a login	Paste the text directly, or upload it as a document
Content inside a scanned PDF image	Use a searchable PDF, or paste the text
Content in a video	Use the YouTube source type
Prices or stock that change hourly	Do not index them; direct visitors to the live page instead

Keeping the crawl current

Your site will change. The index will not, until you tell it to.

→ Keeping content fresh