← All features
Always Fresh. Always Accurate.

# Crawling Interval

Stale search results erode user trust. Keyspider's crawling interval settings let you control exactly how often each source is re-indexed — from continuous monitoring to scheduled daily, weekly, or monthly crawls. And for urgent updates, manual on-demand re-indexing is always one click away.

Request a demo
See AI Search platform
Per-source crawl frequency — set different intervals for different content types
On-demand manual re-crawl triggered from the admin console
Continuous monitoring mode for high-frequency content updates
Scheduled crawls: hourly, daily, weekly, or monthly
Crawl change detection — only re-indexes pages that have actually changed
Crawl reports show what was added, updated, or removed each run

## How it works

1

### Set a baseline crawl schedule

For most knowledge base and documentation sites, a daily crawl is sufficient. For news sites or frequently updated portals, hourly or continuous monitoring keeps results current.

2

### Keyspider detects and indexes changes

Rather than re-crawling every page every time, Keyspider compares page content against the last indexed version. Only changed pages are re-processed — saving time and compute.

3

### Trigger manual re-crawl when needed

Published a critical policy update at 9am? Trigger a manual re-crawl from the admin console and the new content is live in search within minutes — no waiting for the next scheduled run.

## Use cases

### News and editorial sites

Articles published throughout the day need to appear in search immediately. Continuous or hourly crawling ensures nothing sits unindexed for long.

### Government and public sector

Policy documents and service pages are updated infrequently but must be accurate when they are. Daily or weekly crawls with on-demand override handles this perfectly.

### E-commerce catalogues

Product availability, pricing, and descriptions change constantly. Hourly crawls keep the search index aligned with the live catalogue without manual intervention.

## Related

Web Crawler
Multiple Data Sources
Content Type to Crawl

## Ready to give your users better answers?

AI Search, AI Assistant, and Workplace Search. Deployed in days, not months. See it live on your own content.

Book a Demo Talk to Sales

No credit card required · Live in 2 weeks · Cancel anytime