Crawler Health and automatic cron
How to use health actions, source health and the cron worker.
Automatic crawler
DoorFall's cron endpoint should be called regularly by your hosting control panel. The Admin page shows the exact Web cron URL and recommended schedule for your installation.
If PHP CLI is available, calling cron.php directly is preferable because it does not need the web cron key.
The worker schedules sources that are due, recovers abandoned jobs and processes small batches to avoid normal web-request timeouts.
Retry failed jobs
Use this for failures that may have been temporary, such as a server error or connection problem.
Do not endlessly retry permanent 403/404/robots restrictions.
Clear failed jobs
Removes failed queue records after you have reviewed them. It does not make an inaccessible page crawlable.
Clear skipped jobs
Clears recorded skipped jobs such as oversized, slow or content-poor pages.
Source health
Source Health summarizes the overall source: indexed pages, queued work, skipped/failed counts, last successful crawl and next automatic crawl.
A few failed URLs should not make an otherwise successful source unhealthy.
Stuck jobs
A job can appear stuck if a worker was interrupted while it was marked Processing. DoorFall can recover abandoned processing jobs so they can continue safely.