Crawler page and counters
How to run crawls and understand the queue and progress counters.
Website source
Choose one source to work on, or All active sources where available.
Pages this run
Select the size of the manual batch: 10, 25 or 50 pages. Small batches are best for testing a new source.
Counters
Queued means URLs waiting to be processed.
Total Indexed is the number of pages currently stored for the source.
Indexed This Run counts pages successfully added or refreshed during the current run.
Processed This Run includes URLs examined during the run even when some are skipped, blocked or fail.
Failed means a crawl attempt ended with an error that was not classified as a normal skip/block.
Blocked is used for intentional restrictions such as robots.txt or a source exclusion rule.
Retry scheduled means DoorFall recognized a temporary error and queued another attempt.
Skipped means DoorFall deliberately moved past a page that should not stop the batch, such as an oversized, slow or content-poor page.
Crawl percentage
The percentage reflects progress through the current queued work and is useful even at the beginning of a large crawl.