Why a website or page cannot be crawled

Blocked by robots.txt

The website's robots.txt file tells DoorFall not to crawl the URL. DoorFall follows Allow/Disallow rules and the most specific matching rule wins.

This is an intentional website-owner restriction, not a DoorFall error.

HTTP 401

The page requires authentication. A public crawler cannot access content behind a login.

HTTP 403

The server understood the request but refused it. The site may block automated crawlers, restrict IP addresses, use a firewall/CDN rule or require browser-specific access.

Do not repeatedly retry a persistent 403.

HTTP 404 or 410

The page is missing or permanently gone. DoorFall uses repeated missing responses before removing previously indexed content, which protects the index from a single temporary bad response.

HTTP 429

The website is rate limiting requests. Wait and retry later. Frequent 429 responses are a sign that the source should be crawled less aggressively.

HTTP 5xx

The source server has an error or is temporarily unavailable. These errors are normally worth retrying.

Timeout or resolving timeout

The remote server, DNS lookup or connection took too long. DoorFall skips slow pages so one URL does not stop an entire crawl batch. Repeated timeouts mean the website may not be a reliable source.

Page exceeds 2 MB crawl limit

DoorFall intentionally stops downloading/indexing HTML pages larger than 2 MB. The page is marked Skipped and the crawler continues with the next URL.

The limit protects your server from unusually large responses and runaway memory/bandwidth use.

Insufficient indexable content

DoorFall reached the page but could not extract enough useful searchable text, or a usable title was missing.

Common reasons:

• the page is mostly navigation, images or scripts;
• meaningful content is loaded only by JavaScript;
• the page is a thin utility/landing page;
• the HTML does not contain normal readable content.

The page is skipped because indexing it would usually reduce search quality.

Non-HTML content

DoorFall's website crawler is designed primarily for HTML pages. PDFs, media files, downloads and other content types are not treated as normal searchable web pages by this crawler.

Too many or unsafe redirects

DoorFall limits redirect chains and refuses redirects to private/non-public network hosts. This protects the server from unsafe requests and redirect loops.

SSL/DNS/connectivity problems

An invalid certificate, DNS failure, unreachable host or connection failure can prevent fetching. Confirm that the URL opens publicly over normal HTTP/HTTPS and retry.

Source exclusion rule

If the URL matches a pattern configured under Exclude URL patterns, DoorFall marks it blocked intentionally.