TOP

Crawler Settings

This crawler setting contains two properties that have a significant impact when you try to find expired domains. They are Maximum Links per Level and Maximum Crawler Level option. You can play around with their values to get the desired results.

NameExplanation
Maximum Links Per Level

Skip or break the crawler service from collecting links when a maximum number of links is reached. 0 means unlimited.

Please consider with your CPU capability, for example, if you have 512 MB of RAM then 2000 links will be suitable. We should not force our CPU capability, because it might throw a System Out of Memory error.

The default value is 0.

Maximum Crawler Level

On the web crawling world, this option is also known as crawler depth.

The higher value, the longer process will take. The higher value, the opportunity to get more domains is increasingly open.

The default value is 5, and it should between 1-100.

Maximum Search Engine Pages

If you’re not using search queries when creating a crawler task, you can skip this option.

Essentially, this number will tell the software how many search result pages on the search engine should be crawled.

The default value is 10, and it should between 1-1000.

Use Google Search

Tell the software to crawl Google Search Result pages when you're using search queries as seed items.

You should never enable this option if you don't have any proxies. Otherwise, Google will ban your IP due to too many numbers of requests.

The default value is not checked.

Use Bing Search

Tell the software to crawl Bing Search Result pages when you're using search queries as seed items.

The default value is checked.

Use Yahoo Search

Tell the software to crawl Yahoo Search Result pages when you're using search queries as seed items.

The default value is checked.