TOP

Managing Proxies

Proxy is a critical part of web crawling. Yes, in TLDrop you can configure Delay Settings or Cache Settings to avoid IP bans. But still, they won't help you much, especially when the software sends massive requests to the web servers.

Technically, proxies will hide your real machine's IP address. Sure it will help you bypass IP rate-limiting in some popular websites such as Google or Wikipedia. Or, it can also be used to bypass IP restriction.

Once you insert proxies to TLDrop and assign them to the particular services, the software will rotate them automatically on each request.

Supported Proxy Types

To make your proxies working flawlessly, you might need to know what their protocol is. However, if you are not sure, just set them as SOCKS5. See below table for more details.

TypeExplanation
HTTP (S)

This type uses one-way request to retrieve web pages. In general, it is perfect for web browsing.

The standard ports are 80, 8080, 3128.

SOCKS5

Works by routing your internet traffic through a third-party server via TCP. It also offers more degree of anonymity.

The standard ports are 1080, 1081.

SOCKS4

Same as SOCKS5 but doesn't support authorization and UDP.

SOCKS4a

An extension to SOCKS4.

It is much better to have dedicated proxies (private proxies). The primary benefit is that your connection will become much faster than shared proxies (free proxies). In other words, those proxies are yours, and they were not shared with anyone else.

Below are dedicated proxy providers that we could recommend, they have been proven to be reliable and have great support.

Although we are not recommended you to use free proxies, in case you still want to use them, then we could recommend below sources.

Adding Proxies

Click Add or Import button, then the dialog window will show up asking you to enter a proxy list separated by a line. The proxy format per line should be host:port or host:port:username:password.

127.0.0.0:8080
127.0.0.0:8080:user:pass

You can also import proxies from a text file by clicking open from text file link. Then, you need to specify what the protocol or type is (see Supported Proxy Types table). If you are not sure, then just set to SOCKS5. Finally, click the Done button.

Editing Proxies

Just double click the proxy item you want to edit then assign edit or new value(s).

Editing proxy in TLDrop

Assigning Proxies

You will be unable to work with your proxies until you assign them to the specific services. Navigate to the Proxy Usage tab. The checked services will use your proxy list once they are working on their job.

ServiceDetails
Web CrawlerUse proxy when crawling web pages
Search Engine CrawlerUse proxy when crawling search engine result pages
Domain Vanity ServiceUse proxy when making requests to WHOIS servers, commonly used to check domain availability
Web 2.0 Vanity ServiceUse proxy when checking blog (Web 2.0) availability (not recommended)
Statistic ServiceUse proxy when checking domain metrics (not recommended)