Table of contents
Proxy is a critical part of web crawling. Yes, in TLDrop you can configure Delay Settings or Cache Settings to avoid IP bans. But still, they won't help you much, especially when the software sends massive requests to the web servers.
Technically, proxies will hide your real machine's IP address. Sure it will help you bypass IP rate-limiting in some popular websites such as Google or Wikipedia. Or, it can also be used to bypass IP restriction.
Once you insert proxies to TLDrop and assign them to the particular services, the software will rotate them automatically on each request.
Supported Proxy Types
To make your proxies working flawlessly, you might need to know what their protocol is. However, if you are not sure, just set them as SOCKS5. See below table for more details.
This type uses one-way request to retrieve web pages. In general, it is perfect for web browsing.
The standard ports are 80, 8080, 3128.
Works by routing your internet traffic through a third-party server via TCP. It also offers more degree of anonymity.
The standard ports are 1080, 1081.
Same as SOCKS5 but doesn't support authorization and UDP.
An extension to SOCKS4.
Recommended Proxy Providers
It is much better to have dedicated proxies (private proxies). The primary benefit is that your connection will become much faster than shared proxies (free proxies). In other words, those proxies are yours, and they were not shared with anyone else.
Below are dedicated proxy providers that we could recommend, they have been proven to be reliable and have great support.
Although we are not recommended you to use free proxies, in case you still want to use them, then we could recommend below sources.
Click Add or Import button, then the dialog window will show up asking you to enter a proxy list separated by a line. The proxy format per line should be
You can also import proxies from a text file by clicking open from text file link. Then, you need to specify what the protocol or type is (see Supported Proxy Types table). If you are not sure, then just set to SOCKS5. Finally, click the Done button.
Just double click the proxy item you want to edit then assign edit or new value(s).
You will be unable to work with your proxies until you assign them to the specific services. Navigate to the Proxy Usage tab. The checked services will use your proxy list once they are working on their job.
|Web Crawler||Use proxy when crawling web pages|
|Search Engine Crawler||Use proxy when crawling search engine result pages|
|Domain Vanity Service||Use proxy when making requests to WHOIS servers, commonly used to check domain availability|
|Web 2.0 Vanity Service||Use proxy when checking blog (Web 2.0) availability (not recommended)|
|Statistic Service||Use proxy when checking domain metrics (not recommended)|