Configure TrueProxies with Crawl4AI

Crawl4AI takes a proxy per crawl run: pass a ProxyConfig with the TrueProxies server, your username and your proxy password as proxy_config on CrawlerRunConfig. Crawl4AI drives a Chromium browser, so use the HTTP port. Targeting and sticky sessions ride on the username, exactly as in How to connect.

Installation

pip install -U crawl4ai
crawl4ai-setup

crawl4ai-setup installs the browser Crawl4AI drives. The examples use the ProxyConfig class from Crawl4AI 0.9.

Basic usage

import asyncio

from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, ProxyConfig

proxy = ProxyConfig(
    server="http://YOUR_HOST:YOUR_HTTP_PORT",
    username="your_username",
    password="your_password",
)


async def main() -> None:
    run_config = CrawlerRunConfig(proxy_config=proxy)
    async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
        result = await crawler.arun(url="https://httpbin.org/ip", config=run_config)
        print(result.success, result.markdown)


asyncio.run(main())

The printed page shows the exit IP address. Without a session option, each new connection gets a new exit IP. A browser keeps connections to a site open and reuses them, so pages fetched from one site in quick succession can share an exit.

Residential IPv4 country targeting

Add the option to the username:

proxy = ProxyConfig(
    server="http://YOUR_HOST:YOUR_HTTP_PORT",
    username="your_username-country-de",
    password="your_password",
)

Use a separate CrawlerRunConfig for each country when one crawl covers several:

def run_config_for(country: str) -> CrawlerRunConfig:
    return CrawlerRunConfig(
        proxy_config=ProxyConfig(
            server="http://YOUR_HOST:YOUR_HTTP_PORT",
            username=f"your_username-country-{country}",
            password="your_password",
        )
    )


async def crawl_by_country(crawler: AsyncWebCrawler, url: str) -> None:
    for country in ("de", "fr", "us"):
        result = await crawler.arun(url=url, config=run_config_for(country))
        print(country, result.success)

Residential IPv4 sticky sessions

Keep one exit IP for a multi-page crawl, such as a paginated listing, by adding a session value and a lifetime in seconds:

proxy = ProxyConfig(
    server="http://YOUR_HOST:YOUR_HTTP_PORT",
    username="your_username-session-listing01-lifetime-600",
    password="your_password",
)

The session value is 1 to 32 letters or digits. The accepted lifetime range depends on the product; see the options table.

When a crawl fails

A refused proxy connection shows up as result.success being False, with a browser-level message in result.error_message. The browser does not show the proxy’s reason. Send the same username and proxy password with curl -v and read the X-Proxy-Reason header, then look it up in Proxy errors and refusal reasons.