Scrapy

Middleware Setup

Add the proxy middleware to your Scrapy project’s settings.py:

DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
}

PROXY_USER = "your_username"
PROXY_PASS = "your_password"
PROXY_HOST = "connect.trueproxies.com"
PROXY_PORT = 8080

Custom Middleware

Create a middleware for automatic proxy injection:

class TrueProxiesMiddleware:
    def process_request(self, request, spider):
        request.meta["proxy"] = (
            f"http://{spider.settings['PROXY_USER']}:{spider.settings['PROXY_PASS']}"
            f"@{spider.settings['PROXY_HOST']}:{spider.settings['PROXY_PORT']}"
        )

Enable it in settings.py:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.TrueProxiesMiddleware": 100,
}

Country Targeting

Target a specific country in the middleware:

class TrueProxiesMiddleware:
    def process_request(self, request, spider):
        country = request.meta.get("country", "us")
        username = f"{spider.settings['PROXY_USER']}-country-{country}"
        request.meta["proxy"] = (
            f"http://{username}:{spider.settings['PROXY_PASS']}"
            f"@{spider.settings['PROXY_HOST']}:{spider.settings['PROXY_PORT']}"
        )

Use it in your spider:

def start_requests(self):
    yield scrapy.Request(url, meta={"country": "de"})

Example Spider

import scrapy

class IpSpider(scrapy.Spider):
    name = "ip_check"
    start_urls = ["https://httpbin.org/ip"]

    def parse(self, response):
        self.logger.info(f"IP: {response.json()['origin']}")

Run it:

scrapy crawl ip_check