{"id":3047,"date":"2025-11-04T16:44:05","date_gmt":"2025-11-04T08:44:05","guid":{"rendered":"\/blog\/?p=3047"},"modified":"2025-12-09T09:47:10","modified_gmt":"2025-12-09T01:47:10","slug":"python-proxy-requests-scrapy-okkproxy","status":"publish","type":"post","link":"\/blog\/python-proxy-requests-scrapy-okkproxy\/","title":{"rendered":"How to Integrate OkkProxy Dynamic Residential Proxies in Requests & Scrapy (With Code Examples)"},"content":{"rendered":"\n

For any data acquisition or web scraping project, the greatest enemy is \u2014 and always will be \u2014 the IP ban. That\u2019s exactly why a reliable Python proxy solution has become non-negotiable in 2025: whether you\u2019re rotating residential IPs in Requests, building Scrapy middleware, or handling millions of daily requests, the right Python proxy setup is the only thing standing between you and instant blocks from Amazon, Google, TikTok, or any anti-bot system.<\/p>\n\n\n\n

You confidently run your Python scraper, only to be met with a screen full of 403 Forbidden<\/code> or 503 Service Unavailable<\/code> errors minutes later. You’ve hit the wall every scraper engineer knows: the target website has identified high-frequency requests from your single IP and blacklisted you.<\/p>\n\n\n\n

\"Access
Unlock 500M+ Real Unique Residential IPs: 100% Household Residential Proxies for Scraping, Ads & Multi-Account<\/figcaption><\/figure>\n\n\n\n

The solution? A massive, clean, rotating pool of IP addresses that mimics real user behavior.<\/p>\n\n\n\n

This is where OkkProxy Dynamic Residential Proxies<\/a> come in. With a pool of over 100 million real household IPs, OkkProxy allows your scraper to “disappear” among millions of real users across the globe.<\/p>\n\n\n\n

In this technical guide, we’ll show you exactly how to integrate OkkProxy’s residential proxies into the two most popular Python scraping libraries: Requests<\/code> and Scrapy<\/code>, complete with battle-tested anti-banning techniques.<\/p>\n\n\n\n

Preparation: Get Your OkkProxy Credentials<\/h2>\n\n\n\n

Regardless of the library, you’ll need four key pieces of information from your OkkProxy dashboard:<\/p>\n\n\n\n

    \n
  1. Proxy Host:<\/strong> un-residential.okkproxy.com<\/code><\/li>\n\n\n\n
  2. Proxy Port:<\/strong> 16666<\/code><\/li>\n\n\n\n
  3. Username:<\/strong> Your account username<\/li>\n\n\n\n
  4. Password:<\/strong> Your account password<\/li>\n<\/ol>\n\n\n\n

    Pro-Tip:<\/strong> OkkProxy allows you to control sessions by adding parameters to your username. For example, username-country-us-session-abc1234<\/code> would give you a “sticky” session from the US. To achieve “Dynamic Rotation<\/a>” (a new IP for every request)<\/strong>, we will add a random session ID to the username.<\/p>\n\n\n\n

    Method 1: Integrating OkkProxy with Python Proxy Requests<\/h2>\n\n\n\n

    Requests<\/code> is the go-to for simple web requests. Integrating a proxy is as simple as constructing a proxies<\/code> dictionary.<\/p>\n\n\n\n

    Step 1: The Basic Proxy Request<\/h3>\n\n\n\n

    This is the most basic setup. We hard-code the proxy credentials into the proxies<\/code> dictionary.<\/p>\n\n\n\n

    import requests\n\n# --- Your OkkProxy Credentials ---\nOKK_HOST = \"un-residential.okkproxy.com\"\nOKK_PORT = \"16666\"\nOKK_USER = \"YOUR_USERNAME\"\nOKK_PASS = \"YOUR_PASSWORD\"\n# --------------------------------\n\n# Construct the proxy URL\nproxy_url = f\"http:\/\/{OKK_USER}:{OKK_PASS}@{OKK_HOST}:{OKK_PORT}\"\n\n# Set up proxies for both http and https\nproxies = {\n    \"http\": proxy_url,\n    \"https\": proxy_url\n}\n\n# The URL we want to scrape\nurl_to_scrape = \"[https:\/\/httpbin.org\/ip](https:\/\/httpbin.org\/ip)\"\n\ntry:\n    # Send the request\n    response = requests.get(url_to_scrape, proxies=proxies, timeout=10)\n    \n    # Print our current exit IP\n    print(f\"Request successful! Our Exit IP is: {response.json()['origin']}\")\n\nexcept requests.exceptions.RequestException as e:\n    print(f\"Request failed: {e}\")\n<\/code><\/pre>\n\n\n\n
    Step 2: Advanced (IP Rotation & Anti-Banning)<\/h3>\n\n\n\n
    In a real-world scenario, you must do two more things to avoid being banned:<\/p>\n\n\n\n
      \n
    1. Rotate IPs:<\/strong> Ensure every request comes from a different<\/em> IP.<\/li>\n\n\n\n
    2. Fake Your User-Agent:<\/strong> Make your request look like it’s from a real browser, not a Python script.<\/li>\n<\/ol>\n\n\n\n
      We will achieve IP rotation by generating a random session ID<\/strong> for every single request.<\/p>\n\n\n\n
      import requests\nimport random\nimport string\n\n# --- Your OkkProxy Credentials ---\nOKK_HOST = \"un-residential.okkproxy.com\"\nOKK_PORT = \"16666\"\nOKK_USER_PREFIX = \"YOUR_USERNAME\" # Just the username, no session\nOKK_PASS = \"YOUR_PASSWORD\"\n# --------------------------------\n\n# A common browser User-Agent\nHEADERS = {\n    'User-Agent': 'Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/108.0.0.0 Safari\/537.36'\n}\n\ndef get_random_session_proxy():\n    \"\"\"Generates a proxy auth string with a new random session ID\"\"\"\n    \n    # Generate a 10-digit random string for the session ID\n    session_id = ''.join(random.choices(string.ascii_letters + string.digits, k=10))\n    \n    # Construct the dynamic username to rotate IP\n    dynamic_user = f\"{OKK_USER_PREFIX}-session-{session_id}\"\n    \n    proxy_url = f\"http:\/\/{dynamic_user}:{OKK_PASS}@{OKK_HOST}:{OKK_PORT}\"\n    \n    return {\n        \"http\": proxy_url,\n        \"httpsA\": proxy_url\n    }\n\n# --- Let's Start Scraping ---\nurl_to_scrape = \"[https:\/\/httpbin.org\/ip](https:\/\/httpbin.org\/ip)\"\n\nprint(\"Testing IP Rotation (Sending 3 requests)...\")\n\nfor i in range(3):\n    try:\n        proxies = get_random_session_proxy()\n        response = requests.get(\n            url_to_scrape, \n            proxies=proxies, \n            headers=HEADERS, # CRITICAL: Add the User-Agent\n            timeout=10\n        )\n        print(f\"Request {i+1} successful! Exit IP is: {response.json()['origin']}\")\n    \n    except requests.exceptions.RequestException as e:\n        print(f\"Request {i+1} failed: {e}\")\n\n<\/code><\/pre>\n\n\n\n
      (When you run this, you’ll see 3 different IP addresses printed for the 3 requests!)<\/em><\/p>\n\n\n\n
      \n\n\n\n
      Method 2: Integrating OkkProxy with Scrapy<\/code><\/h2>\n\n\n\n
      Scrapy<\/code> is the professional framework for large-scale, asynchronous scraping. Integrating OkkProxy in Scrapy requires configuring Middlewares<\/code>.<\/p>\n\n\n\n
      Step 1: Install scrapy-proxy-pool<\/code> (Recommended)<\/h3>\n\n\n\n
      While you can<\/em> write a custom middleware, the easiest and most robust method is to use the scrapy-proxy-pool<\/code> library.<\/p>\n\n\n\n
      pip install scrapy-proxy-pool<\/code><\/p>\n\n\n\n
      Step 2: Configure settings.py<\/code><\/h3>\n\n\n\n
      This is the most critical step. You need to disable Scrapy’s default downloader middlewares and enable the ones from scrapy-proxy-pool<\/code>.<\/p>\n\n\n\n
      # settings.py\n\n# ... (Your other settings) ...\n\n# --- OkkProxy Configuration ---\n# Enable the Proxy-Pool\nPROXY_POOL_ENABLED = True\n\n# Your list of OkkProxy proxies\n# Format: [http:\/\/YOUR_USERNAME:YOUR_PASSWORD@un-residential.okkproxy.com:16666](http:\/\/YOUR_USERNAME:YOUR_PASSWORD@un-residential.okkproxy.com:16666)\n# You can add multiple OkkProxy credentials or sessions here\nPROXY_POOL_PROXIES = [\n    '[http:\/\/YOUR_USERNAME:YOUR_PASSWORD@un-residential.okkproxy.com:16666](http:\/\/YOUR_USERNAME:YOUR_PASSWORD@un-residential.okkproxy.com:16666)',\n    # Example: If you want to mix in US and German sessions\n    # 'http:\/\/YOUR_USERNAME-country-us:YOUR_PASSWORD@un-residential.okkproxy.com:16666',\n    # 'http:\/\/YOUR_USERNAME-country-de:YOUR_PASSWORD@un-residential.okkproxy.com:16666',\n]\n\n# --- Disable default middlewares and enable Proxy-Pool's ---\nDOWNLOADER_MIDDLEWARES = {\n    # Disable Scrapy's default Retry and HttpProxy middlewares\n    'scrapy.downloadermiddlewares.retry.RetryMiddleware': None,\n    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None,\n    \n    # Enable Proxy-Pool's middlewares\n    'scrapy_proxy_pool.middlewares.ProxyPoolMiddleware': 610,\n    'scrapy_proxy_pool.middlewares.RetryMiddleware': 620,\n}\n\n# --- Anti-Banning: Set a User-Agent ---\nUSER_AGENT = 'Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/108.0.0.0 Safari\/537.36'\n\n# Obey robots.txt (You may want this False for production, but be careful)\nROBOTSTXT_OBEY = True\n\n# Slow down to avoid hammering the site\nDOWNLOAD_DELAY = 1.5\n# -----------------------------------\n<\/code><\/pre>\n\n\n\n
      Step 3: Create Your Spider<\/h3>\n\n\n\n
      After configuring settings.py<\/code>, you don’t need to do anything extra in your spider code<\/strong>. scrapy-proxy-pool<\/code> will handle the proxying for every request automatically.<\/p>\n\n\n\n
      # spiders\/my_spider.py\nimport scrapy\n\nclass IpCheckSpider(scrapy.Spider):\n    name = 'ip_check'\n    allowed_domains = ['httpbin.org']\n    \n    # Scrapy will automatically use different proxies \n    # from your PROXY_POOL_PROXIES for these 3 requests\n    start_urls = [\n        '[https:\/\/httpbin.org\/ip](https:\/\/httpbin.org\/ip)',\n        '[https:\/\/httpbin.org\/ip](https:\/\/httpbin.org\/ip)',\n        '[https:\/\/httpbin.org\/ip](https:\/\/httpbin.org\/ip)',\n    ]\n\n    def parse(self, response):\n        # Log the exit IP\n        self.logger.info(f\"Success! Exit IP is: {response.json()['origin']}\")\n        \n        yield {\n            'ip': response.json()['origin']\n        }\n<\/code><\/pre>\n\n\n\n
      Now, when you run scrapy crawl ip_check<\/code>, Scrapy will route all requests through your OkkProxy residential proxies, handling rotation and retries automatically.<\/p>\n\n\n\n
      \n\n\n\n
      Conclusion<\/h2>\n\n\n\n
      Whether you’re using Requests<\/code> for a quick script or Scrapy<\/code> for a complex AI data<\/a> project, integrating OkkProxy’s Dynamic Residential Proxies<\/a> is the key to bypassing IP bans and ensuring your data collection succeeds.<\/p>\n\n\n\n
      By faking a real User-Agent<\/code> and leveraging OkkProxy’s massive 100M+ IP pool for rotation, your scraper becomes unstoppable.<\/p>\n\n\n\n
      Ready to level-up your scraping projects? Visit OkkProxy<\/a> today and start your free trial<\/strong> (no credit card required) to experience the power of a premium residential proxy network.<\/p>\n","protected":false},"excerpt":{"rendered":"
      For any data acquisition or web scraping project, the greatest enemy is \u2014 and always will be \u2014 the IP ban. That\u2019s exactly why a reliable Python proxy solution has become non-negotiable in 2025: whethe\u2026<\/p>\n","protected":false},"author":3,"featured_media":3287,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2,4],"tags":[],"class_list":["post-3047","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-proxies","category-api"],"_links":{"self":[{"href":"\/blog\/wp-json\/wp\/v2\/posts\/3047","targetHints":{"allow":["GET"]}}],"collection":[{"href":"\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"\/blog\/wp-json\/wp\/v2\/comments?post=3047"}],"version-history":[{"count":5,"href":"\/blog\/wp-json\/wp\/v2\/posts\/3047\/revisions"}],"predecessor-version":[{"id":3551,"href":"\/blog\/wp-json\/wp\/v2\/posts\/3047\/revisions\/3551"}],"wp:featuredmedia":[{"embeddable":true,"href":"\/blog\/wp-json\/wp\/v2\/media\/3287"}],"wp:attachment":[{"href":"\/blog\/wp-json\/wp\/v2\/media?parent=3047"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"\/blog\/wp-json\/wp\/v2\/categories?post=3047"},{"taxonomy":"post_tag","embeddable":true,"href":"\/blog\/wp-json\/wp\/v2\/tags?post=3047"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}