Ever watched a web scraper aggressively smash its head against a crashing website, furiously spamming requests like an impatient toddler demanding ice cream? We’ve all been there. You write a shiny asynchronous script, unleash thousands of concurrent requests, and suddenly the target server buckles under the pressure. Instead of backing off, your script doubles down, hammering the poor server until you get permanently IP-banned.
It is time to teach your scraper some manners.
In this complete guide, we are going to build a high-speed, asynchronous Python web scraper equipped with a Circuit Breaker and rate limiting. Think of it as a fuse box for your code: if things start sparking and failing, it automatically cuts the power before your whole system catches fire.
The Anatomy of a Circuit Breaker
Just like the electrical circuit breaker in your house, our software pattern has three distinct moods:
Closed (All Systems Go): Requests flow freely. Your scraper is living its best life.
Open (Code Red): Too many errors happened. The breaker trips, blocking all outgoing requests instantly without even touching the network. It protects the server and saves your CPU cycles.
Half-Open (The Trust Test): After a cooling-off period, the breaker cautiously lets one trial request sneak through. If it succeeds, we are back in business. If it fails, back to lockdown!
The Complete Production-Ready Engine
Here is the complete, self-contained project implementation combining an async rate limiter, an HTTP fetcher using httpx, and our robust AsyncCircuitBreaker.
Python
import asyncio
import time
import httpx
class AsyncCircuitBreaker:
def __init__(self, failure_threshold: int = 3, recovery_time: int = 5):
self.failure_threshold = failure_threshold
self.recovery_time = recovery_time
self.failures = 0
self.state = "CLOSED"
self.last_failure_time = 0.0
async def execute(self, func, *args, **kwargs):
# Check if the breaker is currently locked down
if self.state == "OPEN":
if time.time() - self.last_failure_time > self.recovery_time:
print("⏳ Cooling period over. Entering HALF-OPEN state for a test run...")
self.state = "HALF-OPEN"
else:
remaining = int(self.recovery_time - (time.time() - self.last_failure_time))
raise RuntimeError(f"🚫 Circuit is OPEN! Request blocked. Try again in {remaining}s.")
try:
# Attempt the async operation
result = await func(*args, **kwargs)
# If we were testing and it worked, close the circuit!
if self.state == "HALF-OPEN":
print("🎉 Test passed! Circuit closed. We are back live.")
self.state = "CLOSED"
self.failures = 0
return result
except Exception as e:
self.failures += 1
self.last_failure_time = time.time()
print(f"⚠️ Failure #{self.failures}: {e}")
if self.failures >= self.failure_threshold:
self.state = "OPEN"
print("🚨 Threshold reached! Circuit flipped to OPEN. Locking down!")
raise e
class RateLimitedScraper:
def __init__(self, requests_per_second: float = 2.0):
self.delay = 1.0 / requests_per_second
self.breaker = AsyncCircuitBreaker(failure_threshold=3, recovery_time=4)
self.semaphore = asyncio.Semaphore(5) # Max 5 concurrent tasks
async def fetch_url(self, client: httpx.AsyncClient, url: str) -> str:
async with self.semaphore:
# Enforce rate limiting pace
await asyncio.sleep(self.delay)
# Define the actual network request closure
async def _make_request():
response = await client.get(url, timeout=5.0)
# Raise an error for server-side failures (e.g., 503, 500)
if response.status_code >= 500:
raise httpx.HTTPStatusError(
f"Server error status: {response.status_code}",
request=response.request,
response=response
)
response.raise_for_status()
return response.text
# Execute via Circuit Breaker wrapper
return await self.breaker.execute(_make_request)
Running the Scraper Project
Let’s test our complete project structure. We will simulate firing requests at an endpoint that throws server errors to watch the circuit breaker trip, cool down, and recover.
Python
async def main():
scraper = RateLimitedScraper(requests_per_second=3.0)
# Using httpbin to test (we'll simulate hitting a failing endpoint)
target_urls = [
"https://httpbin.org/status/503", # Fails
"https://httpbin.org/status/503", # Fails
"https://httpbin.org/status/503", # Trips Circuit Breaker!
"https://httpbin.org/status/200", # Should be blocked by Open Circuit
"https://httpbin.org/status/200", # Should be blocked
]
async with httpx.AsyncClient() as client:
for i, url in enumerate(target_urls, start=1):
print(f"\n--- Scraping Task {i} ({url}) ---")
try:
html = await scraper.fetch_url(client, url)
print(f"✅ Success! Fetched {len(html)} bytes.")
except Exception as err:
print(f"❌ Handled Exception: {err}")
# Pause slightly between loop iterations to observe recovery states
await asyncio.sleep(1)
print("\n⏳ Waiting out the 4-second recovery period...")
await asyncio.sleep(4)
print("\n--- Final Test After Cool-down (Half-Open State) ---")
try:
# Hitting a healthy endpoint to test recovery
html = await scraper.fetch_url(client, "https://httpbin.org/json")
print(f"🎉 Recovery Successful! Response snippet: {html[:60]}...")
except Exception as err:
print(f"❌ Recovery Failed: {err}")
if __name__ == "__main__":
asyncio.run(main())
Why Your Future Self Will Thank You
By plugging this lightweight architecture into your asynchronous data pipelines, you instantly upgrade your scripts from brittle toys to production-ready systems. You get:
Resource Preservation: Your local CPU and network interfaces don’t waste cycles on dead connections.
Server Politeness: You prevent accidental Denial-of-Service (DoS) attacks on target platforms.
Graceful Degradation: Your application fails fast, logs cleanly, and recovers automatically without manual restarts.
