Rate Limits

Understanding API rate limits and how to work with them.

Overview

ScraperCompany enforces rate limits to ensure fair usage and system stability. Rate limits are applied per API key and vary by plan.

Limit Types

1. Monthly Credit Limit

Each plan includes a monthly allocation of credits:

2. Concurrent Request Limit

Maximum number of simultaneous requests your key can make:

3. Per-Second Rate Limit

All plans are subject to a per-second rate limit to prevent abuse:

• Free/Starter: 5 requests/second
• Growth: 20 requests/second
• Scale: 50 requests/second
• Enterprise: Custom

Rate Limit Headers

Every API response includes rate limit information in the headers:

X-RateLimit-Limit: 50000
X-RateLimit-Remaining: 49850
X-RateLimit-Reset: 1698624000
X-Concurrent-Limit: 10
X-Concurrent-Used: 3

429 Too Many Requests

When you exceed a rate limit, you'll receive a 429 response:

{
  "success": false,
  "error": {
    "code": "rate_limit_exceeded",
    "message": "Monthly credit limit exceeded",
    "limit": 50000,
    "remaining": 0,
    "reset_at": "2026-11-01T00:00:00Z"
  }
}

Best Practices

1. Implement Exponential Backoff

When you receive a 429 response, wait before retrying:

import time

def make_request_with_backoff(url, max_retries=3):
    for attempt in range(max_retries):
        response = requests.get(url)
        
        if response.status_code == 429:
            wait_time = 2 ** attempt  # 1s, 2s, 4s
            time.sleep(wait_time)
            continue
            
        return response
    
    raise Exception("Max retries exceeded")

2. Monitor Rate Limit Headers

Check headers to avoid hitting limits:

response = requests.get(url)
remaining = int(response.headers.get('X-RateLimit-Remaining', 0))

if remaining < 100:
    print("Warning: Low credits remaining")
    # Maybe slow down or alert

3. Use Concurrent Requests Efficiently

Don't exceed your concurrent limit:

from concurrent.futures import ThreadPoolExecutor

# Free plan: max 2 concurrent
with ThreadPoolExecutor(max_workers=2) as executor:
    futures = [executor.submit(search_hotel, location) 
               for location in locations]
    results = [f.result() for f in futures]

4. Batch Requests When Possible

Some endpoints support batching to reduce request count. Check endpoint documentation.

Need Higher Limits?

If you're hitting rate limits regularly:

• Upgrade your plan for higher limits
• Contact sales@scrapercompany.com for custom Enterprise limits