Wayback CDX API: Cloud/Datacenter IP Workaround

Problem

The Wayback Machine CDX API (web.archive.org/cdx/search/cdx) silently refuses to serve data to requests from cloud provider IP ranges (Google Cloud Run, AWS, GCE, etc.). There is no HTTP error code — requests either timeout or return 0 bytes. The same queries work perfectly from residential/local IPs.

Context / Trigger Conditions

• CDX API returns empty (0 bytes) or times out from Cloud Run, Lambda, GCE, etc.
• Same query via curl on local machine returns expected data
• No HTTP error codes (not 429, not 403) — just empty or timeout
• Other external APIs (Common Crawl, Arctic Shift, Arquivo.pt) work fine from the same cloud instance
• Even with VPC connector + static IP configured, CDX still fails
• Increasing timeouts and retries doesn't help

Solution

The Pattern: Local Fetch → Cloud Storage → Cloud Import

Since CDX won't respond to cloud IPs, split the workflow:

1. Fetch locally: Run CDX queries from your local machine, save results to a file
2. Upload to cloud storage: gsutil cp the file to GCS (or S3, etc.)
3. Import on cloud: Have your Cloud Run job read from cloud storage instead of CDX

Implementation

Step 1: Local fetch script

# Run this locally — CDX works from residential IPs
import urllib.request, urllib.parse

CDX = 'https://web.archive.org/cdx/search/cdx'
results = set()
resume_key = None

while True:
    params = {
        'url': 'example.com/path/*',
        'output': 'text',
        'fl': 'original',
        'filter': 'statuscode:200',
        'limit': '10000',
        'showResumeKey': 'true',
    }
    if resume_key:
        params['resumeKey'] = resume_key

    url = f"{CDX}?{urllib.parse.urlencode(params)}"
    req = urllib.request.Request(url, headers={'User-Agent': 'MyBot/0.1'})
    with urllib.request.urlopen(req, timeout=120) as resp:
        raw = resp.read().decode('utf-8', errors='replace')

    parts = raw.rstrip().rsplit('\n\n', 1)
    data_lines = parts[0].strip().split('\n')
    resume_key = parts[1].strip() if len(parts) == 2 else None

    for line in data_lines:
        results.add(process_line(line))  # Extract what you need

    if not resume_key or len(data_lines) < 10000:
        break
    time.sleep(3)

# Save to file
with open('/tmp/results.txt', 'w') as f:
    for item in sorted(results):
        f.write(f'{item}\n')

Step 2: Upload to GCS

gsutil cp /tmp/results.txt gs://my-bucket/imports/results.txt

Step 3: Cloud Run job with --from-gcs flag

def load_from_gcs(gcs_path: str) -> set[str]:
    """Load IDs from a GCS text file."""
    import subprocess
    result = subprocess.run(
        ["gsutil", "cp", gcs_path, "/tmp/results.txt"],
        capture_output=True, text=True, timeout=120
    )
    if result.returncode != 0:
        raise RuntimeError(f"gsutil failed: {result.stderr}")

    items = set()
    with open("/tmp/results.txt") as f:
        for line in f:
            items.add(line.strip())
    return items

# In main():
parser.add_argument("--from-gcs", type=str, default=None,
                    help="Import from GCS file instead of CDX")

if args.from_gcs:
    items = load_from_gcs(args.from_gcs)
else:
    items = fetch_from_cdx(delay=args.delay)

Step 4: Deploy with --from-gcs

gcloud run jobs deploy my-job \
    --args="-m,my_module,--from-gcs,gs://my-bucket/imports/results.txt" \
    ...

Verification

• Local fetch completes successfully with expected data volume
• File uploads to GCS without errors
• Cloud Run job reads from GCS and imports to database
• gsutil ls -l gs://my-bucket/imports/results.txt shows expected file size

Example

Real-world case: Fetching 312,427 vine IDs from Wayback CDX oEmbed captures.

• CDX failed from Cloud Run (4 attempts over multiple deploys, all returning 0 bytes)
• Local fetch completed in ~4 minutes with resumeKey pagination
• Uploaded 3.6MB text file to GCS
• Cloud Run imported 170,108 new IDs from GCS in ~10 minutes

Notes

• This affects web.archive.org specifically. Other CDX servers (index.commoncrawl.org) work fine from cloud IPs
• The Internet Archive may whitelist specific IPs on request — contact them if you have a legitimate archival use case
• Even with a whitelisted static IP via VPC connector, the CDX API may still not respond (whitelisting may apply to download, not CDX)
• collapse=urlkey and matchType=prefix are more likely to fail than simple wildcard queries, but even wildcards fail from cloud IPs
• This workaround adds a manual step (local fetch + upload) but is reliable and only needs to be done once per data source
• For frequently-changing data, consider scheduling the local fetch via cron and automating the GCS upload

References

• Wayback CDX API: https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server
• Related skill: wayback-cdx-wildcard-pagination (for the pagination aspect)