Files
community-crawler/crawler
Claude 1ccbc17b79 Improve fetcher with browser-like headers and cookie handling
- Add cookie jar for session management
- Include sec-ch-ua and Sec-Fetch-* headers (Chrome-like)
- Add HTTPS agent with keepAlive
- Log 403 response body for debugging

Result: Still blocked by TLS fingerprinting
- Both Ruliweb and Arcalive return "Access denied"
- Need Puppeteer to bypass advanced bot detection
2025-11-15 17:28:23 +00:00
..