Claude ae85dcbd87 Add Puppeteer support to bypass bot detection ...

Major changes:
- Install puppeteer, puppeteer-extra, puppeteer-extra-plugin-stealth
- Create PuppeteerFetcher class with Stealth plugin
- Update all crawlers to use Puppeteer instead of Axios
- Add browser lifecycle management (init/close)
- Update test.ts and index.ts with browser cleanup

Features:
- Real Chrome browser execution (bypasses TLS fingerprinting)
- Stealth plugin to avoid bot detection
- Headless mode for background operation
- Proper error handling and browser cleanup

Limitations:
- Requires Chrome/Chromium installation
- Higher resource usage (~200MB memory)
- Slower than Axios (browser startup time)
- Cannot test in current environment (Chrome install blocked)

Next steps:
- Test in local environment with Chrome installed
- Adjust HTML selectors based on actual page structure
- Monitor for Cloudflare blocks

2025-11-15 17:39:43 +00:00

94 KiB

Raw Blame History

View Raw