Ruliweb changes:
- Use tr.table_body instead of .table_body tr
- Correct selectors: .subject .subject_link, .writer a, .hit, .recomd, .num_reply, .time
- Remove icons/reply count from title text
- Add debug logging for parsed posts
Arcalive changes:
- Use a.vrow instead of .vrow
- Correct selectors: .col-title .title, .col-author .user-info span[data-filter]
- Selectors: .col-view, .col-rate, .col-time time
- Add debug logging for parsed posts
Both crawlers now match actual HTML structure from pages
- Create debug-html.ts to save actual HTML from pages
- Add npm run debug command to package.json
- Helps identify correct CSS selectors for crawlers
- Saves HTML to debug-ruliweb.html and debug-arcalive.html
Usage: npm run debug
Major changes:
- Install puppeteer, puppeteer-extra, puppeteer-extra-plugin-stealth
- Create PuppeteerFetcher class with Stealth plugin
- Update all crawlers to use Puppeteer instead of Axios
- Add browser lifecycle management (init/close)
- Update test.ts and index.ts with browser cleanup
Features:
- Real Chrome browser execution (bypasses TLS fingerprinting)
- Stealth plugin to avoid bot detection
- Headless mode for background operation
- Proper error handling and browser cleanup
Limitations:
- Requires Chrome/Chromium installation
- Higher resource usage (~200MB memory)
- Slower than Axios (browser startup time)
- Cannot test in current environment (Chrome install blocked)
Next steps:
- Test in local environment with Chrome installed
- Adjust HTML selectors based on actual page structure
- Monitor for Cloudflare blocks
- Test different Ruliweb URLs (search, board, best, main)
- Result: All endpoints return 403 "Access denied"
- Confirms that Ruliweb blocks all bot requests
- Validates that Puppeteer/Selenium is required
- Add cookie jar for session management
- Include sec-ch-ua and Sec-Fetch-* headers (Chrome-like)
- Add HTTPS agent with keepAlive
- Log 403 response body for debugging
Result: Still blocked by TLS fingerprinting
- Both Ruliweb and Arcalive return "Access denied"
- Need Puppeteer to bypass advanced bot detection
- Create crawler project structure
- Implement base crawler class with safety features
- Add crawlers for Ruliweb, Arcalive, DCInside
- Implement utilities: fetcher (with retry logic), logger
- Configure crawling settings (3s delay, max 20 posts/board)
- Add test script and scheduler (30min intervals)
Safety measures:
- 3 second delay between requests
- Exponential backoff retry logic
- Respect robots.txt (DCInside disabled)
- User-Agent and proper headers
Current status:
- Structure complete
- Both Ruliweb and Arcalive return 403 (bot detection)
- Need to decide: Puppeteer, switch targets, or use mock data
