A detailed technical flow diagram of a web scraping pipeline for Thai real estate data. Clean minimal style, white background, light gray connectors, soft shadows on cards. Show two parallel scraping tracks side by side: LEFT TRACK labeled "Facebook Groups": Step 1: "APScheduler triggers job" (clock icon) Step 2: "Playwright launches stealth browser" (browser icon) Step 3: "Load Facebook group page" Step 4: "Dismiss login wall overlay" (shield icon) Step 5: "Scroll feed × 6, expand See More buttons" Step 6: "Extract post text + image URLs per article" Step 7: "Strip header / footer boilerplate" Step 8: "Save to ks_raw_posts" (database icon, teal) RIGHT TRACK labeled "DDproperty Portal": Step 1: "APScheduler triggers job" (clock icon) Step 2: "Playwright launches stealth browser" Step 3: "Navigate zone URL (Sukhumvit, Silom, Rama9...)" Step 4: "Wait for JS render — networkidle" Step 5: "Extract listing cards via data-testid selectors" Step 6: "Paginate through results (page=1..10)" Step 7: "Parse raw_title, raw_price, raw_area, raw_html" Step 8: "Save to ks_raw_posts" (database icon, teal) Both tracks merge at the bottom into one box: "ks_raw_posts — raw_text stored, parse_status = pending" On the right side show a vertical "Proxy Layer" panel with: - Block detector (HTTP 403 / CAPTCHA / selector missing) - IP rotation arrow - Circuit breaker states: CLOSED → OPEN → HALF-OPEN Color: teal for Facebook track, blue for DDproperty track, coral for proxy panel. Rounded rectangle cards. Arrow connectors with labels. Sans-serif font (Inter or similar). 16:9 aspect ratio. High resolution infographic. A clean minimal flowchart diagram on white background showing a web scraper block detection and recovery system. Main flow (left to right): "Make request" → diamond "Response OK?" → YES path: "Record success in Redis" → "Continue scraping" → NO path: leads to detection layer Detection layer (vertical checklist boxes): Box 1: "Check HTTP status — 403 / 429 / 503?" Box 2: "Check page title — contains captcha / blocked?" Box 3: "Check selector — expected card found?" Box 4: "Check result count — less than 50% of 7-day baseline?" Each box has YES/NO branches. Any YES leads to: "Classify block type" (4 labels: HTTP_ERROR / CAPTCHA / SILENT_REDIRECT / SOFT_BLOCK) Then: "Get next proxy from Redis pool" → "Exponential backoff (30s → 5min → 30min)" → "Increment retry counter" → diamond "Retry count > 3?" → YES: "Open circuit breaker → Dead letter queue" → NO: loop back to "Make request" Color: light blue for normal flow, amber for detection, red for blocked states, green for success. Minimal rounded cards, thin gray connectors, clean sans-serif labels. White background. High resolution, 16:9.
Comments
0 B
|👍
/👎
0 B
|👍
/👎
0 B
|👍
/👎
0 B
|👍
/👎
0 B
|👍
/👎
0 B
|👍
/👎
0 B
|👍
/👎
0 B
|👍
/👎
0 B
|👍
/👎
0 B
|👍
/👎
0 B
|👍
/👎
0 B
|👍
/👎
0 B
|👍
/👎
0 B
|👍
/👎
0 B
|👍
/👎
0 B
|👍
/👎
0 B
|👍
/👎