The Data Pulse pipeline processes and transforms web-scraped pageviews using Apache Beam and Google Cloud Dataflow. It reads JSON lines, parses them into PageView objects, filters for "product" post types, enriches with country info, and writes to Google BigQuery. Robust logging and error handling ensure data integrity
java bigquery gcp cloud-storage data-flow data-engineering data-analysis apache-beam data-processing data-projects
-
Updated
Jul 2, 2024 - Java