[{"content":"The Netherlands runs one of the densest rail networks in Europe, with 20+ operators sharing the tracks. Delays are a daily experience, but as a passenger you never learn why: weather, one bad operator, a corridor bottleneck, or a disruption cascading through the network.\nNL Transport Pulse is my attempt to answer that with data: a pipeline that ingests 70M+ train departures (2019 to 2026), 15 years of disruption records, and daily weather observations, transforms them through a medallion architecture on GCP, and serves an interactive dashboard for corridor-level reliability analysis. It started as my capstone for the DataTalksClub DE Zoomcamp 2026.\nCode: github.com/xlsongc/nl-transport-pulse\nArchitecture flowchart LR subgraph Sources NS[NS API\u0026lt;br/\u0026gt;4x daily] RDT[RDT archives\u0026lt;br/\u0026gt;monthly] KNMI[KNMI weather\u0026lt;br/\u0026gt;daily] end subgraph Orchestration[\u0026#34;Airflow (6 DAGs)\u0026#34;] ING[ingest DAGs] DBT[dbt DAG\u0026lt;br/\u0026gt;run + test] end NS --\u0026gt; ING RDT --\u0026gt; ING KNMI --\u0026gt; ING ING --\u0026gt; GCS[(GCS\u0026lt;br/\u0026gt;raw archive)] GCS --\u0026gt; BQ[(BigQuery\u0026lt;br/\u0026gt;raw)] BQ --\u0026gt; DBT DBT --\u0026gt; CORE[(BigQuery\u0026lt;br/\u0026gt;staging / core)] CORE --\u0026gt; ST[Streamlit\u0026lt;br/\u0026gt;4 dashboard pages] TF[Terraform] -.provisions.-\u0026gt; GCS TF -.provisions.-\u0026gt; BQ Everything reproducible from a clean machine: Terraform provisions the GCS bucket, BigQuery datasets, and service account; Docker Compose brings up Airflow and Streamlit; the DAGs and dbt do the rest.\nLayer Tool Infrastructure Terraform Orchestration Airflow 2.8 (Docker Compose) Lake / Warehouse GCS + BigQuery (partitioned, clustered) Transformation dbt Core: staging, intermediate, core + seeds, snapshot, macros Serving Streamlit + Plotly Three decisions worth explaining 1. Idempotent loads: partition-scoped overwrite Every ingestion path writes raw files to GCS first, then loads into BigQuery with the same contract: delete the partition for that service date, then load. Re-running any day, whether after a failure or a backfill, converges to the same state instead of duplicating rows.\ndef load_json_to_bq(gcs_uri, table_id, service_date, partition_field=\u0026#34;_service_date\u0026#34;, skip_delete=False): \u0026#34;\u0026#34;\u0026#34;Partition-scoped overwrite: 1. Delete all rows for the given service_date 2. Load new data from GCS \u0026#34;\u0026#34;\u0026#34; Airflow retries become safe by construction, not by hoping a run never dies halfway.\n2. Polling instead of streaming, deduplication in dbt The NS departures API is a live board, so I poll it 4 times a day rather than streaming. That is a deliberate scope cut: for daily reliability analytics, intraday latency buys nothing. The cost is overlap between polls, which is resolved in the staging layer with row_number() ordered by ingestion time:\nqualify row_number() over ( partition by departure_id order by _ingested_at desc ) = 1 Each staging model also builds a surrogate key and casts every field explicitly, because the raw API mixes JSON strings into otherwise flat records:\ncast(json_extract_scalar(product, \u0026#39;$.operatorName\u0026#39;) as string) as operator_name, timestamp_diff(timestamp(actualDateTime), timestamp(plannedDateTime), MINUTE) as delay_minutes, 3. Tests as part of the DAG, not a chore after it The dbt DAG runs dbt test right after dbt run, with 26 schema tests across the three layers (uniqueness of surrogate keys, accepted values for delay categories, relationships between facts and dimensions). Two data quality bugs this caught early: RDT delay values are documented in minutes but look like seconds at first glance, and disruption durations needed cross-validation against their start and end timestamps.\nThe dashboard Four Streamlit pages on top of the core marts: a network overview (reliability trends, stress map, calendar heatmap), a corridor explorer, a weather impact page (wind, rain, and temperature against on-time performance), and an operator view covering 26 rail companies plus 15 years of disruption history.\nHonest limitations The NS polling path captures snapshots, not a complete event stream; the complete history comes from the monthly RDT archives, and the two are unioned in the intermediate layer. Road traffic data (NDW) is scaffolded as a future extension but not part of the verified end-to-end path. The dashboard queries BigQuery directly with caching; at real scale a serving layer would sit in between. What I took from it The individual tools are all learnable in a weekend each. The actual work, and the actual learning, sits in the seams: making loads idempotent so orchestration retries are safe, deciding where deduplication belongs, and wiring tests into the pipeline so data quality problems surface on day one instead of in a dashboard six weeks later.\n","permalink":"https://xlsong-blog.vercel.app/posts/nl-transport-pulse/","summary":"\u003cp\u003eThe Netherlands runs one of the densest rail networks in Europe, with 20+ operators sharing the tracks. Delays are a daily experience, but as a passenger you never learn why: weather, one bad operator, a corridor bottleneck, or a disruption cascading through the network.\u003c/p\u003e\n\u003cp\u003eNL Transport Pulse is my attempt to answer that with data: a pipeline that ingests 70M+ train departures (2019 to 2026), 15 years of disruption records, and daily weather observations, transforms them through a medallion architecture on GCP, and serves an interactive dashboard for corridor-level reliability analysis. It started as my capstone for the DataTalksClub DE Zoomcamp 2026.\u003c/p\u003e","title":"NL Transport Pulse: an End-to-End Data Pipeline for Dutch Railway Reliability"},{"content":"My partner is a biocatalysis researcher. I am a data engineer who reaches for LLMs daily and mostly likes what I find. She keeps her distance from all of it, on solid professional grounds: experimental science runs on reproducibility, verifiability, and traceability, and raw LLM output guarantees none of the three.\nTo be precise about the skepticism: she had never actually used an agent, a workflow, or a data pipeline. The distance came from the failure mode everyone has heard about, hallucination. For a scientist whose day job is separating verified claims from plausible ones, a tool famous for confident fabrication starts in deep debt.\nI still wanted to build her something useful. Doing it taught me that a skeptic judges a tool by its worst day, not by its best demo.\nPicking the entry point Her daily friction was triage. She follows journal and preprint feeds in The Old Reader, and every day dozens of new items arrive, of which maybe one or two matter. Scanning titles and abstracts eats a fixed slice of every day, and the fear of missing something important makes skipping feel unsafe.\nA few more impressive ideas got rejected first:\nHelp writing literature reviews. Directly on her red line. Hard to verify, and hallucinations there are expensive. A paper Q\u0026amp;A agent over her library. Technically sexy, but she never asked for it. A hammer looking for nails, and the first wrong answer would end the experiment. Automating experimental data analysis. That is the core of her professional judgment. Highest trust cost, touch it last if ever. Literature triage won on a simple filter: frequency of pain times verifiability, divided by cost of error. It happens daily. Every summary links to the original abstract, so checking the AI takes one click. A bad summary costs her thirty seconds. And the final judgment (what to read, how to interpret it, what to cite) never leaves her hands.\nThe architecture is deliberately boring flowchart LR TOR[The Old Reader] --\u0026gt;|fetch| GHA[GitHub Actions\u0026lt;br/\u0026gt;daily cron] GHA --\u0026gt;|abstracts| CL[Claude API] CL --\u0026gt;|digest| TG[Telegram] GHA --\u0026gt;|commit seen.json| REPO[(git repo\u0026lt;br/\u0026gt;= state)] A cron-triggered pipeline: fetch RSS entries, deduplicate, one Claude call to summarize and classify, push the digest to Telegram. No agent, no loop, no tool use.\nThat was the most consequential decision in the project. The task is deterministic: fixed input, fixed processing, fixed output. Nothing in it requires a model to decide what happens next. An agent here would only add ways to fail: loops that refuse to terminate, tool calls that error, behavior that drifts from day to day. \u0026ldquo;Arrives at eight every morning, same format every time\u0026rdquo; builds more trust with a skeptic than any amount of visible intelligence. Since this project I default to a pipeline before a workflow, and a workflow before an agent.\nThe other decisions follow the same logic:\nSummarize, never extrapolate. The prompt restricts the model to the title and abstract in the feed. No outside knowledge, no commentary, classification from a fixed category list. A dull summary costs nothing; a hallucinated one costs weeks of accumulated trust. Telegram, not a web app. She already uses Telegram every day. A web app would ask her to build a new habit around a thing she doubts. Tools should bend around people. GitHub Actions, not a server. Zero cost, zero ops, secrets built in. Personal projects die of maintenance fatigue, so the target was \u0026ldquo;still running after I ignore it for three months.\u0026rdquo; Traded away: real-time delivery. Papers are not stock prices. Her subscriptions stay hers. The bot reads the feed list she already curates in The Old Reader. Building my own subscription manager would have moved control of \u0026ldquo;what to follow\u0026rdquo; from her side of the table to mine. State is one JSON file committed back to the repo by the workflow. A few dozen articles a day cannot justify a database. One implementation detail carries most of the philosophy. Articles are marked as seen only after Telegram confirms delivery:\nif not send_telegram(digest): return # not marked seen; tomorrow retries for art in articles: seen.add(art[\u0026#34;id\u0026#34;]) save_seen(seen) If delivery fails, she gets a duplicate tomorrow. A duplicate is a visible, thirty-second annoyance; a silently dropped paper is exactly the failure she was worried about in the first place. Same reasoning inside the LLM call: if Claude errors out, the digest degrades to a plain list of titles rather than failing the run.\nCost, for completeness: roughly 20 articles a day at about 700 tokens each lands around one dollar a month. Cheap enough to never enter the discussion.\nResults It has run unattended since launch. No missed mornings, no manual restarts; the failure handling above has stayed theoretical, which is the best outcome for code like that.\nThe API bill came in under even my low estimate. The digest runs on Claude Sonnet, at most one call per day, and interesting papers do not publish on a schedule: on days when her feeds have nothing new, the pipeline skips the LLM entirely and sends a one-line \u0026ldquo;nothing today\u0026rdquo;. Quiet days cost zero.\nThe strongest result arrived as a feature request. Her lab onboards a steady stream of new bachelor and master students. Supervisors repeat the same lab rules and instrument procedures for every new person; students forget and ask again, stall when no supervisor is around, and occasionally run an experiment without asking at all. It is private domain knowledge that no public chatbot has ever seen, and questions about it block real work. She asked me whether an agent could answer them.\nI never pitched her that idea. She brought it to me.\nThe balance sheet Trust with a skeptic works like a balance sheet. Every accurate summary is a small deposit. Every hallucination is a large withdrawal. Feature lists appear nowhere on it.\nThe LLM in this pipeline does exactly one thing that regex and rules cannot: compress an English abstract into a three-second Chinese summary with a category label. Before handing a step to an LLM I now ask two questions: is the model genuinely irreplaceable there, and are its failures cheap? Here both answers were yes, so it earned its single daily call.\nShe reads the digest every morning, and she is still not sold on AI agents in general. Both can be true, and I suspect both should be. What she adopted was a reliable tool that happens to contain one well-caged LLM call. The next tool in this house will be an agent she asked for herself, and I cannot think of a better review than that.\n","permalink":"https://xlsong-blog.vercel.app/posts/ai-tool-for-an-ai-skeptic/","summary":"\u003cp\u003eMy partner is a biocatalysis researcher. I am a data engineer who reaches for LLMs daily and mostly likes what I find. She keeps her distance from all of it, on solid professional grounds: experimental science runs on reproducibility, verifiability, and traceability, and raw LLM output guarantees none of the three.\u003c/p\u003e\n\u003cp\u003eTo be precise about the skepticism: she had never actually used an agent, a workflow, or a data pipeline. The distance came from the failure mode everyone has heard about, hallucination. For a scientist whose day job is separating verified claims from plausible ones, a tool famous for confident fabrication starts in deep debt.\u003c/p\u003e","title":"Building an AI Tool for an AI Skeptic"},{"content":" I\u0026rsquo;m Xiaolong Song, a data engineer based in the Netherlands.\nI build data platforms and apply LLMs in production. Recent work includes PySpark pipelines processing 120M+ rows per day, lakehouse architectures on GCP and Microsoft Fabric (Delta Lake, BigQuery, dbt, Airflow), and AI engineering: LLM cost engineering, prompt caching, and coding-agent workflows.\nBefore data, I spent four years as an infrastructure engineer, then moved into this field through an MSc in Statistics and Data Science at KU Leuven. That detour shaped how I work: data problems are engineering problems, and they deserve versioning, tests, and honest trade-off analysis.\nWhat I write about here:\nData engineering: pipelines, lakehouse patterns, orchestration, data testing AI engineering: LLM cost engineering, prompt caching, agentic workflows Fundamentals: occasional deep dives into the CS underneath the tools Find me on GitHub and LinkedIn, or reach me at andrewsong93@gmail.com.\n","permalink":"https://xlsong-blog.vercel.app/about/","summary":"\u003cimg src=\"/images/avatar.jpg\" alt=\"Xiaolong Song\" width=\"180\" style=\"border-radius: 50%;\" /\u003e\n\u003cp\u003eI\u0026rsquo;m Xiaolong Song, a data engineer based in the Netherlands.\u003c/p\u003e\n\u003cp\u003eI build data platforms and apply LLMs in production. Recent work includes PySpark pipelines processing 120M+ rows per day, lakehouse architectures on GCP and Microsoft Fabric (Delta Lake, BigQuery, dbt, Airflow), and AI engineering: LLM cost engineering, prompt caching, and coding-agent workflows.\u003c/p\u003e\n\u003cp\u003eBefore data, I spent four years as an infrastructure engineer, then moved into this field through an MSc in Statistics and Data Science at KU Leuven. That detour shaped how I work: data problems are engineering problems, and they deserve versioning, tests, and honest trade-off analysis.\u003c/p\u003e","title":"About"}]