Line of work

Real-time Data Systems

Event-driven data systems in Python and Rust, with ML research on top — time-series foundation models, PPO and evolution strategies.

  • #work
  • #freelance
  • #data
  • #streaming
  • #ml

One of my four lines of Independent AI & Full-Stack Engineer (Freelance) work: real-time, event-driven data systems, plus the machine-learning research I run on the data they collect. The systems consume continuous streams of events from external sources, process them with low latency, store them for analysis, and replay recorded history through the same processing logic.

Architecture

  • Event-driven core — Python asyncio. I build services around asynchronous event loops, so many concurrent streams are handled without a thread per connection. Components communicate through events rather than direct calls, which keeps them independently restartable.
  • Streaming I/O — WebSockets and FastAPI. Long-lived WebSocket connections ingest and publish streams; FastAPI exposes the control plane and query endpoints with typed request and response models.
  • Low-latency path — Rust. I wrote the WebSocket feed handling and an internal message bus in Rust, built with release-profile optimizations (LTO). Microbenchmarks in C compared fan-out strategies before I committed to a design. Everything off the hot path stays in Python for development speed.
  • Columnar storage — DuckDB and Apache Parquet & Arrow. Events are normalized into Arrow/Parquet and queried with DuckDB, an embedded engine that needs no database server. Large histories are baked into flat, memory-mapped NumPy arrays so research jobs fit in modest RAM.
  • Replay and simulation. Recorded history runs through the same decision code used live, behind a parity gate: a replay has to reproduce the live numbers before its results are trusted.
  • Operations — Linux & systemd. Long-running services and daemons run on a Linux VPS under systemd, with SQLite in WAL mode and scheduled checkpoints, plus Cloudflare Workers used to probe network latency from different edges. Live dashboards are built in Svelte, and a small Swift companion app runs on macOS.

Machine learning on top

  • Fine-tuning a time-series foundation model. I fine-tuned IBM Granite TSPulse on GPU (Google Colab) with PyTorch, with hyperparameter grids and evaluation against baselines — including a tabular foundation model and gradient-boosted trees from the scikit-learn ecosystem.
  • Reinforcement learning in custom simulators. I implemented PPO directly in PyTorch (no RL framework) against a purpose-built simulation environment, always benchmarked against trivial baseline policies to tell whether the model learned the real dynamics or just the simulator.
  • Evolution strategies when gradients fail. When per-step rewards were too noisy for PPO to assign credit, I retrained the policy with OpenAI-style evolution strategies and antithetic sampling, optimizing the episode outcome directly with a deliberately small parameter count to resist overfitting.
  • Research discipline. I pre-register experiments, compare every result with a baseline, and keep decisions, evidence and known pitfalls in a linked knowledge base operated with Claude Code — the same note-graph idea as this site.

Compute

I run heavy batch research as self-terminating jobs on AWS EC2 Spot instances (code and results through S3, secrets from SSM, a hard time cap per job); datasets are versioned on Hugging Face; GPU work runs on Google Colab.

Why these choices

My design separates concerns by latency budget: Python where iteration speed matters, Rust where latency matters, and a columnar store where scanning large histories matters. Replaying recorded events through production code turns "will this work?" into a repeatable test — the same discipline of Observability and measurement I apply to agent systems.

Related