Oct 2024 - Present
Superleap
Software Development Engineer 2 | Founding Member | Back End
Asynchronous Architecture
Asynchronous Architecture
- Architected and built data pipelines to capture all audit-data changes from main DB (Postgres) to secondary DB (Cassandra) using Kafka for message queues and Go consumers to transform data and insert.
- Built Kafka consumers in Go capable of processing over 1 million events in 30 seconds.
- Added and maintained monitoring of all programs deployed along with infrastructure monitoring using Prometheus and Grafana.
- Created APIs for clients to poll their data changes using a cursor-and-limit method instead of webhooks.
- Set up River for async tasks and added graceful shutdowns to all backend applications.
Performance — Redis & Local Cache
Performance — Redis & Local Cache
- Leveraged Redis client tracking to build local cache for distributed systems with invalidation logic, reducing latency from milliseconds (Redis) to nanoseconds (local cache).
- Built APIs to transform and run client queries while adding multi-tenancy support and core business logic.
Analytics — Clickhouse
Analytics — Clickhouse
- Set up data pipeline from Postgres to Clickhouse for clients to run analytics on.
- Built APIs to transform and run client queries with multi-tenancy support and internal business logic.
Workflow Builder
Workflow Builder
- Designed workflow builder and workers in Go using Temporal.
- Built and maintained a drag-and-drop workflow system with SLAs of 99.9% successful runs, running over 2.2 million complex workflows per day.