All posts

Deep, practical engineering stories — browse everything.

Posts

Microservices Auth: We Tried 4 Patterns, Here's What Actually Worked

Authentication between microservices is one of those things that seems simple until you actually try to implement it. We went through 4 different patterns over 18 months. Each one solved some problems but created new ones. Here’s what we learned, and what we eventually settled on. The problem You have 15 microservices. API Gateway authenticates users with JWT. But how do internal services verify requests from each other? Service A needs to call Service B. Service B needs to know: ...

Read

Node.js Memory Leak: Two Weeks to Find One Missing removeListener()

Our Node.js API was restarting every 6 hours due to memory leaks. Took me two weeks to find the bug. It was a single missing removeListener() call. Here’s how I found it, and what I learned about debugging Node memory leaks that actually works. The symptom Memory usage graph looked like this: Memory │ ╱╱╱╱ │ ╱╱ │ ╱╱ └────────────> Time Classic memory leak pattern. Process starts at 200MB, grows to 2GB over 6 hours, then OOM kills it. Kubernetes restarts it. Repeat. ...

Read
Posts

Redis Caching: The Mistakes That Cost Us $12K/Month

We were spending $12,000/month on Redis before I realized we were doing it completely wrong. Not “slightly inefficient” wrong. Full-on “why is our cache bigger than our database” wrong. Here’s what I learned after three months of firefighting and optimization. The problem nobody warned us about Our Redis instance hit 32GB of memory. Our actual PostgreSQL database? 8GB. Something was very, very wrong. Started digging through our caching logic. Found this gem: ...

Read
Posts

PostgreSQL JSONB: Three Months of Pain and What I Learned

So here’s the thing nobody tells you about JSONB in Postgres: it’s fast until it’s not. We migrated our user preferences table to use JSONB columns three months ago. The pitch was simple - no more ALTER TABLE every time marketing wants to track a new preference. Just stuff it in JSON and call it a day. Seemed smart at the time. The honeymoon phase First two weeks were great. Engineers loved it. Product loved it. We shipped features faster because we didn’t need schema migrations for every little thing. Our user_settings column grew from tracking 5 fields to 23. No problem, right? ...

Read
Posts

React Server Components: I Get It Now

Took me six months to actually understand RSC. Not the “I read the docs” understanding - the visceral “oh THAT’S why” moment that only comes from shipping code. The docs make it sound simple: server components render on the server, client components run in the browser. Cool. Useless explanation. Here’s what actually clicked for me. The mental model that stuck Stop thinking about server vs client components. Think about where the data lives. ...

Read
Posts

Docker Build Times: From 8 Minutes to 40 Seconds

Our CI pipeline was embarrassing. Every PR took 8+ minutes to build a Docker image for a Next.js app. Developers complained. I ignored them for two months because “it’s just CI, ship faster code.” Then we hit 50+ PRs per day and our CI bill jumped $400/month. Time to actually fix it. The original Dockerfile (the bad one) FROM node:18 WORKDIR /app COPY . . RUN npm install RUN npm run build CMD ["npm", "start"] Looks innocent. Builds every time though. Every. Single. Time. ...

Read

Why Our API Rate Limiter Failed (And How We Fixed It)

Rate limiting seems simple until it isn’t. We thought we had it figured out - Redis counters, sliding windows, the works. Then a client with a distributed system hit our API and everything fell apart. What we built initially Standard stuff. Token bucket in Redis: def check_rate_limit(api_key: str) -> bool: key = f"rate_limit:{api_key}" current = redis.get(key) if current and int(current) >= 100: # 100 req/min return False pipe = redis.pipeline() pipe.incr(key) pipe.expire(key, 60) pipe.execute() return True Worked great in testing. 100 requests per minute per API key, clean reset every minute. ...

Read
Posts

TypeScript Generics: A Practical Guide (No PhD Required)

Generics in TypeScript are one of those things that click suddenly or not at all. I spent months copy-pasting generic syntax without understanding it. Then I refactored a data fetching hook and everything made sense. Here’s the path that worked for me, skip the academic explanations. Start with the problem We had a bunch of API fetching hooks that looked like this: function useUserData() { const [data, setData] = useState<User | null>(null); const [loading, setLoading] = useState(false); useEffect(() => { setLoading(true); fetch('/api/user') .then(r => r.json()) .then(setData) .finally(() => setLoading(false)); }, []); return { data, loading }; } function usePostsData() { const [data, setData] = useState<Post[] | null>(null); const [loading, setLoading] = useState(false); useEffect(() => { setLoading(true); fetch('/api/posts') .then(r => r.json()) .then(setData) .finally(() => setLoading(false)); }, []); return { data, loading }; } function useCommentsData() { // ... you get the idea } Same logic, different types. Classic duplication. First instinct: make it a function. ...

Read

Database Indexing: The Stuff Nobody Tells You

Indexes are supposed to make queries fast. Sometimes they make them slower. Here’s what I wish someone told me before I tanked production performance trying to “optimize” our database. The query that started it all Support said the admin dashboard was timing out. Found this in the slow query log: SELECT * FROM orders WHERE customer_id = 12345 AND status IN ('pending', 'processing') AND created_at > '2025-12-01' ORDER BY created_at DESC LIMIT 20; Took 8 seconds on a table with 4 million rows. Obviously needs an index, right? ...

Read
Posts

Git Rebase: Stop Being Scared of It

Everyone tells you to avoid git rebase. “It’s dangerous!” “You’ll lose commits!” “Just merge!” I used to think that too. Then I joined a team that rebases everything and our git history is actually readable. Here’s what changed my mind. The merge commit mess This was our git log before rebasing: * Merge branch 'feature/user-auth' |\ | * Fix typo in button text | * Update tests | * Merge main into feature/user-auth | |\ | |/ |/| * | Merge branch 'fix/header-spacing' |\ \ | * | Adjust header margin * | | Merge branch 'feature/notifications' |\| | | |/ |/| See those merge commits? They add zero value. Just noise. Finding where a feature was added means clicking through 5 merge commits. ...

Read