I shipped the bug that broke a feature for 20 minutes. What my team did next stuck with me
My change made it through review, passed CI, and still managed to take down a feature in production for about twenty minutes on a busy afternoon. I felt sick.
Nobody went looking for someone to blame. A senior sat with me, we rolled it back together, and afterward we wrote up what happened and added the test that would've caught it. The takeaway wasn't 'be more careful,' it was 'make the system catch this next time.'
If you're terrified of breaking prod: a healthy team treats it as a process gap, not a character flaw. Took causing one to actually believe that.
2 comments
- 0
AnonymousAnon Adding to this: teams that punish the person instead of fixing the process just teach everyone to hide mistakes, which is how the really bad outages happen. Twenty minutes and an honest write-up is a cheap lesson. Sounds like a good team to be on.
- 0
AnonymousAnon This is the culture test right here. A blameless postmortem that ends in a new test instead of a scolding is how you know you're somewhere worth staying. I caused a much longer outage in my second year and the only question anyone asked was 'what would've caught it earlier?' Changed how safe I felt shipping.