u/Midnight_LurkerERP Analystin r/dataengineering • Recommends
Developer Deletes Entire Production Database (r/videos xpost)
0:00 / 0:00
#gitlab#databaseoutage#postmortem
Dev Deletes Entire Production Database, Chaos Ensues
Kevin Fang
206K subscribers • 1 video on Hallway
- Recommended by 2 people: 1 on Hacker News and 1 in engineering communities.
- Presented by Kevin Fang
Beginner. No background in the topic needed.
3.6M views3 years ago2 recommendations
Databases and dataCloud and DevOpsPostgresSRE and incidents
Kevin Fang reconstructs the infamous 2017 GitLab outage where an engineer accidentally deleted the primary production database during troubleshooting.
If you're tasked with deleting a database, make sure you delete the right one. Sources: about.gitlab.com/blog/2017/02/10/postmortem-of-database-outa… about.gitlab.com/blog/2017/02/01/gitlab-dot-com-database-inc… Notes: - The middle bullet point about the account that had 47,000 IPs was never mentioned in the postmortem (there was an initial report the day of and a more detailed postmortem a bit over a week after that). Perhaps that was a red herring which they figured out later on didn't really matter. - I made the error say too many open connections since it's easier to understand than semaphores - This part was confusing, since the postmortem and the initial report conflicted. The postmortem said the engineers believed pg_basebackup was failing because previous attempts created some files in the data directory, but the initial report said the theory was because the data directory existed (despite being empty). So for some reason the engineers really wanted to delete the data directory, but for what reason who knows. - They probably didn't check for backups in this order. I'm sure team-member-1 immediately called out he had taken a backup 6 hours earlier, and then they just had to verify the other backups in case there was a better one. - Being reported by a troll will not automatically remove a user, but flag it for manual review. It was then incorrectly deleted after review. Chapters: Seconds before disaster Part 1: Database issues Part 2: The rm -rf moment Part 3: Restore from backup Part 4: Post incident discoveries Lessons learned The fate of team-member-1 ??? Music: - Thriller Trailer Teaser Tense by Cold Cinema youtube.com/watch?v=r9uV7rjM5Ec - Finding the Balance by Kevin MacLeod - Eyes Gone Wrong by Kevin MacLeod - Desert City by Kevin Macleod - Jane Street by TrackTribe Twitter: twitter.com/kevinfaang/ Instagram: instagram.com/kevinfaang_yt/Why it's here
2 people recommend thisWhat people say. Recommended as an entertaining and educational breakdown of GitLab's famous database deletion incident and backup failures.
xyston Hacker News • Recommends
On “Gitlab is reportedly up for sale”
Every time GL is mentioned, I am reminded of the 2017 incident where a person 'rm -rf' the primary pg data logs. The reenactment by Kevin Fang is always a good source of comedy on a bad day. link to this video It’s a shame they are selling out, but I suppose it was bound to happen sooner or later. GH is vacuuming up all of the market share. I kind of liked the ability to self host my own GL. But for small/personal use cases. It was a bit overkill.
sovietmudkipzon Hacker News • Mentions
On “Gitlab.com is down”
Here is a retelling of the time when gitlab went down because someone deleted a production database: "Dev Deletes Entire Production Database, Chaos Ensues" link to this video
laserDinosauron Hacker News • Mentions
On “Nginx Unit – Universal web app server”
"When a program has nothing surprising to say, it should say nothing." Which sounds good in theory, until you run into issues like this: link to this video TLDW: Most of the Gitlabs outage where they deleted the primary prod and backup prod database could have been solved if they just waited for the command line op to complete - but because of a lack of feedback they thought it had frozen, leading to several other plans which made the situation worse.
renegade-otteron Hacker News • Mentions
On “ElephantSQL Is Shutting Down”
If the primary fails, will the replica fail over? How do you know? Are the backups working? Who's checking? Can anyone delete a directory when tired at 6PM on a Friday? link to this video