In March of 2021, two weeks before we were supposed to be wired $10m for selling my first bootstrapped startup, one of our engineers accidentally deleted the entire code base - and WE DIDN’T HAVE A BACKUP!!!
Here’s the full story and how NOT to run your IT at a tiny startup:
March 26, 2021 was a day like any other.
I woke up, went for a run, made some coffee, and plugged in for the day.
At around 10am, I got a text from Tate, our CTO.
Tate: “Something really bad just happened.”
This wasn’t immediately alarming.
It's a startup. Things always go wrong.
Me: What happened?
Tate: Gustavo (name changed to protect the innocent) deleted the entire code base.
Me: WHAT?!?!?! How did he do that?
Tate: One of our four nodes crapped out, which has been happening all the time. I taught the guys to go into Amazon and 'reset' it. The problem is there is an option to 'stop' the node directly below the 'reset' option. He chose to 'stop' that one, and just then, by chance, a second node crapped out. He stopped that one two. When you stop two nodes on a 4 node distributed database, it’s all gone.
Me: Well, that sucks, but I guess he didn’t mean to. We have a backup, right?
Tate: No, because to save money, I wasn’t writing a brand new backup file every day because I didn’t want to store two 1TB files. Instead, I would update the old file with the new backup information. There is a two hour window every day where it was writing where if the code base was deleted we wouldn’t have a backup.
I finally began to internalize the meaning of what I was hearing.
My heart started to accelerate and my palms began to perspire.
TIMELINE:
10am: Find this out.
10:02am: Call co-founder, James, we panic.
10:15am: Call Amazon, they say there is nothing they can do. Ask for someone
higher up, tell them millions of dollars are on the line.
10:30am: Speak to the higher up person at Amazon, they tell us there is nothing they can do.
10:32am: Tate attempts to reach out to the backup company.
11:00am: Acquirer reaches out and says “your website has been down for a couple hours. Do you know what’s going on?”
11:30am: Tate makes contact with the backup company. They tell him he is an idiot and pennywise and pound foolish for having any time exposed without a complete backup file. Say they can’t help.
11:45am: Tate explains that we are about to sell this for $10m - in literally two weeks - and is able to convince them to connect him with a very senior engineer who actually built the product to see if they can dig into the file somehow. But they had never done this before, so they had no idea. They gave it us a 20% shot they would be able to get the code base back.
12:00pm: I go to my friend Brad’s house - a VC - who says he has never heard anything like this ever in 30 years of tech, and that Tate had to be plotting something. I said he had 25% of the equity. No way.
12:30pm: Go to my friend Lou’s house who worked at Vista for 11 years, also never heard of anything like this, also assumes it must be some sort of malicious attack, maybe by the developers?
1pm: I realize talking to my friends isn’t helping. I grab my 6lb dog and start walking around Austin.
I walked until 8pm. Couldn’t eat, couldn’t drink.
I was calling up our bankers and telling them the transaction was off because we didn’t have a website to sell, and having to explain this absurdity to them.
I had already internalized that I was personally going to get $4m in two weeks - and I basically had ZERO savings at the time.
I was going to pay off my mortgage. I was going to buy a new truck, and buy a boat because I had just found a slip on Lake Austin. After taxes I was still going to bank $1.5m and put it in the equity market.
What did I do?
I thought the smartest strategy was to accept the most likely and worst case scenario, and if I got surprised, great. If not, I was already on my way to healing.
I walked around Austin for 8 hours and told myself, over and over again:
“It’s only money. You will make way more money in the future. It’s only money. You’ll make way more money in the future.”
But it wasn’t just money.
Internalizing that all of the work everybody who had ever worked on it had put in over the years - even though we never got it past $3m ARR - literally brought me to tears.
By 6pm there had still been zero progress breaking into this backup file.
I kept walking.
At 10pm Austin time, Tate called.
“We’re in. They aren’t sure how the process I was using actually works, so they don’t know what we will find, but we are in.”
I was so exhausted from the crushing devastation I was actually able to pass out at around 11pm.
I woke up at 4am Austin time, after having nightmares about backups and databases all night.
I checked my email. There was one from Tate, from 15 minutes prior.
“WE FOUND A BUNCH OF FOLDERS!!!”
I called him right then, and he explained that they were able to dig up what looked like all of the folders that made up the code base.
He didn’t have anything to compare it to, so he didn’t know the completeness of it, nor did he know if there were actually any files in them, but they started with the folders that started with
“A’s” and it appeared as though most of the critical files were in it.
After spending eight hours preparing myself for total loss, I felt a rush of relief that I had never felt before or since.
Slowly, over the next 8 hours, I got update after update that they were into more folders with more files, and by noon, they had recovered 97% of the code base.
It would take two days for the website to be up and running again, but the only problem would be that some reports would be off from the day before.
Jesus.
Adam
P.S.
MoltSets just crossed 300 paid users in our beta. Enrich your prospect lists with verified emails, mobile numbers, and LinkedIn data, right inside your AI stack from only $27/mo! Join the beta waitlist here.