So Expensify released their S-1, and this section caught the internet’s eye:
Here, I’ll embed the picture too:
This was my reaction:
If any of this seems normal to you, I feel for you. Lemme know where to send the fifth of Jack, because God knows you need it.
Now let me start off by saying there could be an entirely logical chain of events that got us to this point with Expensify. Bad decisions — at least how it looks to the outside — are often the result of a lot of little decisions stacked up. No one that should be practicing in tech gets up one day and says, “I Know, I’ll build a tech stack that would make Hacker News — the place that seems to adore nonsensical architectures — to go, ‘Hrm. That’s a bold idea. A bit too bold for us'”.
What’s wrong with this architecture? I’m going to assume for a second that selling this architecture in this way in the S-1 was the goal from day one, and not just the most innane bullshit written in the first draft that somehow looked good to the CEO and it was kept in even though the CTO wrote it as a first-draft joke to stave off writers block talking about architecture. The architecture itself is way too complicated for anyone’s needs, let alone Expensify’s. It’s like going to the grocery story and buying all the chocolate candy from the checkout lane to melt down and make chocolate ganache from. Sure, you can do it, but why would you want to?
SQLite is a very fast database. It’s very fast because it’s a single writer database. That’s how it’s so fast. It doesn’t have to worry about all those sticky little problems of multiple users and multiple results sets and locking because there should only ever be one connection with one thing happening at a time. It’s meant to be used in situations where an embedded database is a really good idea, and that’s not in a high-volume internet application where billions of dollars is managed in expenses daily.
and so the architect (Again, I really bet this was an accidental architecture, but in case it wasn’t) likely knew this, and knew this little prototype needed to be rewritten, but the business folks said “no”, and so they decided to… shard it. Literally. Shard the unshardable database using Paxos.
There are lots of problems I have with this description of their stack, but not the least of which is that they are heavily susceptible to job lock-in. How many people in the world have experience with Paxos and Sqlite in this setting? For my money, I bet it’s just the folks at Expensify — because, and I’ll say this again, this architecture is unmoored from reality. Let’s say the chief architect leaves for greener pastures. You can’t just hire off the street for this. No one has 1 year of experience trying to make Sqlite and Paxos work together in this manner, let alone several. You’d better hope you have a good succession plan in place and every single part of this system needs to be documented to the nines and out of anyone’s head to even have a hope against a critical person leaving. As of this writing, there are 99,000 total results from Google about the words “Paxos” and “Sqlite” together. For comparison, DB2, a database that hopefully isn’t still in use in the tech industry for new projects, has 27,300,000 hits.
At some point, any successful business needs to scale its tech. This is normal: V1 isn’t meant to last, it’s meant to get money in the door, either through VC or customer adoption, and the problems you’re solving with Version 1 aren’t the same problems you have when you’re filing for your S-1. Scaling isn’t easy, but if you pick a path no one else has ever taken before, that’s not a competitive advantage, that’s a risk with lots of long term bagged you now carry. A rewrite may not be viable, but if you want to be a world-class chef, making ganache from the checkout aisle isn’t an alternative unless you want to be the checkout-aisle ganache chef.
Does Expensify want to be a public company or do they want to be that quirky “we doubled down on bad decisions and make them sound good” company? It’s hard to say.