Posts

Booklet is a modern email group software from Contraption Company. Think "Google Groups," but with a modern interface and features. If you haven't tried Booklet, here is a 3-minute video showing how it works.

In this post, I'll share technical details about how I built Booklet. I'll cover the architecture, the technology stack, and the infrastructure. I'll also share some of the lessons I've learned.

Background

I'm Philip - owner of Contraption Company, which makes dependable software tools such as Booklet. I've been building software products for over a decade, ranging from dorm room hacks to enterprise software to startups. I've coded in dozens of different languages and frameworks. Over time, I've developed an appreciation for boring technology that just works.

I built my last startup, Moonlight, with Go, gRPC, Kubernetes, and a Vue.js single-page app. While it was fun to use cutting-edge technologies, it was a pain to maintain them, and new developers had to learn a lot of new tools before becoming productive. I wasted time rebuilding common patterns and integrations that I could have gotten for free with a more established stack. I also found that the cutting-edge tools were often less reliable, and I spent much time debugging issues I wouldn't have had with more mature tools. In particular, maintaining the versions and dependencies of frontend frameworks can feel like a full-time job (especially when using Next.js).

In developing Booklet, I aimed for simplicity in tool selection, aligning with Contraption Company's philosophy of dependability. I seek to create software that remains functional for decades and is easily maintainable by future developers. Hotwire, a minimalistic framework for dynamic and real-time frontends from the creator of Rails, largely influenced my adoption of Rails. In most modern web applications, the backend and frontend are separate in language, framework, and logic. Hotwire feels like an extension of the backend, which reduces the context from two frameworks to one - making development faster and more enjoyable. (Knowledge workers grossly underestimate the costs of context switching.)

Contraption Company's other main product, Postcard, is also built in Rails. It's a similar stack, but I learned some valuable lessons from Postcard. Specifically, Postcard hosts on Render, and I chose a different host for Booklet because I found that Render's DDOS protections slowed down requests by hundreds of milliseconds each, which didn't work well with server-rendered HTML. I recommend Render for apps with simple frontends, but I wanted something faster and more flexible for Booklet.

The secret is that Booklet started before Postcard, back in 2020. The original iteration of Booklet hosted all communities on the same domain. This year, I decided to restart Booklet with a multi-tenant architecture, taking more of a B2B approach to the product and adding the core functionality of custom domains. As you browse the code, you'll see that all of the code references bklt instead of booklet. The naming follows this pattern because booklet was the old repo, and bklt the new one. (I was listening to bzrp when I picked the name.)

Basics

Ruby on Rails is the coding framework powering Booklet. I first used Ruby on Rails in 2011, then didn't return to it for ten years. When I did, I discovered a newfound appreciation for its omakase approach because everything worked together, including testing, caching, and internationalization. In addition, Booklet has a slightly complex permissions system. I prefer to use server-rendered HTML with complex permissions systems because it avoids duplicating logic between the backend and frontend.

Instead of a complicated frontend, I rely on Stimulus and Turbo from the Hotwired framework. These tools lightly enhance server-rendered HTML. But, the simplicity allows me to go deeper - for instance, by building real-time support into Booklet, which I would never have done with a more complicated frontend.

Here is a simple architecture diagram of Booklet. In this section, I'll explain the basic setup, and later I'll go into more advanced details about the configuration.

• Requests route to bklt, the main Rails application where the business logic lives.
• bklt-db is the Postgres database powering Booklet.
• A queue for background jobs runs in a separate Redis instance, bklt-q, with disk-backed persistence.
• Rate limiting is applied using bklt-throttle, a Redis instance with no persistence.
• bklt-cache is a Memcached instance for caching. Memcached is simpler and more predictable than Redis for simple caching.
• bklt-bg is a Rails application for background jobs. It reads jobs of bklt-q using Sidekiq. Most of its work is asynchronous email delivery and AI analysis.
• Postmark sends emails. I've used Postmark for years, and I like it because it's simple and reliable.
• bklt-logs forwards logs to Mezmo for storage and analysis.
• AWS Cloudfront is the CDN that caches static assets.

Having two distinct Redis instances and a separate Memcached instance might seem more complex than just using one Redis for all tasks. However, in engineering terms, complexity is measured by the interdependence between systems. By employing independent systems for different functions, we make the system less complexed, thus simplifying scaling and troubleshooting. Each system has unique requirements, particularly in responding to memory or CPU shortages. Managing these responses is more straightforward when the systems are isolated. For example, bklt-q is set up to preserve data, bklt-cache is designed to remove old data regularly, and bklt-throttle is optimized for speed.

Advanced

Booklet hosts on Fly.io. I chose Fly.io because it allows you to distribute an application globally. I currently operate data centers in New Jersey and Los Angeles. Data centers are traditionally named using airport codes, so I refer to the data centers as ewr and lax in the diagrams. As traffic increases, I can add additional data centers around the world. So, if you're using Booklet in San Francisco, your requests route to Los Angeles instead of New Jersey, speeding up the response times. And, if I add many customers in Tokyo, I can spin up a nrt data center to speed up requests for those customers.

The issue with a distributed application is the CAP theorem, which means that real-time data syncing between data centers is nearly impossible to do safely. Google figured out how, but most common database technologies have yet to catch up. So, I opted to use a hybrid approach recommended by Fly.io: ewr is my primary data center, and lax is a read-only copy. So, if you're reading a Booklet post from San Francisco, you're reading from lax. But, if you're writing a post from San Francisco, that request forwards to ewr. Most requests are for reading data, not writing it - so this approach keeps the app snappy. Write requests are already slower than read requests because they have to write to a physical disk, so the additional latency of forwarding to ewr is less noticeable for these infrequent requests.

Fortunately, setting up this distributed application in Rails is trivial with the fly-ruby gem. It automatically handles all requests routing within the Fly.io network to make this hybrid approach work.

Here's a diagram of the more advanced architecture:

Because lax is a read-only data center, it only has a read-only copy of the Postgres database. I maintain local cache and rate-limiting infrastructure locally to speed up requests. However, I don't duplicate the queue logic right now - background jobs run out of ewr only for simplicity. The speed of enqueuing jobs from lax is a potential area of improvement.

Additional tools

• Docker for deployments.
• Sentry for error reporting.
• AWS S3 for file storage.
• OpenAI for AI and moderation.
• Helicone provides OpenAI monitoring and caching.
• Fly.io handles custom domains and TLS certificates.
• AWS CloudWatch for monitoring uptime.
• Pagerduty wakes me up for every single error or downtime.
• Stripe for payments.
• TailwindCSS and TailwindUI for styling.
• Skylight for performance monitoring.

How it's working

Overall, this stack has been fast. As a user, clicking around a community is snappy and pleasant.

Overall, I'm happy with Fly.io. I appreciate the low-level access it gives to machines. They're a startup with growing pains - I've had to manually restart machines or reroute traffic a few times due to a lack of auto-healing. But, with proper monitoring and alerting, I've made the system more stable.

The primary source of instability within Booklet has been occasional database connectivity issues. After encountering recurring database connectivity issues, I discovered that Fly.io uses HAProxy in front of Postgres but doesn't document well that their HAProxy has a 30m timeout. So, occasionally, HAProxy would terminate an active database connection, Rails would not immediately reconnect, and the result would be 500 errors from a particular container. Fortunately, with some health check improvements and connection configuration changes, I've prevented the issue from affecting customers.

What's next

The next feature with infrastructure implications is search. Booklet users should be able to search posts, replies, and profiles.

I haven't finalized the approach yet, but it will likely involve Elasticsearch. However, I'm also investigating vector-based search as an alternative using OpenAI Embeddings and pgvector.

Eventually, I could see moving toward CockroachDB for a distributed database to improve latency and reliability further. But, the additional complexity is not worth it right now.

Conclusion

If Booklet's architecture meaningfully changes in the future, I'll publish a follow-up post.

If you have any questions or feedback, email me at philip@contraption.co. And, if you're interested in trying Booklet, you can join an existing group or create your own for free.

With the launch of Indiehackers.com in 2016, a counter-culture of startups emerged seeking to build businesses without venture capital funding. The trend followed established independent companies such as 37Signals, but also emerging makers such as Levels.io and Danny Postma.

Seven years since the start of the trend, a polarizing discussion between Pieter Levels and Arvid Kahl asked whether indie hacking is dead. The evidence they present is that it's now mainstream, competitive, and saturated.

I have a different perspective. I believe that AI is causing a convergence of traditional startups and indie companies because AI decreases the cost of starting new software businesses dramatically. As "startup" and "indie" converge, we'll instead recognize a divergence of "micro" versus "growth" technology companies. AI will decrease the cost of starting new software companies so much that many will become profitable before pitching VCs, and these companies be able to decide whether to stay small or to raise money for growth. With these changes, "Micro" will become the new "Indie."

In Oslo this week, I enjoyed visiting one of my favorite coffee companies, Tim Wendelboe. The business started in 2007 and has grown to be an influential coffee brand served by discerning customers such as top restaurant Noma. With this success, Tim Wendelboe still has a single cafe location. I saw his company described as a "Micro roaster," and that's where the inspiration for this essay came from. Between quantity and quality, this company chose a craft mindset - to stay small and focus on quality instead of growth. Compare this to a coffee roaster like Starbucks, which scaled to multiple locations and gradually offered commoditized products instead of specialty ones. I'm sure Tim Wendelboe could pursue more locations and scale, but doing so would compromise the quality - and its customers might change. "Micro" companies already exist outside of technology, and represent a focus on quality instead of scale that customers appreciate.

The word "Indie" describes companies that don't raise external money. But, the definition is blurry - The Hustle described itself as "Bootstrapped" because, though it raised money, it was a relatively small amount from individuals instead of from institutional venture capitalists. Then, The Hustle stayed small and capital-efficient, which made its exit a major success for the founders. At the other end of the scale are businesses that achieved scale without raising money, but later chose to raise investment. Take GitHub, which raised no external money for years, building a large book of recurring revenue as an "indie" company. But, this indie approach gave GitHub access to minimally dilutive capital, and in 2012 they raised a first round of funding of $100m at nearly a billion-dollar valuation. Companies can raise some money, but stay indie - or, inversely, start indie but then raise growth capital.

The nature of software businesses is that they are expensive to design and develop, but inexpensive to scale. They have high upfront costs and zero marginal costs. For example, if I make an iPhone app for editing photos, then that app has the same development cost whether 1, 1k, or 1m people buy and use it - but the revenue is a function of the number of users.

Software businesses typically have two phases - a development phase, where they iteratively make a product until they have figured out something that customers want. What follows is a scaling phase, where the unit economics of the business are positive, so the company needs to decide and how to grow. Within growth, the general idea is to "spend money to make money" - whether using ads, marketing, sales, or new features to attract new customers or increase revenue from existing customers. Typically, the scaling phase is where you see companies raise large amounts of venture capital - any investment over $5m is typically for growth, not initial development.

"Indie" has typically referred to businesses that try to raise little or no money to develop their product. This attracts motivated builders, who typically make a product themselves, whether by coding or using low-code tools. Capital constraints filter for founders who have a specific vision of what to build and can fulfill that vision. Indie founders have a reputation for being scrappy and bohemian in their approaches to business, in contrast to VC-backed founders who want to project a more polished facade to potential investors.

What has changed in the last year is AI. While casual users may see ChatGPT as an amusing tool for writing emails or memos, it has begun to dramatically change how software gets written. Experienced engineers who embrace tools such as Cursor can more efficiently design and build complicated systems in much less time, turning them into "cyborgs". In the past, a software team would be built with two senior developers to design the systems, four junior developers to implement the code, a project manager to handle work distribution, and a people manager to help with hiring and teamwork. With AI, a senior engineer can give directions to an AI agent instead of junior developers - thus making the rest of the team redundant. In short, AI is turning senior engineers into one-person teams. This isn't the future - it's happening right now, and my one-person software company is an example of the productivity gains that AI can afford.

With AI productivity advances, the amount of capital required to start most software businesses is dramatically plummeting. Unless you're building physical products or developing moonshot intellectual property, I believe the cost of making a new software startup will decrease by 50-90%. This means that most startups will have pressure to raise significantly less capital if any at all. People with compelling ideas can just go build a prototype to validate their hypothesis, instead of needing millions in funding just to launch a minimally viable product. People will no longer waste lots of money and time pursuing bad ideas - the market will be able to offer concrete feedback much earlier in the process

With the decrease in upfront costs due to AI, the majority of technology companies will look like what we call "indie" today. These companies may still raise a small amount of money, but it will primarily be to cover the salaries of three or four people, rather than 20. My prediction is that investors will write smaller seed checks more frequently, but that the total amount of seed funding will decrease.

As "indie"-style building becomes the norm, most companies will first consider raising venture capital after they have initial traction. Among startups that get traction, this capital can be spent to double down on growth. However, the smaller headcounts of these more efficient companies mean that companies can achieve profitability sooner in their lifecycles. With a smaller team, you need less revenue to break even on costs. Over time, the timing of profitability will converge with opportunities to raise growth funding - meaning that the need for capital will no longer be urgent or existential. Historically, $10k in monthly recurring revenue is a Schelling point among seed investors that a startup that has achieved reasonable traction to raise additional capital. But, that $10k MRR line can now mean profitability for a tiny team.

The convergence of profitability and growth funding opportunities means that startups can decide whether to stay "micro" or chase growth. And, increasingly - instead of a division between "traditional startups" and "indiehackers", we'll see a division between "micro" versus "growth" startups. Micro companies will choose to stay small and focus on profitability, craft, quality, and lifestyle. Growth companies will invest external capital into more revenue, more markets, more products, and more team members in pursuit of a larger outcome.

This shift toward smaller seed rounds will catch many repeat founders off guard. Over the past decade, building a startup required raising a lot of money upfront. So, the system naturally selected founders who were good at impressing investors and raising money. But, some of these founders lacked the skills to build a business and make something that customers wanted. I've personally observed founders raise and spend over $10m before investors realized that no product traction was being made.

Lower upfront funding and more growth funding create a more egalitarian building environment, where founders will have to compete for customer attention instead of investor adoration. And, this happens to be exactly what the Indie community exemplifies today - founders who put their energy into impressing customers instead of impressing investors. With this shift, I think you will see the "indie" outcasts of today become the same companies that go on to raise large growth rounds of capital in the future. More companies will follow the GitHub model of proving market demand with minimal capital, and investors will seek evidence of traction before meaningfully investing. Innovative venture capital funds are already exploring this market - such as Indie.vc and Calm Company Fund. But, soon all the traditional funds will be competing for these deals.

Indie isn't dead - it's becoming the norm. AI means that founders need significantly less capital to start a company. Investors will expect firm evidence of traction to raise venture capital. But, traction will also converge with profitability for most of these startups. So, more companies will have the choice of whether to stay "micro", or whether to raise money in pursuit of a growth path. This change in incentives will benefit founders who can build and show traction with minimal or no funding, which happens to be the indie community today. But, more "indie" founders may begin to raise venture capital in pursuit of growth - while others may make the intentional decision to stay micro.

It will be an exciting few years for software companies. But, I'm excited to see more small, craft-focused software companies. Tim Wendelboe's coffee is fantastic because he chose to stay small, and pursuing a niche created a customer community passionate about his product. I'm excited that AI will lead to more, smaller software companies - which will increase the diversity of products and business models available to customers.