LiveStore: SQLite-based data layer for local-first apps

This is a guest post from Johannes Schickling - the orginal founder of Prisma, who is currently building Overtone and LiveStore. He is also the host of a podcast called localfirst.fm.

...

• LiveStore, a cross-platform, SQLite-based, local-first data layer, is now available in early access for Expo
• LiveStore comes with first-class devtools (incl. live databrowser, query inspector, DB import/export, …)
• You can request access via this form or take a look at the Linearlite example app & demo video

Hi, my name is Johannes Schickling (prev. founder of Prisma) and I’m super happy to share some news about a new project I’ve been working on for the past 3 years called LiveStore.

And more specifically, an Expo integration which was developed in collaboration with the Expo team for the past 6 months. I’m really excited about it, so let’s get into it!

Building great apps has become a lot simpler over the years - particularly mobile apps with tools such as Expo helping with many aspects of development. One aspect however is still commonly a big barrier when building great apps: data management.

User expectations are particularly high for mobile apps requiring apps to work offline in most cases as well as sync data in real-time across devices and users when possible. Designing and implementing a data architecture which fulfills those requirements in a correct and performant way can be very challenging with traditional architectures (e.g. three-tier architecture) as online/offline data can easily get “out of sync” leading to inconsistencies/conflict/data loss.

An increasingly popular approach which offers an elegant solution to this problem is called local-first. A typical implementation involves a sync engine which synchronizes data across clients (usually through a central server) which replaces request/response-based API patterns.

As local-first apps usually hold a full copy of the (relevant) data locally, client-side state management becomes even more important. Luckily Expo already offers a powerful building block for this scenario: SQLite. When building cross-platform applications it’s particularly desirable to re-use (parts of) your data layer across various platforms. SQLite is a great fit here as it works on every platform including the web via WASM (Expo web support pending).

Building your own local-first data layer can be very challenging (yet is still done by companies such as Linear). In order to provide a great user and developer experience (i.e. to be fast, correct and ergonomic to work with) it requires solving many hard problems.

Those challenges usually involve:

• Implementing (or integrating) a distributed syncing engine
• Schema management / migrations for your SQLite database
• Reactive and performant SQLite queries

Unless you have very specific reasons to build your own data layer, it’s strongly recommended to build on top of an existing local-first technology.

The rest of this post will dive into LiveStore but note that there are also other great local-first(ish) technologies available (such as Jazz, ElectricSQL, Zero, TinyBase…) which are fantastic options. However, none of those options were meeting the specific requirements / tradeoffs LiveStore offers (e.g. SQLite-based, event-sourcing-based syncing, …).

LiveStore is a local-first data layer based on SQLite built for demanding applications. LiveStore is based on years of research (including the Riffle project which I presented at App.js Conf 2023) and has been the foundation for Overtone, a local-first music app.

LiveStore is aims to provide a principled data layer enabling complex and high-performance scenarios with a simple API and mental model. The following demo illustrates the achievable performance (even in a web app).

LiveStore is built on top of SQLite which does the heavy lifting of state management by providing a proven performant and efficient query/storage engine. LiveStore enhances SQLite by adding a high-performance, signal-based reactivity layer enabling live query results.

LiveStore also provides a powerful schema management system including advanced type-safety and automatic schema migrations.

Different from other local-first systems (which are often based on CRDTs), LiveStore is designed around the idea of event-sourcing by embracing read/write-model separation. This means all writes are stored/synced as “mutation events” in a separate eventlog (write model) and simultaneously applied/reduced on top of the SQLite database (read model).

💡 There are some similarities to how Redux works where LiveStore mutations can be seen as Redux actions but instead of a Redux reducer updating a JS object, LiveStore translates mutation events into SQLite writes.

LiveStore offers first-class devtools to assist with common development workflows. Similar to how tools like Chrome devtools or React devtools have become essential for developing frontend applications, the LiveStore devtools provide visibility into your app data in real time.

Some of the top features include:

• A live data browser with two-way data editing support
• A query inspector to analyse and debug your app queries
• Import/export database workflows
• A SQLite playground
• And many more…

While LiveStore aims to keep a minimal API surface and its responsibilities focussed on app data management, it does a lot of heavy lifting and provides many benefits such as:

• Instant local queries/mutations with synchronous API
• Deep view library integrations for React
  • Includes optimizations to minimize re-renders
  • Packages for Solid & web components are experimental
• Cross-platform support for web & mobile
  • Web offers cross-tab syncing & OPFS-based persistence
  • Electron/Tauri support are on the roadmap
• Works with Expo Go to speed up development for simple apps
• Coming soon: Local-first syncing (cross-device/cross-user)
  • LiveStore syncing is not yet fully implemented for Expo but is on the roadmap.

To get a better feeling for what’s possible with LiveStore - both from a user and developer experience point-of-view - we’ve built a comprehensive example app: A Linear clone built in Expo, called Linearlite. (Shout out to Linear for setting the quality bar high for new apps!)

A few noteworthy aspects of this app:

• It’s fast!
• Works entirely offline
• Persists most app / UI state (including current navigation route & text inputs)

Check out the demo video below to see it in action or take a look at the example source code.

A special thanks to Beto who built this example app in collaboration with me over the past couple months and provided amazing feedback which led to multiple improvements in LiveStore.

If you’ve made it this far in this blog, then you’re probably curious about how to get started with LiveStore. As of today (November 2024), LiveStore is still in private preview to limit the amount of initial feedback and keep development sustainable.

So far only GitHub sponsors had early access to LiveStore but as of today, I’m excited to announce that EAS subscribers (production plan and higher) will also be able to request early access by filling out this form.

With early access you’ll be able to access the following:

• LiveStore repository including the source for all packages and examples (excl. devtools source)
• LiveStore Devtools
• Discord community

I want to also share a brief note on the early access/sponsorship-driven approach:

LiveStore is a very ambitious project and has already seen many years of upfront research and development (incl. a MIT PhD thesis) while not being VC-funded or incubated in a larger company (like React by Meta).

Keeping the continuous research, development and maintenance sustainable is very important to me and the long-term health of the project. I’m planning to work on this project for the years to come and hopefully (with more support from the community) will even be able to bring someone on to work on LiveStore full-time. I deeply appreciate all the support and contributions.

If LiveStore sounds like the right fit for your commercial app and you’re interested in discussing a potential partnership, please get in touch.

Even though LiveStore has already been in development for over three years and has come a long way, in some way the project is only getting started. While LiveStore is not yet ready for production, it’s already a great time to start developing apps with it.

The current plan is to open up access to LiveStore more over the coming months based on feedback and stability moving the project into public preview. You can follow along the progress on Bluesky or X.

In terms of features, the current priority is implementing a new git-style syncing system which will enable powerful and flexible collaborative experience while keeping development simple.

Specifically related to the Expo integration for LiveStore, there are plans to also support LiveStore via Expo for Web as well as a high-performance list component.

I couldn’t have come so far without the help of countless other individuals and companies who have supported LiveStore along the way. A huge thank you to everyone!

A special thanks to Expo for this wonderful collaboration. It was a lot of fun and I’m really excited about the result. In particular I want to say thank you to Brent, Beto and Kudo for our close collaboration and continuous help along the way.

I can’t wait to see what you’ll build with LiveStore and I’m looking forward to your feedback! Oh and if you’re interested in learning more about local-first, then you can check out the localfirst.fm podcast.