How I optimized Firestore read/write by 2000 times

If you use Firebase and Firestore for long enough, you gonna realize that the read/write times are the biggest impact on your invoice every month.

I have this problem for my pet project https://boostmequotes.vercel.app/

Basically, I have about 65k quotes that gonna send to the browser for running offline mode. The app is doesn't have any users, except me, and let's think about this. If 1 user takes 65k read every day, what if I have 50 users?


65k * 50 * 30 days = 97 500k

Which cost me about 17.5$ month

That's a lots for just 50 users and a pet project, if I use Cloud computing, it is more affordable solutions.

How can I reduce read/wite to 15 times

When I learn NEAR Protocal I found that they used sharding - which is an approach to optimization that comes from database.

Understanding Database Sharding | DigitalOcean

Sharded databases have been receiving lots of attention in recent years, but many don't have a clear understanding of what they are or the scenarios in which it might make sense to shard a database. This conceptual article goes over what sharding is,

https://www.digitalocean.com/community/tutorials/understanding-database-sharding

By changing the structure of #FireStore from each item into a new doc, we can combine multiple items into a doc and saving thousand of reading/writing time.

I store about 200 quotes in each documents, you can think that each my document is the same as pagination query.

So that instead of costing reading 2000 records, It just cost 1 read. 2000 times decreasing!

What does it cost?

Well, the most cost is, you need to programing to get your desire result, no more query, filter, or order supported by the SDK, you need to do it by yourself

But believe me, you gonna do that by yourself if you use Firestore, it's suck 🙂

Ref: https://firebase.google.com/docs/firestore/query-data/queries#query_limitations

Firestore looks awesome when we first start and then it's suck when you found out those limitations

You can't even count how many documents you have in a collection

How should I split my documents?

It depends, this is the hardest question when you implement sharding. For my application, I just separate each shard by 200 quotes. You have more about your application context, so you're the best to answer on How to separate your documents

After having a strategy on separating your document, you also need some effort when inserting/updating your document. Which is hard or easy is based on your strategy

When should I use sharding for Firestore?

You are happy on coding to query documents

You have a strategy on sharding that can answers

How can I separate my document?

By your separate strategy, how can I find the document I desire

How can I insert/update new documents? Which shard should I put it in

If you have a clear answer to those questions? Get ready on saving your billing by 2000 times