How to automatically create Cloud Firestore database backups at regular intervals.
I can't emphasize enough on the fact how important it is to backup your application data. I see a shift in thought as more and more applications are relying on cloud for all their data need. We are under a false assumption that our data is safe and easily recoverable in the cloud. However the truth is, using a cloud infrastructure doesn't necessarily guarantee a disaster recovery plan. It's our own responsibility to ensure that our data is safe and in case of failures, we are able to recover as much of data and information as we could. The use case intensifies with incidents like the recent AWS disaster that left many applications with massive data losses.
Using a cloud infrastructure doesn't necessarily guarantee a disaster recovery plan.
Luckily if you are using Firebase Cloud Firestore database as your data source, you can take a backup of your entire database from the command-line using the
gcloud tool by following their instructions. However, it's still a manual process, and in this era of automation, we would like to have this backup happen on its own, so we can have a peace of mind and spend our time and efforts building great applications. Let's see how we can achieve this automation in few simple steps and a bit of code.
Some background to facilitate the automation
gcloudcommand-line tool uses the Cloud Firestore REST/RPC APIs under the hood.
- There is google-auth-library which can be used to send signed requests to the Google APIs (in our case the export endpoint from Cloud Firestore REST API).
- We can schedule Cloud Functions to run any task periodically, much like a cron job.
- Create a folder inside your firebase storage bucket, eg:
firestore_backups. This folder will contain all your backups that would be automatically created by the setup at regular intervals.
- Enable the
Cloud Datastore Import Export Adminpermission for the App Engine default service account of your project. Make sure to have this step executed before trying the further steps, or else the backup process will fail.
npm install google-auth-library in the functions folder of your firebase project and create a file
backup.js with the following code.
4. Create a scheduled cloud function by adding the following code in your
5. Deploy the function with
firebase deploy --only functions which will deploy the cloud function and also create the necessary Pub/Sub trigger for the cloud scheduler to run the backup function at the specified intervals.
Voilla! We have our automated Cloud Firestore database backup :)
To ensure if everything is working as expected, you can manually trigger the scheduled function by going to the Cloud Scheduler settings in the Google Cloud console and clicking on
Run Now button next to the function name. Wait until the
dbBackup function has finished executing (check cloud functions logs from the Firebase console). If everything went well, you'd see a new folder inside of the
firestore_backups folder of your firebase storage bucket. The folder will have the name as the current timestamp and would contain the backup of your entire Cloud Firestore database from the same timestamp.
If you happen to run into issues or spot any mistakes with this article, please feel free to comment and I'd try my best to help you out / correct the mistakes.