My SRE thought, sure, there might be some benefits for using SQLite but massive downside is you cannot easily examine what server is serving up.
So let's go over the claims about SQLite:
>Transactional Updates : This is the main benefit. Updating multiple files can be done in one transaction. Isolation ensures that there are no broken webapps during the update.
Yep, big one
>Deployment Rollbacks: Another of the transactional benefits is the ability to roll back deployment in case of errors. If multiple apps are being updated, all of them can be rolled back in one go. Rolling back a database transaction is much easier than cleaning up files on the file system.
I guess if you don't keep old artifact of what you are serving up. We zip up our entire static site as part of pipeline and dump into Azure Storage before deployment.
>File De-duplication Across Versions: Clace automatically versions all updates. This can lead to lots of duplicate files. The file data is stored in a table with the schema
INTERNAL SCREAMING Deduplication gives me nightmares from cache issues and so forth. Show me SQLite level of tests here to save a few kilobytes.
>De-duplication Across Apps : Each production app in Clace has an staging app. Apps can have multiple previews apps. This can lead to lots of duplication of files. Using the database helps avoid all the duplication. Even across apps, there are files which have the same contents. Files are de-duplicated across apps also.
INTERNAL SCREAMING Oh god, I hope this section of the code is extremely battle tested because de-duplication at this level gives me SRE nightmares.
>Easy Backups: Using SQLite means that backups are easy. The state of the whole system, metadata and files can be backed up easily using SQLite backup tools like Litestream.
tar directory is difficult?
>Content Hashing: For content caching on the browser, web servers return a ETag header. Using the database for files makes it easy to save the content SHA once during file upload without having to recompute it later.
Most web servers handle this for you.
>Compression: The file contents are saved Brotli compressed in the SQLite table. The database approach has the advantage that contents can be saved in multiple formats easily. GZip compressed data and uncompressed data can be added by just adding a column in the files table.
Cool, now you have to uncompress it every time you want to serve it? Is there massive shortage on disk space and I missed the memo?
I mean, I don't know what Clace does, seriously OP, reading your front page and I have no idea but disk space/speed is rarely a factor. Hell, most stuff probably gets loaded into RAM and is served from there at the end. Meanwhile, your content is being served out this proprietary file system and you are kind of locked in.
Great to get a SRE perspective. The de-duplication uses a SQLite table using a primary key that is the SHA256 of the file contents. There is not much code on top of that. The one complexity is knowing when it is safe to actually delete a file, since it could be referenced in multiple apps.
Clace is built for use cases where teams want to be able to easily deploy Streamlit/Gradio/FastAPI type internal apps (python is supported out of the box, other languages work if they have a Dockerfile present). Clace adds the image building, GitOps, OAuth access control, blue-green staged deployment, secrets management etc on top. So use cases where Kubernetes and an IDP would be overkill for.
> The one complexity is knowing when it is safe to actually delete a file, since it could be referenced in multiple apps.
If this person is anything like me, it's this "feature" that is giving them nightmares. This code has to be rock solid and impossible for anyone to mess up. It only takes one little bug and EVERY SINGLE APP GOES DOWN. Congrats on inventing a new single-point-of-failure!
If it helps, that cleanup is not implemented currently. I did not want the cleanup to be done as part of regular app update/delete. The plan is to add a vacuum/purge type of command, which expressly does this cleanup, and document that a database backup is good to have before running that.
> Cool, now you have to uncompress it every time you want to serve it?
Not necessarily! If the client supports Brotli encoded data (Accept-Encoding: br), you can pass it straight through without decompressing. Ditto for gzip, or zstd for newer browsers.
So let's go over the claims about SQLite:
>Transactional Updates : This is the main benefit. Updating multiple files can be done in one transaction. Isolation ensures that there are no broken webapps during the update.
Yep, big one
>Deployment Rollbacks: Another of the transactional benefits is the ability to roll back deployment in case of errors. If multiple apps are being updated, all of them can be rolled back in one go. Rolling back a database transaction is much easier than cleaning up files on the file system.
I guess if you don't keep old artifact of what you are serving up. We zip up our entire static site as part of pipeline and dump into Azure Storage before deployment.
>File De-duplication Across Versions: Clace automatically versions all updates. This can lead to lots of duplicate files. The file data is stored in a table with the schema
INTERNAL SCREAMING Deduplication gives me nightmares from cache issues and so forth. Show me SQLite level of tests here to save a few kilobytes.
>De-duplication Across Apps : Each production app in Clace has an staging app. Apps can have multiple previews apps. This can lead to lots of duplication of files. Using the database helps avoid all the duplication. Even across apps, there are files which have the same contents. Files are de-duplicated across apps also.
INTERNAL SCREAMING Oh god, I hope this section of the code is extremely battle tested because de-duplication at this level gives me SRE nightmares.
>Easy Backups: Using SQLite means that backups are easy. The state of the whole system, metadata and files can be backed up easily using SQLite backup tools like Litestream.
tar directory is difficult?
>Content Hashing: For content caching on the browser, web servers return a ETag header. Using the database for files makes it easy to save the content SHA once during file upload without having to recompute it later.
Most web servers handle this for you.
>Compression: The file contents are saved Brotli compressed in the SQLite table. The database approach has the advantage that contents can be saved in multiple formats easily. GZip compressed data and uncompressed data can be added by just adding a column in the files table.
Cool, now you have to uncompress it every time you want to serve it? Is there massive shortage on disk space and I missed the memo?
I mean, I don't know what Clace does, seriously OP, reading your front page and I have no idea but disk space/speed is rarely a factor. Hell, most stuff probably gets loaded into RAM and is served from there at the end. Meanwhile, your content is being served out this proprietary file system and you are kind of locked in.