A long time ago, I wrote an ORM to serialize/deserialize object data seamlessly into SQLite, with massive arrays of floats being stored as blobs.
In code, I could effectively mark which class members need to be stored/restored, and optionally provide a custom serialization function for them if needed.
The latter was effectively never necessary, because all the bases types and multi-dimensional arrays were handled by templates.
Really wish I open-sourced that thing then, but the corporate bureaucracy around that was tricky.
I remember sufficiently little about implementation details now that I think I can get to writing it again, without producing a copypasta of that code - and maybe I should :)
I am using SQLite as document db for a side project for years now. Made a custom repository base class that can also store blobs in separate columns, so this type of data is not part of the json document. Today there is also jsonb [1], as far as I remember all functions work the same for json and jsonb.
Also the repo class stores write and delete timestamps as separate columns so I can have CDC. CDC is used for building cached view models and is pushed to object storage every 5 minutes for backup as NDJSON. Another process on home server is restoring the db every couple of minutes for second backup and ready to use DB in case it's needed.
I know there are things like Litestream, I wanted something in process and something that can send alerts on failed backups.
> (Aside: The hard bit may be getting a new enough SQLite, at the time of writing Homebrew on macOS has it, else you likely need to use an unstable source like nixpkgs-unstable.)
That was the part I really missed from my Google days.
That, insane achievement badges, and terrible-ideas-discuss (if anyone at Google is reading this, I have one word for you: dirigibles). It was like /r/NonCredibleDefense but for Google.
In code, I could effectively mark which class members need to be stored/restored, and optionally provide a custom serialization function for them if needed.
The latter was effectively never necessary, because all the bases types and multi-dimensional arrays were handled by templates.
Really wish I open-sourced that thing then, but the corporate bureaucracy around that was tricky.
I remember sufficiently little about implementation details now that I think I can get to writing it again, without producing a copypasta of that code - and maybe I should :)
Also the repo class stores write and delete timestamps as separate columns so I can have CDC. CDC is used for building cached view models and is pushed to object storage every 5 minutes for backup as NDJSON. Another process on home server is restoring the db every couple of minutes for second backup and ready to use DB in case it's needed.
I know there are things like Litestream, I wanted something in process and something that can send alerts on failed backups.
[1] https://sqlite.org/jsonb.html
https://sqlite-tutorial-pycon-2023.readthedocs.io/en/latest/...
https://simonwillison.net/2021/Jul/28/baked-data/
So I can have cassette.propxyz = {1, 2, “abc”, true}
And
local anotherprop = cassette.leprop
Or just download the source and build it! (Gasp!)
https://youtu.be/b2F-DItXtZs?is=HlayyJ_DPb8NzbS4
(lol! couldn’t resist)
That was the part I really missed from my Google days.
That, insane achievement badges, and terrible-ideas-discuss (if anyone at Google is reading this, I have one word for you: dirigibles). It was like /r/NonCredibleDefense but for Google.