All you need is Postgres until you scale into TBs of data. We use Postgresql as a durable workflow engine, vector search, time-series data, BM25 search, OLTP/OLAP engine, and a queue. It's basically the only dependency we have for https://lobu.ai
The main benefit is centralizing all the data in one place so we don't need to worry about copying data in between multiple systems. Once something becomes the bottleneck, you can eventually migrate to a purpose specific tool to scale out.To be honest, LISTEN/NOTIFY in my opinion is the most fragile part of PG but it's fine as start until you scale out.
Use different “databases” besides public at the very start. No joins between them. You will be in a good position to just split the postgres instance by those at a later date. They will have different usage patterns than the merged version you have now, and will be easier to optimize and will buy you some time. And time is all you need.
conversely, startups that start scaling for tbs of data never make it to needing tbs of data. They burn too much energy scaling when they don't yet have a product people want yet.
But when you hit that wall, it is hard to stop and convince people to use different patterns and systems. I've seen so many tables go from "it will only be a few thousand rows" to suddenly several TB and then people are looking confused when performance and db admin tasks get really difficult.
I'm working at a scale where almost every day I have to ask people "are you use you need to treat that as relational data? It doesn't seem relational"
In pg19 https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit... will land, which significantly improves NOTIFY performance. Right now LISTEN/NOTIFY doesn't scale to very busy instances because a `NOTIFY` within a transaction takes a global lock.
I'm in the same camp. Do you use any specific extensions? Especially for OLAP and time series (partitioned tables + related extensions work fine, but curious if you use anything else)
The native extensions are fine but I don't have good experience with any third party extensions, so far tried Timescale, pg_lake, citus, and pgvectorscale. They look very appealing but it's usually a trap as you can't get the value without using the vendor's cloud offerings.
I think if you grow enough to look for these extensions, it's usually better to bet on purpose-specific tooling. For example, I use DuckDB/Iceberg combination extensively for columnar data and connect DuckDB to PG when I need it.
I don't see logs mentioned. I agree with most those applications but would keep my OLAP stuff (metrics, logs, traces) in a separate store like VictoriaMetrics, both for capacity and read activity.
If you don't need a ton of throughput I think `absurd` (and our Rust derivative `durable`) are very nice options that keep the client side extremely simple. It's also lightweight enough that a coding agent can keep the entire thing in its head easily and just run queries to look up state as needed.
As someone who uses dbos.dev, restate.dev, cf workflows here is a snippet from our Agents.md:
Restate.dev:
for payment integrations on northflank since its faster than cf workflows, independent of cf and its downtime and self-hostable vendor-lock-in free,
Cloudflare workflows:
for non critical stuff like csv/pdf report generations since it's very cheap.
DBOS.dev:
for workflows that need atomic messaging tied to a postgres db transaction for 100% reliabilty/durabilty(for example populating a materialized row or sending out critical email/push to a merchant).
DBOS and Restate are similar on surface but Restate requires a central "orchestrator" which has pros and cons but makes it easy to build with serverless workers on cf/vercel.
It also has VirtualObject which is a nice vendor-lock-in-free OSS alternative to CF's single threaded DurableObject.
Where DBOS absolutely shines is
1) Atomic messaging in the same db tx as your business logic via dbos.enqueue_workflow! This is often the most brittle part of any solution and doing it atomically and durably with same tx that ran your business logic drastically reduces lots of complexity.
2) Since DBOS stores workflow state in db it should be easy to build dashboard for observability from metabase/looker(I wish restate exposed its rocksdb instance so it could be hooked up to metabase).
My dream is, instead of separating data storage, state machines, valid state constraints, and the logic that transitions between valid states, we can actually unify these into some kernel of app state. Honestly, Postgres already has a lot of these capabilities, but I don’t see an obvious story on the app or product level, providing provably correct sets of states that apps can transition between, and which they can automatically expose to clients in informative ways (this user can like this post, but not edit). It looks colored Petri net shaped to me, but I don’t yet see a simple app state paradigm in the same way that the database has obvious successful boundaries.
Curious to know experience of people using DBOS and Temporal.
I have used Temporal in the past, works really good, my only problem with it was some limits on request payload or event sizes, created some inconveniences to us when building solutions. It also enforces good engineering practices, but sometimes you don't want to write special logic if your CSV file is larger than 2Mb, upload it to S3, pass link, then download it in the workflow.
What is your experience with DBOS? How does it compare to Temporal in terms of operational complexity, feature parity and anything else
I thought Temporal was overly complex, but as you said the best part is it does enforce good engineering practices.
Then I tried their Cloud offering and was appalled at their pricing. I burned through the $1,000 free credits before I even got something to production. Didn't want to bother with running a local Temporal, either.
Best solution is to just take inspiration from their architecture and then do it yourself in Postgres, IMO.
They've just released an external storage approach to solve the large payload issue. I don't 100% love it (it's bolted on, not an intrinsic part), and it's an early release right now - but you can consider this effectively solved for now.
That's good because back in the day if you were putting entire documents in a message queue I would laugh people out the door, putting something in object storage + linking is much more useful (though the distributed system part/backup current state part can be annoying!)
we're using dbos for ai gen workflows and processing video files. understanding how to migrate from celery took time, but for our case it was worth it.
I run a large on-prem temporal setup - throwaway acct as they will likely out me.
Temporal is, in my opinion having run it in prod for over a year - poorly designed, slow and ridicliously heavy infra wise.
If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely suck.
Try running their own benchmarks, the numbers are pathetic.
Their sales team is also absolutely appalling and desperate.
From a Developer standpoint, the SDK is quite nice though.
Don't get trapped into nexus, and if the sales team call you make sure legal is in the room.
> If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely suck.
Where are the “millions” on infra going? It’s a handful of services and a Postgres?
> Their sales team is also absolutely appalling and desperate.
You said “on-prem”. It’s open source; why are you dealing with their sales team?
> If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day…
If “millions” were required to obtain such tiny scale, I’d agree there’d be a massive problem. No one would use Temporal; it would be a complete waste of resource. If this were true.
Agree. Have worked in a codebase using Temporal, and is pretty much a nightmare. I don't know about the infra side, but from the developer side, all the abstractions they bring to the table are poorly designed. Wouldn't recommend
https://github.com/agentspan-ai/agentspan which is essentially an agentic SDK layer for Conductor can convert any of your langgraph, openAI, vercel, or ADK agent and makes it durable and adds orchestration with no code changes.
I completely get the concept and agree - this is great way to build this kind of durability in a workflow system.
That said, my gamer-brain wants to call this "Save-scumming at scale." Which is to say, a lot of people already know that this approach works, but maybe they haven't made the connection to abstract CS stuff.
Another strategy that can be used to build robustness is to build your workflow out of idempotent operations. That can be useful for situations where the workflow state is too large to back up. Instead, you just run the job from the top and it's a bunch of no-ops until you start making progress again.
Since DBOS doesn't support Rust, we implemented a very minimal Rust version of this at https://github.com/tensorzero/durable. It has been quite stable and extensible but of course you need to be very careful with the SQL implementations. Hope this is interesting to readers here.
Continuously amazed by what you can do with few tools, as long as Postgres is a part of your toolkit.
I recently developed a distributed queue and it works really great - benchmarks great too, with no race conditions or conflicts. I used SKIP LOCKED so that workers can compete safely.
You can also have multiple workers across nodes avoid conflict by using session wide mutexes i.e. pg advisory lock.
We work on disk log based architecture for workflows at Unmeshed (https://unmeshed.io/) which helps it to scale at a fraction of the cost of traditional workflow systems that are based on expensive databases.
Postgres is not cheap to run in the cloud at scale. We went for the cheapest infra, which is basically the disk storage.
Having inherited a few of these - you tend to home-grow an ad-hoc version of many of the existing OSS tools, but with less of the patterns baked in.
Not sure where the NIH ends and where you're actually better off with a supported orchestration approach. I suppose if you expect your program to be around a while (or need advanced features), maybe think about using something a bit more battle tested?
We have a durable queue built into postgres to handle some complex notification-ish logic. It's worked excellently and while there are services various cloud providers would love to sell us to do that it's extremely cheap to run.
For that particular usage, the volume we process and business criticality make it a good choice for inventing here - but for other durable processes we just use off the shelf tools since the cost of maintenance would quickly outstrip the value.
Postgres is a great tool to use and far more powerful than most people give it credit for - but there's always the balance of in-house maintenance vs. paying rent for someone else's solution.
You pay for everything you build - the more complexity you put into it the more that costs over time. Dependencies need to be updated, language/framework upgrades usually break something, new features/requirements introduce additional complexity and code to manage. Software just costs money every day - not a lot, our industry is much lower margin than, say, stamping sheets of metal into tools - but it still has operational costs beyond just the money to operate the hardware we run our products on.
I know that. This looks like some lib you update once a year/every new CVE, and it is compared to a lib from cloud vendor and also update once a year/every new CVE, which is why I asked what it costed YOU in this particular case.
I feel it's way too hand wavy on consistency and correctness. My opinion as someone who've implemented marketing workflows that breaks all the time (and tons of painful lessons).
Strong correctness guarantee is something that should not be undermine. Even more important than availability.
The examples on the website is simple but heavily undermines the importance of correctness. Anyone who implement similar pseudo-code directly will eventually suffer from data correctness issue in crashes.
As you said, the example is simple and it might not be obvious to people without prod experience what the problems can be. Postgres can give you all the primitives you need to solve this at the application layer. Durable workflows on Postgres is an effective way to access these primitives.
The efforts we've undergone to make Oban (and Pro) work with CRDB have been ridiculous. Feature detection all over because of a lack of common operators and functions that can't be used in indexes. The worst is the rampant "serialization_failure" errors that force continual transaction retries. Not how I'd suggest scaling Postgres.
That said, as a predecessor to dbos in building durable workflows just using Postgres, I concur with the overall sentiment.
Can you expand on why you chose to use CRDB with Oban? I have no opinion here, I’m genuinely curious as someone using Oban myself (with Postgres). I haven’t hit the point of really needing to scale it out yet and I’d rather avoid the traps others have figured out.
Convex has a workpool component that gives the ability to compose big complicated flows in an understandable way, and give you realtime updates on status of various pieces: https://www.convex.dev/components/workflow
But that's not what we are saying; we're suggesting use Postgres until you truly need something else. 90% of applications aren't "web scale", keep the stack simple and portable.
Unless you have a very specific use case, you wouldn't want to store in db or in any message you use in any workflow like this. Usually whatever does the actual work has a way to get the secret.
I am not convinced that using a special software for "durable workflows" is necessary. If one has a stateful message queue or job task queue, e.g. RabbitMQ or Celery, one can use it. Irrespective, many jobs can be made idempotent. The most that you ought to residually need is a column in an existing table of your own database which keeps track of what remains to be done.
Given the above, it would seem that durable workflow software is pushed forward by those who have a surplus of VC money to spend. As for the vendors, there is no shortage of people trying to sell you things that you don't need.
I find it strange that some think in terms of AWS architecture as the default. You could replace nearly the entire AWS stack with an Elixir (Erlang) monolith + Postgres.
The main benefit is centralizing all the data in one place so we don't need to worry about copying data in between multiple systems. Once something becomes the bottleneck, you can eventually migrate to a purpose specific tool to scale out.To be honest, LISTEN/NOTIFY in my opinion is the most fragile part of PG but it's fine as start until you scale out.
I'm working at a scale where almost every day I have to ask people "are you use you need to treat that as relational data? It doesn't seem relational"
Is this intended to be "you sure you need..."?
I think if you grow enough to look for these extensions, it's usually better to bet on purpose-specific tooling. For example, I use DuckDB/Iceberg combination extensively for columnar data and connect DuckDB to PG when I need it.
https://lucumr.pocoo.org/2025/11/3/absurd-workflows/
https://github.com/earendil-works/absurd
https://earendil-works.github.io/absurd/
I've not used it, but it's worth comparing to other options
You might consider another name for it, that one is wholly ungoogle-able! Looks neat though
It also has VirtualObject which is a nice vendor-lock-in-free OSS alternative to CF's single threaded DurableObject.
Where DBOS absolutely shines is
1) Atomic messaging in the same db tx as your business logic via dbos.enqueue_workflow! This is often the most brittle part of any solution and doing it atomically and durably with same tx that ran your business logic drastically reduces lots of complexity.
2) Since DBOS stores workflow state in db it should be easy to build dashboard for observability from metabase/looker(I wish restate exposed its rocksdb instance so it could be hooked up to metabase).
I have used Temporal in the past, works really good, my only problem with it was some limits on request payload or event sizes, created some inconveniences to us when building solutions. It also enforces good engineering practices, but sometimes you don't want to write special logic if your CSV file is larger than 2Mb, upload it to S3, pass link, then download it in the workflow.
What is your experience with DBOS? How does it compare to Temporal in terms of operational complexity, feature parity and anything else
Then I tried their Cloud offering and was appalled at their pricing. I burned through the $1,000 free credits before I even got something to production. Didn't want to bother with running a local Temporal, either.
Best solution is to just take inspiration from their architecture and then do it yourself in Postgres, IMO.
Temporal is, in my opinion having run it in prod for over a year - poorly designed, slow and ridicliously heavy infra wise.
If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day, you're going to spend millions on infra, and it's still going to absolutely suck.
Try running their own benchmarks, the numbers are pathetic.
Their sales team is also absolutely appalling and desperate.
From a Developer standpoint, the SDK is quite nice though.
Don't get trapped into nexus, and if the sales team call you make sure legal is in the room.
Where are the “millions” on infra going? It’s a handful of services and a Postgres?
> Their sales team is also absolutely appalling and desperate.
You said “on-prem”. It’s open source; why are you dealing with their sales team?
> If you're doing anything non-trivial (say, 200+ events/workflow) and you need to run only a couple hundred of them concurrently all day…
If “millions” were required to obtain such tiny scale, I’d agree there’d be a massive problem. No one would use Temporal; it would be a complete waste of resource. If this were true.
https://github.com/temporalio/temporal/blob/e22e6304b3c4a409...
https://github.com/temporalio/temporal/blob/e22e6304b3c4a409...
Temporal does a crazy amount of database operations and all of these are behind that mutex.
Oh, and you can't change the shard count on existing clusters.
Great stuff.
https://github.com/agentspan-ai/agentspan which is essentially an agentic SDK layer for Conductor can convert any of your langgraph, openAI, vercel, or ADK agent and makes it durable and adds orchestration with no code changes.
That said, my gamer-brain wants to call this "Save-scumming at scale." Which is to say, a lot of people already know that this approach works, but maybe they haven't made the connection to abstract CS stuff.
Another strategy that can be used to build robustness is to build your workflow out of idempotent operations. That can be useful for situations where the workflow state is too large to back up. Instead, you just run the job from the top and it's a bunch of no-ops until you start making progress again.
I recently developed a distributed queue and it works really great - benchmarks great too, with no race conditions or conflicts. I used SKIP LOCKED so that workers can compete safely.
You can also have multiple workers across nodes avoid conflict by using session wide mutexes i.e. pg advisory lock.
Edit: Actually I checked this again and apparently the advice has now changed to the inverse.
Postgres is not cheap to run in the cloud at scale. We went for the cheapest infra, which is basically the disk storage.
Not sure where the NIH ends and where you're actually better off with a supported orchestration approach. I suppose if you expect your program to be around a while (or need advanced features), maybe think about using something a bit more battle tested?
For that particular usage, the volume we process and business criticality make it a good choice for inventing here - but for other durable processes we just use off the shelf tools since the cost of maintenance would quickly outstrip the value.
Postgres is a great tool to use and far more powerful than most people give it credit for - but there's always the balance of in-house maintenance vs. paying rent for someone else's solution.
Strong correctness guarantee is something that should not be undermine. Even more important than availability.
The examples on the website is simple but heavily undermines the importance of correctness. Anyone who implement similar pseudo-code directly will eventually suffer from data correctness issue in crashes.
That said, as a predecessor to dbos in building durable workflows just using Postgres, I concur with the overall sentiment.
Given the above, it would seem that durable workflow software is pushed forward by those who have a surplus of VC money to spend. As for the vendors, there is no shortage of people trying to sell you things that you don't need.
Typescript: https://www.pgflow.dev
Elixir: https://github.com/agoodway/pgflow/blob/main/docs/COMPARISON...