Years ago I was making the case that instead of digging ourselves into the Amazon eco-system with S3 storage, EC2 instances, DynamoDB and various other Amazon specific cloud products... we should just host virtual machines and have everything in there using open source products.
People looked at me like they saw water burning but that would have made the dependency on the US a lot easier to sever. Just move the VM's.
I've operated at companies using both models, and have observed similar reactions to suggestions of using the cloud.
To me it's like anything else in engineering, are the costs, risks, and benefits fully understood, and worth the tradeoff in the particular context.
I worked for a startup doing internet of things, the consumer would buy a device and get lifetime service baked in. And that company was a step further, just renting space in a colo was incredibly cost efficient, which supported the sales model and competitive landscape of that product. But it was also very costly to attention, one of the most valuable resources. But it can also get costly in non-intuitive ways, an example that comes to mind is we started to get interviews where a generation of candidates no longer had experience with metal, it was a foreign world to them.
With more experience, I find it's really the costs that get severely underestimated, both for and against the suggestion.
Especially in larger organizations, it's easy to lose track of all the distributed soft costs that DIY can bring (and all the bus factors that may be involved). There are lots of people that kinda want to get paid and get benefits and which require some level of management structure.
At some point, you have people (on here and elsewhere) questioning what all these people in an organization do. PART of the answer is that they're doing internal work that could have been outsourced in various ways.
I am running my startup out of a self build GPU server from our office with a backup to the cloud.
I only pay for the IP address as electricity is included in the rent.
If the startup fails, Ill have thousand other potential use case for it and in the worst case, it will make for a awesome gaming machine.
The machine is a beast and I can serve a lot of users with it. In fact, and quite funnily, I already serve much more users with it than a lot of my older clients do with their software running on expensive k8s setup because „scale“ :-)
And last, but not least, I had a lot of fun building it. Its just nice to hear that thing humming away in the corner.
> The machine is a beast and I can serve a lot of users with it. In fact, and quite funnily, I already serve much more users with it than a lot of my older clients do with their software running on expensive k8s setup because „scale“ :-)
Honestly even if you have a single server, running k8s (or maybe Docker Compose for really simple cases) on it is still the simplest way to manage it (assuming you have more than 1 service, anyway). One configuration file format, one CLI tool, zero special paths to memorize, no filesystem permissions to configure, pretty good security out of the box, access to a whole bunch of helm charts and operators (for example, cert-manager, external-dns, prometheus, alert-manager, some logging operator for centralized logging with a decent UI and search, and a postgres operator for backups / replication / failover), etc.
The whole business model is around “Optimization through custom tools”.
We can go with your idea, sure: a few months in, an Account Manager from the cloud provider shows up and says your bill could be reduced by 50% if you just adopt some changes, using their custom, super optimized tools (“minor changes” will be the mantra).
And now you have your own company looking back to you on how can they get those savings, people who don’t understand what a VM is and cannot differentiate salesforce from an elastic container, as everything is “cloud”, but heard “50% off”.
Preventing this from happening requires a clued-in CTO and equivalent senior level leadership who can defend against such 'attack' methods and knows the difference between, for instance, paying a monthly recurring cost to host a Linux/KVM virtual machine and paying for some totally 'cloud' SaAs.
Further, it needs people in decision making roles who understand and value the strategic differences between having an infrastructure concept that is trapped in one provider's proprietary software tooling ecosystem (aws, azure, etc), vs things built on open standards that are portable.
> Preventing this from happening requires a clued-in CTO and equivalent senior level leadership
Most CTOs (and increasingly M2s and M3s) I've met are what I call "box architects". You know the ones who love drawing boxes, moving one box inside another box, drawing a line between 2 boxes or changing a unidirectional arrow into a bidirectional one, then declaring the hard part is done and now we need any random engineer to implement that or "Is there an AWS service that does that? I just don't see the value in us doing it in house".
A "super optimized tools" is just a box that you swap for another box and the "minor changes" will be just a couple of arrows than need to change or another box to swap for another box. You get them to feel good about doing architect stuff plus the 10x reduction in the bill. They can always replace that box with another box later after all.
> Preventing this from happening requires a clued-in CTO and equivalent senior level leadership who can defend against such 'attack' methods and knows the difference between, for instance, paying a monthly recurring cost to host a Linux/KVM virtual machine and paying for some totally 'cloud' SaAs.
And the reality is eventually you'll get a clueless one, and everything will revert to the mean.
And the mean is heavily influenced by marketing propaganda.
This is true, but making money in any business is constantly fighting against the entropy and regression to the mean. Also, maybe, just maybe it's an example of relative competitive advantage and paying more for the AWS is the right call.
2018 - I see you are hosting your own PostgreSQL in EC2, you can use our managed solution
2020 - you are already using 18 our services (note, at this point you might still be using non-vendor products, like VMs, managed DB, and so on), why not use our IAM instead of rolling out your own auth.
2024 - you are now deeply locked, lets add more lock-in, why don't you use this tool to optimize your costs (welcome DynamoDB)
At this point, no one would ever question next tool from salesman. Because engineers see that company doesnt have strategy to move to another cloud, why should they reject this new tool?
also consider the people who are involved, a lot of times after 2 years you have totally new people in your team, they won't have context and constraints you had in the past when deciding to buy "just VM", they see it as "we already use AWS"
I had many conversations with a former boss about the Azure sales team. They would come in, say they can do it cheaper, simpler and better — he was immediately convinced.
I would do a calculation based on their public price plan and come up with a 5-10x price compared to the bare metal OVH solution that perfectly fit our use case. I would then ask the sales team where I made a mistake in my calculation and hear nothing back.
A few months later, they would come back with the same pitch and the whole process would repeat...
AWS has been (blatantly) using Microsoft method of making their way in. Redis, Elasticsearch, whatnot, all follow the same procedure: 1. Here is a managed service. 2. Here is a fork of the managed service where we manage the server (you don't see) with 15% off in price/credits. Easier backups with clicks etc. 3. We are dropping support of managed-X, move to our fork. 4. Due to the market conditions, our forked service is now 50% more expensive. 5. Ah also, you cannot export/download your backups because they are in proprietary format. 6. Locked-in.
You'd be wrong to laugh at them, because different cars of the same general size can indeed vary 50% or more in fuel efficiency. It's fair to be skeptical of promises of huge savings, and question why your counterparty would benefit from giving you those savings, but sometimes there's a good reason.
> Do people actually take claims like that from glorified salesmen seriously?
People who know the tech, no
Non-technical middle management types, yes. It produces revenue when done aggressively enough, google "solarwinds sales people" for many anecdotal examples of extreme persistence. Not that I agree with it.
I prefer not using managed services but I kind of understand the appeal. Instead of paying several engineers, that you have to vet first, to configure and maintain the services adjacent to your product you can just pay AWS or Azure or someone else to maintain the service. Then you can concentrate your whole manpower on your product. In case the service goes down you can blame someone else and maybe even recover some money. On the other hand it of course makes you dependent on the provider.
> Instead of paying several engineers, that you have to vet first, to configure and maintain the services adjacent to your product you can just pay AWS or Azure or someone else to maintain the service.
Your engineers who all have to possess AWS or similar certs before you hire them, work for free?
A move off VPS to managed services doesn't reduce your headcount or labour costs.
In my experience it doesn’t take long until you use such complex offerings from the cloud vendors, you need those ops engineers anyways. Just with slightly different skillsets.
There was a period when development and system adminstrators were really concerned about vendor lock in and would choose on the basis of the ease of moving to a different platform, Java and J2EE was clearly based on this mindset. I have always found it odd people have been willing to adopt AWS with no apparent easy route off given its price.
Projects like Ceph and Minio have existed for years, though?
Beyond that, I just don't understand your point of view at all. Do people unironically think there is some super special dark magic being done in the bowels of Amazon, as opposed to just...code that runs on (virtual and physical) machines? The open source community yielded Linux but it's just sooo impossible for it to yield an object storage service? What a strangely shackled view of the world.
> No it’s using an army of extremely well paid engineers, something I guarantee the parent comment has no access to
That's a different argument to the one I replied to, and the reply to "they have expensive infra people" is "you have to have expensive product-trained people to use them anyway".
The suggestion was to replace DBB and S3 with some VMs. Presumably those VMs would be managed by the engineers part of the parent commenter’s organization. They do not have access to as many engineers as AWS, nor do they pay them as well.
Not arguing about cost effectiveness here. Just pointing out how silly it is to suggest that you can replace DDB/S3 with some VMs ran by a midsize organization
That's genuinely my baseline, then I ask 'why do we want to manage this dependency?'
I can appreciate the desire to close gaps on expertise deficiency and make a vendor responsible, but the whole schtick of 'outsource everything and focus on your business for advantage' always rang to me as just an excuse to give our money to vendors.
Its almost as if the whole case for vertical integration is just taken as a wash
Most cloud VMs have network-attached storage working through a billing layer, and its IOPS numbers are pathetic. This makes running your own DB in a cloud VM much less reasonable. Now you can use local NVMe, but you still have to set up your own failover.
The original promise of the cloud is "you pay us less than you pay your sysadmins", which is not entirely unreasonable, especially at early stages.
Of course running on bare metal from Europe's own Hetzner is even more cost-efficient, if you already have a lot of sysadmin chops.
Ok so nothing has actually happened. These migrations are difficult and expensive, and often fail. It will be interesting to see an update in 5 years on how this went.
I've got experience with the LIDL cloud aka STACKIT and work for a STACKIT partner. Just drop me a message if you are interested. Two fun solutions implemented (fully automated via Terraform):
Site-to-Site VPN between STACKIT and Azure using a LibreSwan VM and an Azure VPN Gateway
FortiGate HA cluster in STACKIT - not a single ICMP packet got lost during failover
Yeah, kind of. Lidl and Kaufland is owned by the Schwarz Group. They have been busy replicating the AWS orgin story. Their cloud is called StackIT. I've worked with them. Still some room to grow but a solid foundation. I like that competition is back on
Schwarz seems to be obsessed with how Amazon (book seller) created AWS and they are trying to do the same... with 5 people. Also Aleph Alpha + Cohere is a Lidl work as the current CEO of the former led Lidl digital division.
Schwarz Gruppe includes Schwarz Digits, which include StackIT. 7500 is the number of employees at Digits, which also includes online marketplaces like Kaufland e-commerce, so definitely not all of them work on the sovereign cloud.
Makes sense. I never worked with this particular provider, but I must say that for many (many) use cases, Europe has very capable providers, and the big US players are not necessarily the best choices.
Isn't this how AWS also started? They built internal devops tools for them that were so good and expandable that they decided to give others access to them.
Yeah, though in this case the selling point is less about scale and more about data sovereignty. German companies are pretty touchy about storing data cross borders
The title is heavy clickbait. To say I just bought a Porsche when it was actually a Volkswagen is also wrong. Just because they belong to the same owner doesn't make it the same brand.
However it was more complicated than that. Porche owned 50+% of Volkswagen at the time of Volkswagen buying them. Porche got over extended and leveraged buying Volkswagen . The management family is closely connected since the start and at the time in early 2010s 20% government ownership rule was just getting stuck down by European courts .
And Germany is better? It's government is almost a copy of the 10 commandments of war propoganda. I don't know but it seems to be a dangerous place to put your stuff.
Ok so nothing has actually happened. It's also not specified whether this is in addition to their AWS footprint, or if it's a migration. It will be interesting to see an update in 5 years on how this goes.
The missing background piece is that the European Commission awarded a 180m EUR sovereign cloud contract to 4 European providers [0]. This framework agreement made the choice of national banks a lot more straight forward.
the parent company (Schwarz Group) has over half a million employees and makes something like 200 billion in revenue per year, I think calling it a discount grocer is underselling it a bit lol.
> DNB Director Steven Maijoor announced last October that he intended to “set a good example” and switch to a European cloud, though he acknowledged that it “is not yet as robust or high-quality as the one from the U.S.”
> Last year, the Dutch Central Bank (DNB) and the Netherlands Authority for the Financial Markets (AFM) warned that the Dutch financial sector had become too dependent on foreign IT service providers
I wonder how much if this is a personal choice, and how much is pressure from the government. Banks are famously the first target of politicians, and it's common in China for exec's to publicly choose a national option under pressure from the CPP.
If they cannot provide it nationally, Germany seems a good place to have it, especially as they are both EU.
At the very least a country dependent cloud services from multiple other countries is less dependent on any one of them than a country predominantly dependent on one (and most of Europe is currently dependent on US cloud providers).
People looked at me like they saw water burning but that would have made the dependency on the US a lot easier to sever. Just move the VM's.
To me it's like anything else in engineering, are the costs, risks, and benefits fully understood, and worth the tradeoff in the particular context.
I worked for a startup doing internet of things, the consumer would buy a device and get lifetime service baked in. And that company was a step further, just renting space in a colo was incredibly cost efficient, which supported the sales model and competitive landscape of that product. But it was also very costly to attention, one of the most valuable resources. But it can also get costly in non-intuitive ways, an example that comes to mind is we started to get interviews where a generation of candidates no longer had experience with metal, it was a foreign world to them.
With more experience, I find it's really the costs that get severely underestimated, both for and against the suggestion.
At some point, you have people (on here and elsewhere) questioning what all these people in an organization do. PART of the answer is that they're doing internal work that could have been outsourced in various ways.
The machine is a beast and I can serve a lot of users with it. In fact, and quite funnily, I already serve much more users with it than a lot of my older clients do with their software running on expensive k8s setup because „scale“ :-)
And last, but not least, I had a lot of fun building it. Its just nice to hear that thing humming away in the corner.
Honestly even if you have a single server, running k8s (or maybe Docker Compose for really simple cases) on it is still the simplest way to manage it (assuming you have more than 1 service, anyway). One configuration file format, one CLI tool, zero special paths to memorize, no filesystem permissions to configure, pretty good security out of the box, access to a whole bunch of helm charts and operators (for example, cert-manager, external-dns, prometheus, alert-manager, some logging operator for centralized logging with a decent UI and search, and a postgres operator for backups / replication / failover), etc.
We can go with your idea, sure: a few months in, an Account Manager from the cloud provider shows up and says your bill could be reduced by 50% if you just adopt some changes, using their custom, super optimized tools (“minor changes” will be the mantra).
And now you have your own company looking back to you on how can they get those savings, people who don’t understand what a VM is and cannot differentiate salesforce from an elastic container, as everything is “cloud”, but heard “50% off”.
Further, it needs people in decision making roles who understand and value the strategic differences between having an infrastructure concept that is trapped in one provider's proprietary software tooling ecosystem (aws, azure, etc), vs things built on open standards that are portable.
Most CTOs (and increasingly M2s and M3s) I've met are what I call "box architects". You know the ones who love drawing boxes, moving one box inside another box, drawing a line between 2 boxes or changing a unidirectional arrow into a bidirectional one, then declaring the hard part is done and now we need any random engineer to implement that or "Is there an AWS service that does that? I just don't see the value in us doing it in house".
A "super optimized tools" is just a box that you swap for another box and the "minor changes" will be just a couple of arrows than need to change or another box to swap for another box. You get them to feel good about doing architect stuff plus the 10x reduction in the bill. They can always replace that box with another box later after all.
And the reality is eventually you'll get a clueless one, and everything will revert to the mean.
And the mean is heavily influenced by marketing propaganda.
If a car salesman told me I could save 50% of my fuel bill from driving their special car a certain way I'd laugh at them.
2016 - lets use EC2, its just VM, we can move off
2018 - I see you are hosting your own PostgreSQL in EC2, you can use our managed solution
2020 - you are already using 18 our services (note, at this point you might still be using non-vendor products, like VMs, managed DB, and so on), why not use our IAM instead of rolling out your own auth.
2024 - you are now deeply locked, lets add more lock-in, why don't you use this tool to optimize your costs (welcome DynamoDB)
At this point, no one would ever question next tool from salesman. Because engineers see that company doesnt have strategy to move to another cloud, why should they reject this new tool?
also consider the people who are involved, a lot of times after 2 years you have totally new people in your team, they won't have context and constraints you had in the past when deciding to buy "just VM", they see it as "we already use AWS"
I would do a calculation based on their public price plan and come up with a 5-10x price compared to the bare metal OVH solution that perfectly fit our use case. I would then ask the sales team where I made a mistake in my calculation and hear nothing back.
A few months later, they would come back with the same pitch and the whole process would repeat...
People who know the tech, no
Non-technical middle management types, yes. It produces revenue when done aggressively enough, google "solarwinds sales people" for many anecdotal examples of extreme persistence. Not that I agree with it.
Your engineers who all have to possess AWS or similar certs before you hire them, work for free?
A move off VPS to managed services doesn't reduce your headcount or labour costs.
It's the new "Nobody ever got fired for buying IBM".
I don’t blame people for being skeptical
Beyond that, I just don't understand your point of view at all. Do people unironically think there is some super special dark magic being done in the bowels of Amazon, as opposed to just...code that runs on (virtual and physical) machines? The open source community yielded Linux but it's just sooo impossible for it to yield an object storage service? What a strangely shackled view of the world.
Well... yes?
What do you think the AWS S3 and DDB is running on? Fairy dust?
That's a different argument to the one I replied to, and the reply to "they have expensive infra people" is "you have to have expensive product-trained people to use them anyway".
The suggestion was to replace DBB and S3 with some VMs. Presumably those VMs would be managed by the engineers part of the parent commenter’s organization. They do not have access to as many engineers as AWS, nor do they pay them as well.
Not arguing about cost effectiveness here. Just pointing out how silly it is to suggest that you can replace DDB/S3 with some VMs ran by a midsize organization
I can appreciate the desire to close gaps on expertise deficiency and make a vendor responsible, but the whole schtick of 'outsource everything and focus on your business for advantage' always rang to me as just an excuse to give our money to vendors.
Its almost as if the whole case for vertical integration is just taken as a wash
Another advantage of AWS is permission management, automatic RDS snapshots, cloudwatch comes out of the box...
You can do everything with VMs, but in practise it's probably much harder.
The original promise of the cloud is "you pay us less than you pay your sysadmins", which is not entirely unreasonable, especially at early stages.
Of course running on bare metal from Europe's own Hetzner is even more cost-efficient, if you already have a lot of sysadmin chops.
Unlike most VPSes
Ok so nothing has actually happened. These migrations are difficult and expensive, and often fail. It will be interesting to see an update in 5 years on how this went.
Site-to-Site VPN between STACKIT and Azure using a LibreSwan VM and an Azure VPN Gateway
FortiGate HA cluster in STACKIT - not a single ICMP packet got lost during failover
https://stackit.com/en
https://news.ycombinator.com/item?id=17541092
Yes, when you pay you have to print a receipt with QR code, and then have to scan it to log out.
https://accounts.stackit.cloud/ui/login/user
VW bought Porsche
However it was more complicated than that. Porche owned 50+% of Volkswagen at the time of Volkswagen buying them. Porche got over extended and leveraged buying Volkswagen . The management family is closely connected since the start and at the time in early 2010s 20% government ownership rule was just getting stuck down by European courts .
German government is certainly slow and overly limited by bureaucracy, but dangerous?
Who are you comparing to?
The last war the US started is still ongoing and was started by them a few weeks ago.
It’s very much not a discount cloud provider. They are costly unlike their physical discount grocery stores.
Ok so nothing has actually happened. It's also not specified whether this is in addition to their AWS footprint, or if it's a migration. It will be interesting to see an update in 5 years on how this goes.
[0] https://commission.europa.eu/news-and-media/news/commission-...
> Last year, the Dutch Central Bank (DNB) and the Netherlands Authority for the Financial Markets (AFM) warned that the Dutch financial sector had become too dependent on foreign IT service providers
I wonder how much if this is a personal choice, and how much is pressure from the government. Banks are famously the first target of politicians, and it's common in China for exec's to publicly choose a national option under pressure from the CPP.
At the very least a country dependent cloud services from multiple other countries is less dependent on any one of them than a country predominantly dependent on one (and most of Europe is currently dependent on US cloud providers).