Twitter spaces are awful however and are also not easily recorded and can't be used easily on any device. (You need to go to some kind of youtube link from one of the participants to listen to a recording.) Also their twitter talks are more unfocused rants by the participants with little focus. So it's hard to listen to. I'm sure it's more fun for the participants, but not so much for the listeners. You also lose the comfort of consistency of participants.
Some of them have even been delving into some of the participants personal politics as well which is just something I'm not interested in hearing.
We've turned it into a podcast as well, so you should be able to enjoy it wherever you consume podcasts.[0]
While there may be some unfocused rants in there (sorry, I guess?), there's also a lot of extraordinary technical content -- certainly, if anyone else has described their board bringup experiences in as explicit technical detail as we have in [1] and [2], I would love to be pointed to it!
Thanks for the link for a podcast version. That's helpful. I did listen to those two talks earlier and those were probably the most interesting. I did dislike that there seemed to be a constant process of interrupting each other in the middle of someone telling a story so the story became quite fragmented as it took some time to come back to the topic (or the topic was forgotten entirely and never elaborated on after the interruption).
Another thing that ended up being frustrating is repeated references to some image and I had to pause and go digging through people's personal twitter accounts to try and find the image that was being talked about. There seemed to be an assumption that the listener follows all employees personal twitter accounts.
There's also audio quality issues in general as it's not nearly as good a quality of media equipment as was used for the original on the metal podcast. (Maybe it's similarly cheap equipment, but it was better quality before regardless.)
I love those little interruptions. Like when a guest is speaking on a topic and they bring up a name or company of the past. And that segues into another bit of SV lore.
The Oxide Twitter spaces are recorded and distributed as a podcast [1]. If you're not a fan of the content it won't help but it's at least more convenient to consume than through Twitter.
That's such a shame. I've tried listening to those recordings and it's hard to follow without the context of the "Twitter Space" which I'm not even sure what is because I don't use Twitter. The audio quality is also quite grating to listen to.
Would love recommendations for in depth, tech related podcasts that stand on their on, like On The Metal.
IMO the recordings stand just fine on their own. Even when I'm in the Twitter space live, I'm certainly not looking at tweets or anything else that's happening on the screen, unless I want to talk.
We're (obviously!) huge fans of On the Metal[0] too (and the episode you cite with Ken was indeed extraordinary[1]!) -- but we have come to like our Oxide and Friends[2] Twitter Space even more. The reasons why are manifold, and really merit their own long-form piece, but as a few examples of episodes that show why we find it so compelling, see "Tales from the Bringup Lab"[3], "Theranos, Silicon Valley, and the March Madness of Tech Fraud"[4], "Another LPC55 Vulnerability"[5], "The Sidecar Switch"[6], "The Pragmatism of Hubris"[7], "Debugging Methodologies"[8], or "The Books in the Box"[9].
There's tons more where that came from; if you are a fan of On the Metal, I don't think you'll be disappointed -- and it has the added advantage that you join a future conversation!
Yes, that's in the works: we have a website redesign coming up, and this was included in it. So stay tuned!
And the 5 minute announcement on the On the Metal feed is a great idea; thanks for the idea -- and for the kind words.
Finally, I hasten to add: we have a really exciting Space coming up on Monday[0], where we'll be joined by Jon Masters to talk about the importance of integrating hardware and software teams; join us!
If Oxide computers had existed when I was working on sorting out the infrastructure for in-country domestic payments at Visa, I'd have lobbied pretty hard to adopt them. Assembling basically the same thing from a bunch of random vendors and terribly integrated software was an infuriating, expensive process that only achieved great results because we invested the time and money into it that I'd have rather budget allocated to writing core payments & settlement applications.
Plenty of companies still run hardware. A lot of midsize enterprise and a lot of non-tech/non-software F500. And quite a few won’t move to the cloud - between retailers who don’t want to fund big bad Amazon, hyperactive security departments who (rationally or irrationally) see physical ownership as a necessity, and organizations who have needs which are poorly suited to cloud offerings, it’s not like the datacenter hardware market is completely dead by a long shot.
You can proxy this demand by looking at VMWare sales. Anyone running a large amount of baremetal VMWare virtualization has a large on premise deployment and could find managed rack products like Oxide interesting.
> who will spend the premium for a rack server that is easier to maintain?
Not my area, but based on other areas: "easier to maintain" can translate directly to "fewer engineer/admin hours consumed", which can translate directly to saved money (or resources that are now freed up to be put toward something more useful)
Banks, hedge funds, governments, … anyone who needs a lot of grunt and cannot offload to the cloud. I would say they target the same segment of buyers who go for Oracle’s “engineered” / ExaData / Exalogic machines.
For normal businesses cost is important, for smart governments, hedge funds etc. one may have leeway to spend a little more for a better product or better tail risk etc.
People who have to maintain servers in house for various reasons. Maybe they need to run some software that needs to run even if they lose Internet connectivity. Or maybe they're working on secret stuff and their CIO/CTO has decided that this needs to be on prem instead of on the cloud.
You know how you can buy pizza in a cardboard box?
What happens if you need to feed a party of ~80 people? You end up getting dozens of pizza boxes delivered.
Oxide Pizza delivers all that pizza in a single box.
you are correct, margins are very thin in hosting company business and the absolute most server performance/number of nodes for the least money is very important.
have been on the manufacturing side of this before.
bulk x86-64 servers such as you could buy from the vendors you meet at the computex taipei are a big part of it.
things along the same general idea as the supermicro that is 2U high and has four discrete nodes in it, but cheaper.
long/narrow motherboards with dual socket xeons
and other similar stuff...
if the price for this is not market competitive with that, or just a bunch of 1U commodity dual socket servers, the only people buying it will be enterprise/specialty end user and not hosting operations.
What is missing is a middle ground: an improvement over the IPMI broken-state-of-things that is aimed for automated deployments, without demanding a certain network environment for PXE booting and configuration of IP addresses for management.
I haven't used RedFish but it seems it does not address the problems of IPMI in this regard: To be fully automatable there shouldn't be an assumption about the network environment - rather I would like that to have an L2-based protocol (one has to use the MAC address as identifier anyway) or if IPv4/v6 is used then at least demand mandatory link-local auto-addressing by default.
However, RedFish seems to support emulating a CD-ROM drive for booting an installation media, this is a nice idea but what I really would like to see is directly writing an image to the hard disk. Then of coure the hardware manufacturers would need to write high-quality firmware and not the regular quirk- and bug-ridden stuff that we know from IPMI implementations.
Your last sentence reveals what you probably already know to be true: there isn't a middle ground. We tried for years to get hyperscaler-class software running on commodity-class servers -- and came to the conclusion that the hyperscalers came to (and as Alan Kay famously observed): to be really serious about elastic infrastructure, one needs to do one's own hardware. Now, ~2.5 years into Oxide, I can substantiate that with more detail that I could have ever imagined: there is no middle ground -- and indeed, this is why the infrastructure world has become so acutely bifurcated!
Do you think there's at least a place for a holistic system that's not a whole rack? Or do you think that those of us with smaller computing needs will just have to either rent from a hyperscaler or put up with PC servers?
Definitely, but it's not a market that Oxide intends to go after (at least not for the foreseeable future). Part of the challenge is that it really does require diverging from the reference designs (as these have effectively enshrined a proprietary BMC and everything that comes with it) -- and once you are away from reference designs, you are talking about a very different economic proposition. This is part of why we are so transparent and open about everything we're doing: our hope is that we can enable others to pick up the baton and serve those markets that we do not intend to address.
I have put our implementation of _exactly_ this in production last week.
Boot a "CD-ROM" via IPMI into a custom installer, flash image (generated from a Dockerfile!) to disk, reboot.
Install takes ~15 seconds, full process (going via POST once) takes ~4 minutes. This also allows us to have a "hardware validation" image (that one doesn't get persisted to disk).
Not sure if there's any plans to make it public right now, but I'll ask around. Feel free to contact me via email (in profile)
Yes, automation with IPMI can be done (for a particular hardware due to quirks), and I recommend doing it (and I have done it) with a separate DHCP environment for the BMC NICs to be at least agnostic about the OS NIC's network environment.
So, I share your excitement when it works but I'm not happy with the out-of-the box experience of IPMI. I wish there was something better but unfortunately this is just one case of a split/mismatch of what HW people provide and what software people need.
I wrote down the kernel of the idea quite a while ago here[0], however, the actual version we run nowadays is using UEFI and written in Go (installer) + Python (API that generates the ISO on demand)
Fly.io[1] also does this (although they boot the result in a VM, the concepts are the same)
I saw an IBM-funded research project that fit 1,024 ARM cores and 2-4 TB of memory into a 3U box. The box didn't include the liquid cooling system you needed to keep the thing running.
Looking at the hardware, I'd wag somewhere in the $2-3M range. Depends on how big of a profit margin they want to make or how scrappy they feel like being (could probably go as low as ~$1M if they can make it up in volume).
If they're following the typical 1/5x pricing model for support, that'd be roughly $500k/yr/rack. But it's also hard to do that while simultaneously describing Dell as "rapacious".
I suspect they are not going to be cheap at all lol, aside from being a very nice hardware package they also have an entire custom hypervisor platform which is intended to be one of the big plusses for the platform.
The business model is straightforward: we sell servers, a rack at a time. You put them in your data center and enjoy a cloud deployment model but a “I own the metal” ownership model.
according to the website for oxide theyre "Hardware, with the software baked in" so I'm worried this is going to be more like Cisco's hamfisted UCS platform or dell/HP lock in with drac and ilo and "hyperconverged" embedded Java fuckery that can never be patched.
I’ve been relatively satisfied with HP’s iLO offering, in that it’s been sufficient for my needs, avoids many pointless fly/drives to the colo (or dealing with the comic misnomer that is “smart hands”), and is still getting the occasional update on 8-year old servers.
Even for servers in my building, I find it’s more convenient than a crash cart for most things (and of course my desk is a better work environment).
Didn't VCE still use consumer rack computers? All they were was basically a hardware reseller with their own racks. They re-sold Cisco stuff. They just repackaged everyone else's stuff as a value added service.
The NSA tries to shut down Bryan Cranston so he goes underground, building illegally powerful servers in a commandeered Pixar storage room at night and sleeping under the stage of youth music venue 924 Gilman by day.
I wish the Oxide web site had a little more detail on the software running on this. Does
This use a custom hypervisor? If so what is it based on, and what are its capabilities?
Just to add some additional detail: the control plane (Omicron[0] -- named before the COVID variant!), the user-level portion of the hypervisor (Propolis[1]) and the storage subsystem (Crucible[2]) are all open source. And while our focus has been building them for our own product, we know that they are at least usable outside of Oxide![3][4]
This company was created because somebody that actually built a large could with Dell/Supermicro suffered years and years of pain.
If you do as you suggest you will have to do a whole lot more, at the minimum you need to set-up (and pay) VMWare.
Then you will need to figure out how you do automated firmware upgrades, low level monitoring and if security is a concern you likely want to configure secure booting with attestation and things like that.
Consider how much work it is to do what you suggest, vs buying this rack. And then consider how good (and repeatable) the end result is.
Windows Server yes -- as a hardware virtualized guest. HyperV and VMware, no: we provide the hypervisor and control plane (and that that's built into the cost of the rack is very much part of the value proposition!).
Seems like this company has gotten a lot of hype but what so they actually do? Cloud hosting? Or selling physical servers? Both of those things sound like a terrible business tho..
Even a cursory glance of their website or listen to the episode pretty clearly explains they're building server hardware for hyperscalers. I.e. they want to build and sell the hardware that your hosting provider would buy to host your applications. I would guess they see quite a market for on-premises computing, particularly with companies that through legal or competitive reasons cannot or do not want to lose control of their data by hosting it on a cloud service.
I think Outpost is a great hybrid-ish solution for those companies with workloads in AWS that want to supplement that with on-premise workloads using the same APIs and tools they are already used to.
The use cases I see for Oxide are, like other have said, are for hyperscalers or hosting providers who want to have their own on-premise infrastructure that isn’t tied to another entity like AWS. Whether that’s because you have the in-house talent to manage it or for compliance reasons it’s required it does have a niche.
Currently, Outpost doesn’t provide any discounts over hosting the same workloads in the cloud, so it seems only useful if you can’t host in the public cloud. For now, they could easily be undercut and a solution like oxide has advantages.
But if outpost was nearly the same price as oxide, wouldn’t you always want access to the full AWS or GCP infrastructure and world class control plane versus something bespoke that only works in your own data center.
I think what Oxide has to bank on is that this isn’t a market that Google, Microsoft or AWS want to really compete in (e.g. it may require these companies to undercut themselves too much).
People who design their data centers in at least rack-sized units and ideally data center sized units, and creates custom hardware fit for that purpose. Would be my personal definition.
AWS Outposts are tragically limiting in so many ways. If you are a small shop they are great but if you don’t fit their model exactly you are SOL.
Same problem with Anthos.
You can work around the limitations by having a million points of presence with them and splitting up your workloads but there’s real costs associated with that model. Running 1 rack in 100 DCs is a lot harder than 10 racks in 10 DCs.
Fwiw I work with an engineer that came from a shop that used them. Long story short: they’re not remotely cost effective, are quite limited, and existing AWS tool chains run into all sorts of hair-pulling snags.
My gut is that they are for places that need on-Prem for compliance purposes for certain parts of their stack
It's a private cloud rack aka hyperconverged infrastructure rack but they're allergic to standard terminology for some reason. I assume the market is cloud repatriation for startups.
Wikipedia says that hyperconverged infrastructure typically uses commercial off-the-shelf computers. So that's likely why they're allergic to it. As they're designing their own computers so the term isn't accurate.
From what I gather, it’s extremely beefy, physical servers with DPUs built-in. I.e. perfect for large on-premise deployments while maintaining the high performance that DPUs enable, all as a single appliance.
It’s the kind of thing I would expect Nvidia or Dell to become big at as well, and Bryan Cantrill has enough leverage to pique my interest significantly here.
Also known as: SmartNIC, and a bunch of other names, basically a NIC with an embedded CPU (usually ARM of some sort), so can offload some workload into the NIC.
An example is nvida's bluefield, there are others
That sort of thing is being used in a few areas, can even run ESXi on them.
