Rendered at 20:08:28 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
colonwqbang 10 hours ago [-]
When you think about it, 24 years is not such a long time for a machine to keep doing what it's supposed to be doing. It says something about our profession that we find a 24-year service life surprising.
There are airliners that have been in service for over 50 years.
VectorLock 6 hours ago [-]
>There are airliners that have been in service for over 50 years.
Are there any airliners that have flown for 24 years non-stop?
serf 6 hours ago [-]
this is why I find comparing mostly-solid-state stuff with big mechanical contraptions a frustrating exercise.
it's more impressive to me that the electric fans within the Stratus server still operate than it is that the chips still work, and if we're going with the plane metaphor it's the silicon that puts the thing into the air, not the simple fans.
So really it just boils down to "well, the planes work is harder." as to why it fails, which is of course abstract and unsatisfying; comparing the comparative lifetime workloads of a chip that has shifted trillions of bits versus some jet engines that have moved millions of kilograms around the world.
and then another layer comes and makes it even weirder : the vast majority of hardware around the world that has been EoLd and replaced has been in that situation due to software, not the hardware itself; a paradigm that really doesn't exist in the physical engineering world in the same was as it does CS.
semi-extrinsic 2 hours ago [-]
Mechanical integrity in industrial processes is often divided into two categories: static equipment (tanks, pipes, heat exchangers...) and rotating equipment.
These two have very distinct failure modes. The failure mechanisms for solid state electronics are much more similar to those for static equipment (corrosion, thermal fatigue, creep, ...)
dlisboa 6 hours ago [-]
Yeah, it’s not a good comparison. Give me endless parts and a maintenance window of 6 months every so often and I can make any server last a century.
russianGuy83829 6 hours ago [-]
That would not be a useful airliner.
literalAardvark 1 hours ago [-]
Mr Bones wild ride
But really didn't the NSA have something that was roughly an airliner in perpetual flight?
giancarlostoro 3 hours ago [-]
Was going to say, there's servers that last several decades, they receive routine maintenance, most people buy a computer, and rarely replace parts, and if enough years pass, you opt to buy a new one instead.
jrootabega 2 hours ago [-]
I'm not sure, but I think there are a few submarines that have remained underwater for that long.
literalAardvark 1 hours ago [-]
Absolutely not. They come up for shift changes at the very least, and more.
loloquwowndueo 2 hours ago [-]
Name one.
jrootabega 1 hours ago [-]
I think most of the time the navy does that.
tyrabound 8 hours ago [-]
You raised in my mind that we are looking at this all wrong, the server is not actually one component, it’s really a system of systems, and just like an airliner is a system of systems too, the conflict arises in that the comparison is at the wrong level.
It seems when people think of a server today, they think of a single computer, if not some software server somewhere in the cloud. These subject kinds of servers are far closer to complex systems of separate components, more like a network system, it’s why components of what are really separate networked computers can and need to be replaced.
An airliner is of course also made up of components that are also systems, however at least in my mind, the difference is the actual expected use case. An airliner is not a good comparison because it is never expected to remain in continuous operation, e.g., that at least one engine is always running even when it is being overhauled or repaired, to satisfy a requirement of continuous operation.
pitched 7 hours ago [-]
FTA, this is a fault-tolerant server where all hardware is redundant and can be hot swapped while running. The servers you’re thinking about are redundant at the software level so rebooting one server won’t cause the service to go down.
What is remarkable is that apparently that VOS thing has never crashed. I wonder if they also do software redundancy under the hood to keep uptime going during reboots. If the CPU is hotswappable, it must have something.
wildzzz 6 hours ago [-]
Probably uses formally verified code along with plenty of housekeeping processes to keep any failures from shitting the whole bed. In critical system design, you build in redundancies that work in parallel such that any one failure will not interrupt the system.
The main computer system in the Space Shuttle is an excellent example of this. It had 5 identical IBM System/4 Pi machines. Three of them ran identical code and handled the same work. The fourth ran a completely different codebase to handle the same work, preventing a bug in the main code from killing the whole system. A fifth computer handled other tasks but could be swapped over to the critical role if needed. You could lose 2/5 computers and still have insurance against a cosmic ray flipping a bit.
theamk 4 hours ago [-]
It's really not that hard to keep computers non-crashing, as long as you have good hardware and run a limited subset of software.
Many servers I've owned had multiple years of uptime, and the only reason they'd go down is because they will get decommissioned or because of power outage.
The article says:
> disk drives, power supplies and some other components have been replaced but Hogan estimates that close to 80% of the system is original.
so I am guessing there was no reboots, nor CPU replacements.
serf 6 hours ago [-]
VOS is a parallel lockstep OS. You drop nodes and replace them to keep the whole operational.
afavour 9 hours ago [-]
I think it just speaks to how quickly the tech has developed. Airliners work more or less the same as they did 50 years ago, computers are vastly different (and vastly more powerful).
Leonard_of_Q 8 hours ago [-]
Airliners may look the same but they don't work as they did 50 years ago when the was no fly by wire in these planes, no high-bypass engines, etc.
zrail 45 minutes ago [-]
Eh, sorta. Fifty years ago was 1976 (I know, right?!). The 747 debuted in 1970 with high bypass engines. The A320 came out in 1988 with fly-by-wire and all Airbus aircraft since have been the same.
The biggest change from then to now is probably the amount of solid state electronics onboard. That's ridden roughly the same curve as other industrial and commercial applications.
yCombLinks 6 hours ago [-]
Computers are many thousands of times faster and more efficient than they were 50 years ago. Airplanes still fly at the same speeds, and with only marginal safety improvements. Fuel efficiency has doubled however. Overall significantly less changes.
Leonard_of_Q 6 hours ago [-]
Faster but for the rest basically the same. More megahertz, more megabytes, more transistors per cm2 of silicon, more processors but that's basically it. Von Neumann architecture, processing separated from memory, block/file-based storage, input and output through keyboards and displays.
Yes, this drastically understates the change in computing but the actual technology hasn't changed that much, it just got denser and faster.
asah 9 hours ago [-]
yabut... airliners require constant maintenance and periodic rebuilds.
andrew_lettuce 8 hours ago [-]
Not to mention service disruptions and unplanned downtime.
bitwize 7 hours ago [-]
One of the major differences between like, your PC, and a high-availability server is that high-availability servers require those things as well—and get them, piecewise, while still running. It's like doing a complete overhaul while the plane is flying.
IBM mainframes, famously, phone home if they detect a faulty component. Service personnel will be on site, same day, to replace it before you even knew it was there, let alone had the opportunity to ask.
organsnyder 4 hours ago [-]
I wonder how many datacenters bother with those sorts of things anymore, given how many workloads are no longer tied to individual nodes. Seems like it would just be easier to accept that a certain percentage of nodes will be down at any one time, and go through and repair/replace them as needed.
7 hours ago [-]
lp92 8 hours ago [-]
Those aircraft have maintainance windows/refurbishment/refits and aren't flying 24/7 though.
debo_ 7 hours ago [-]
This server also had planned downtime maintenance. The article specifically mentions "no unplanned downtime."
vanviegen 9 hours ago [-]
It's not so much surprising that the machine hasn't broken, but that someone apparently considered it worthwhile to keep it around, having about the compute of a throwaway vape.
verzali 4 hours ago [-]
Isn't mostly capacitors that limit the lifespan? With a proper maintenance plan (as airliners follow) you could probably keep servers running for a long time.
Transformanshen 8 hours ago [-]
That may well be the case, but such a long period still seems extraordinary
wazoox 8 hours ago [-]
There are industrial machines such as steam hammers that have been in constant use for more than a century.
Aardwolf 3 hours ago [-]
This is the most dwarven thing I've read today
PunchyHamster 2 hours ago [-]
Those are getting regular maintenance and overhauls while not flying, it's not really comparable.
On other side only moving element would be hard drives and only wear element really would be the electrolytic caps.
streetfighter64 9 hours ago [-]
Airliners are up against physical limitations that don't incentivize upgrading. If there was an airliner that could fit millions of people and cost less than the 1980s version, I don't think you'd see many 80s aircraft around anymore, except in niece applications such as this computer. Perhaps now that we're at the end of Moore's law, you might see a larger amount of long-lived computers. But on the other hand, we'll probably find new ways to innovate in computing power.
StilesCrisis 6 hours ago [-]
Computers are still evolving rapidly. Instead of raw GHz increases, it will be things like tensor cores or other extremely-wide/GPU-type processing units. And of course, more RAM and more caches to feed them.
glitchc 6 hours ago [-]
> There are airliners that have been in service for over 50 years.
...with a very aggressive maintenance cycle (typical part life is 1000-10000 hrs of service). Most airliners of that vintage resemble the "Ship of Theseus" in real life.
9 hours ago [-]
mech422 10 hours ago [-]
I started working on Stratus machines right out of high school... They took me from backwoods New England to Wall St.
The were Awesome machines - don't let those filthy hobbitsis...err..tandem users cloud your judgement. Stratus was pure hardware FT, and tandem... wasn't :-P
Vos was pretty nice too - decent macro language - not all the utils you expect with a unix box, but in terms of the core OS and shell it was pretty good machine .. 40 years later I still love those machines and the career they helped get me into :-D
mkovach 8 hours ago [-]
I spent six months programming on VOS after a gig on a VAX. Yes, I'll be outside telling the kids to get off my lawn as soon as I find my reading glasses and have a nap.
VOS was intuitive and did what I needed. Nothing flashy. Just useful. I miss that sometimes. Give me a tool that works; if it also happens to be great, lovely.
I didn't work much with the Stratus hardware, but reading about it was fascinating. I wish I remembered more of it.
keiferwiseman 8 hours ago [-]
Oh Tandem, you don’t hear that much anymore. They still have plenty of them still running banks and what not under the name HP non-stop.
mech422 3 hours ago [-]
yeah - the big difference was Tandem was software (+ hardware) fault tolerant while Stratus was full hardware level fault tolerant. This gave Tandem a real price edge.
Both were really popular with banks and brokerage houses (back when I worked with stratus at brokerages, HFT was just emerging and down time was umm... frowned upon :-D
arscan 7 hours ago [-]
Hey, I worked there in high school as an intern! But just on the intranet, which for its time (‘96?) was really quite good.
nineteen999 12 hours ago [-]
We use to run Stratus VOS on Continuum series hardware at a lottery/poker machine company I worked at in the mid 2000's. It handled the jackpot system which decided which of 13,500 poker machines in our state would jackpot at any given time.
A reliable beast - never failed once in the entire time I worked there. Based on PA-RISC if I recall correctly.
jll29 12 hours ago [-]
Great story, thanks. I wish it said more about the architecture and also VOS.
Speaking of reliable beasts, I own(ed) a HP PA-RISC workstation that never crashed in six years, and it was even purchased refurbished (second hand, and educational rebate, and still the cost of a car in 1996 - but worth every penny).
I'd add "R.I.P. HP 9000 715/75" but of course she is STILL working now, twenty years later, and she is sitting right next to me (also got a second 715/100XC for spare parts in case on day they are needed).
sdcfgy 10 hours ago [-]
The old workstations were always pretty damn reliable. I’ve got a Sun Ultra 30 in storage. I may grab it and boot it up and see if it still works. Suspect the NVRAM and power supply caps are duff though.
mech422 10 hours ago [-]
IIRC, Stratus was M68K (like 12 per cpu board organized into 3 logical cpu's for fault checking)
kjs3 7 hours ago [-]
The OG Stratus FT was mc68k; what a beast. They moved to Intel i860 in the early 90s (XA series...I had a processor board from one hanging in my conference room back in the day), then HP-PA (Continuum series) in the 2000s and these days it's apparently virtualized on x86. Still very popular in the credit card processing world.
mech422 3 hours ago [-]
Ohh...I guess I had moved off them before cpu switches. I heard they ported Unix to Stratus as well - Did you ever get a chance to play with it ? was it any good ?
I think VOS being PL/1 based pretty cool - Stratus (and Intel fab PL/M stuff) was the only time I've ever gotten to work with PL/1...
Nursie 7 hours ago [-]
That's where I came across them in around 03/04, though the company was moving away from them at the time. Two intel processors somehow kept in lockstep, allowing either one (or its RAM) to fail and be replaced with no interruption to the system.
mech422 3 hours ago [-]
the originals ran like 3 'cpu groups' or whatever they called it...
all instructions ran on all 3 and the results were compared - one of the cool things done with this was taking faulty 'cpu groups' out of service without down time
zorked 9 hours ago [-]
The poker machines are connected? Why wouldn't they just jackpot locally and indepdendently?
rafaelvasco 9 hours ago [-]
Can't have several machines "jackpotting" at the same time or similar time. The prizing algorithm is very strict.
andyjohnson0 8 hours ago [-]
Is this widely known? Presumably the jackpots are sufficiently geographically distributed so that a player who observes one wouldn't be incentivised to pause playing?
wildzzz 6 hours ago [-]
These are probably the kind of machines you sometimes see in gas stations and bars, there's only a few of them in any location at most so there's never too many players nearby to be dissuaded from playing. Plus, a jackpot isn't the only way to win money.
freeopinion 7 hours ago [-]
I don't think it is widely known how much casino gaming machines are allowed to control outcome. Indeed, in some cases they are required by law to control outcome.
toast0 6 hours ago [-]
Networked jackpots are a user visible and aparently user desirable feature.
When the jackpot is fed by all the machine in the state, they can grow much faster.
voidUpdate 12 hours ago [-]
> "This system runs an older version Stratus proprietary VOS operating system, which Hogan believes hasn’t been updated since the early 2000s. “It’s been extremely stable,’ he said."
That's not particularly surprising to be honest. If it's a server operating system, on hardware that doesn't change, there's not really a reason to update the OS unless there's a security problem found
mrweasel 12 hours ago [-]
It's not really a security strategy, but running an obscure proprietary operating system on a system which probably isn't connected to the internet means that you're only susceptible to highly targeted attacks by a dedicated foe.
Apparently the latest VOS release was in 2025, which means that somewhere there's a team of developers maintaining it. That's really the thing I find the most interesting, that there exists teams developing operating systems like VOS, strange DOS clones or UnixWare (perhaps less so these days). Must be a strange anonymous life.
throwaway173738 7 hours ago [-]
> Must be a strange anonymous life.
This perfectly characterizes most embedded systems work. Competent software development using bespoke or off the shelf components that are not mainstream. Nobody ever realizes they’re using your computer software if you’ve done your job right. One day someone throws your appliance away because it’s obsolete.
voidUpdate 11 hours ago [-]
The article is very vague about what it actually does, so I don't know if it's connected to the internet or not :/
rbanffy 9 hours ago [-]
Most likely not directly connected. I'm assuming this one is doing process control and any internet access it might have must be mediated by other systems.
K0balt 11 hours ago [-]
Story time: I got handed a ticket to go change a UPS in a “server room”.
I was in Fairbanks visiting family, I had some downtime, so I decided to do some “slumming” to break up the monotony and relive some of the good old days before I was more boardroom than server room.
Besides, a cool grand for what I figured was 2 hours work appealed to me on some “miss the struggles” level, and it was better than walking around the tourist traps or doing video calls at some depressingly fecund coffee shop.
So, I drive up in my borrowed SUV with peeling paint, grab my trusty tool bag I leave with family for adventures like this, and walk boldly through the doors of this apparently normal retail establishment, clipboard in hand. It was 2019, and the smell of denim and factory pressed shirts wafted through my sensorium as I walked through the awkwardly desynchronized automatic doors.
It was a perfectly normal, somewhat 90s nostalgic retail store.., but walking up the stairs (the elevator was out of service) to the second floor, I could just sense that something was a little off.
The office had a smell. I can’t say what it was, it was businesslike, and not unpleasant, but it spoke to me in an unknown language and I was at that moment curiously heartened that my contract was being paid by a 3rd party. It smelled of stagnant ideas, missed opportunities, and a fossilized board of directors.
I’m not just being colorful here in prose. All of these thoughts were projected into my forebrain by some lizard sense I had developed in 40 years of skulking the halls of enterprises large and small. I was involuntarily imagining a half used tin of bengay on the boardroom table, right by the shiny beige speakerphone with the long stroke buttons and the filthy woven grill.
Curious by now about the flood of intuitions, I treaded the slightly dingy burnt orange carpet tiles up to the POC office, checked in, and was shown the door tot the server room. Through there she said, gesturing at a plain flat golden pine door with the elbow thingy at the top. “Through there, straight through, the door at the back. You can’t miss it. I think it’s unlocked.”
Unconcerned with the casual lack of escort (pretty common really), I opened the door and was immediately transported to surreal alternative world. It was dark, there were tens of flickering screens across to the right and one half hanging, flashing fluorescent fixture dangling precariously from the uni-strut above. The suspended ceiling tiles were about 50 percent in attendance.
Fragments littered the floor, and the door closed behind me with a loud thump. for just a moment I was touched by a singular thought….. (run). I shook it off, by now, far too curious to turn back, come what may.
Briefly thinking of my heavy tool bag as a potential weapon, then realizing the ridiculousness of the notion, I let my eyes adjust as I took in the scene with what I confess could only be described as some kind of unhealthy delight.
This was a large room, easily 800 square feet, and a was clearly dedicated to loss prevention surveillance. There was an epic workstation with 2 giant flat screen plasma monitors front and center, and ten smaller monitors on either side. Most of them were dark, and most of the others were flickering with static.a singular monitor had a poor but recognizable image of a small nondescript room with a small desk and two metal chairs, one on each side of the desk in an overtly adversarial position. The image smelled of stale sweat and musty cardboard.
There was seating for two pilots at the helm of this derelict craft, replete with an impressive array of buttons, strange industrial joysticks, and an oddly out of place black dell workstation keyboard with the cord ripped in half. As I made my way up to admire the helm, I nearly tripped on an extension cord, only catching myself on a water filled trashcan with plastic sheeting aspiring to the heavens up through the dark abyss of the half-intact ceiling, which I only now noticed was comprised entirely (if sparsely) of black painted tiles.
Somehow this jolted me out of my entranced state, and I looked for the door I was supposed to find, quickly locating it towards the back of the room, just as I had been told.
Somehow it seemed like not an ordinary door now. It was with great anticipation and not a little trepidation that I examined the door. It was quite ordinary, really, and the knob was less dusty than it might have been. This was encouraging.
I tried the door, and it opened easily. Opened, that is, to a time 30 years in the past.
The buzzing fluorescent lamps were on and radiating furiously, instantly shattering the wreckage of the surveillance spaceship to smithereens. There was a gigantic, wheeled, beige machine in the middle of the room, about the size of an industrial floor scrubber. It hummed patiently, and featured a dim 10 inch green CRT, a beautifully sculpted beige keyboard, and a little less than a quarter inch of dust. I froze like a tourist gaping at a stegosaur’s nose, but quickly recovered my composure as I remembered I was here on purpose.
Conveniently, someone had swept a path through the center of the room, back to the newly installed floor rack with a couple of ciscos, a backup cellular telemetry box, and a pile of screws and a throwaway screwdriver scattered haphazardly on top of one of the routers.
The UPS I was to replace was actually pretty complex to configure, but I very carefully completed it in about 45 minutes including some firmware regressions to accommodate some nonstandard plugin the vendor installed. By very carefully I mean without stepping too hard on the floor, dropping any tools, and afraid to sneeze even though I was under intense respiratory assault from the unavoidable clouds of fine carpet dander that had accumulated on the linoleum tile over years of janitorial malfeasance.
I was terrified to make any sudden moves, because sitting on a stackable plastic chair next to the pliestocene PBX was a beige box. Its fans groaned with the wails of a thousand parched capacitors, and its hard drive made a crunchy rattling sound that seemed entirely improbable for a functioning device. Just visible under the dust on the front was a badge proudly displaying “pentium pro” and a giant cable was fed from the back of the artifact, like an implant from the borg, something I strangely remember as “IBM type 1” token ring wire, though I can’t be sure anymore that means anything at all.
At any rate, it was obvious that the entire on-prem server infrastructure was this ancient and faltering nightmare box, and I knew that a fly landing on it could bring it down. The easy money contract was starting to look like a setup.
With terror in my heart, I finished my work, carefully closed the door, and put as much distance between me and that roiling cauldron of liability as I could. I don’t even remember going back through the wrecked starship, just checking out, and thanking Voltaire that I made it out of there in time.
It now made abundant sense that the job had been left on the board for nearly a month and the contract had kept rising… god only knows how many sensible technicians had noped out of there before I was hypnotized into compliance by the sheer bizarre fascination of the thing.
Poking around afterward, I found that the box apparently was the server that ran the terminals in the tire service area. They also had that fat cable that looked more like waveguide than wire, and it must have been an absolute nightmare to install. It was definitely token ring, humming along in 2019.
Sears closed for good a few months later.
20after4 9 hours ago [-]
Great story! This reminded me of the urban explorer videos taken inside of the last Sears headquarters building. It was (is?) a pretty amazing place and it's somewhat tragic that it was abandoned and left to rot.
ErroneousBosh 10 hours ago [-]
> The office had a smell. I can’t say what it was, it was businesslike, and not unpleasant
Timber that was very expensive when it was fitted, and carpet tiles that have repeatedly got damp and not dried out properly. Very old cigarette smoke from when people were allowed to smoke in the office that is now embedded in the blinds and suspended ceiling tiles.
I was recently in a large suite of government offices, fitted out in the 1970s and largely unchanged since, and that was the exact smell. We're imagining the same smell right now.
jeffrallen 10 hours ago [-]
If E.A. Poe was a sysadmin.
Hugsbox 6 hours ago [-]
It's not often I feel disappointed that I've reached the end of a comment, wishing I had more of it to read.
frumplestlatz 8 hours ago [-]
The real marvel is that you tortured both the English language and your readers that much just to describe a UPS swap.
ajb 10 hours ago [-]
Great story! Was expecting it to be Bernie Madoff's, not Sears lol
qingcharles 5 hours ago [-]
This is peak HN. Bravo!
nathell 9 hours ago [-]
Reminds me of a C64C that, as of 2016, was used in production in a car workshop in Poland:
I know a guy who is very proud of the fact that he has a Flask based web app, fully exposed on the public web, running with >99.9% uptime on a cheap hosted VM with zero updates to either the OS or application code since 2016.
asa400 4 hours ago [-]
Definitely cool about the uptime, but as I have no experience on systems like this, I'm also curious about the economics of this as a business strategy.
How do the maintenance, development, and operations costs of a system like this compare to - for sake of argument - a typical line-of-business Java system running on commodity servers you own in a DC somewhere?
Can anyone with experience in both comment?
mech422 3 hours ago [-]
Stratus 'servers' were originally 'super minis' - 19" racks full of gear with multi-million dollar price tags - looking at the picture in the article, I was surprised to see what look liked racks of desktops ?
kev009 2 hours ago [-]
You can find random PDP, VAX, even X86 systems that have lifespans 20+ years. As for a single boot probably a much smaller corpus, maybe some forgotten Sun SPARC in a telco on -48V DC power bus.
A VAXCluster or Sysplex(S/390) would I guess be more analogous but those can be a bit Ship of Theseus since you can change the hardware out over time.
mveety 2 hours ago [-]
For VMSClusters 8 to 10 years seems to be a pretty common uptime for long lived clusters. I worked a gig a few years ago that had a VMSCluster that was up for 19 years at that point. Started on alpha, migrated to itanium, then I helped migrate it to x86. It was the oldest I've personally seen, though I have heard legends of much older.
My current contract has a PDP-11/70 running RSX-11 that's been up since 1992. It's uptime is longer than mine! Poor thing is getting retired next year though.
Neil44 12 hours ago [-]
Well Great Lakes Works is still operating, and you can still buy Stratus ftServers, how nice is that after all this time?
jll29 12 hours ago [-]
"Stratus ztC Endurance®
Stratus ztC Endurance extends fault tolerance to modern enterprise and operational environments with intelligent, predictive technology that prevents failures before they impact operations. Delivering up to *99.99999%* [my emphasis] availability, the platform combines proactive health monitoring, automated failover, redundancy, and simplified serviceability to keep applications continuously available and ensures that data stays protected.
Stratus ztC Endurance is ideal for organizations running mission-critical workloads that cannot tolerate downtime or data loss."
That's 3.15 seconds of downtime a year, or about 104 seconds since 1993. Quite impressive for a steel products maker.
kjs3 7 hours ago [-]
When your swinging a crucible of 2500F molten steel, you really don't want to hear "hey y'all, one sec...I gotta reboot the server...".
lloydatkinson 12 hours ago [-]
Why even write an article with such little detail?
killingtime74 10 hours ago [-]
Clicks
kjs3 7 hours ago [-]
Ads don't get eyeballs all be themselves...
pjmlp 7 hours ago [-]
Same applies to most IBM and Unisys, mainframe and micros.
thomasmarton 9 hours ago [-]
Times when Windows Update was not a thing
jungkeunkim 5 hours ago [-]
This is really calming. Is the music generated fresh for each city, or do cities share a set of loops? I'd love a rainy Seoul alley at 2am as a preset.
UltraSane 9 hours ago [-]
It is interesting how long server uptime used to be considered a good thing but now the shorter the average runtime the better as it indicates the system was designed to tolerate failures.
Suzuran 7 hours ago [-]
No, a long uptime is evidence of the crime of not sacrificing sufficiently on the altar of security. You must be constantly interrupting service for security patches, firmware updates, release upgrades, etc. so your users are constantly reminded that security is your paramount concern.
UltraSane 7 hours ago [-]
Patching vulnerabilities IS important and well designed systems can keep running while one server reboots.
tosti 6 hours ago [-]
Linux may get a kexec handover protocol some years from now. You can reboot the kernel and your userland daemon seperately :)
6 hours ago [-]
api 8 hours ago [-]
I wonder if doing this to hardware would have overall been cheaper than doing the baroque Rube Goldberg shit we do to software to make it somewhat kind of fault tolerant… or at least convince ourselves it is.
(You can write real fault tolerant software but a wad of tangled crap and hidden dependencies crammed into Kubernetes is not it.)
kjs3 7 hours ago [-]
For a compare-and-contrast, Tandem is the Stratus competitor that went more toward software FT (though it does have redundant hardware). You can also look at various n-of-m voter redundancy setup that are very common in aerospace computers. Definitely not rube goldberg, but also not cheap if for no other reason than you need 3+ of everything (e.g. the primary Space Shuttle computer ran 4 synchronized voters).
advisedwang 1 hours ago [-]
Fault tolerance is expensive, even when it's a single box. This kind of hardware is the wrong tool for the job because most use cases can tolerate some downtime and it's not worth paying 10x cost to get a few nines you don't need.
There are airliners that have been in service for over 50 years.
Are there any airliners that have flown for 24 years non-stop?
it's more impressive to me that the electric fans within the Stratus server still operate than it is that the chips still work, and if we're going with the plane metaphor it's the silicon that puts the thing into the air, not the simple fans.
So really it just boils down to "well, the planes work is harder." as to why it fails, which is of course abstract and unsatisfying; comparing the comparative lifetime workloads of a chip that has shifted trillions of bits versus some jet engines that have moved millions of kilograms around the world.
and then another layer comes and makes it even weirder : the vast majority of hardware around the world that has been EoLd and replaced has been in that situation due to software, not the hardware itself; a paradigm that really doesn't exist in the physical engineering world in the same was as it does CS.
These two have very distinct failure modes. The failure mechanisms for solid state electronics are much more similar to those for static equipment (corrosion, thermal fatigue, creep, ...)
But really didn't the NSA have something that was roughly an airliner in perpetual flight?
It seems when people think of a server today, they think of a single computer, if not some software server somewhere in the cloud. These subject kinds of servers are far closer to complex systems of separate components, more like a network system, it’s why components of what are really separate networked computers can and need to be replaced.
An airliner is of course also made up of components that are also systems, however at least in my mind, the difference is the actual expected use case. An airliner is not a good comparison because it is never expected to remain in continuous operation, e.g., that at least one engine is always running even when it is being overhauled or repaired, to satisfy a requirement of continuous operation.
What is remarkable is that apparently that VOS thing has never crashed. I wonder if they also do software redundancy under the hood to keep uptime going during reboots. If the CPU is hotswappable, it must have something.
The main computer system in the Space Shuttle is an excellent example of this. It had 5 identical IBM System/4 Pi machines. Three of them ran identical code and handled the same work. The fourth ran a completely different codebase to handle the same work, preventing a bug in the main code from killing the whole system. A fifth computer handled other tasks but could be swapped over to the critical role if needed. You could lose 2/5 computers and still have insurance against a cosmic ray flipping a bit.
Many servers I've owned had multiple years of uptime, and the only reason they'd go down is because they will get decommissioned or because of power outage.
The article says:
> disk drives, power supplies and some other components have been replaced but Hogan estimates that close to 80% of the system is original.
so I am guessing there was no reboots, nor CPU replacements.
The biggest change from then to now is probably the amount of solid state electronics onboard. That's ridden roughly the same curve as other industrial and commercial applications.
Yes, this drastically understates the change in computing but the actual technology hasn't changed that much, it just got denser and faster.
IBM mainframes, famously, phone home if they detect a faulty component. Service personnel will be on site, same day, to replace it before you even knew it was there, let alone had the opportunity to ask.
On other side only moving element would be hard drives and only wear element really would be the electrolytic caps.
...with a very aggressive maintenance cycle (typical part life is 1000-10000 hrs of service). Most airliners of that vintage resemble the "Ship of Theseus" in real life.
The were Awesome machines - don't let those filthy hobbitsis...err..tandem users cloud your judgement. Stratus was pure hardware FT, and tandem... wasn't :-P
Vos was pretty nice too - decent macro language - not all the utils you expect with a unix box, but in terms of the core OS and shell it was pretty good machine .. 40 years later I still love those machines and the career they helped get me into :-D
VOS was intuitive and did what I needed. Nothing flashy. Just useful. I miss that sometimes. Give me a tool that works; if it also happens to be great, lovely.
I didn't work much with the Stratus hardware, but reading about it was fascinating. I wish I remembered more of it.
Both were really popular with banks and brokerage houses (back when I worked with stratus at brokerages, HFT was just emerging and down time was umm... frowned upon :-D
A reliable beast - never failed once in the entire time I worked there. Based on PA-RISC if I recall correctly.
Speaking of reliable beasts, I own(ed) a HP PA-RISC workstation that never crashed in six years, and it was even purchased refurbished (second hand, and educational rebate, and still the cost of a car in 1996 - but worth every penny).
I'd add "R.I.P. HP 9000 715/75" but of course she is STILL working now, twenty years later, and she is sitting right next to me (also got a second 715/100XC for spare parts in case on day they are needed).
I think VOS being PL/1 based pretty cool - Stratus (and Intel fab PL/M stuff) was the only time I've ever gotten to work with PL/1...
When the jackpot is fed by all the machine in the state, they can grow much faster.
That's not particularly surprising to be honest. If it's a server operating system, on hardware that doesn't change, there's not really a reason to update the OS unless there's a security problem found
Apparently the latest VOS release was in 2025, which means that somewhere there's a team of developers maintaining it. That's really the thing I find the most interesting, that there exists teams developing operating systems like VOS, strange DOS clones or UnixWare (perhaps less so these days). Must be a strange anonymous life.
This perfectly characterizes most embedded systems work. Competent software development using bespoke or off the shelf components that are not mainstream. Nobody ever realizes they’re using your computer software if you’ve done your job right. One day someone throws your appliance away because it’s obsolete.
I was in Fairbanks visiting family, I had some downtime, so I decided to do some “slumming” to break up the monotony and relive some of the good old days before I was more boardroom than server room.
Besides, a cool grand for what I figured was 2 hours work appealed to me on some “miss the struggles” level, and it was better than walking around the tourist traps or doing video calls at some depressingly fecund coffee shop.
So, I drive up in my borrowed SUV with peeling paint, grab my trusty tool bag I leave with family for adventures like this, and walk boldly through the doors of this apparently normal retail establishment, clipboard in hand. It was 2019, and the smell of denim and factory pressed shirts wafted through my sensorium as I walked through the awkwardly desynchronized automatic doors.
It was a perfectly normal, somewhat 90s nostalgic retail store.., but walking up the stairs (the elevator was out of service) to the second floor, I could just sense that something was a little off.
The office had a smell. I can’t say what it was, it was businesslike, and not unpleasant, but it spoke to me in an unknown language and I was at that moment curiously heartened that my contract was being paid by a 3rd party. It smelled of stagnant ideas, missed opportunities, and a fossilized board of directors.
I’m not just being colorful here in prose. All of these thoughts were projected into my forebrain by some lizard sense I had developed in 40 years of skulking the halls of enterprises large and small. I was involuntarily imagining a half used tin of bengay on the boardroom table, right by the shiny beige speakerphone with the long stroke buttons and the filthy woven grill.
Curious by now about the flood of intuitions, I treaded the slightly dingy burnt orange carpet tiles up to the POC office, checked in, and was shown the door tot the server room. Through there she said, gesturing at a plain flat golden pine door with the elbow thingy at the top. “Through there, straight through, the door at the back. You can’t miss it. I think it’s unlocked.”
Unconcerned with the casual lack of escort (pretty common really), I opened the door and was immediately transported to surreal alternative world. It was dark, there were tens of flickering screens across to the right and one half hanging, flashing fluorescent fixture dangling precariously from the uni-strut above. The suspended ceiling tiles were about 50 percent in attendance.
Fragments littered the floor, and the door closed behind me with a loud thump. for just a moment I was touched by a singular thought….. (run). I shook it off, by now, far too curious to turn back, come what may.
Briefly thinking of my heavy tool bag as a potential weapon, then realizing the ridiculousness of the notion, I let my eyes adjust as I took in the scene with what I confess could only be described as some kind of unhealthy delight.
This was a large room, easily 800 square feet, and a was clearly dedicated to loss prevention surveillance. There was an epic workstation with 2 giant flat screen plasma monitors front and center, and ten smaller monitors on either side. Most of them were dark, and most of the others were flickering with static.a singular monitor had a poor but recognizable image of a small nondescript room with a small desk and two metal chairs, one on each side of the desk in an overtly adversarial position. The image smelled of stale sweat and musty cardboard.
There was seating for two pilots at the helm of this derelict craft, replete with an impressive array of buttons, strange industrial joysticks, and an oddly out of place black dell workstation keyboard with the cord ripped in half. As I made my way up to admire the helm, I nearly tripped on an extension cord, only catching myself on a water filled trashcan with plastic sheeting aspiring to the heavens up through the dark abyss of the half-intact ceiling, which I only now noticed was comprised entirely (if sparsely) of black painted tiles.
Somehow this jolted me out of my entranced state, and I looked for the door I was supposed to find, quickly locating it towards the back of the room, just as I had been told.
Somehow it seemed like not an ordinary door now. It was with great anticipation and not a little trepidation that I examined the door. It was quite ordinary, really, and the knob was less dusty than it might have been. This was encouraging.
I tried the door, and it opened easily. Opened, that is, to a time 30 years in the past.
The buzzing fluorescent lamps were on and radiating furiously, instantly shattering the wreckage of the surveillance spaceship to smithereens. There was a gigantic, wheeled, beige machine in the middle of the room, about the size of an industrial floor scrubber. It hummed patiently, and featured a dim 10 inch green CRT, a beautifully sculpted beige keyboard, and a little less than a quarter inch of dust. I froze like a tourist gaping at a stegosaur’s nose, but quickly recovered my composure as I remembered I was here on purpose.
Conveniently, someone had swept a path through the center of the room, back to the newly installed floor rack with a couple of ciscos, a backup cellular telemetry box, and a pile of screws and a throwaway screwdriver scattered haphazardly on top of one of the routers.
The UPS I was to replace was actually pretty complex to configure, but I very carefully completed it in about 45 minutes including some firmware regressions to accommodate some nonstandard plugin the vendor installed. By very carefully I mean without stepping too hard on the floor, dropping any tools, and afraid to sneeze even though I was under intense respiratory assault from the unavoidable clouds of fine carpet dander that had accumulated on the linoleum tile over years of janitorial malfeasance.
I was terrified to make any sudden moves, because sitting on a stackable plastic chair next to the pliestocene PBX was a beige box. Its fans groaned with the wails of a thousand parched capacitors, and its hard drive made a crunchy rattling sound that seemed entirely improbable for a functioning device. Just visible under the dust on the front was a badge proudly displaying “pentium pro” and a giant cable was fed from the back of the artifact, like an implant from the borg, something I strangely remember as “IBM type 1” token ring wire, though I can’t be sure anymore that means anything at all.
At any rate, it was obvious that the entire on-prem server infrastructure was this ancient and faltering nightmare box, and I knew that a fly landing on it could bring it down. The easy money contract was starting to look like a setup.
With terror in my heart, I finished my work, carefully closed the door, and put as much distance between me and that roiling cauldron of liability as I could. I don’t even remember going back through the wrecked starship, just checking out, and thanking Voltaire that I made it out of there in time.
It now made abundant sense that the job had been left on the board for nearly a month and the contract had kept rising… god only knows how many sensible technicians had noped out of there before I was hypnotized into compliance by the sheer bizarre fascination of the thing.
Poking around afterward, I found that the box apparently was the server that ran the terminals in the tire service area. They also had that fat cable that looked more like waveguide than wire, and it must have been an absolute nightmare to install. It was definitely token ring, humming along in 2019.
Sears closed for good a few months later.
Timber that was very expensive when it was fitted, and carpet tiles that have repeatedly got damp and not dried out properly. Very old cigarette smoke from when people were allowed to smoke in the office that is now embedded in the blinds and suspended ceiling tiles.
I was recently in a large suite of government offices, fitted out in the 1970s and largely unchanged since, and that was the exact smell. We're imagining the same smell right now.
https://www.neoteo.com/en/a-commodore-64-stands-the-test-of-...
How do the maintenance, development, and operations costs of a system like this compare to - for sake of argument - a typical line-of-business Java system running on commodity servers you own in a DC somewhere?
Can anyone with experience in both comment?
A VAXCluster or Sysplex(S/390) would I guess be more analogous but those can be a bit Ship of Theseus since you can change the hardware out over time.
My current contract has a PDP-11/70 running RSX-11 that's been up since 1992. It's uptime is longer than mine! Poor thing is getting retired next year though.
Stratus ztC Endurance extends fault tolerance to modern enterprise and operational environments with intelligent, predictive technology that prevents failures before they impact operations. Delivering up to *99.99999%* [my emphasis] availability, the platform combines proactive health monitoring, automated failover, redundancy, and simplified serviceability to keep applications continuously available and ensures that data stays protected.
Stratus ztC Endurance is ideal for organizations running mission-critical workloads that cannot tolerate downtime or data loss."
Source: https://www.penguinsolutions.com/en-us/solutions/fault-toler...
(You can write real fault tolerant software but a wad of tangled crap and hidden dependencies crammed into Kubernetes is not it.)