The short answer is "yes" due to the way the configuration management works. Other infrastructure changes or service upgrades might get undone, but it's possible. Or otherwise revert the commit that introduced the package bump with the new code and force that to rollout everywhere rather than waiting for progressive rollout.
There shouldn't be much chance of bringing the system to a novel state because configuration management will largely put things into the correct state. (Where that doesn't work is if CM previously created files, it won't delete them unless explicitly told to do so.)
> I collaborated a lot with a collage - James (Jog). I asked him loads of questions, from "how to login to a server", via "what is anycast" to "tell me how you mitigated this one, give me precise instructions you've run".
Hi, that's me! There were definitely a lot of fun conversations.
I liked that a culture of internal blogs became a thing too. It was good to see people brain dumping their experiments and findings. I think people learnt a lot from following all the internal blogs.
Always funny to see these sort of missed connections on HN.
> internal blogs
In my personal experience the problem is the total lack of writing culture at non-premiere companies.
Put differently: unless you’re working on a great team at a great organization roughly 90% of people cannot be expected to write/read well as a component of technical collaboration. Any thoughts on that? I may just be too cynical
From my personal experience, internal Wikis are 90+% written by about 5% of the workforce. The vast majority of internal Wiki users are "read-only". That is fine. Really: Read that twice. It is OK that the vast majority of docs are written by the few who are highly motivated to do it. When I write for an internal Wiki, I am trying to reduce the number of future questions that people will ask me. If they do come with a question, my first answer is always: Did you search our Wiki? It only takes once or twice for them to change.
I'm not sure it's really cynical at all, in my experience, even with a LOT of wiki documentation, it is almost always severely out of date and many people don't know what is where... after a while, there's new, new, new versions of docs because people didn't want to refactor existing docs someone else created. And, in the end, almost nobody reads it and just ask other, more senior people within the team/org.
This is, of course, without explicit management directives to update documentation as a task/chore. Much like in many (most?) orgs, churning through technical debt is rarely a priority.
That said, no place is perfect and it takes only a handful of people to improve a culture, but without at least management support and/or fostering such a culture, it doesn't tend to happen.
But the non-premiere companies also do not hire the talent that is capable of doing this (and that might be fine, they might not really need that specific talent).
I've many friends that worked most of their carreers in small to medium companies where this wasn't a thing and stuff mostly worked. We can't expect to replicate the high end tech experience everywhere, I doubt it is a good investment of resources.
Reading comprehension is declining. Emphasis on individual "impact" and deadline pressure, common at both premiere companies and "non-premiere" companies mindlessly copying the former, both consume time that could have been spent thinking, writing, and engaging with optional things others wrote. People who avoid documents because they might get outdated create systems with no documentation at all. And now, LLMs promise a world where documents don't need to exist a priori -- a model will look at your code and generate a plausible description of possible intent and system architecture on demand, if someone implausibly happens to ask for documentation. If nobody happens to ask, that's even more time saved - a win-win!
Leadership can either support or inhibit the culture of writing and reading. However, modern managers are not immune from the rising pressure. Their response is to shift from thinking and deliberate information processing to rapid-fire pattern matching. Some of them don't see documents as "real output" to begin with, and operate solely in meetings without any written record or documentation whatsoever. Of coures their staff will pick up on the pattern. I have a vivid memory of engaging with a full team working on a substantial project in the absence of their senior leader, getting the tactical picture and then asking the team about the project's goal. You could have heard a pin drop. None of the people working on the project could speak about anything but their immediate assigned tasks.
Internal blog is another thing, but indeed related to chat. Once we figured some techical thing out, I often put it under "personal blog" on wiki under something like ~marek/date-slug. (I'm actually quite upset about jira removing slug from the url and replacing it with random number, this makes url completion much worse. I often titled the wikis funny/witty to make it easier to remember and find)
These were usually quite short notes. Like I did this and that. Or here are instructions I've run. Pretty much a "lab notebook".
Again, this was often useful to people. Folks new what I was working on at that time. Over time more people adapted this style of public note taking (within company), but I have the impression I was more consistent than others.
I did it for many reasons. I genuinely forget things. It was always nice to see comments (like James complaining that I shouldn't do `dpkg -i x.deb`, and instead finally install the package repo in apt-sources!). So again - searchability (a form of documentation), reach (getting feedback), and work log (for planning).
The format was also nice - short, without specific audience in mind. Zero drama, zero effort. Plentiful and low quality. Because it was "blog" and not "pages", this meant it was obvious when the note was written and nobody expected old blogs to be up to date. The lower the friction in note taking - the better.
Finally, I was working with an SRE team, which was tightly knit and communicated very often over daily checkins (which I wasn't invited to) and other informal channels. And I worked from home a bit. This meant I had to find an asynchronous way to communicate with the team. My personal blogs worked nicely. I highly recommend this style to anyone working remotely.
Although I must admit the personal "lab notebook" wiki thing is not scaleable. It's impossible to "follow" more than a handful of people.
I kept statistics. It's fairly obvious when I was most productive. Here it is, my internal blogs on the wiki per quarter:
It should more properly be written as dæmon. The æ ("ash") character is usually pronounced more like "ee", as in encyclopædia. I've never heard anyone say "encycloPAYdia" :-)
Fascinating! This is why I stick with nice, clean structural linguistics, this applied stuff gets sticky. I just confirmed on Youtube that the (some?) British people do indeed pronounce "Aesthetic" as "ah-stet-ic" not "ee-stet-ic", and upon diving a bit, it seems that the rule is "don't ask for a rule, you fool! It's 'e' now except for when it isn't." Thanks for the interesting tidbit!
The letter æ was used in Old English to represent the vowel that's pronounced in Modern English ash, fan, happy, and last: /æ/. Mostly we now spell that vowel with the letter a, because of the Great Vowel Shift.
When æ appears in writing Modern English, it's meant to be a typographic variant of ae, and is pronounced the same as that sequence of vowel letters would be. So Encyclopaedia or Encyclopædia, no difference.
Highly recommend the protracted arguments in the comments, that's a wonderfully pedantic StackExchange. Big shoutout to someone in 2012 defining "NLP" as an unusual word -- how the world has changed! It's only a matter of time before they open an AP/IB course in NLP...
Hmm, I'm a Californian and I pronounce daemon as demon, understanding the first vowel as the same vowel as for Aesop. Indistinguishable from the vowel in "beam" and "niece".
But I pronounce the first vowel in aesthetic differently. For me, it's somewhat in between the vowels in "bed" and "bad" but closer to the former.
…TIL how to pronounce “Aesop”! Thanks for saving me eventual embarrassment - now I know why other people don’t mix up Aesop Rock and ASAP Rocky!
As a fellow Californian, I’d say we have authority anyway - I was taught in school that Ohio has the least specialized dialect, but that’s based on newscasters and such. The 21st century is the Californian century!
…that is, assuming Brussels’ English is out of the running, I suppose ;)
"Aesthetic" gets even stickier! In the UK I tend to more commonly hear it pronounced as "es-thetic".
The Great Vowel Shift indeed makes written English much more confusing than it perhaps should be. English is already a messy hodge-podge of a language, then our writing system started to get standardised (or standardized, if you're American!) right as pronunciation started to change, leading to the written version of words suddenly no longer being anything like the pronunciation.
The letter "æ" as used in Old English does indeed correspond to /æ/, but we don't use that letter (or even digraph) for this purpose anymore. In all the words where it is still occasionally used, it corresponds not to Old English "æ", but to Latin "ae", which is [ae̯].
The original pronunciation of ae/æ in words originating from Latin or Greek is basically like "I". As usual, English molded it into something else in many cases, which is why we write "demon" these days. But if you insist of "daemon", then it really ought to be pronounced like the original Greek δαίμων.
Yes. Daemon is just the archaic spelling of demon. The ae is a vowel sound that didn't survive to modern times. The word was NEVER pronounced "damon." To my knowledge.
Since the introduction of APFS I've taken to creating a new APFS volume formatted as case-sensitive, and put my git repositories there.
This has mostly been useful for working on shared repositories where, say, a Linux user (or other user on a case-sensitive filesystem) pushes two branches, say `feature/foo` and `feature/Foo` which works fine for them, but on a case-insensitive filesystem, git gets very upset.
Managed Kubernetes exists because Kubernetes is so mind-bendingly complex.
Nomad, on the other hand, is pretty simple and easy to run, and a single binary. There's probably no benefit to having a managed offering.
That said, I wouldn't be surprised if HashiCorp add Nomad to HashiCorp Cloud Platform, which currently lets you deploy Consul and Vault to cloud providers via their system.
Managed Kubernetes exists for the same reason managed PostgreSQL, MySQL, Redis, ElasticSearch so on and so forth exist.
Which is to say it's useful enough to a large amount of people to make it viable as a product offering. Nomad would fail as a managed offering here like like Deis, Flynn, Convox and probably 100 other container management platforms that came before.
As a niche tool to manage sub-1000 boxen it's probably ok. But k8s has won the greater war.
Disclaimer: Worked on Flynn, still run it for my own personal stuff by $DAY_JOB is all k8s.
You got that sub-1000 boxen wrong. Nomad is usually employed where the infrastructure is huge. IMO, K8s is driven by hype and marketing, not by technical merit.
Your enumeration of the managed things is very particular, in that it includes only stores. There's a reason for things that have keeping persistent state as their reason to exist are managed: reliably maintaining persistent state in the cloud is a lot more difficult than reliably orchestrating things that just have to run.
If public cloud providers would offer Nomad, then it would only be a question of time until managed Nomad would be a thing.
Managed _something_ does not mean it's "mind-bendingly complex", but rather that people don't want to take care about it and focus on their own stuff like building applications.
Actual deployments take hours to propagate worldwide.
(Disclosure: former Cloudflare SRE)