I can be in a room looking at something with my eyeballs and listening with my ears perfectly legally... But it would not be legal if I replaced myself with a humanoid mannequin with a video camera for a head.
You can even write down what you are looking at and listening to, although in some cases, dissemination of, e.g. verbatim copies in your writing could be considered copying.
But it is automatically copying if you use a copier.
I always find these articles really interesting, maybe it's just the way my brain works but I can't ever find any meaningful differences or issues with any of the examples.
> but as noted we found this quite obvious to the eye as well
... and then 2 images that look exactly the same to me :(
I am a huge monorepo supporter, including "no development branches".
However there's a big difference between development and releases. You still want to be able to cut stable releases that allow for cherrypicks for example, especially so in a monorepo.
Atomic changes are mostly a lie when talking about cross API functions, i.e. frontend talking to a backend. You should always define some kind of stable API.
Can you explain this comment? Are you saying to develop directly in the main branch?
How do you manage the various time scales and complexity scales of changes? Task/project length can vary from hours to years and dependencies can range from single systems to many different systems, internal and external.
The complexity comes from releases. Suppose you have a good commit 123 were all your tests pass for some project, you cut a release, and deploy it.
Then development continues until commit 234, but your service is still at 123. Some critical bug is found, and fixed in commit 235. You can't just redeploy at 235 since the in-between may include development of new features that aren't ready, so you just cherry pick the fix to your release.
It's branches in a way, but _only_ release branches. The only valid operations are creating new releases from head, or applying cherrypicks to existing releases.
That's where tags are useful because the only valid operations (depending on force push controls) are creating a new tag. If your release process creates tag v0.6.0 for commit 123 your tools (including `git describe`) should show that as the most recent release, even at commit 234. If you need to cut a hotfix release for a critical bug fix you can easily start the branch from your tag: `git switch -c hotfix/v0.6.1 v0.6.0`. Code review that branch when it is ready and tag v0.6.1 from its end result.
Ideally you'd do the work in your hotfix branch and merge it to main from there rather than cherry picking, but I feel that mostly because git isn't always great at cherry picking.
> Suppose you have a good commit 123 were all your tests pass for some project, you cut a release, and deploy it.
And you've personally done this for a larger project with significant amount of changes and a longer duration (like maybe 6 months to a year)?
I'm struggling to understand why you would eliminate branches? It would increase complexity, work and duration of projects to try to shoehorn 2 different system models into one system. Your 6 month project just shifted to a 12 to 24 month project.
The reason I said it would impact duration is the assumption that the previous version and new version of the system are all in the code at one time, managed via feature flags or something. I think I was picturing that due to other comments later in the thread, you may not be handling it that way.
Either way, I still don't understand how you can reasonably manage the complexity, or what value it brings.
Example:
main - current production - always matches exactly what is being executed in production, no differences allowed
production_qa - for testing production changes independent of the big project
production_dev_branches - for developing production changes during big project
big_project_qa_branch - tons of changes, currently being used to qa all of the interactions with this system as well as integrations to multiple other systems internal and external
big_project_dev_branches - as these get finalized and ready for qa they move to qa
Questions:
When production changes and project changes are in direct conflict, how can you possibly handle that if everyone is just committing to one branch?
How do you create a clean QA image for all of the different types of testing and ultimately business training that will need to happen for the project?
It depends a lot on a team by team basis as different teams would like different approaches
In general, all new code gets added to the tip of main, your only development branch. Then, new features can also be behind feature flags optionally. This allows developers to test and develop on the latest commit. They can enable a flag if they are interested in a particular feature. Ideally new code also comes with relevant automated tests just to keep the quality of the branch high.
Once a feature is "sufficiently tested" whatever that may mean for your team it can be enabled by default, but it won't be usable until deployed.
Critically, there is CI that validates every commit, _but_ deployments are not strictly performed from every commit. Release processes can be very varied.
A simple example is we decide to create a release from commit 123, which has some features enabled. You grab the code, build it, run automated tests, and generate artifacts like server binaries or assets. This is a small team with little SLAs so it's okay to trust automated tests and deploy right to production. That's the end, commit 123 is live.
As another example, a more complex service may require more testing. You do the same first steps, grab commit 123, test, build, but now deploy to staging. At this point staging will be fixed to commit 123, even as development continues. A QA team can perform heavy testing, fixes are made to main and cherry picked, or the release dropped if something is very wrong. At some point the release is verified and you just promote it to production.
So development is always driven from the tip of the main branch. Features can optionally be behind flags. And releases allow for as much control as you need.
There's no rule that says you can only have one release or anything like that. You could have 1 automatic release every night if you want to.
Some points that make it work in my experience are:
1. Decent test culture. You really want to have at least some metric for which commits are good release candidates.
2. You'll need some real release management system. The common tools available like to tie together CI and CD which is not the right way to think about it IMO (example your GitHub CI makes a deployment).
TL:Dr:
Multiple releases, use flags or configuration for the different deployments. They could all even be from the same or different commits.
> As another example, a more complex service may require more testing. You do the same first steps, grab commit 123, test, build, but now deploy to staging. At this point staging will be fixed to commit 123, even as development continues. A QA team can perform heavy testing, fixes are made to main and cherry picked, or the release dropped if something is very wrong. At some point the release is verified and you just promote it to production.
But how would you create that QA environment when it involves thousands of commits over a 6 month period?
It totally depends on how you want to test releases. You can have nightlies that deploy the latest green commit every day, do some QA there, then once it is feature complete promote it to a stable release that only cherry picks fixes, then finally promote it to production.
It will be highly dependent on the kind of software you are building. My team in particular deals with a project that cuts "feature complete" releases every 6 months or so, at that point only fixes are allowed for another month or so before launch, during this time feature development continues on main. Another project we have is not production critical, we only do automated nightlies and that's it.
> It totally depends on how you want to test releases.
For a big project, typically it involves deploying to a fully functioning QA environment so all functionality can be tested end to end, including interactions with all other systems internal to the enterprise and external. Eventually user acceptance testing and finally user training before going live.
I don't see how you're avoiding development branches. Surely while a change is in development the author doesn't simply push to main. Otherwise concurrent development, and any code review process—assuming you have one—would be too impractical.
So you can say that you have short-lived development branches that are always rebased on main. Along with the release branch and cherry-pick process, the workflow you describe is quite common.
They don’t do code reviews or any sort of parallel development.
They’re under the impression that “releases are complex and this is how they avoid it” but they just moved the complexity and sacrificed things like parallel work, code reviews, reverts of whole features.
You lose atomic deployment and have a distributed system the moment you ship Javascript to a browser.
Hell, you lose "atomic" assets the moment you serve HTML that has URLs in it.
Consider switching from <img src=kitty.jpg> to <img src=puppy.jpg>. If you for example, delete kitty from the server and upload puppy.jpg, then change html, you can have a client with URL to kitty while kitty is already gone. Generally anything you published needs to stay alive for long enough to "flush out the stragglers".
They just refresh the page, it's not a big deal. It'll happen on form submission or any navigation anyway. Some people might be caught in a weird invalid state for, like, a couple minutes absolute maximum.
Right, there's level of solutions. You can't sit here and say that a few seconds of invalid state on the front-end only for mayyyyybe .01% of your users is enough to justify a sprawling distributed system because "well deployments aren't atomic anyway!1!".
IMO, monorepos are much easier to handle. Monoliths are also easier to handle. A monorepo monolith is pretty much as good as it gets for a web application. Doing anything else will only make your life harder, for benefits that are so small and so rare that nobody cares.
Monorepo vs not is not the relevant criteria. The difference is simply whether you plan your rollout to have no(/minimal) downtime, or not. Consider SQL schema migration to add a non-NULL column on a system that does continuous inserts.
Again, that's trivial if you use up and down servers. No downtime, and to your users, instant deployment across the entire application.
If you have a bajillion services and they're all doing their own thing with their own DB and you have to reconcile version across all of them and you don't have active/passive deployments, yes that will be a huge pain in the ass.
So just don't do that. There, problem solved. People need to stop doing micro services or even medium sized services. Make it one big ole monolith, maybe 2 monoliths for long running tasks.
Magical thinking about monorepos isn't going to make SQL migrations with backfill instantaneous and occur simultaneously with the downtime you have while you switch software versions. You're just not familiar with the topic, I guess. That's okay. Please just don't claim the problem doesn't exist.
And yes, it's often okay to ignore the problem for small sites that can tolerate the downtime.
Not sure what GP had in mind, but I have a few reasons:
Cherry picks are useful for fixing releases or adding changes without having to make an entirely new release. This is especially true for large monorepos which may have all sorts of changes in between. Cherry picks are a much safer way to “patch” releases without having to create an entirely new release, especially if the release process itself is long and you want to use a limited scope “emergency” one.
Atomic changes - assuming this is related to releases as well, it’s because the release process for the various systems might not be in sync. If you make a change where the frontend release that uses a new backend feature is released alongside the backend feature itself, you can get version drift issues unless everything happens in lock-step and you have strong regional isolation. Cherry picks are a way to circumvent this, but it’s better to not make these changes “atomic” in the first place.
Do you take down all of your projects and then bring them back up at the new version? If not, then you have times at which the change is only partially complete.
I would see a potentially more liberal use of atomic, that if the repo state reflects the totality of what I need to get to new version AND return to current one, then I have all I need from a reproducibility perspective. Human actions could be allowed in this, if fully documented. I am not a purist, obviously.
Blue/green might allow you to do (approximately) atomic deploys for one service, but it doesn't allow you to do an atomic deploy of the clients of that service as well.
Why that? In a very simple case, all services of a monorepo run on a single VM. Spin up new VM, deploy new code, verify, switch routing. Obviously, this doesn't work with humongous systems, but the idea can be expanded upon: make sure that components only communicate with compatible versions of other components. And don't break the database schema in a backward-incompatible way.
So yes, in theory you can always deploys sets of compatible services, but it's not really workable in practice: you either need to deploy the world on every change, or you need to have complicated logic to determine which services are compatible with which deployment sets of other services.
There's a bigger problem though: in practice there's almost always a client that you don't control, and can't switch along with your services, e.g. an old frontend loaded by a user's browser.
The notion of external clients is a smell. If that’s the case, you need a compat layer between that client and your entrypoints, otherwise you’ll have a very hard time evolving anything. In practice, this can include providing frontend assets under previously cached endpoints; a version endpoint that triggers cache busting; a load balancer routing to a legacy version for a grace period… sadly, there‘s no free lunch here.
The only way I could read their answer as being close to correct is if the clients they're referring to are not managed by the deployment.
But (in my mind) even a front end is going to get told it is out of date/unusable and needs to be upgraded when it next attempts to interact with the service, and, in my mind atleast, that means that it will have to upgrade, which isn't "atomic" in the strictest sense of the word, but it's as close as you're going to get.
If your monorepo compiles to one binary on one host then fine, but what do you do when one webserver runs vN, another runs v(N-1), and half the DB cluster is stuck on v(N-17)?
A monorepo only allows you to reason about the entire product as it should be. The details of how to migrate a live service atomically have little to do with how the codebase migrates atomically.
That's why I mention having real stable APIs for cross-service interaction, as you can't guarantee that all teams deploy the exact same commit everywhere at once. It is possible but I'd argue that's beyond what a monorepo provides. You can't exactly atomically update your postgres schema and JavaScript backend in one step, regardless of your repo arrangement.
Adding new APIs is always easy. Removing them not so much since other teams may not want to do a new release just to update to your new API schema.
But isn't that a self-inflicted wound then? I mean is there some reason your devs decided not to fix the DB cluster? Or did management tell you "Eh, we have other things we want to prioritize this month/quarter/year?"
This seems like simply not following the rules with having a monorepo, because the DB Cluster is not running the version in the repo.
Maybe the database upgrade from v(N-17) to v(N-16) simply takes a while, and hasn't completed yet? Or the responsible team is looking at it, but it doesn't warrant the whole company to stop shipping?
Being 17 versions behind is an extreme example, but always having everything run the latest version in the repo is impossible, if only because deployments across nodes aren't perfectly synchronised.
This is why you have active/passive setup and you don't run half-deployed code in production. Using API contracts is a weak solution, because eventually you will write a bug. It's simpler to just say "everything is running the same version" and make that happen.
each deployment is a separate "atomic change". so if a one-file commit downstream affects 2 databases, 3 websites and 4 APIs (madeup numbers), then that is actually 9 different independent atomic changes.
I can guarantee that your codebase is spaghetti of conditional functionality that no developer understands, and that most of those conditionals are leftovers that are no longer needed, but nobody dares to remove.
Feature flags are a good idea, but they require a lot of discipline and maintenance. In practice, they tend to be overused, and provide more negatives than positives. They're a complement, but certainly not a replacement for VCS branches, especially in monorepos.
Not OP, but I think building feature flags yourself really isn’t hard and worth doing. It’s such an important component that I wouldn’t want to depend on a third party
I agree, but it's hard to get the nuances right. It's easy to roll out a feature to half of your user base. It's a bit harder to roll a feature out to half of users who are in a certain region, and have the flag be sticky on them.
We use Unleash at work, which is open source, and it works pretty well.
I generally agree, but see some more nuance. I think feature-flagging is an overloaded term that can mean two things.
First, my philosophy is that long-lived feature branches are bad, and lead to pain and risk once complete and need to be merged.
Instead, prefer to work in small, incremental PRs that are quickly merged to main but dormant in production. This ensures the team is aware of the developing feature and cannot break your in-progress code (e.g. with a large refactor).
This usage of "feature flags" is simple enough that it's fine and maybe even preferable to build yourself. It could be as simple as env vars or a config file.
--
However, feature flagging may also refer to deploying two variants of completed code for A/B testing or just an incremental rollout. This requires the ability to expose different code paths to selected users and measure the impact.
This sort of tooling is more difficult to build. It's not impossible, but comparatively complex because it probably needs to be adjustable easily without releases (i.e. requires a persistence layer) and by non-engineers (i.e. requires an admin UI). This becomes a product, and unless it's core to your business, it's probably better to pick something off the shelf.
Something I learned later in my career is that measuring the impact is actually a separate responsibility. Product metrics should be reported on anyway, and this is merely adding the ability to tag requests or other units of work with the variants applied, and slice your reporting on it. It's probably better not to build this either, unless you have a niche requirement not served by the market.
--
These are clearly two use cases, but share the overloaded term "feature flag":
1. Maintaining unfinished code in `main` without exposing it to users, which is far superior than long-lived feature branches but requires the ability to toggle.
2. Choosing which completed features to show to users to guide your product development.
(2) is likely better served by something off the shelf. And although they're orthogonal use cases, sometimes the same tool can support both. But if you only need (1), I wouldn't invest in a complex tool that's designed to support (2)—which I think is where I agree with you :)
I like keeping old branches but a lot of places ditch them, never understood why. I also dislike git squash, it means you have to make a brand new branch for your next PR, waste of time when I should be able to pull down master / dev / main / whatever and merge it into my working branch. I guess this is another reason I prefer the forking approach of github, let devs have their own sandbox and their own branches, and let them get their work done, they will PR when its ready.
Squashing only results in a cleaner commit history if you're making a mess of the history on your branches. If you're structuring the commit history on your branches logically, squashing just throws information away.
I’m all ears for a better approach because squashing seems like a good way to preserve only useful information.
My history ends up being:
- add feature x
- linting
- add e2e tests
- formatting
- additional comments for feature
- fix broken test (ci caught this)
- update README for new feature
- linting
With a squash it can boil down to just “added feature x” with smaller changes inside the description.
If my change is small enough that it can be treated as one logical unit, that will be reviewed, merged and (hopefully not) reverted as one unit, all these followup commits will be amends into the original commit. There's nothing wrong with small changes containing just one commit; even if the work wasn't written or committed at one time.
Where logical commits (also called atomic commits) really shine is when you're making multiple logically distinct changes that depend on each other. E.g. "convert subsystem A to use api Y instead of deprecated api X", "remove now-unused api X", "implement feature B in api Y", "expose feature B in subsystem A". Now they can be reviewed independently, and if feature B turns out to need more work, the first commits can be merged independently (or if that's discovered after it's already merged, the last commits can be reverted independently).
If after creating (or pushing) this sequence of commits, I need to fix linting/formatting/CI, I'll put the fixes in a fixup commit for the appropriate and meld them using a rebase. Takes about 30s to do manually, and can be automated using tools like git-absorb. However, in reality I don't need to do this often: the breakdown of bigger tasks into logical chunks is something I already do, as it helps me to stay focused, and I add tests and run linting/formatting/etc before I commit.
And yes, more or less the same result can be achieved by creating multiple MRs and using squashing; but usually that's a much worse experience.
That seems better as long as you can keep it standard across the team. I don’t usually check each commit when reviewing since frequent iterative commits mean folks change their mind and I’d review already removed logic when looking at early commits.
I’ve been scraping by on basic git usage so didn’t know about fix-up commits, that’s excellent.
You can always take advantage of the graph structure itself. With `--first-parent` git log just shows your integration points (top level merge commits, PR merges with `--no-ff`) like `Added feature X`. `--first-parent` applies to blame, bisect, and other commands as well. When you "need" or most want linear history you have `--first-parent` and when you need the details "inside" a previous integration you can still get to them. You can preserve all information and yet focus only on the top-level information by default.
It's just too bad not enough graphical UIs default to `--first-parent` and a drill-down like approach over cluttered "subway graphs".
stacked diffs are the best approach and working at a company that uses them and reading about the "pull request" workflow that everyone else subjects themselves to makes me wonder why everyone is not using stacked diffs instead of repeating this "squash vs. not squash" debate eternally.
every commit is reviewed individually. every commit must have a meaningful message, no "wip fix whatever" nonsense. every commit must pass CI. every commit is pushed to master in order.
Not everyone develops and commits the same way and mandating squashing is a much simpler management task than training up everyone to commit in a similar manner.
Besides, they probably shouldn't make PR commits atomic, but do so as often as needed. It's a good way to avoid losing work. This is in tension with leaving behind clean commits, and squashing resolves it.
The solution there is to make your commit history clean by rebasing it. I often end my day with a “partial changes done” commit and then the next day I’ll rebase it into several commits, or merge some of the changes into earlier commits.
Even if we squash it into main later, it’s helpful for reviewing.
At work there was only one way to test a feature, and that was to deploy it to our dev environment. The only way to deploy to dev was to check the repo into a branch, and deploy from that branch.
So one branch had 40x "Deploy to Dev" commits. And those got merged straight into the repo.
Good luck getting 100+ devs to all use the same logical commit style. And if tests fail in CI you get the inevitable "fix tests" commit in the branch, which now spams your main branch more than the meaningful changes. You could rebase the history by hand, but what's the point? You'd have to force push anyway. Squashing is the only practical method of clean history for large orgs.
Also rebasing is just so fraught with potential errors - every month or two, the devs who were rebasing would screw up some feature branch that they had work on they needed and would look to me to fix it for some reason. Such a time sink for so little benefit.
I eventually banned rebasing, force pushes, and mandated squash merges to main - and we magically stopped having any of these problems.
We squash, but still rebase. For us, this works quite well. As you said, rebasing needs to be done carefully... But the main history does look nice this way.
True but. There's a huge trade-off in time management.
I can spend hours OCDing over my git branch commit history.
-or-
I can spend those hours getting actual work done and squash at the end to clean up the disaster of commits I made along the way so I could easily roll back when needed.
Squash loses the commit history - all you end up with is merge merge merge
It's harder to debug as well (this 3000line commit has a change causing the bug... best of luck finding it AND why it was changed that way in the first place.
I, myself, prefer that people tidy up their branches such that their commits are clear on intent, and then rebase into main, with a merge commit at the tip (meaning that you can see the commits AND where the PR began/ended.
"squash results in a cleaner commit history" Isn't the commit history supposed to be the history of actual commits? I have never understood why people put so much effort into falsifying git commit histories.
Here is how I think of it. When I am actively developing a feature I commit a lot. I like the granularity at that stage and typically it is for an audience of 1 (me). I push these commits up in my feature branch as a sort of backup. At this stage it is really just whatever works for your process.
When I am ready to make my PR I delete my remote feature branch and then squash the commits. I can use all my granular commit comments to write a nice verbose comment for that squashed commit. Rarely I will have more than one commit if a user story was bigger than it should be. Usually this happens when more necessary work is discovered. At this stage each larger squashed commit is a fully complete change.
The audience for these commits is everyone who comes after me to look at this code. They aren’t interested in seeing it took me 10 commits to fix a test that only fails in a GitHub action runner. They want the final change with a descriptive commit description. Also if they need to port this change to an earlier release as a hotfix they know there is a single commit to cherry pick to bring in that change. They don’t need to go through that dev commit history to track it all down.
“Falsifying” is complete hyperbole.
Git commit history is a tool and not everyone derives the same ROI from the effort of preserving it. Also squashing is pretty effortless.
There are several valid reasons to "falsify" commit history.
- You need to remove trash commits that appear when you need to rerun CI.
- You need to remove commits with that extra change you forgot.
- You want to perform any other kind of rebase to clean up messages.
I assume in this thread some people mean squashing from the perspective of a system like Gitlab where it's done automatically, but for me squashing can mean simply running an interactive (or fixup) and leaving only important commits that provide meaningful information to the target branch.
I'm very fortunate to not have to use PR style forges at work (branch based, that is). Instead each commit is its own unit of code to review, test, and merge individually. I never touch branches anymore since I also use JJ locally.
Nothing lasts forever. The second you decide to use a new 3P API you have to understand it might disappear one hour after your production launch, and that's okay.
You also have to understand that there will be APIs that eagerly rugpull and APIs that don't, and if you're offering the former then your users will end up moving to your competitors all else being equal.
Unless I'm mistaken, electoral college representatives are assigned based on state borders and individual state laws - usually either winner take all or proportional. The electoral college itself can't be gerrymandered in the same way congress can.
Electoral college representatives are currently elected on a statewide winner-take-all basis except Maine and Nebraska, each of which assign two electors (corresponding to the votes due to two Senate seats each state gets) based on the statewide winner while assigning the other electors based on the winner in each Congressional district.
So, in those states only, electoral votes can be gerrymandered in exactly the same was as Congressional seats, because they are exactly the same districts. (Of course, both states are small enough that the gerrymandering opportunities are fairly limited, and would have limited impact on Presidentual elections, as Maine has only two CDs and Nebraska only 3.)
reply