A non handwavy explanation of how does evolution result in such complicated and carefully orchestrated mechanisms, way beyond our engineering capability. The computer analogy of genetic algorithms certainly doesn't explain much, they are not very effective, and if they were, we should be able to generate marvels of engineering with GAs and current computational power.
> complicated and carefully orchestrated mechanisms
Yeah, I'd definitely second that. How could evolution result so quickly in something as "rudimentary" as Chlorella (i.e. the simplest plant). https://en.wikipedia.org/wiki/Chlorella
I am being serious. I just purchased Dawkin's book "Blind Watchmaker". My point is no explanation should amount to "magic X did it" whether X is god, evolution, the earth spirit, aliens, etc. All the above are bad explanations.
It is not scientific to substitute one bad explanation for another. The scientific approach is to say we don't know, and then look for a good explanation.
I'll take a shot at trying to explain that to the best of my ability. I'm starting from the point where we have organisms and dna, since nobody knows how exactly abiogenesis happened yet.
First, two general points.
The most important thing to keep in mind is time. Life has had insane amounts of time. Billions of years. Beyond human comprehension amounts of time.
The second most important thing is that complicated != carefully orchestrated or optimal. Life is pretty cool, but it doesn't hold itself to a very high standard. It's the survival of the good enough, and is full of so many random hacks and poor design choices it's insane. Things get easier to accomplish when you lower the standard.
Now an attempt at an explanation.
Evolution by natural selection works on two principles. First, generation of diversity. Second, selective pressure.
DNA can and does mutate frequently. One important type of mutation is a duplication, since it let's you gain new functionality. You make two versions of the same gene, one keeps it's original function, and the other does some new function. This theme of repurposing existing things comes up again and again. Take something you have, make another version of it, change it a bit. If you've worked out how to grow a vertebra as a lizard and want to become a snake, turn off legs amd make more vertebra. Use the same genes, and just modify how you control them. This video (https://youtu.be/ydqReeTV_vk) is actually pretty good at running through the science behind evolutionary development, and how evolution can quickly reuse and modify existing parts. Basically, once smaller features evolve, you start modifying them in a modular way and can start making really big changes really easily. Keep in mind again that you are helped by enormous, mind boggling amounts of time randomly generating these features originally as well. Heres a summary of how our eyes evolved by repurposing neurons (https://youtu.be/ygdE93SdBCY). Our eye is a good example of how the standards are just good enough, we have veins and nerves on top of our light detecting cells instead of behind and just poke a hole to get them through to the other side. Doesn't that leave a blind spot? Yep, and we just hallucinate something to fill in the space. There are a couple other major ways we generate more diversity. You have things like viruses transferring DNA, but a really powerful one is sex. Sexual reproduction lets you combine and generate new combinations of Gene's to speed up how quickly diversification happens.
For selective pressure, think about it purely statistically. You have sets of arrangements of atoms, some of which are good at making new sets of arrangements of atoms that look like then, and others less so. Each tick of the clock, versions that are able to make more increase, and versions that dont decrease. This basically provides a directionality for evolution - whatever is good at replicating is successful. This weeds out mutations that dont over time while keeping mutations that do. This means the next round of mutations are building on ones that were good enough, and not ones that weren't. This lets evolution be more cumulative than a random search.
Neither of those are complete descriptions by any stretch, I'm just trying to give you a taste of the mechanisms behind it, but it goes a lot deeper. The most important things do just boil back down to what I started with. Survival of the good enough - lower your standards. And there's just so much time for these to happen. Evolutionary step only has a 1 in a million chance to happen in a given year? Then it's happened about 65 times since dinosaurs went extinct.
The more I learn about the genetic code, how proteins get translated from DNA, the nature of DNA itself, bioinformatics, biochemistry, etc., the less like a random jumble it seems.
I also wonder why we cannot reproduce evolution's effectiveness computationally. Genetic algorithms and the like are not very good, no where near capable of matching biological systems, even though we can match evolution timescales and populations with our computing power today.
As far as I can tell, we have absolutely no idea why and how evolution works, let alone works so well.