Yes, it depends a lot. Typical signal-processing tasks you will definitely want to hand-optimize, no matter how good the compiler is, it will never be able to get close to hand-crafted SIMD code in combination with careful memory-layout, reducing the working set, eliminating dependencies, maximizing cache usage etc. There's a lot involved with that kind of optimizations that supposes semantic knowledge of the data going through the algorithm to be able to make transformations to optimize the code.
For the other 99% of code it's hard (and definitely not worthwhile to try) to beat the compiler though. I sometimes watch programming streams where people go very deep into the compiler-generated code to analyze runtime behavior of their code, and it's almost scary to see the kinds of things compilers can do these days.
> For the other 99% of code it's hard (and definitely not
> worthwhile to try) to beat the compiler though.
Bernstein's thesis is that the other 99% of the code is so cold that it's not really worth running the optimizer, and my experience (and evidence from industry) suggests that he is right about that. In fact, you might be better off with an interpreter and a really compact byte-code.
Examples: Tcl used for "steering" supercomputer computations, Squeak Smalltalk doing fine for multi-media work with a simple bytecode interpreter + primitives, Objective-C, Ruby on Rails and Sinatra for a huge swathe of web-sites, the (incorrect) perception of Swift as "fast", even in Debug mode (without optimizations), Python/Numpy, etc.
Certainly in my experience it's about 90-9-1; where 90% of the runtime is "cold", 9% is "warm" (optimising compiler is good enough), and 1% is "hot". I'm literally writing some code right now where -O2 pushes us from "not fast enough in the average case" to "just about fast enough in the worst case". I fully accept that things might be different in Web Development, but are optimising compilers really that common in Web Dev?
>> [99% of all code is cold]
> in my experience it's about 90-9-1
I don't think such numbers can be meaningfully presented as representative data. They are probably true for certain specific application domains such as crypto or signal processing, if you squint at the data just the right way (i.e., by benchmarking a crypto library with a tiny driver program, not as part of a real application).
I did some profiling on the SPEC CPU 2000 benchmarks some years ago. I don't have the exact numbers lying around. But I recall large variation among the different benchmarks, with anything from 90% of execution time in the top function to the top 60% of execution time spread evenly (i.e., 20% each) over the top three functions. So some things certainly follow the patterns above, and others... don't.
I've also several times optimized programs until their profile was so flat that you could say that 100% of the code was lukewarm.
There's an impressive amount of resources poured into making Javascript faster; we often have to think about what the interpreter will do with the code we write in order to let it do some optimisations.
That thing on the link above easily does almost everything you described as unattainable for compiler and does it in automatic manner. AFAIK, it even allows your code to MPI parallelized, with optimal communication.
Handmade hero [1] mostly, and sometimes Jonathan Blow's compiler programming streams [2].
Not all of these video's are equally interesting, but there's some real gems in there, e.g. where Casey (of HH) hand-optimizes his bilinear bitmap interpolation code (somewhere between streams ~115 and ~120 if I remember correctly).
I cringe a lot about their coding style and complete disregard of architecture and basic code hygiene they appear to have, but there's still a lot of interesting things to learn from watching these streams.
For the other 99% of code it's hard (and definitely not worthwhile to try) to beat the compiler though. I sometimes watch programming streams where people go very deep into the compiler-generated code to analyze runtime behavior of their code, and it's almost scary to see the kinds of things compilers can do these days.