If there was demand for more high performance 68k processors, presumably Motorola would have made more. But by the time of the 68060 customers for high end 68k had either already moved onto various RISC architectures (Mac, UNIX workstations) or were dead / dying (ST, Amiga).
A related question would be if Motorola could pull off what Intel did with x86, and drag the architecture to be competitive with RISC. Part of Intel's trick was that x86 isn't the most CISCy of chips, so it was easy to implement a processor that takes a "RISCy subset" of instructions and translate them into one or two µops. 68k is perhaps more on the VAX side of things, so whether they'd be able to pull off the same trick is a bit unclear. Clearly we need to check the parallel universes where IBM chose the 68008 for Chess instead of the 8088...
I checked one of the adjacent ones, and there Motorola used traps and microcode internally for emulation of some of the CISCier instructions and a fast RISCy core for the rest.
Everyone was encouraged to move on to the fast subset and the new multimedia instructions.
The problem was mainly in x86, existing code already mostly used instructions that belonged to the fast subset, and on 68k, it didn't. So while you could evolve the instruction set towards a saner design (and the 68060 was well on the way towards that), only new code would benefit from it.
And of course that's exactly what they did with Coldfire - rounding off the inconvenient corners of the ISA to produce CPUs with lower power requirements and able to run at higher clock speeds.
They did it with the 68030 before Coldfire. They discarded a number of things (e.g. addressing modes) that seemed like good ideas for the <=68020 but didn't end up being used in practice.
on the m68k, the "cisc-y-ness" is in the many many addressing modes, whereas x86 in that particular aspect of the architecture has always been rather "risc-y" (read: rather limited compared to other CISC architectures, including m68k).
The core instruction set of the m68k, as far as ALU/FPU is concerned, is simple enough. But converting the addressing modes to "risc building blocks" (μops or whatever term you like to use) is harder.
Not everyone has that sort of luxury of access to source etc., people want their existing binaries to run faster.
Also, if you need to recompile to get a performance boost, why not recompile for a cleaner modern architecture? You can always use an emulator for legacy code, if it isn't going to run fast on a modern CPU either way...
Motorola wanted to replace the 68000 series with their RISC design, the 88000, so it was probably always going to end there. Their customers were also all in full RISC hype fever and were looking for RISC chips to move to.
In Apple's search for a RISC chip, didn't want to be stuck being single-sourced again so they had IBM and Motorola work together to launch PowerPC instead, so Motorola gave up on the 88k.
I have seen mentioned a couple of times that the 88K would have had some inherent design flaw that made the architecture a dead end. I have never found an explanation what that would have been though.
Perhaps this had just been a misunderstanding of a second-hand statement. Its firstiteration was flawed in that it was an expensive multi-chip solution with few buyers. The second was more integrated though.
My understanding is that by the time the second iteration came around, the AIM Alliance had been formed and Motorola simply abandoned the 88k and started focusing on PowerPC.
If NeXT hadn’t killed their hardware division in 93, the RISC Workstation would’ve run an MC88110.
