Partly because, historically (e.g. on feature phones), the GSM baseband serves as the primary processor (with real-time responsiveness requirements), with "applications" as a subordinate function. The concept of GSM being "just" a modem peripheral is a more recent development, coming more from the laptop arena; pushing that model down into phones (especially cheap phones) will take work.
Even on top of that, the concept of not trusting your peripherals is a recent one as well. Ideally, all hardware peripherals would have no more permissions than they need; for instance, no ability to DMA except to specific pre-arranged regions. In practice, most systems don't actually set up that level of security.
Is this true of say, the iPhone? It seems like a wifi iPad or iPod Touch is exactly the same as an iPhone, but without the baseband. If the baseband were a peripheral of the A8 SoC, this would seem like a trivial difference. But it seems if that's not the case, the iPad A8 would have considerable architectural differences compared to the iPhone one.
It's less true in some modern smartphones (disclaimer: not an expert on the iOS/iDevice architecture in particular), but a shocking amount of code still ends up on the baseband, and the baseband still has as much trust as the kernel. For example, the baseband processor often serves as an offloading engine for power efficiency reasons, to avoid waking up the main processor; thus, the baseband processor might have direct access to the audio hardware, so that phonecall audio doesn't need to wake up the host CPU.
Even on top of that, the concept of not trusting your peripherals is a recent one as well. Ideally, all hardware peripherals would have no more permissions than they need; for instance, no ability to DMA except to specific pre-arranged regions. In practice, most systems don't actually set up that level of security.