100% agree with most of this. Most large organisations in Australia are just clueless about their needs and ride hype cycles with execs acting on FOMO. And they bought the lie that you need all these separate distinct engineers who are professionals in a niche. They think they are building a high end kitchen by having specialists and think they are hiring sous-chefs etc, but most of the time they are hiring line cooks, and in fact most of the time they should just be hiring short order cooks.
Most teams and businesses I know who are doing great things with data are mostly smaller companies and tech built up start-ups and hire one of two types of people. Generalists who can fill the gaps and are interested in everything, and staff software engineers who are amazing developers and asked to put together data pipelines with consideration of the whole infrastructure.
When I was looking for work, all the good companies were not hiring data engineers or machine learning engineers. They were hiring Senior Software Engineers with a remit to build data infrastructure, or build machine learning models. And they immediately removes 90% of the noise applicants.
Most teams and businesses I know who are doing great things with data are mostly smaller companies and tech built up start-ups and hire one of two types of people. Generalists who can fill the gaps and are interested in everything, and staff software engineers who are amazing developers and asked to put together data pipelines with consideration of the whole infrastructure.
When I was looking for work, all the good companies were not hiring data engineers or machine learning engineers. They were hiring Senior Software Engineers with a remit to build data infrastructure, or build machine learning models. And they immediately removes 90% of the noise applicants.