There are job objects which are similar to Linux cgroups, including the ability to set a limit on the number of processes. But I'm not sure if that limit will be tripped in this case or not because the child processes have exited, whereas the job object parameter is specifically called LIMIT_ACTIVE_PROCESS
OpenProcess retrieves a handle to an existing process rather than creating a process so it won't be governed by JOB_OBJECT_LIMIT_ACTIVE_PROCESS, the bug here is that it's leaking handles, not processes.