If I understand correctly, the primary advantage of using a secrets manager is that you can log all of the requests.

If your security is compromised, it's compromised, but having at least some kind of auditing can be hugely beneficial from a security standpoint.