Because then you can be vulnerable to csrf attack. For example if someone tricks you into clicking www.mysite.com/api/delete-account

CSRF attacks can be prevented using same-site policy with cookies.

That is true but it will not protect against all forms of CSRF, for example you'll be vulnerable if you have user generated content that's not sanitized properly. On the refresh_token cookie I have sameSite and httpOnly set.