I borrowed this from Milligram[0] because it seemed like a sane thing to do at the time. Would your recommendation be to not anything to the base font-size and adjust the REM sizes accordingly?
In relation to this since you're using rem for borders and progress bar height, it's all going to scale along with browser-defined font size. You might not want that, as people typically increase this setting to have just the text larger.
[0] https://github.com/milligram/milligram/blob/d895f179623b56f3...