That's fair, and perhaps my example wasn't the best, but my point is that just as the continuum hypothesis is an artifact of the models we use to describe our universe (we use continuum-sized sets to describe physical space), more basic properties like "how numbers ought to work" could also be artifacts of our models. In particular, it wouldn't be inconceivable for an alternate universe to be better described by entirely different models from the ground up, which could be fairly described as "different math".

> we use continuum-sized sets to describe physical space

But... we don't. We use integers to describe physical space. We have real numbers as a mathematical construct, but we have never applied them to even a single physical problem. That's impossible to do, because specifying a real number takes an infinite amount of information.