That's a 'statechart'. [0] They are actually different things.
Your classic 'finite state machine' (FSM) is pretty simple. Boxes with arrows coming out of them leading to other boxes.
You can't store additional state in a FSM. Your state is what box you're in. As a result, more complex processes get really complex fast in a FSM.
Hence statecharts. This is what most people these days actually think of as state machines. They have history, you can implement guard conditions, you can store state (hence removing the finitude), and so on.
So, actually, yes! Infinite state machines are very common. They're very, very useful. I love programming using statecharts.
The "infinite" descriptor is an imprecise one: by definition, a computer with limited memory has finite state. While we can make pretensions to infinite capacity, that's not really the thrust of the description when we say "finite state machine", which is to put emphasis not on the limitations of the computer, but on the definition of a quantified and labelled set of states and transitions.
What all of these terms really describe are formalisms over finite state - the linked paper is precise in this respect. "You can't store state in an FSM" misses the point: you can store as many concepts as you have enumerated states, and the act of storage itself is a change from one state to another state. Breaking with this concept and allowing unlabelled storage is indeed a useful quality because it helps with dealing with bulk permutations.
But every programming language is intricately tied to their models of finite state and how it's presented to the user. Most PLs, and likewise hardware instruction sets, hide away state bookkeeping with notions of "assignment" and "branching", so when the formalisms appear it's usually because you really need to explicitly enumerate more of your state. The formalized FSM goes hand-in-hand with concurrent programming because it clarifies how coordination occurs and when state changes are allowed. And real implementations of the FSM usually piggyback in some degree on the base programming environment, which has always been a source of muddiness: oftentimes a "goto" is exactly what you need to describe a transition, and yet in structured programs, we've eliminated goto.
