The fact that your heat pump setup is also taking longer to cool suggests there's something fundamentally different between the setup on your different floors, not that there is something bad with heat pumps in general.

A heat pump in cooling mode works exactly like an AC unit, because that's exactly what it is. So if your AC unit on the first floor cools more quickly than you AC unit (i.e. heat pump) on your second floor, it's because A) your floors are different sizes or insulated differently or something else is different about their construction, B) your units are sized differently, or C) your heat pump has some mechanical problem. But the fact that it's a heat pump should make no difference to its cooling performance.

Sounds like you have one of two problems: 1) you are changing the temperature too often. Set it and forget it. 2) you have a refrigerant leak