Agreed. For most people it would be more reasonable to lose the three-four kg in body fat and keep the aluminium frame rather than upgrading to carbon fibre.
Unless you have found a way to spend a few thousand dollars to make 3-4 kg of body fat magically disappear with no complications or effort, losing weight and spending more on a bike are not things you're choosing between.
