0.6 percent of a model with 890 million parameters is still 5.34 million parameters.

That's still pretty big. Maybe big enough to fake normals, learn some albedo smoothing functions, and learn a depth estimator perhaps??