One weird trick to avoiding nuget breakage: treat packages as immutable. If you need a new or local build of a package, you must bump the version number (use -alphaNNN or increment the patch number) for every rebuild.

Or, if you're trying to temporarily use a local source tree, swap out <PackageReference> for <ProjectReference>.