To my knowledge git-lfs is only really designed to store your large files on a central server. I think it also uses its own out-of-band (from the perspective of git) protocol and connection to talk to that server. So it doesn't work with a standard ssh remote, and breaks git's distributed nature.

For an actually distributed large file tracking system on top of git you could take a look at git-annex. It works with standard ssh remotes as long as git-annex is installed on the remote too (it provides its own git-annex-shell instead of git-shell), and has a bunch of additional awesome features.