To use the smart card with a new gnupg home, i.e. different host/user, you have to import the public keys first.

$ gpg --card-edit
…
gpg/card> fetch
…

This will download the public keys from the URL provided on the smart card.