To use the smart card with a new gnupg home, i.e. different host/user, you have to import the public keys first.
$ gpg --card-edit … gpg/card> fetch …
This will download the public keys from the URL provided on the smart card.