That’s why all fields from the user table are retrieved. The code does not have any clue on which of the fields are actually used by your smart field.
That’s why for the moment, there is no other way than requesting all the fields.
If you choose to display a real field that is present in your table, then only this field will be requested for each user.