The solution here is to create a segment that match the records that are eligible for that smart action.
Then, you can restrict this smart action to this segment.The smart action then will be available only through that segment list or directly on the given records (if they match the segment condition).
You can choose to treat the records differently in your smart action code (and either send a success or an error message for instance).
However, for the UI action availability, it can be only done the way I told you.