The search does behave exactly the way you described it.
This is a design choice: search & filters do persist while navigating from one collection segment to the other.
It would be extremely painful to redefine the filtering context on each segment switch (it is a use case to consider).
I am afraid that you’ll have to manually reset the filtering context when not needed anymore.
An alternative is to navigate to another collection (naturally resets the filtering context) and then go to the segment you want.