Yes, that is the big problem with using scenes that way.
Higher end desks often allow what is being saved as part of a scene to be customised so recalls don’t overwrite certain control changes, but it is still possible to accidentally land in a mess unless you are very careful.
When I was doing corporate sound gigs that had an entertainment element we’d usually have separate desks for the band and conference. That meant the band desk was under full control of the band engineer and saves/recalls there wouldn’t affect the rest of the event or the system config. (E.g. zones off a matrix etc.)