By adding a UpdateVisibilityNextTick flag against every FrozenActor to track when a visibility update is required, we can remove the dirtyFrozenActorIds set in FrozenActorLayer. In the Tick method we can now avoid a set lookup. Also, don't create the frozenActorsToRemove list until we need one to avoid an allocation.