I think this is reasonable. To get the best performance you would probably have to start a clean slate otherwise you'd likely need to maintain some crazy white/blacklist, and handle closing all those desktop processes gracefully, which is more or less a restart. Given the boot will probably be faster in this new mode I think it's a reasonable compromise for the flexibility it brings, and the fact that you can jump back into game mode with/without a restart is great as well.
When this eventually hits desktop computers, I'd personally be using game mode just for the UI benefits, because I'd want my desktop processes to continue and performance wouldn't be a concern