The problem is that session_set_user() is doing unrelated loading of capabilities and enrolment checks:
1/ we need the USER->access only in has_capability() and friends
2/ the $USER is not yet available inside session_set_user() which complicates everything in accessslib.php
3/ enrolment checks may be very slow in some plugins, it does not make much sense to do it every time we try to fake global $USER

Solution is:
1/ delay loading of $USER->access till the time it is really necessary
2/ do enrol_check_plugins($user); only in login-stlye scripts
3/ clean up all places that are related to these changes