To reproduce..

• go through your roles and switch off 'activity modules' for all instances of 'Context types where this role may be assigned'.
• now go to any activity as admin and note that 'Locally assigned roles' link has vanished from the activity's admin menu.

For a site admin (I think) it should still be there.

Analysis:
The display of 'Locally assigned roles' in the navigation relies on the number of records returned from get_assignable_roles($context) being greater than zero. This in turn checks for the 'role_context_levels' table (amongst other criteria) containing entries for the relevant context level (module / 70). Because the activity modules option has been removed from all roles there are no level 70 entries at all and even the admin will get an empty array returned.