As of Moodle 3.1, the CSV export of responses returns all of the responses even if group selections are in effect (see https://moodle.org/mod/forum/discuss.php?d=346659).
This probably happened when the CSV export was improved in 3.1.
If group selections are in effect, or if the user performing the operation is limited by group rules, the responses should only be for users in the appropriate groups.