Preparations

This is about inspecting the Apache logs for complete behat runs and fixing as many notices/problems as possible, aggregating information from all supported themes.

Download the webserver artifact for the following Behat runs on the main view in CI.

• Behat - NonJS (Boost)
• Behat - Boost
• Behat - NonJS (Classic)
• Behat - Classic

Extract these webserver logs in a folder and combine them into an "aggregated.txt" file.

cat webserver* > aggregated.txt

Used this regexp against Non-JS and Chrome webserver logs, for reference:

ag -v 'POST|GET|OPTIONS|\] (checking plan|(saving|calculating|loading|setting) controller|(applying|loading|instantiating) (backup|restore|plan)|(activity|section) skipped|setting file inclusion|processing file aliases|Unknown setting: (questionbank|overwrite_conf|course_shortname|course_fullname|course_startdate))|Failed Login: ' aggregated.txt > results.txt

(where aggregated.txt is the union of Non-JS and Chrome web server logs of all core themes)

Trick: When it's difficult to find which exact behat test is causing the problem, it's useful to run the tests with the "--pretty" command, getting the date prepended. That way, we can match them later with the dates in the Apache error logs. To do so, add the following to your behat run command:

vendor/bin/behat ......  --format pretty --no-colors | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'

You can see the actual results @ [^results.txt]

Trick: Check the issue from the previous release from which this issue was cloned and see if the error was already detected there and considered okay.