Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-67334

Behat not correctly checking column values with Selenium

XMLWordPrintable

    • MOODLE_38_STABLE

      It appears that when testing for values in a table when using Selenium, identifying columns by their title string rather than column number does not restrict the check to the column specified. The result is that it is possible to have a false positive result, if any other column on that row contains the value in the test.

      Proof of concept:

      1. Add the attached MDL_67334_poc.feature file to the /mod/forum/report/summary/tests/behat/ directory.
      2. Re-initialise your site's behat.
      3. Modify your /lib/tests/behat/behat_general.php file's "row_column_of_table_should_contain" method, so that the TESTCODE section below is inserted. This will print the HTML node that the test is matching against:

            // Looks for the requested node inside the container node.
            $coumnnode = $this->getSession()->getDriver()->find($columnvaluexpath);
         
            /////START TESTCODE/////
            error_log("Node HTML for value '{$value}':");
            foreach($coumnnode as $thing) {
                error_log(print_r($thing->getOuterHtml(), true));
            }
            /////END TESTCODE/////
         
            if (empty($coumnnode)) {
                $locatorexceptionmsg = $value . '" in "' . $row . '" row with column "' . $column;
                throw new ElementNotFoundException($this->getSession(), "\n$columnvaluexpath\n\n".'Column value', null, $locatorexceptionmsg);
            }
        

      4. Run the behat test from step 1, using the following command (update the path your behat config):

        vendor/bin/behat --config /path/to/your/moodles/stable_master/moodledata_behat/behatrun/behat/behat.yml --tags="@MDL67334test1" --stop-on-failure
        

      5. Observe that the test passes. It should have failed, as Teacher 1 only has 2 replies posted, but the test checks for 3.
      6. Look at the output from the behat test, and observe that the test is matching each of the table values against all columns in the row that contain the specified value, not only the one it should be checking.
      7. Re-run behat, changing the --tags to test 2, ie "@MDL67334test2". This is the same test, but run without the javascript tag (ie running via Goutte instead of Selenium).
      8. Observe that the test fails, and also observe that the test is only matching the values against a single node (the correct behaviour).
      9. Re-run behat, changing the --tags to test 3, ie "@MDL67334test3". This replaces the text column headings with their column numbers.
      10. Observe that this is also working correctly (failing on the last value, the same as seen in step 7). This confirms that the bug only occurs when using column names to identify the column to be tested.
      11. Re-run behat, changing the --tags to test 4, ie "@MDL67334test4". This runs the same test (using column numbers), but with the correct testing values. This confirms the test does work when valid values are used with this method.

       

            Unassigned Unassigned
            michaelh Michael Hawkins
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 5 hours, 21 minutes
                5h 21m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.