Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-28516

Add support for course category synchronisation in External DB Enrolment Plugin

XMLWordPrintable

    • Icon: New Feature New Feature
    • Resolution: Duplicate
    • Icon: Minor Minor
    • None
    • 2.1
    • Enrolments
    • Any
    • MOODLE_21_STABLE
    • Moderate
    • Hide

      To test this:

      At present it is not possible to synchronise course categories from an external database in moodle.

      After applying this patch, create an external database. The following assumes that you're using postgres, and have a user 'moodle'.:

        1. Creating our test dataset
          sudo su - postgres
          createdb -O moodle -EUTF8 externaldb

      cat <<EOF | psql externaldb
      CREATE TABLE categories (
      id integer NOT NULL,
      title character varying(255) NOT NULL,
      description text,
      parent integer
      );

      ALTER TABLE public.categories OWNER TO moodle;

      EOF

        1. Create some data in the categories
          cat <<EOF | php5 | psql externaldb
          <?php
          echo "INSERT INTO categories (id, title, description, parent) VALUES (1, 'Top Level', 'Description for Top Level', null);\n";

      \$i = 2;
      while (\$i <= 6) {
      echo "INSERT INTO categories (id, title, description, parent) VALUES (\$i, 'Faculty \$i', 'Description for Faculty \$i', 1);\n";
      \$j = 1;
      while (\$j <= 5) {
      echo "INSERT INTO categories (id, title, description, parent) VALUES ({\$i}00{\$j}, 'Department {\$j}', 'Description for Department {\$i}', {\$i});\n";
      \$j++;
      }
      \$i++;
      }
      EOF

        1. Iniial Configuration
          Now configure the enrolment/database module as follows:
          Site administration
          -> Plugins
          -> Enrolments
          -> Manage enrol plugins

      Enable "External database"
      Navigate to Settings page for External database

      Set database driver, host, user, password, and name as you've set them for your database

      Under "Creation of new Course Categories"
      Set "External course categories table" to:
      categories

      Set "Course Category id field" to:
      id

      Set "Course Category name field" to:
      title

      Leave the parent, and description fields blank for the moment as we'll use these to test updates, and re-parenting.

      Now run a cli sync:

        1. Test Initial sync
          Open a shell, and switch to your web server user. Under debian I do:
          sudo su - www-data
          bash

      cd ~/git/moodle
      php enrol/database/cli/sync.php

      In Moodle, you should now have lots of top level categories.
      Open one of the categories in the category editor and verify that it has:

      • the correct name;
      • a category id number as appears in the database (e.g. Faculty 2 has idnumber 2; Faculty 2 -> Department 3 has idnumber 2003); and
      • no description.

        1. Test Setting a Category Parent

      Navigate back to the plugin's settings page and

      Set "Course Category parent field" to:
      parent

      Run the cli sync again

      In Moodle, the categories should now be re-parented. You should now have a top level category, with five sub-categories, each having five further sub categories

        1. Test Setting a Category Description

      Navigate back to the plugin's settings page and

      Set "Course Category description field" to:
      description
      Set the "Format of Course Category description field" to "Plain text format"

      Run the cli sync again

      Open the same category again and verify that it now has the correct description and format.

        1. Testing Deletion

      echo "DELETE FROM categories;" | psql externaldb

      Run the cli sync again

      All categories should be deleted with the exception of the "Top Level" category. It is not possible to delete this using the standard moodle category_delete_move

      All courses should have been re-parented to the "Top Level" category.

        1. Testing a stupdily large dataset
          Please note that the Moodle interface doesn't actually work at all well with this many. This test merely proves that the category sync doesn't hog memory

      Ensure that you have the following set in your moodle config:
      define('MDL_PERF' , true);
      define('MDL_PERFDB' , true);
      define('MDL_PERFTOLOG' , true);
      define('MDL_PERFTOFOOT', true);

      I stress, moodle cannot handle this number of course categories. It cannot render the Edit Courses page with this many. This is a purely a test to demonstrate that this patch is really pretty memory efficient

      cat <<EOF | php5 | psql externaldb
      <?php
      echo "INSERT INTO categories (id, title, description, parent) VALUES (1, 'Top Level', 'Description for Top Level', null);\n";

      \$i = 2;
      while (\$i <= 99) {
      echo "INSERT INTO categories (id, title, description, parent) VALUES (\$i, 'Faculty \$i', 'Description for Faculty \$i', 1);\n";
      \$j = 1;
      while (\$j <= 99) {
      echo "INSERT INTO categories (id, title, description, parent) VALUES ({\$i}00{\$j}, 'Department {\$j}', 'Description for Department {\$i}', {\$i});\n";
      \$j++;
      }
      \$i++;
      }
      EOF

      echo "SELECT COUNT FROM categories;" | psql externaldb

      There should be 9801 course categories

      Run the sync again – it may take some time and it does generate output.
      After creating all of the categories it may appear to hang – It's just commiting the transaction

      In my test setup (exactly as above) with a fresh Moodle installation on master, I see the following perf data:

      PERF: time: 87.208004s memory_total: 38125852B (36.4MB) memory_growth: 37229628B (35.5MB) memory_peak: 81938716B (78.1MB) includecount: 75 contextswithfilters: 0 filterscreated: 0 textsfiltered: 0 stringsfiltered: 0 langcountgetstring: 9807 langcountmemcache: 9804

      Memory peaks at 78MB which is relatively low for such a large dataset.

      The number of reads/writes is regrettable, but is only high for the initial sync

      Try performing a second sync and you'll see far lower reads/writes and memory usage:
      PERF: time: 2.722942s memory_total: 39761160B (37.9MB) memory_growth: 38864936B (37.1MB) memory_peak: 57756268B (55.1MB) includecount: 75 contextswithfilters: 0 filterscreated: 0 textsfiltered: 0 stringsfiltered: 0 langcountgetstring: 7 langcountmemcache: 4 langco

      Show
      To test this: At present it is not possible to synchronise course categories from an external database in moodle. After applying this patch, create an external database. The following assumes that you're using postgres, and have a user 'moodle'.: – Creating our test dataset sudo su - postgres createdb -O moodle -EUTF8 externaldb cat <<EOF | psql externaldb CREATE TABLE categories ( id integer NOT NULL, title character varying(255) NOT NULL, description text, parent integer ); ALTER TABLE public.categories OWNER TO moodle; EOF Create some data in the categories cat <<EOF | php5 | psql externaldb <?php echo "INSERT INTO categories (id, title, description, parent) VALUES (1, 'Top Level', 'Description for Top Level', null);\n"; \$i = 2; while (\$i <= 6) { echo "INSERT INTO categories (id, title, description, parent) VALUES (\$i, 'Faculty \$i', 'Description for Faculty \$i', 1);\n"; \$j = 1; while (\$j <= 5) { echo "INSERT INTO categories (id, title, description, parent) VALUES ({\$i}00{\$j}, 'Department {\$j}', 'Description for Department {\$i}', {\$i});\n"; \$j++; } \$i++; } EOF – Iniial Configuration Now configure the enrolment/database module as follows: Site administration -> Plugins -> Enrolments -> Manage enrol plugins Enable "External database" Navigate to Settings page for External database Set database driver, host, user, password, and name as you've set them for your database Under "Creation of new Course Categories" Set "External course categories table" to: categories Set "Course Category id field" to: id Set "Course Category name field" to: title Leave the parent, and description fields blank for the moment as we'll use these to test updates, and re-parenting. Now run a cli sync: — Test Initial sync Open a shell, and switch to your web server user. Under debian I do: sudo su - www-data bash cd ~/git/moodle php enrol/database/cli/sync.php In Moodle, you should now have lots of top level categories. Open one of the categories in the category editor and verify that it has: the correct name; a category id number as appears in the database (e.g. Faculty 2 has idnumber 2; Faculty 2 -> Department 3 has idnumber 2003); and no description. — Test Setting a Category Parent Navigate back to the plugin's settings page and Set "Course Category parent field" to: parent Run the cli sync again In Moodle, the categories should now be re-parented. You should now have a top level category, with five sub-categories, each having five further sub categories — Test Setting a Category Description Navigate back to the plugin's settings page and Set "Course Category description field" to: description Set the "Format of Course Category description field" to "Plain text format" Run the cli sync again Open the same category again and verify that it now has the correct description and format. — Testing Deletion echo "DELETE FROM categories;" | psql externaldb Run the cli sync again All categories should be deleted with the exception of the "Top Level" category. It is not possible to delete this using the standard moodle category_delete_move All courses should have been re-parented to the "Top Level" category. — Testing a stupdily large dataset Please note that the Moodle interface doesn't actually work at all well with this many. This test merely proves that the category sync doesn't hog memory Ensure that you have the following set in your moodle config: define('MDL_PERF' , true); define('MDL_PERFDB' , true); define('MDL_PERFTOLOG' , true); define('MDL_PERFTOFOOT', true); I stress, moodle cannot handle this number of course categories. It cannot render the Edit Courses page with this many. This is a purely a test to demonstrate that this patch is really pretty memory efficient cat <<EOF | php5 | psql externaldb <?php echo "INSERT INTO categories (id, title, description, parent) VALUES (1, 'Top Level', 'Description for Top Level', null);\n"; \$i = 2; while (\$i <= 99) { echo "INSERT INTO categories (id, title, description, parent) VALUES (\$i, 'Faculty \$i', 'Description for Faculty \$i', 1);\n"; \$j = 1; while (\$j <= 99) { echo "INSERT INTO categories (id, title, description, parent) VALUES ({\$i}00{\$j}, 'Department {\$j}', 'Description for Department {\$i}', {\$i});\n"; \$j++; } \$i++; } EOF echo "SELECT COUNT FROM categories;" | psql externaldb There should be 9801 course categories Run the sync again – it may take some time and it does generate output. After creating all of the categories it may appear to hang – It's just commiting the transaction In my test setup (exactly as above) with a fresh Moodle installation on master, I see the following perf data: PERF: time: 87.208004s memory_total: 38125852B (36.4MB) memory_growth: 37229628B (35.5MB) memory_peak: 81938716B (78.1MB) includecount: 75 contextswithfilters: 0 filterscreated: 0 textsfiltered: 0 stringsfiltered: 0 langcountgetstring: 9807 langcountmemcache: 9804 Memory peaks at 78MB which is relatively low for such a large dataset. The number of reads/writes is regrettable, but is only high for the initial sync Try performing a second sync and you'll see far lower reads/writes and memory usage: PERF: time: 2.722942s memory_total: 39761160B (37.9MB) memory_growth: 38864936B (37.1MB) memory_peak: 57756268B (55.1MB) includecount: 75 contextswithfilters: 0 filterscreated: 0 textsfiltered: 0 stringsfiltered: 0 langcountgetstring: 7 langcountmemcache: 4 langco

      Using the enrolment/database plugin, you can currently synchronise courses, but not categories. Many institutions may want to define their course category structure based upon an external database too.

            poltawski Dan Poltawski
            dobedobedoh Andrew Lyons
            Petr Skoda Petr Skoda
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.