Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-85445

Improving the performance of the messaging system when it is heavily used

    • MOODLE_401_STABLE
    • main
    • Moderate
    • Hide

      0 - Prerequisites

      • Ensure that 1000 users have been created. Each user must have a minimum of 10 conversations, and each conversation a minimum of 15 messages. Pick three users at random and they will enter 200 conversations of 50 messages each.

      A - On Main, 405, 404, 403, 401, 500, 501 (all supported and security supported branches):

      Note:

      Usually this needs to be tested on all the databases

      1. Login with one of three users that contains the 200 conversations
      2. Go to message drawler
        1. VERIFY that It loads in less than 5 seconds

      B - Verify, for all branches, that all units tests are passed (easier with MDK):

      B-1 Using minimum supported version

      1. Install Moodle 4.1.18
        1. Uninstall your current instance

          mdk uninstall -y

        2. Checkout the 4.1.18 tag

          git checkout v4.1.18

        3. Install

          mdk install

      2. VERIFY: that all phpunit tests of message system are passed

        mdk phpunit -r -u message/tests/api_test.php
        mdk phpunit -r -u message/tests/externallib_test.php
        mdk phpunit -r -u message/tests/helper_test.php
        mdk phpunit -r -u message/tests/inbound_test.php
        mdk phpunit -r -u message/tests/messagelib_test.php
        mdk phpunit -r -u message/tests/search_received_test.php
        mdk phpunit -r -u message/tests/search_sent_test.php
        

      3. Attempt upgrade to main
        1. Checkout the main branch.

          git checkout main

        2. Upgrade either via the browser or via CLI (mdk upgrade)
        3. VERIFY: that upgrade works
      4. VERIFY: that all phpunit tests are passed

        mdk phpunit -r -u message/tests/api_test.php
        mdk phpunit -r -u message/tests/externallib_test.php
        mdk phpunit -r -u message/tests/helper_test.php
        mdk phpunit -r -u message/tests/inbound_test.php
        mdk phpunit -r -u message/tests/messagelib_test.php
        mdk phpunit -r -u message/tests/search_received_test.php
        mdk phpunit -r -u message/tests/search_sent_test.php

      C-2 Performance check will be manual

      1. Install Moodle 4.1.18
        1. Uninstall your current instance

          mdk uninstall -y

        2. Checkout the 4.1.18 tag

          git checkout v4.1.18

        3. Install

          mdk install

        4. Add 1000 users, for every user make 40 messages. Add 3 new users, make 200 conversations, each conversation contains 50 messages

          mdk run users --users=1000

          1. For each created user:
            1. Login
            2. Go to messaging system > view all
            3. Select 200 random users, for each user:
            4. Add contact
            5. Sent a message with a random text
            6. For each received message:
              1. Response with a random text
      2. VERIFY: that the messaging system response in less than 5 seconds
      3. Attempt upgrade to main
        1. Checkout the main branch.

          git checkout main

        2. Upgrade either via the browser or via CLI (mdk upgrade)
      4. VERIFY: that upgrade works
      5. VERIFY: that the messaging system response in less than 5 seconds
      Show
      0 - Prerequisites Ensure that 1000 users have been created. Each user must have a minimum of 10 conversations, and each conversation a minimum of 15 messages. Pick three users at random and they will enter 200 conversations of 50 messages each. A - On Main, 405, 404, 403, 401, 500, 501 (all supported and security supported branches): Note : Usually this needs to be tested on all the databases Login with one of three users that contains the 200 conversations Go to message drawler VERIFY that It loads in less than 5 seconds B - Verify, for all branches, that all units tests are passed (easier with MDK): B-1 Using minimum supported version Install Moodle 4.1.18 Uninstall your current instance mdk uninstall -y Checkout the 4.1.18 tag git checkout v4.1.18 Install mdk install VERIFY : that all phpunit tests of message system are passed mdk phpunit -r -u message /tests/api_test .php mdk phpunit -r -u message /tests/externallib_test .php mdk phpunit -r -u message /tests/helper_test .php mdk phpunit -r -u message /tests/inbound_test .php mdk phpunit -r -u message /tests/messagelib_test .php mdk phpunit -r -u message /tests/search_received_test .php mdk phpunit -r -u message /tests/search_sent_test .php Attempt upgrade to main Checkout the main branch. git checkout main Upgrade either via the browser or via CLI (mdk upgrade) VERIFY : that upgrade works VERIFY : that all phpunit tests are passed mdk phpunit -r -u message /tests/api_test .php mdk phpunit -r -u message /tests/externallib_test .php mdk phpunit -r -u message /tests/helper_test .php mdk phpunit -r -u message /tests/inbound_test .php mdk phpunit -r -u message /tests/messagelib_test .php mdk phpunit -r -u message /tests/search_received_test .php mdk phpunit -r -u message /tests/search_sent_test .php C-2 Performance check will be manual Install Moodle 4.1.18 Uninstall your current instance mdk uninstall -y Checkout the 4.1.18 tag git checkout v4.1.18 Install mdk install Add 1000 users, for every user make 40 messages. Add 3 new users, make 200 conversations, each conversation contains 50 messages mdk run users -- users =1000 For each created user: Login Go to messaging system > view all Select 200 random users, for each user: Add contact Sent a message with a random text For each received message: Response with a random text VERIFY : that the messaging system response in less than 5 seconds Attempt upgrade to main Checkout the main branch. git checkout main Upgrade either via the browser or via CLI (mdk upgrade) VERIFY : that upgrade works VERIFY : that the messaging system response in less than 5 seconds
    • Show
      Fails against automated checks. Checked MDL-85445 using repository: https://github.com/tmas0/moodle main (1 errors / 2 warnings) [branch: main | CI Job ] overview (0/0) , phplint (0/0) , phpcs (0/0) , js (0/0) , css (0/0) , phpdoc (0/0) , commit (1/2) , savepoint (0/0) , thirdparty (0/0) , externalbackup (0/0) , grunt (0/0) , shifter (0/0) , mustache (0/0) , gherkin (0/0) , Should these errors be fixed? Built on: Mon May 12 12:16:23 UTC 2025

      Hi,

      In order to improve the performance of the messaging system when it's heavily used, I have found that the following query have a very bad performance when the messages table have more than 500K messages.

      $sql = "SELECT m.id as messageid, mc.id as id, mc.name as conversationname, mc.type as conversationtype, m.useridfrom,
                             m.smallmessage, m.fullmessage, m.fullmessageformat, m.fullmessagetrust, m.fullmessagehtml, m.timecreated,
                             mc.component, mc.itemtype, mc.itemid, mc.contextid, mca.action as ismuted
                        FROM {message_conversations} mc
                  INNER JOIN {message_conversation_members} mcm
                          ON (mcm.conversationid = mc.id AND mcm.userid = :userid3)
                  LEFT JOIN (
                                SELECT m.conversationid, MAX(m.id) AS messageid
                                  FROM {messages} m
                            INNER JOIN (
                                            SELECT m.conversationid, MAX(m.timecreated) as maxtime
                                              FROM {messages} m
                                        INNER JOIN {message_conversation_members} mcm
                                                ON mcm.conversationid = m.conversationid
                                         LEFT JOIN {message_user_actions} mua
                                                ON (mua.messageid = m.id AND mua.userid = :userid AND mua.action = :action)
                                             WHERE mua.id is NULL
                                               AND mcm.userid = :userid2
                                          GROUP BY m.conversationid
                                       ) maxmessage
                                     ON maxmessage.maxtime = m.timecreated AND maxmessage.conversationid = m.conversationid
                               GROUP BY m.conversationid
                             ) lastmessage
                          ON lastmessage.conversationid = mc.id
                  LEFT JOIN {messages} m
                         ON m.id = lastmessage.messageid
                  LEFT JOIN {message_conversation_actions} mca
                         ON (mca.conversationid = mc.id AND mca.userid = :userid4 AND mca.action = :convaction)
                      WHERE mc.id IS NOT NULL
                        AND mc.enabled = 1 $typesql $favouritesql
                    ORDER BY (CASE WHEN m.timecreated IS NULL THEN 0 ELSE 1 END) DESC, m.timecreated DESC, id DESC";
              $params = array_merge($favouriteparams, 
      $typeparams, ['userid' => $userid, 'action' => 
      self::MESSAGE_ACTION_DELETED,
                  'userid2' => $userid, 
      'userid3' => $userid, 'userid4' => $userid, 'convaction' => 
      self::CONVERSATION_ACTION_MUTED]);
              $conversationset = $DB->get_recordset_sql($sql, $params, $limitfrom, $limitnum); 

      Analysing this query, on the following code fragment appears the problem:

      SELECT m.conversationid, MAX(m.timecreated) as maxtime
      FROM {messages} m
      INNER JOIN {message_conversation_members} mcm
      ON mcm.conversationid = m.conversationid
      LEFT JOIN {message_user_actions} mua
      ON (mua.messageid = m.id AND mua.userid = :userid AND mua.action = :action)
      WHERE mua.id is NULL
      AND mcm.userid = :userid1
      GROUP BY m.conversationid 

      The LEFT JOIN using mua.id IS NULL means that you need to get all the messages that have not been DELETED.

      My proposal is change this LEFT JOIN with mua.IS NULL for a WHERE IF EXISTS to avoid all DELETED messages. With this change, it looks like:

      SELECT m.conversationid, MAX(m.timecreated) as maxtime
         FROM {messages} m
      INNER JOIN {message_conversation_members} mcm
                     ON mcm.conversationid = m.conversationid AND mcm.userid = :userid1
      WHERE NOT EXISTS (
               SELECT 1
                 FROM {message_user_actions} mua
                WHERE mua.messageid = m.id
                  AND mua.userid = :userid
                  AND mua.action = :action
             )
             GROUP BY m.conversationid 

      The changes are:

      1. Drop LEFT and change for WHERE NOT EXISTS
      2. Move mcm.userid to ON clause.

      Regards

            Unassigned Unassigned
            tmas Toni Mas
            Votes:
            46 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.