Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-30439

Restore runs out of memory

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Won't Fix
    • Icon: Critical Critical
    • None
    • 1.9.14, 2.1.2
    • Backup
    • MOODLE_19_STABLE, MOODLE_21_STABLE
    • Hide

      lib/xmlize.php

      // near the end of function xmlize() - change it to use result of xml_depth() by reference
          $array[$tagname]["#"] =& xml_depth($vals, $i);
          return $array;
       
      // Then here is a more memory-efficient xml_depth():
      /**
       * @internal You don't need to do anything with this function, it's called by
       * xmlize. It's a recursive function, calling itself as it goes deeper
       * into the xml levels.  If you make any improvements, please let me know.
       * @param array $vals array of associative arrays, one per XML element. See 3rd parameter of PHP's xml_parse_into_struct().
       * This function removes entries from $vals which were processed already, to save memory.
       * @param int $i 0-based index in $vals, where processing starts
       * @param int $valssize Max. index in $vals + 1. That will differ from count($vals)
       * after the 1st call to xml_depth() for given $vals, since it removes entries from $vals as it goes.
       * It's optional only when calling xml_depth() for the first time at the uppermost
       * level (first time for given $vals). Basically, it can be set to size of $vals only before
       * $vals was processed by any previous/upper calls to xml_depth().
       * @return array of mixed-level structure, as used by Moodle restore
       * @access private
       */
      function &xml_depth(&$vals, &$i, $valssize=-1) {
          if( $valssize<0 ) {
              $valssize= count($vals);
          }
          $children = array();
       
          if ( isset($vals[$i]['value']) )
          {
              array_push($children, $vals[$i]['value']);
          }
       
          while (++$i <$valssize ) {
       
              switch ($vals[$i]['type']) {
                 case 'open':
                      if ( isset ( $vals[$i]['tag'] ) )
                      {
                          $tagname = $vals[$i]['tag'];
                      } else {
                          $tagname = '';
                      }
       
                      if ( isset ( $children[$tagname] ) )
                      {
                          $size = sizeof($children[$tagname]);
                      } else {
                          $size = 0;
                      }
       
                      if ( isset ( $vals[$i]['attributes'] ) ) {
                          $children[$tagname][$size]['@'] = $vals[$i]["attributes"];
       
                      }
       
                      $children[$tagname][$size]['#'] =& xml_depth($vals, $i, $valssize);
                      break;
       
                  case 'cdata':
                      array_push($children, $vals[$i]['value']);
                      break;
       
                  case 'complete':
                      $tagname = $vals[$i]['tag'];
       
                      if( isset ($children[$tagname]) )
                      {
                          $size = sizeof($children[$tagname]);
                      } else {
                          $size = 0;
                      }
       
                      if( isset ( $vals[$i]['value'] ) )
                      {
                          $children[$tagname][$size]["#"] = $vals[$i]['value'];
                      } else {
                          $children[$tagname][$size]["#"] = '';
                      }
       
                      if ( isset ($vals[$i]['attributes']) ) {
                          $children[$tagname][$size]['@']
                                                   = $vals[$i]['attributes'];
                      }
                      break;
       
                  case 'close':
                      unset( $vals[$i] );
                      return $children;
              }
              unset( $vals[$i] );
          }
          return $children;
      }

      Show
      lib/xmlize.php // near the end of function xmlize() - change it to use result of xml_depth() by reference $array[$tagname]["#"] =& xml_depth($vals, $i); return $array;   // Then here is a more memory-efficient xml_depth(): /** * @internal You don't need to do anything with this function, it's called by * xmlize. It's a recursive function, calling itself as it goes deeper * into the xml levels. If you make any improvements, please let me know. * @param array $vals array of associative arrays, one per XML element. See 3rd parameter of PHP's xml_parse_into_struct(). * This function removes entries from $vals which were processed already, to save memory. * @param int $i 0-based index in $vals, where processing starts * @param int $valssize Max. index in $vals + 1. That will differ from count($vals) * after the 1st call to xml_depth() for given $vals, since it removes entries from $vals as it goes. * It's optional only when calling xml_depth() for the first time at the uppermost * level (first time for given $vals). Basically, it can be set to size of $vals only before * $vals was processed by any previous/upper calls to xml_depth(). * @return array of mixed-level structure, as used by Moodle restore * @access private */ function &xml_depth(&$vals, &$i, $valssize=-1) { if( $valssize<0 ) { $valssize= count($vals); } $children = array();   if ( isset($vals[$i]['value']) ) { array_push($children, $vals[$i]['value']); }   while (++$i <$valssize ) {   switch ($vals[$i]['type']) { case 'open': if ( isset ( $vals[$i]['tag'] ) ) { $tagname = $vals[$i]['tag']; } else { $tagname = ''; }   if ( isset ( $children[$tagname] ) ) { $size = sizeof($children[$tagname]); } else { $size = 0; }   if ( isset ( $vals[$i]['attributes'] ) ) { $children[$tagname][$size]['@'] = $vals[$i]["attributes"];   }   $children[$tagname][$size]['#'] =& xml_depth($vals, $i, $valssize); break;   case 'cdata': array_push($children, $vals[$i]['value']); break;   case 'complete': $tagname = $vals[$i]['tag'];   if( isset ($children[$tagname]) ) { $size = sizeof($children[$tagname]); } else { $size = 0; }   if( isset ( $vals[$i]['value'] ) ) { $children[$tagname][$size]["#"] = $vals[$i]['value']; } else { $children[$tagname][$size]["#"] = ''; }   if ( isset ($vals[$i]['attributes']) ) { $children[$tagname][$size]['@'] = $vals[$i]['attributes']; } break;   case 'close': unset( $vals[$i] ); return $children; } unset( $vals[$i] ); } return $children; }
    • Hide

      Details:

      I had a backup of my course and 1 custom module, no users included; 1.1MB zipped, 13MB unzipped backup XML file. The custom module took most of the backup, based on data of scale c.a.
      2600 rows x 47 columns
      1500 rows x 6 columns
      2200 rows x 5 columns
      8400 rows x 14 columns

      The unzipped backup is a proper XML (it opens in browsers, and it gets restored properly once I applied the suggested fix).

      I've tried to restore the backup, and it failed - even when I indicated not to restore my custom module.

      Apache log contained:

      [Thu Nov 24 13:44:18 2011] [error] [client 10.10.6.1] PHP Fatal error:  Allowed memory size of 272233752 bytes exhausted (tried to allocate 37558705 bytes) in /var/www/html/pkehl/lalala/moodle/lib/xmlize.php on line 96, referer: http://server-name/pkehl/lalala/moodle/backup/restore.php

      On Linux 2.6.18, Apache/2.2.3 (CentOS), PHP 5.1.6 that caused out of diskspace problem. Apache wrote to Apache log file same stack trace repeatedly (c.a. 9GB log increase). That unrobust behaviour is probably an Apache bug, but it may make an existing Moodle installation unstable/sick. That's the reason why I request you include the fix for branch 1.9, too.

      I didn't test in 2.1.2 since I haven't got it installed, but the problem code is still there, so I expect it would run out of memory in the same configuration, too.

      I can't include my backup file since it's company data. If you really need one, I may try to generate a dummy one.

      Show
      Details: I had a backup of my course and 1 custom module, no users included; 1.1MB zipped, 13MB unzipped backup XML file. The custom module took most of the backup, based on data of scale c.a. 2600 rows x 47 columns 1500 rows x 6 columns 2200 rows x 5 columns 8400 rows x 14 columns The unzipped backup is a proper XML (it opens in browsers, and it gets restored properly once I applied the suggested fix). I've tried to restore the backup, and it failed - even when I indicated not to restore my custom module. Apache log contained: [Thu Nov 24 13:44:18 2011] [error] [client 10.10.6.1] PHP Fatal error: Allowed memory size of 272233752 bytes exhausted (tried to allocate 37558705 bytes) in /var/www/html/pkehl/lalala/moodle/lib/xmlize.php on line 96, referer: http://server-name/pkehl/lalala/moodle/backup/restore.php On Linux 2.6.18, Apache/2.2.3 (CentOS), PHP 5.1.6 that caused out of diskspace problem. Apache wrote to Apache log file same stack trace repeatedly (c.a. 9GB log increase). That unrobust behaviour is probably an Apache bug, but it may make an existing Moodle installation unstable/sick. That's the reason why I request you include the fix for branch 1.9, too. I didn't test in 2.1.2 since I haven't got it installed, but the problem code is still there, so I expect it would run out of memory in the same configuration, too. I can't include my backup file since it's company data. If you really need one, I may try to generate a dummy one.

      In short: In lib/xmlize.php, function xml_depth() is memory-inefficient. If may run out of memory for a legitimate and realistic backup.

            stronk7 Eloy Lafuente (stronk7)
            peter.kehl Peter Kehl
            Votes:
            2 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.