How to build an update CLI script without memory problem ?

Tuesday 05 May 2009 2:16:31 am

Hi Hakim,

You can set memory limit on execution like this:

php -d memory_limit=128M yourscript.php

http://www.linkedin.com/in/ivolukac
http://www.netgen.hr/eng/blog
http://twitter.com/ilukac

Tuesday 05 May 2009 3:55:16 am

Hi,

You are fetching ALL users objects from DB without limit. I would rebuild your script to fetch e.g by 25 users using limit and offset. That should do the trick.

Personal website -> http://serwatka.net
Blog (about eZ Publish) -> http://serwatka.net/blog

Friday 08 May 2009 4:34:33 am

Yes you are right, for large fetches ez publish do not load data_map for object during treenode fetch. and loads it only when you ask for it first time

Friday 08 May 2009 11:58:57 am

I've ran into the same problem some days ago. adding more "memory-limit" to the script could do the trick, but it's not the best solution.

my situation is like yours. we have to export some user data to cvs files. we started with 128M but when more users registered into the site, the problems started. now we have more than 65.000 registered users and even if you set memory-limit to 1GB the memory get exhausted.

and it seems that the problem is not really an eZ one but a php problem with the memory management.

i suggest to you do this. add a

$cli->output(memory_get_usage()) ;

at the end of every loop. you will see how this is value is increased every step. i mean if you expect a lot of users in your site next months, you'll probably get the problem again.

Event if you unset all variables used, no matter if you change your foreach to for $i=0 ; $i<count($users)... memory will get exhausted.

in a ideal world, php should free memory in every step, but it seems that it's not doing well.

My workaround for this was to build some kinda batch process.

i have two scripts, script1.php and script2.php.

script1 is the one executed with the cron. it it something like

//get the total of users

do 
{
//stuff here
} while $offset < total of users

in every loop script queries a record of a dummy table. at the first step that record is 0.

script1 get that value and pass it to the script to via exec command. i mean something like

exec("script2.php --offset=$offset limit=100);

script2 get the offset param and do the subTreeByNodeId fetch.
when offset is 0, it opens a file to write the results. when is not 0, it opens it for adding results.

when script2 end, it updates the dummy table and set the value to value + 100
script 1 get this value and pass it to script to until the value of the table is bigger than the total users we have.

hope it helps

Monday 01 June 2009 11:49:10 am

Sorry, but this is not working for me. i just tried the next piece of code. it would be nice if anyone can do the same and tell everybody about the results

$nodes = eZContentObjectTreeNode::subTreeByNodeId( array("Limit" => 100) , 2 );    
for ( $i = 0; $i < count( $nodes ); $i++ )
{
    $data = $nodes[$i]->dataMap();
    echo memory_get_usage() . "<br />";    
    unset( $node );      
    unset( $GLOBALS['eZContentObjectContentObjectCache'] );
    unset( $GLOBALS['eZContentObjectDataMapCache'] );
    unset( $GLOBALS['eZContentObjectVersionCache'] );
}

As just see, i'm "unsetting" everything in each loop. but that call to echo memory_get_usage();
shows something like

7211636
7214944
7237280
7268092
7328240
7373416
7390356
7457684
7500012
7516584
...

Every time memory usage is increased. It's easy to think that if limit is big enough, memory will get exhausted...

Tuesday 02 June 2009 1:22:29 am

<i>PHP is not perfect and memory leak happens very offen</i>

Sorry, but this is just not true.

"Memory leaks" happen because
- variables are still referenced somewhere and thus not destroyed
- there are circular variable references
- there is an aggressive caching strategy

all of this is part of the application in question, not in the engine itself.

The upside is that it is extremely easy in php to get a dump of the complete list of vars in memory. You can then split up your code in some looping construct, and every 10, 100 or 1000 iterations dump memory contents to a file. After running the script you can compare the dumps and find the runaway variable.

I did once and found out that I was simply appending a value to an array that I forgot to reset at every iteration step. Easy to miss, and would only give problems when run 100.000 times in a row...

Principal Consultant International Business
Member of the Community Project Board

Tuesday 02 June 2009 10:02:09 am

don't forget to clear the user cache
unset( $GLOBALS["eZUserObject_".$userID] );

on ez4.x before the end of the foreach add
eZContentObject::clearCache();
unset( $GLOBALS["eZUserObject_".$userID] );

http://www.kouz-cooking.fr

Wednesday 03 June 2009 3:12:56 am

If you need to extract a lot of data this will work

<?php 
$userId = eZUser::currentUser()->id();  
$limit=1000;
$offset=0;
do{
$nodes = &eZContentObjectTreeNode::subTreeByNodeId( array("Limit" => $limit,'Offset',$offset) , 2 );     

foreach($nodes as $node){
       
$data = &$node->dataMap();
eZContentObject::clearCache();    
unset( $GLOBALS['eZContentObjectContentObjectCache'] );    
unset( $GLOBALS['eZContentObjectDataMapCache'] );    
unset( $GLOBALS['eZContentObjectVersionCache'] );    
unset( $GLOBALS['eZContentClassAttributeCache'] );    
unset( $GLOBALS["eZUserObject_".$node->ContentObjectID] );  
unset( $data ); 
} 

$offset+=$limit;

echo memory_get_usage() . "\n"; 
}while( !empty( $nodes) );

?>

mem usage
59751392
59814408
59946280
59950608
59946168
59952120
59957768
59936880

http://www.kouz-cooking.fr

Tuesday 17 November 2009 1:31:41 am

Just a fix in Stéphane's code

$nodes = &eZContentObjectTreeNode::subTreeByNodeId( array("Limit" => $limit,'Offset',$offset) , 2 );

>

$nodes = &eZContentObjectTreeNode::subTreeByNodeId( array("Limit" => $limit,'Offset' =>$offset) , 2 );

Wednesday 18 November 2009 3:16:58 am

Interesting topic, as I faced this problem myself several times.

What about introducing a specialized implementation of PHP's Iterator interface that hides all the logic of fetching with offset and limit and clearing caches?

The example below shows how the name is displayed of all folders under node 2. After each batch of 100 nodes the iterator takes care of clearing the in-memory caches and fetches the next 100 nodes from the database.

$params = array( 'ClassFilterType' => 'include', 'ClassFilterArray' => array( 'folder' ) );

$nodeList = new eZContentObjectTreeNodeSubTreeIterator( 2, $params, 100 );

foreach ($nodeList as $node)
{
    echo $node->attribute('name');
}

independent eZ Publish developer and service provider | http://blog.coomanskristof.be | http://ezpedia.org