Yii - Looping through a model in chunks

Or Simple Pagination

I recently needed to run through all records in a table and recalculate some variables.  However, the lovely php was giving me a memory exceeded error after about 2000 records.

I assumed that all the model objects were not being cleared down properly, so I wanted to try and process batches of records and then use unset after each batch to clear down all the model entries.

But I struggled to find out how to simple implement the MySQL "LIMIT from,chunk" syntax.

The solution, as often, was quite simple.

$chunk=1000; for ($i = 0; $i <= $total; $i=$i+$chunk) { $criteria=new CDbCriteria; $criteria->offset=$i; $criteria->limit=$chunk; $recs=Article::model()->findAll($criteria); foreach ($recs as $rec) { /** Processing here **/ } unset ($rec); unset ($recs); }

Note:  It didn't actually get rid of my memory exceeded messages but I could process many more records!

Or a more compact way

for ($i = 0; $i < Table::model()->count(); $i += $chunkSize) { foreach (Yii::app()->db->createCommand() ->from('table') ->limit($chunkSize) ->offset($i) ->queryAll() as $item) { // process here } } thanks to mmshfe for the contribution

Did you know you can hire me?

I take on projects of all sizes. From Consulting to large Development Projects.

If you're starting a new Yii project and would like some help to get setup and running or you need some help with a particular module or you just need someone to develop the whole dang thing, then just ask ...


Leave a Comment

twitterfacebookgooglelinkedin https://me.yahoo.com