Multiple foreach loops exhausting memory for large data

Question

We might consider foreach to be painful and a headache while using it to deal with thousands or millions of records. Even more so if we have a nested foreach.

For example,

foreach ($parent as $parentData) {
    // few conditional added here
    $parentObj = $this->extract();
    foreach ($chilren as $childrenData) {
        if ($childrenData['id'] === $parentData['id']) {
            $childrenObj = $this->extract();
            $parentObj->setData($childrenObj);
            // and even more evil things come here....
        }
    }
    $parentObj->save();
}

In my situation, I have two foreach loops. And each one contains approximate 50,000 ~ 70,000 records. $parent and $children are parameters passed to the method.

The raw data source both of $parent and $children are CSV files. And I'm using yield to transform them to be traversable with foreach.

There is no trouble with yield.

If you concern about the code: https://stackoverflow.com/a/37342184/2932590

I tried to unset both of $parentObj and $childrenObj at the end of first foreach, but unfortunately it's not working. I also tried to use the references &$parentData, but the result is same.

How could I make this work?

I was advised to use SPL Iterators in this case. Can anyone please explain me how it works?

@hkBst: Using multiple foreach and save objects inside foreach and PHP will throw error: Allowed memory exhausted.... — Toan Nguyen, Jun 01 '16 at 03:39
So now that you are reading the data row by row, are you still trying to reconstruct the entire dataset in your $parentObj? — hkBst, Jun 01 '16 at 06:30
@hkBst: Yes. I updated my new code. Check the question updates. — Toan Nguyen, Jun 01 '16 at 07:02
what does/should `$this` refer to in each case and what does `save()` do? — hkBst, Jun 01 '16 at 07:48
@hkBst: I won't reveal the whole code. `$this` refer to current class as it always be. `save()` will save the data to the database. — Toan Nguyen, Jun 01 '16 at 10:30
Can't you use the database to define the parent-child relationship and query the related data you need? — Kwebble, Jun 04 '16 at 21:13

score 0 · Answer 1 · answered May 29 '16 at 08:29

0

To use SPL iterator:

// create a new ArrayIterator and pass in the array
$parentDataIter = new ArrayIterator($parentData);

// loop through the object one item at a time memory-wise
foreach ($parentDataIter as $key => $value) {
    // ...
}

answered May 29 '16 at 08:29

j4hangir

2,532
1
13
14

1

I'm not asking for this. Btw, your code might not work without `IteratorIterator` – Toan Nguyen Jun 03 '16 at 10:21

score 0 · Answer 2 · edited May 23 '17 at 12:18

0

SPL iterators are not a uniform thing but just the collective name for them (Standard PHP Library iterators). The ones available works quite differently but the following stack-overflow thread has already tried to explain their uses and pros/cons quite well. PHP - Reasons to use Iterators?.

edited May 23 '17 at 12:18

Community

1
1

answered May 31 '16 at 12:06

Björn Fyrvall

231
1
7

I need to reduce my memory usage. That is why I choose Iterators, and yes I read that question before create this one. – Toan Nguyen Jun 03 '16 at 10:20

altcatalin · Answer 3 · 2016-06-01T19:19:07.583

0

probably, $parentObj = $this->extract(); doesn't use a reference and $parentObj->setData($childrenObj); makes PHP to create a copy from initial array for each element that is changed.

another approach is to move the pointer in a while loop:

reset($array);
while (list($key, $val) = each($array)) {
    $array[$key] = ++$val;
    echo "$array[$key]\n";
}

edited Jun 01 '16 at 19:19

answered Jun 01 '16 at 18:54

altcatalin

11
3

based on the information you provided it seems like there is a problem with the way you update an array - [explanation](http://stackoverflow.com/a/14854568/1539119). for a better response, we need a more details about these methods $this->extract(), $parentObj->setData($childrenObj), `code`$parentObj->save()`code` at least in pseudocode – altcatalin Jun 07 '16 at 20:50

Multiple foreach loops exhausting memory for large data

3 Answers3