How to open list of pages synchronously

Question

let downloadPageLinks = [];
fetchStreamingLinks.forEach(async (item) => {
    page = await browser.newPage();
    await page.goto(item, { waitUntil: "networkidle0" });
    const fetchDownloadPageLinks = await page.evaluate(() => {
        return loc4;
    });

    console.log(fetchDownloadPageLinks);
});

I have an array of links(fetchStreamingLinks). Above function opens all the links simultaneously present in fetchDownloadPageLinks. Suppose the array contains 100 links then it opens all the 100 links simultaneously.

Now what I want to do is, open all the links one by one present in fetchStreamingLinks, perform some logic in page context's and close it then open next link.

Does this answer your question? [Crawling multiple URLs in a loop using Puppeteer](https://stackoverflow.com/questions/46293216/crawling-multiple-urls-in-a-loop-using-puppeteer) — ggorlen, Sep 01 '22 at 20:46
Terminology nitpick: there's a difference between "synchronous", "sequential async tasks" and "parallel async tasks". What you're asking for here isn't synchronous (async vs sync is something Puppeteer forces on you by presenting an API), it's sequential async tasks that run one after the other. [Crawling multiple URLs in a loop using Puppeteer](https://stackoverflow.com/questions/46293216/crawling-multiple-urls-in-a-loop-using-puppeteer) should explain everything you need. — ggorlen, Sep 01 '22 at 21:40

score 2 · Answer 1 · edited Sep 01 '22 at 21:08

2

.forEach() is not promise-aware so when you pass it an async callback, it doesn't pay any attention to the promise that it returns. Thus, it runs all your operations in parallel. .forEach() should be essentially considered obsolete these days, especially for asynchronous operations because a plain for loop gives you so much more control and is promise-aware (e.g. the loop will pause for an await).

let downloadPageLinks = [];
for (let item of fetchStreamingLinks) {
    let page = await browser.newPage();
    await page.goto(item, { waitUntil: "networkidle0" });
    const fetchDownloadPageLinks = await page.evaluate(() => {
        return loc4;
    });
    await page.close();
    console.log(fetchDownloadPageLinks);
}

FYI, I don't know the puppeteer API really well, but you probably should close the page (as I show) when you're done with it to avoid pages stacking up as you process.

edited Sep 01 '22 at 21:08

Ouroborus

16,237
4
39
62

answered Sep 01 '22 at 19:47

jfriend00

683,504
96
985
979

I disagree with your assertion that `.forEach` is obsolete. Once upon a time `for` loops were all there was. The addition of `.forEach` and related (hereafter just `.forEach`) has done much to simplify code, making it easier to read and less prone to bugs. (Though I say this in the face of the common mistake of trying to use `async` functions with `.forEach` but I feel that's more about not understanding how `async`/`await` works than any fault with `.forEach`.) – Ouroborus Sep 01 '22 at 21:15
@Ouroborus - You can use `.forEach()` if you want, but `for` has so many advantages over `.forEach()` including loop control (such as `break`, `continue`, `return`) which `.forEach()` has none of, easier for the compiler to optimize because there's no extra function, `await` support to suspend the loop iteration, regular iterator support, async iterator support, and so on... I never use `.forEach()` anymore and my code is better for it. I just find a `for` loop to be easier, more powerful and more flexible. – jfriend00 Sep 01 '22 at 21:25
If you want loop control, use `.some` or `.every`. – Ouroborus Sep 01 '22 at 21:27
@Ouroborus - Now you're reaching a bit. Those are not even close to general purpose loop control. Try returning from the parent function from within one of those callbacks. Those are special cases for certain purposes only. And, I don't see why I need a whole bunch of different methods to accomplish what a single loop can already do. And, without creating callback functions and with full iterator support and with full loop control and with full support of `await`. You're welcome to your opinion if you want, but I don't use `.forEach()` any more. – jfriend00 Sep 01 '22 at 21:31

How to open list of pages synchronously

1 Answers1