How to Avoid Missing Target Network Requests When Scraping Google Maps

Regarding how to avoid missing the target network requests when scraping Google Maps, we often encounter a situation where, after opening the page with Puppeteer, the desired data cannot be captured, leading to failed interception.
Why Does This Happen?
This occurs due to the "open page first, then listen" pattern. A timeout can happen where the page loads and potentially closes the connection for the required data before the listener is actively set up to capture it.
How to Solve It?
The solution is to switch to a "listen first, then open the page" pattern. This ensures the listener is active the moment the page starts loading, effectively preventing the scenario where the desired data is missed.
How to Modify the Code?
Set up the listener first:
Call the page.waitForResponse() function and store its Promise.
Then navigate to the webpage:
Call page.goto() to open the page and navigate to the target URL.
Wait for the listener to complete:
Await the resolution of the stored Promise.
Code Example:
1.Before navigating to the target page, set up the listener for the network request.

console.log(`[${searchQuery}] Task started，Setting up API response listener...`);  	
const responsePromise = page.waitForResponse(res => res.url().includes(apiQueryFragment));

2.Navigate to the target page.

console.log(`[${searchQuery}] Navigating to: ${postUrl.substring(0, 80)}...`);
await page.goto(postUrl, { waitUntil: "networkidle2" });

3.Wait for the result; allow the Promise (listener) to complete.

console.log(`[${searchQuery}] Page loaded, waiting for API data to return....`);
const response = await responsePromise;
console.log(`[${searchQuery}]API response has been intercepted!`);