Puppeteer Interview Questions and Answers

Answer

The puppeteer package is puppeteer-core plus an installer. On npm install it runs @puppeteer/browsers and downloads a pinned Chrome for Testing build into a cache directory outside node_modules, typically ~/.cache/puppeteer on Linux and macOS. puppeteer-core exposes the identical API but downloads nothing and refuses to launch unless you give it an executablePath or a channel, which is what you want inside a Docker image that already installed Chromium through apt, or on a machine where a platform team pins the browser version centrally. Three environment variables control the behaviour: PUPPETEER_SKIP_DOWNLOAD=true skips the postinstall fetch, PUPPETEER_CACHE_DIR relocates the cache, and PUPPETEER_EXECUTABLE_PATH points at a system binary.

The version-controlled equivalent is a .puppeteerrc.cjs file at the repo root, which is the right answer in a team setting because every developer and every CI runner picks it up automatically. You can also install browsers explicitly with npx puppeteer browsers install chrome. The follow-up interviewers almost always ask: why does the same script work on a laptop and fail in the container with 'Could not find Chrome (ver. 1xx.x.xxxx.xx)'?

Nearly always because the download was skipped, or because a multi-stage Dockerfile downloaded the browser as root into /root/.cache during build and the runtime stage runs as a non-root user who cannot read it. Setting PUPPETEER_CACHE_DIR to a shared path such as /opt/puppeteer in both stages fixes it permanently.

// .puppeteerrc.cjs (repo root, picked up by every install)
const { join } = require('path');

module.exports = {
  cacheDirectory: join(__dirname, '.cache', 'puppeteer'),
};

// Dockerfile: use the distro browser with puppeteer-core
// RUN apt-get install -y chromium
// ENV PUPPETEER_SKIP_DOWNLOAD=true
// ENV PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium

import puppeteer from 'puppeteer-core';

const browser = await puppeteer.launch({
  executablePath: process.env.PUPPETEER_EXECUTABLE_PATH,
  args: ['--no-sandbox', '--disable-dev-shm-usage'],
});

Answer

This is the single most common version-knowledge question. Historically headless: true launched the old headless implementation, a separate lightweight binary path that shared limited code with the browser real users run, while headless: false launched full Chrome with a window. Puppeteer v18 added headless: 'new' as an opt-in to Chrome's new headless mode, which is the same browser binary running without a visible window, so extensions, permissions, printing and the renderer behave the same as headful.

In v22 the defaults flipped: headless: true now means new headless, headless: 'new' became a deprecated alias, and the old implementation is available only as headless: 'shell', which pulls a separate chrome-headless-shell binary. Practically, the shell starts faster and uses less memory, so high-volume rendering and scraping fleets still deliberately choose it, but it does not support extensions and differs from real Chrome in a few print and media paths. If a suite started failing or got slower immediately after a Puppeteer upgrade, this default change is the first thing to check.

One more detail interviewers like: new headless is not stealth. It still reports navigator.webdriver as true and, unless you override it with setUserAgent, its user agent string still contains the HeadlessChrome token, so any site doing basic automation detection sees it.

import puppeteer from 'puppeteer';

// Real Chrome, no window (default since v22)
const modern = await puppeteer.launch({ headless: true });

// Old lightweight implementation, opt-in only
const shell = await puppeteer.launch({ headless: 'shell' });

// Debugging locally
const visible = await puppeteer.launch({
  headless: false,
  slowMo: 100,
  devtools: true,
});

const page = await modern.newPage();
console.log(await modern.version());
console.log(await page.evaluate(() => navigator.userAgent));

Answer

A Browser is one Chrome process (plus its renderer children) that Puppeteer either launched or connected to. Inside it sit BrowserContexts, which are isolated cookie jars, storage partitions and permission sets, the programmatic equivalent of separate profiles. Every browser has one default context, and you create extra ones with browser.createBrowserContext().

Recent versions renamed this from createIncognitoBrowserContext, so older tutorials and Stack Overflow answers will throw 'browser.createIncognitoBrowserContext is not a function' on a current install. Inside a context sit Pages, which map to tabs. Inside a page sits a tree of Frames: page.mainFrame() plus one Frame per iframe, reachable through page.frames() or elementHandle.contentFrame().

Finally, ElementHandle is a Node-side reference to a live DOM node in a specific frame's JavaScript execution context. The practical reason to know this hierarchy is isolation and cost. Launching a browser costs hundreds of milliseconds and 80 to 150 MB of resident memory before you open a single tab, so spinning one up per request is the classic beginner mistake in a rendering service.

Creating a context is nearly free and gives you a clean session, which is what you want when running parallel tests that must not share a login cookie. Contexts also let you close everything a job created in one call: context.close() disposes all its pages, which is far more reliable than tracking page handles yourself. Remember that browser.close() kills the process while browser.disconnect() leaves it running, which matters when you connect to a shared remote browser.

const browser = await puppeteer.launch();

// Isolated session per job: separate cookies and storage
const context = await browser.createBrowserContext();
const page = await context.newPage();

await page.goto('https://example.com');
console.log(page.frames().length); // main frame + iframes

const input = await page.$('input[name=q]'); // ElementHandle
await input?.type('puppeteer');
await input?.dispose();

await context.close(); // closes every page in this context
await browser.close(); // kills the Chrome process

Answer

page.goto resolves when the chosen lifecycle event fires. domcontentloaded resolves when the HTML is parsed, before stylesheets, images and most scripts have run. load waits for the load event, so subresources referenced in the initial HTML are done. networkidle0 waits until there have been zero network connections for 500 ms, and networkidle2 until there are no more than two, which exists because analytics beacons and long-polling connections keep a couple of sockets alive on many real sites. The default is load. Every one of these can lie.

On a React or Angular single-page app, load fires when the empty shell and the bundle are fetched, long before the app has rendered anything, so a script that screenshots right after goto captures a blank page. networkidle0 fails the other way on sites with a websocket, an SSE stream, a chat widget or a poller, because the connection count never reaches zero and you eat the full 30 second navigation timeout, surfacing as 'Navigation timeout of 30000 ms exceeded'. The reliable pattern in production is to navigate on domcontentloaded and then wait for something semantic: a selector that only exists once data has rendered, a specific API response through page.waitForResponse, or a predicate through page.waitForFunction. Interviewers ask this because candidates who reach for networkidle0 everywhere have usually never run a scraper against a site with a live chat widget.

// Fragile on an SPA: load fires before data renders
await page.goto(url, { waitUntil: 'load' });

// Reliable: navigate fast, then wait for real content
await page.goto(url, { waitUntil: 'domcontentloaded', timeout: 45_000 });
await page.locator('[data-testid=order-total]').wait();

// Or wait on the API call the UI actually depends on
const [res] = await Promise.all([
  page.waitForResponse(
    (r) => r.url().includes('/api/orders') && r.status() === 200,
  ),
  page.click('#load-orders'),
]);
console.log((await res.json()).length);

page.setDefaultNavigationTimeout(45_000);

Answer

page.$(selector) runs document.querySelector inside the page and returns a Promise of an ElementHandle or null. page.$$(selector) is querySelectorAll and returns an array of handles. Both hand you Node-side references to live DOM nodes, which means every interaction after that is a round trip over the DevTools Protocol, and every handle you do not dispose keeps the corresponding renderer object alive. page.$eval(selector, fn) and page.$$eval(selector, fn) are different: they run your function inside the browser with the matched element (or array of elements) as the first argument, and return only the serialized result. Nothing crosses the boundary except JSON-compatible data, so there is no handle to leak and only one protocol round trip.

The rule of thumb is simple. If you need to interact with an element (click, type, hover, upload, screenshot a clip), you need a handle, so use $ or better a Locator. If you only need to read data out (text, attributes, a table's contents), use $eval or $$eval and do the extraction in one pass in the browser.

Scrapers that pull a hundred rows by looping over page.$$ and calling evaluate on each handle are typically ten to fifty times slower than the equivalent single $$eval, and that performance difference is a favourite interview probe. Two traps: page.$ returns null rather than throwing when nothing matches, so null checks are mandatory, and $$eval throws if the selector matches zero elements only in the sense that your callback receives an empty array, which silently produces empty output.

// Slow: one protocol round trip per row
const rows = await page.$$('table tbody tr');
const slow = [];
for (const row of rows) {
  slow.push(await row.evaluate((el) => el.innerText));
  await row.dispose();
}

// Fast: one round trip, extraction happens in the browser
const fast = await page.$$eval('table tbody tr', (trs) =>
  trs.map((tr) => ({
    name: tr.querySelector('.name')?.textContent?.trim() ?? '',
    amount: Number(tr.dataset.amount ?? 0),
  })),
);

const title = await page.$eval('h1', (el) => el.textContent);
const missing = await page.$('.does-not-exist'); // null, not a throw

Answer

page.evaluate takes your function, converts it to a string, and ships it to the browser where Chrome compiles and runs it in the page's JavaScript context. That context is a different process from your Node script, so nothing from the surrounding Node scope exists there. A closure variable like a config object or a filename is simply undefined inside the callback, producing the classic 'ReferenceError: myVar is not defined' that every Puppeteer beginner hits.

You pass data explicitly as extra arguments after the function, and those arguments are serialized: JSON-compatible values, plus ElementHandle and JSHandle objects which are unwrapped into the real DOM node or object on the other side. Anything that cannot be serialized breaks. Functions become undefined, Dates come back as ISO strings unless you rehydrate them, Map and Set flatten to empty objects, circular references throw, and undefined inside an object is dropped.

The return value travels back the same way, so returning a DOM element from evaluate gives you an empty object; use evaluateHandle if you want a reference instead. The other half of the boundary is exposeFunction, which installs a Node-backed function on the page's window object so browser code can call back into your script, useful for streaming results out of an infinite-scroll page or for logging. Interviewers probe this because it separates people who copied a snippet from people who understand that two processes are involved.

const minPrice = 500;

// Wrong: minPrice does not exist in the browser context
// await page.evaluate(() => document.querySelectorAll(minPrice));

// Right: pass it explicitly, it gets serialized
const items = await page.evaluate((min) => {
  return [...document.querySelectorAll('.product')]
    .map((el) => ({
      title: el.querySelector('h3')?.textContent ?? '',
      price: Number(el.dataset.price ?? 0),
    }))
    .filter((p) => p.price >= min);
}, minPrice);

// Browser calls back into Node
await page.exposeFunction('reportRow', (row) => console.log('got', row));
await page.evaluate(() => window.reportRow({ ok: true }));

// Returning a DOM node needs evaluateHandle, not evaluate
const bodyHandle = await page.evaluateHandle(() => document.body);
await bodyHandle.dispose();

Answer

page.waitForTimeout() was deprecated and then removed in the v22 line, alongside the XPath helpers page.$x() and page.waitForXPath(). The Puppeteer team removed it deliberately because it made flakiness easy to write: a fixed sleep is either too short on a loaded CI runner, where it produces intermittent failures nobody can reproduce locally, or too long everywhere else, where it silently adds minutes to a suite. There is no correct number.

The replacements are all condition-based. waitForSelector waits for a node to appear, optionally with visible: true or hidden: true. waitForFunction polls an arbitrary predicate inside the page, which is what you use for 'the spinner is gone and the table has more than zero rows'. waitForResponse and waitForRequest wait on specific network traffic. waitForNetworkIdle waits for quiet. Locators, the recommended API in recent versions, fold waiting into the action itself, so page.locator('button.pay').click() waits for the element to exist, be visible, be stable (not animating) and be enabled before clicking, and retries the whole thing until the timeout. If you genuinely need a delay, for example to respect a crawl rate limit, use a plain Node timer and make it obvious that it is a politeness delay, not a synchronisation mechanism. Interviewers ask this specifically to see whether a candidate's scripts are timing-based or state-based, because the first kind never survives contact with a shared CI machine.

// Removed in v22: await page.waitForTimeout(3000);

// State-based waits
await page.waitForSelector('.results-table tr', { visible: true, timeout: 15_000 });

await page.waitForFunction(
  () =>
    !document.querySelector('.spinner') &&
    document.querySelectorAll('.results-table tr').length > 0,
  { polling: 'mutation', timeout: 15_000 },
);

// Locators wait, retry and check actionability for you
await page.locator('button.pay').click();

// Deliberate politeness delay between crawl requests
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
await sleep(1200);

Answer

page.screenshot({ fullPage: true }) captures the whole scrollable document rather than just the viewport. Useful options are type ('png', 'jpeg' or 'webp'), quality for the lossy formats, omitBackground: true to get a transparent PNG when the page has no opaque background, clip to capture a rectangle, and path to write straight to disk. For a single component, elementHandle.screenshot() is better than computing a clip yourself because Puppeteer scrolls the element into view and measures its bounding box for you.

The breaking change worth naming: in the v23 line page.screenshot() and page.pdf() return a Uint8Array instead of a Node Buffer. Code that did buf.toString('base64') or piped the result straight into an S3 upload that expects a Buffer will break, and the fix is Buffer.from(bytes). Practical gotchas dominate this topic. fullPage on a page with position: fixed headers repeats the header down the image on some layouts.

Lazy-loaded images below the fold are blank unless you scroll the page first or force loading. Very tall pages hit Chrome's texture size limit and come back truncated or blank, so for anything past roughly 16000 pixels you should capture in slices or switch to PDF. And on CI, screenshots differ from a developer machine because the container lacks the fonts the design uses, which is why a fonts package in the image is part of every serious visual-testing setup.

await page.setViewport({ width: 1440, height: 900, deviceScaleFactor: 2 });
await page.goto(url, { waitUntil: 'domcontentloaded' });

// Force lazy images to load before a full-page capture
await page.evaluate(async () => {
  await new Promise((resolve) => {
    let y = 0;
    const step = setInterval(() => {
      window.scrollBy(0, 600);
      y += 600;
      if (y >= document.body.scrollHeight) {
        clearInterval(step);
        window.scrollTo(0, 0);
        resolve(null);
      }
    }, 80);
  });
});
await page.evaluate(() => document.fonts.ready);

const bytes = await page.screenshot({ fullPage: true, type: 'webp', quality: 85 });
const buffer = Buffer.from(bytes); // v23+ returns Uint8Array

const card = await page.$('#invoice-card');
await card?.screenshot({ path: 'card.png', omitBackground: true });

Answer

page.pdf() drives Chrome's print pipeline, so it only works in headless mode; calling it on a headful browser throws. The options that matter are format ('A4' is the default for Indian invoicing and reporting work), printBackground: true (off by default, which is why the first PDF everyone generates has white boxes where the branded backgrounds should be), margin, and displayHeaderFooter with headerTemplate and footerTemplate. Those templates are small HTML fragments that must carry their own inline styles because page CSS does not apply to them, and they support the special classes pageNumber, totalPages, date, title and url.

Setting preferCSSPageSize: true makes Chrome honour an @page rule in the document instead of the format option, which is the cleaner approach when the design team already owns the print stylesheet. Two behaviours catch people out. First, Chrome applies print media by default, so any element hidden behind @media print disappears; call page.emulateMediaType('screen') if you want the on-screen rendering.

Second, page breaks are pure CSS: break-inside: avoid on table rows and cards, break-after: page on section boundaries, and a repeated thead for long tables. Also wait for fonts and web images before printing, because Chrome will happily print a document mid-load. In a rendering service, keep the HTML template local and inline the CSS and fonts as data URIs so a slow CDN cannot make an invoice render without its logo.

await page.setContent(html, { waitUntil: 'domcontentloaded' });
await page.evaluate(() => document.fonts.ready);
await page.emulateMediaType('screen');

const pdf = await page.pdf({
  format: 'A4',
  printBackground: true,
  margin: { top: '18mm', bottom: '18mm', left: '12mm', right: '12mm' },
  displayHeaderFooter: true,
  headerTemplate: '<div style="font-size:9px;width:100%;padding:0 12mm;color:#666">Tax Invoice</div>',
  footerTemplate:
    '<div style="font-size:9px;width:100%;padding:0 12mm;text-align:right;color:#666">' +
    'Page <span class="pageNumber"></span> of <span class="totalPages"></span></div>',
  timeout: 60_000,
});

// In the document CSS
// tr, .line-item { break-inside: avoid; }
// .terms { break-before: page; }
// thead { display: table-header-group; }

Answer

page.click(selector) does more than dispatch a synthetic event: it resolves the selector, scrolls the node into view, computes its centre point, and sends real CDP mouse events at those coordinates. That is why it fails in ways a JavaScript .click() never would. If a sticky header, a cookie banner, a chat bubble or a modal overlay covers the computed point, the click lands on the overlay.

If the element animates, the coordinates were measured before it settled. If the element exists but has zero size, Puppeteer throws 'Node is either not clickable or not an HTMLElement'. Options help: page.click(sel, { delay: 50 }) holds the button down, { button: 'right' } and { clickCount: 2 } cover context menus and double clicks.

For typing, page.type(sel, text, { delay: 30 }) fires keydown, keypress, input and keyup for each character, which matters on inputs with masking or autocomplete, while elementHandle.evaluate(el => el.value = 'x') bypasses React's synthetic event system and leaves component state stale. The modern answer is Locators, which check actionability (visible, stable, enabled, receives events) and retry until the timeout, and locator.fill() which picks the right strategy for inputs, textareas and select elements. When a click still refuses to land, the pragmatic escapes are elementHandle.evaluate(el => el.click()) for the DOM-level dispatch, or dismissing the overlay first, which is usually the real bug.

import { Locator } from 'puppeteer';

// Auto-waits for visible, stable, enabled before acting
await page.locator('#email').fill('candidate@example.com');
await page.locator('button[type=submit]').click();

// Character-by-character typing for masked or autocomplete inputs
await page.type('#otp', '482913', { delay: 40 });

// Whichever appears first wins, the other is aborted
await Locator.race([
  page.locator('#accept-cookies'),
  page.locator('#close-banner'),
]).click();

// Last resort when an overlay keeps eating the real mouse event
const el = await page.$('#hidden-cta');
await el?.evaluate((node) => node.click());

Answer

page.setViewport({ width, height, deviceScaleFactor, isMobile, hasTouch, isLandscape }) controls the rendered surface. deviceScaleFactor is the one people forget: at the default 1 your screenshots look soft next to a retina design review, and at 2 or 3 the image is sharp but four to nine times heavier, which matters when a service stores thousands of them. isMobile changes the meta viewport handling and hasTouch enables touch event support, so a responsive layout that keys off pointer capabilities behaves correctly. For full device emulation, recent versions export KnownDevices, which replaced the older puppeteer.devices export; import it and call page.emulate(KnownDevices['iPhone 15 Pro']) to set viewport, scale factor, touch and user agent together. Beyond the screen, Puppeteer can emulate other environment conditions that matter for testing Indian traffic patterns: page.emulateTimezone('Asia/Kolkata'), page.emulateMediaFeatures for prefers-color-scheme and prefers-reduced-motion, page.emulateCPUThrottling(4) to simulate a mid-range Android device, and network throttling through a CDP session using Network.emulateNetworkConditions.

Those two together are how you reproduce the 'works on my machine, unusable on a 4G phone in Jaipur' class of bug. A caveat worth stating in an interview: emulation is not a device. The rendering engine is still desktop Chrome, so Safari-specific and real GPU-specific bugs will not show up, and for those you need a real device cloud.

import puppeteer, { KnownDevices } from 'puppeteer';

const page = await browser.newPage();
await page.emulate(KnownDevices['iPhone 15 Pro']); // replaces puppeteer.devices

await page.emulateTimezone('Asia/Kolkata');
await page.emulateMediaFeatures([
  { name: 'prefers-color-scheme', value: 'dark' },
]);
await page.emulateCPUThrottling(4);

// Throttled network via raw CDP
const cdp = await page.createCDPSession();
await cdp.send('Network.emulateNetworkConditions', {
  offline: false,
  downloadThroughput: (1.6 * 1024 * 1024) / 8,
  uploadThroughput: (750 * 1024) / 8,
  latency: 150,
});

await page.setViewport({ width: 390, height: 844, deviceScaleFactor: 3, isMobile: true, hasTouch: true });

Answer

Chrome blocks the renderer when a native dialog opens, so if you do not handle it your script hangs until the timeout with a message like 'Navigation timeout of 30000 ms exceeded' that gives no hint about the real cause. Puppeteer surfaces them through the page-level 'dialog' event, and you must call either dialog.accept(promptText) or dialog.dismiss(). The Dialog object exposes type() ('alert', 'confirm', 'prompt' or 'beforeunload'), message() and defaultValue().

Register the listener before triggering the action, not after, because the event fires synchronously with the block. A subtlety that separates candidates: Puppeteer auto-dismisses dialogs only when no listener is attached, so adding a listener that conditionally does nothing on some dialogs will hang the page permanently. Handle every dialog your listener receives. beforeunload is a special case: it is only shown if the page has received a user gesture, and page.close({ runBeforeUnload: true }) is what triggers it during teardown, so leaving that option off is the simplest way to avoid unsaved-changes prompts blocking cleanup. In tests, the strongest pattern is to assert on the dialog message rather than blindly accepting, because 'a confirm appeared' and 'the correct confirm appeared' are different assertions and a silent accept will happily pass a broken flow.

page.on('dialog', async (dialog) => {
  console.log(dialog.type(), dialog.message());
  if (dialog.type() === 'prompt') {
    await dialog.accept('Saksham');
  } else {
    await dialog.accept();
  }
});

await page.click('#delete-account'); // fires confirm()

// Assert on the message instead of blindly accepting
const dialogMessage = await new Promise((resolve) => {
  page.once('dialog', async (d) => {
    resolve(d.message());
    await d.dismiss();
  });
  page.click('#risky-action');
});

// Skip a beforeunload prompt during teardown
await page.close({ runBeforeUnload: false });

Answer

Uploads are the easy half. Get an ElementHandle for the input and call elementHandle.uploadFile(absolutePath), which sets the files property and dispatches the change event the app listens for. It works even when the input is visually hidden behind a styled label, so do not waste time trying to click the pretty button.

When the app opens the chooser from JavaScript instead of exposing an input, start page.waitForFileChooser() before the click and call fileChooser.accept([path]). The most common failure is a relative path: it resolves against the Node process working directory, which is the repo root on a laptop and something else entirely on a CI runner, so build paths with path.resolve. Downloads have no first-class API.

You enable them over the DevTools Protocol by sending Browser.setDownloadBehavior with behavior 'allow' (or 'allowAndName', which names each file by a GUID so you can correlate it with the click that caused it), a downloadPath, and optionally eventOnly. Use the browser-level domain; the older Page.setDownloadBehavior is deprecated and does not apply to contexts you create later. The second gotcha is timing.

The click promise resolves immediately, the bytes arrive later, and Chrome writes the file as name.crdownload until the transfer finishes, so you poll the directory until a file without that suffix appears rather than sleeping for a guessed number of seconds. In a container, create the download directory up front and make it writable by the user Chrome runs as, because a permission failure produces no error on the Node side at all. For tests there is often a cleaner route: read the anchor's href and fetch it from inside the page so session cookies apply, then assert on the bytes without involving the download manager.

import path from 'node:path';
import fs from 'node:fs/promises';

// Upload: works even when the input is display:none
const input = await page.$('input[type=file]');
await input?.uploadFile(path.resolve(process.cwd(), 'fixtures/resume.pdf'));

// Native chooser opened by page JavaScript
const [chooser] = await Promise.all([
  page.waitForFileChooser(),
  page.click('#attach-btn'),
]);
await chooser.accept([path.resolve('fixtures/offer-letter.png')]);

// Downloads need raw CDP
const dir = path.resolve('downloads');
await fs.mkdir(dir, { recursive: true });
const cdp = await browser.target().createCDPSession();
await cdp.send('Browser.setDownloadBehavior', {
  behavior: 'allow',
  downloadPath: dir,
});

await page.click('#export-csv');

// Poll until Chrome stops writing the .crdownload temp file
const settled = async () =>
  (await fs.readdir(dir)).find((f) => !f.endsWith('.crdownload'));
let name;
while (!(name = await settled())) {
  await new Promise((r) => setTimeout(r, 250));
}
console.log('downloaded', name);

Answer

Puppeteer ships its own query engine layered on top of CSS, and it matters more since page.$x() and page.waitForXPath() were removed in the v22 line. ::-p-text(Place order) matches elements whose rendered text contains that string, with whitespace normalised, and it resolves to the smallest matching element, so it picks the span inside the button rather than the whole body. ::-p-aria(Place order) queries Chrome's accessibility tree by accessible name, and you can narrow it by role, which makes it the most resilient selector on a well-built app because it only breaks when the user-visible semantics break. ::-p-xpath(//td[2]) embeds an XPath expression inside an ordinary selector string and is the migration path for old $x() code. The >>> combinator is a deep descendant that pierces open shadow roots and >>>> is its direct-child equivalent, which is how you reach into web components without manually walking shadowRoot properties in evaluate. Closed shadow roots remain invisible to all of these.

The pieces compose, so a single string can start in light DOM, cross a shadow boundary and then match by text, and they work anywhere a selector is accepted including waitForSelector and page.locator. You can register your own with Puppeteer.registerCustomQueryHandler when a design system needs it. Two caveats interviewers listen for.

This syntax is Puppeteer-specific, so a suite built around it does not port cleanly to Playwright or Selenium. And text and aria lookups walk far more of the tree than an id lookup, which becomes measurable when you run them thousands of times in a scraping loop, so keep CSS for volume work and reserve aria and data-testid for UI tests.

// Text match: resolves to the smallest element containing the string
await page.locator('::-p-text(Place order)').click();

// Accessibility tree: name plus role
await page.locator('::-p-aria([name="Continue"][role="button"])').click();

// XPath, the replacement for the removed page.$x()
const cell = await page.$('::-p-xpath(//table//tr[2]/td[3])');
console.log(await cell?.evaluate((el) => el.textContent));

// Pierce an open shadow root with the deep combinator
await page.waitForSelector('gs-checkout >>> input[name=upi-id]');
await page.type('gs-checkout >>> input[name=upi-id]', 'test@okaxis');

// Compose: CSS, then shadow boundary, then text
await page.locator('#cart >>> ::-p-text(Apply coupon)').click();

// This now throws: page.$x is not a function
// const nodes = await page.$x('//button');

Answer

Turning interception on makes Chrome pause every outgoing request and emit a 'request' event that your Node handler must resolve with request.continue(), request.respond() or request.abort(). Nothing proceeds until you answer, so a handler with a branch that throws before calling continue leaves the page hanging until the navigation timeout, and that is the single most common bug in intercepting code. The cost is real.

Every request becomes a round trip to Node, and enabling interception disables Chrome's HTTP cache, so a page that was previously served from memory refetches everything. On a heavy site a pass-through handler alone can add twenty to forty percent to load time, which is why you only enable it when you get something back. What you get back is usually large for scraping and document work: abort images, media, fonts and known analytics or ad hosts by checking request.resourceType() and the URL, and a listing page that took four seconds loads in one.

For a rendering service, blocking third-party scripts also removes a whole class of nondeterminism from screenshots. You can rewrite traffic too, with request.continue({ url, method, postData, headers }), which is how teams point a production build at a staging API or inject an auth header without touching the app. Two operational notes.

Call request.isInterceptResolutionHandled() first when more than one listener is attached, and remember that requests served from a service worker never surface unless you call page.setBypassServiceWorker(true). If all you need is blocking, the cheaper option is CDP Network.setBlockedURLs, which filters inside the browser with no Node round trip at all.

await page.setRequestInterception(true);

const BLOCKED_TYPES = new Set(['image', 'media', 'font']);
const BLOCKED_HOSTS = ['googletagmanager.com', 'doubleclick.net', 'hotjar.com'];

page.on('request', (req) => {
  if (req.isInterceptResolutionHandled()) return;
  const url = req.url();
  if (BLOCKED_TYPES.has(req.resourceType())) return req.abort();
  if (BLOCKED_HOSTS.some((h) => url.includes(h))) return req.abort();
  if (url.includes('api.example.com')) {
    return req.continue({
      url: url.replace('api.example.com', 'staging-api.example.com'),
      headers: { ...req.headers(), authorization: 'Bearer ' + process.env.TOKEN },
    });
  }
  req.continue();
});

// Blocking only, filtered inside Chrome with zero Node round trips
const cdp = await page.createCDPSession();
await cdp.send('Network.enable');
await cdp.send('Network.setBlockedURLs', {
  urls: ['*.woff2', '*doubleclick*', '*google-analytics*'],
});

Answer

request.respond({ status, contentType, headers, body }) fulfils a paused request from Node without it ever reaching the network. This is how you test states that are painful to reproduce with real data: an empty search result, a 500 from the payments API, a partially failed bulk upload, a response that arrives after eight seconds. Because you control the body, the test asserts on rendering logic rather than on whatever the staging database happens to contain that morning, which removes the whole category of failures caused by someone else editing seed data.

Practical details: set contentType explicitly, add the CORS headers the browser expects if the stubbed URL is on a different origin (access-control-allow-origin at minimum), and remember that preflight OPTIONS requests hit your handler too and need their own response. To simulate latency, await a timer inside the handler before calling respond, which exercises loading states honestly. respond cannot produce a redirect chain, so for 302 behaviour you continue the request and let the server do it. Cooperative intercept resolution is the mechanism that lets several independent listeners share one request.

Each of continue, respond and abort accepts a priority number as a second argument; Puppeteer collects the intents from all listeners and applies the highest priority once every handler has run, with abort winning ties over respond and respond over continue. Without priorities the first listener to resolve wins and the others throw 'Request is already handled'. In a codebase where a fixtures module and a metrics module both attach listeners, using priorities is the difference between a working suite and a race.

await page.setRequestInterception(true);

// Fixture layer: stub the orders API, priority 1
page.on('request', (req) => {
  if (!req.url().includes('/api/orders')) return req.continue({}, 0);
  if (req.method() === 'OPTIONS') {
    return req.respond(
      { status: 204, headers: { 'access-control-allow-origin': '*' } },
      1,
    );
  }
  req.respond(
    {
      status: 200,
      contentType: 'application/json',
      headers: { 'access-control-allow-origin': '*' },
      body: JSON.stringify({ orders: [], nextCursor: null }),
    },
    1,
  );
});

// Failure-path test: force a 500 with a higher priority
page.on('request', (req) => {
  if (req.url().includes('/api/payments')) {
    return req.respond({ status: 500, body: 'gateway down' }, 5);
  }
  req.continue({}, 0);
});

await page.goto(url);
await page.locator('::-p-text(No orders yet)').wait();

Answer

Selectors never cross a frame boundary, so page.$('#card-number') will not find an input that lives inside a payment iframe no matter how long you wait. You have to get the Frame object first. The reliable ways are elementHandle.contentFrame() from the iframe element, page.frames().find(f => f.url().includes('checkout')), or page.waitForFrame(predicate) when the frame is injected asynchronously.

Once you hold the Frame, it exposes the same surface as a page: frame.$, frame.waitForSelector, frame.type, frame.locator and frame.evaluate, all scoped to that document. Cross-origin frames add a wrinkle. Under site isolation Chrome puts them in a separate renderer process (an out-of-process iframe), so the parent document's JavaScript genuinely cannot see inside them; anything based on document.querySelector from the top frame is hopeless.

Puppeteer models the OOPIF as its own target and still gives you a Frame, so the API path works, but evaluate tricks do not. This is exactly what you hit with Razorpay and PayU checkout widgets, reCAPTCHA, embedded Google Maps and most third-party support chat. The second wrinkle is lifetime.

Frames detach on navigation and on any re-render that swaps the iframe element, and a stale reference throws 'Attempted to use detached Frame' or 'Execution context was destroyed'. Never cache a Frame across a reload or across an SPA route change; resolve it again right before you use it. If a frame appears and disappears while a wizard advances, use page.waitForFrame with a URL predicate at each step rather than indexing into page.frames(), because the index shifts.

// Resolve from the iframe element
const el = await page.waitForSelector('iframe.razorpay-checkout-frame');
const frame = await el.contentFrame();
await frame.locator('input[name=card_number]').fill('4111111111111111');
await frame.locator('button[type=submit]').click();

// Or wait for it by URL, which survives re-renders
const otpFrame = await page.waitForFrame(
  (f) => f.url().includes('/otp'),
  { timeout: 20_000 },
);
await otpFrame.type('#otp-input', '482913');

// Enumerate for debugging
for (const f of page.frames()) {
  console.log(f.name() || '(anon)', f.url(), f.isDetached());
}

// Wrong: selectors do not cross frame boundaries
// await page.click('input[name=card_number]');

Answer

Because you attached the listener too late. If you write await page.click(link) followed by await page.waitForNavigation(), the click can dispatch, the navigation can start and the load event can fire before the second statement even runs. waitForNavigation then sits waiting for the next navigation, which never comes, and you burn the full thirty second default before seeing 'Navigation timeout of 30000 ms exceeded'. On a fast local server this happens often; on a slow staging box it almost never does, which is why the bug reaches CI.

The fix is to create the waiter first and await both together, either with Promise.all where the wait is listed before the action, or by holding the promise in a variable. The same reasoning applies to waitForResponse, waitForFileChooser, waitForRequest and waitForFrame. Two related failures are worth naming.

First, single-page apps that route with history.pushState do not fire a document navigation at all, so waitForNavigation never resolves; wait on the rendered result instead, with a selector or a waitForFunction on location.pathname. Second, 'Execution context was destroyed, most likely because of a navigation' means an evaluate or a handle operation was in flight when the document swapped underneath it. That is usually a symptom of not synchronising with the navigation, and the cure is the same ordering fix, or re-querying after the new document has settled. In modern code you can often skip explicit navigation waits altogether: page.locator(...) on the destination waits for the element in whichever document is current, which is both shorter and harder to get wrong.

// Race: the navigation can finish before the waiter is attached
// await page.click('a.details');
// await page.waitForNavigation();

// Correct: start the waiter first
const nav = page.waitForNavigation({ waitUntil: 'domcontentloaded' });
await page.click('a.details');
await nav;

// Equivalent idiom
await Promise.all([
  page.waitForNavigation({ waitUntil: 'domcontentloaded' }),
  page.click('a.details'),
]);

// SPA route change: no document navigation fires at all
await page.click('a[href="/dashboard"]');
await page.waitForFunction(
  () => location.pathname === '/dashboard',
  { polling: 'mutation' },
);

// Simplest of all: just wait for what the new page shows
await page.locator('h1::-p-text(Dashboard)').wait();

Answer

There are three levels, and picking the wrong one is what makes auth flaky. For HTTP basic auth, page.authenticate({ username, password }) is enough; it is implemented through request interception and is scoped to that page. For normal cookie sessions, log in once in a setup step, snapshot the cookies with browser.cookies() and write them to disk, then restore them with browser.setCookie(...saved) in a fresh context before the first navigation.

Note that recent versions moved these to the Browser object and deprecated page.cookies() and page.setCookie(), so old snippets warn or break. Cookies alone are often not enough: most React and Angular apps keep the access token in localStorage, which cookies do not cover. To restore that you must already be on the origin, so navigate to a cheap path first, write the keys through evaluate, then reload.

The sturdiest option is userDataDir, which persists the whole Chrome profile including cookies, localStorage, IndexedDB and service worker caches, so the second run simply starts logged in. Its constraint is exclusivity: two browsers cannot share one profile directory, and the second launch fails on SingletonLock, so copy a template profile per worker. Whichever you choose, validate the session at the start of the run and fall back to a real login when it has expired, otherwise an expired token turns into a mid-test redirect and a confusing selector timeout. Keep the snapshot out of git and in CI secrets, and for Indian products with OTP-only login, ask the backend team for a seeded test account or a test-hook endpoint rather than automating SMS.

import fs from 'node:fs/promises';

// One-time setup: log in, then snapshot cookies + localStorage
const cookies = await browser.cookies();
const storage = await page.evaluate(() =>
  JSON.stringify(Object.entries(localStorage)),
);
await fs.writeFile('.auth/state.json', JSON.stringify({ cookies, storage }));

// Every later run: restore into a clean context
const { cookies: saved, storage: savedStorage } = JSON.parse(
  await fs.readFile('.auth/state.json', 'utf8'),
);
const ctx = await browser.createBrowserContext();
const p = await ctx.newPage();
await browser.setCookie(...saved);

// localStorage needs the origin loaded first
await p.goto('https://app.example.com/health', { waitUntil: 'domcontentloaded' });
await p.evaluate((entries) => {
  for (const [k, v] of JSON.parse(entries)) localStorage.setItem(k, v);
}, savedStorage);
await p.goto('https://app.example.com/dashboard');

// Sturdiest option: persist the whole profile
// await puppeteer.launch({ userDataDir: './.profiles/worker-1' });

Answer

Two processes leak independently, and you have to say which one you are looking at. On the Node side, every ElementHandle and JSHandle you create pins the corresponding renderer object until you call dispose() or the execution context is destroyed, and every page.on(...) listener registered inside a loop accumulates, which eventually prints MaxListenersExceededWarning. On the Chrome side, pages that are never closed keep their renderer processes alive, so browser.pages().length climbing over time is the clearest single signal, and a count of chrome child processes is the second.

The fixes are structural rather than clever. Create a fresh BrowserContext per job and close it, because one context.close() disposes every page, handle and cookie it owned. Never reuse a page across unrelated jobs.

Prefer $$eval and evaluate, which create no handles at all, over looping across $$ results. Use once() for one-shot listeners and remove listeners in teardown. Then accept that Chrome itself leaks slowly and recycle: restart the browser after N jobs or when RSS crosses a threshold, which no amount of hygiene replaces.

Instrument all of it: process.memoryUsage().rss on the Node side, page.metrics() for JSHeapUsedSize, Nodes and JSEventListeners on the browser side, and the page count as a gauge. Container settings matter too. Chrome writes shared memory to /dev/shm, which Docker defaults to 64 MB, so either pass --disable-dev-shm-usage or run with a larger --shm-size, otherwise you get renderer crashes that look like leaks. Finally, trap SIGTERM and close the browser, or every deploy leaves orphaned Chrome processes behind.

let jobsSinceRestart = 0;
let browser = await puppeteer.launch({ args: ['--disable-dev-shm-usage'] });

async function render(url) {
  const ctx = await browser.createBrowserContext();
  try {
    const page = await ctx.newPage();
    page.once('pageerror', (e) => console.warn('page error', e.message));
    await page.goto(url, { waitUntil: 'domcontentloaded' });
    return await page.$$eval('.row', (rows) => rows.map((r) => r.innerText));
  } finally {
    await ctx.close(); // disposes pages and handles in one call
    jobsSinceRestart += 1;
  }
}

setInterval(async () => {
  const rssMb = process.memoryUsage().rss / 1024 / 1024;
  const pages = (await browser.pages()).length;
  console.log({ rssMb: Math.round(rssMb), pages, jobsSinceRestart });
  if (jobsSinceRestart > 500 || rssMb > 1500) {
    const old = browser;
    browser = await puppeteer.launch({ args: ['--disable-dev-shm-usage'] });
    jobsSinceRestart = 0;
    await old.close();
  }
}, 30_000);

process.on('SIGTERM', async () => {
  await browser.close();
  process.exit(0);
});

Answer

There is no magic number, but there is a way to reason about it. A launched Chrome costs roughly 80 to 150 MB resident before any tab, and each open page costs another 30 to 100 MB depending on how heavy the site is, more for anything canvas or video driven. Rendering is CPU bound, so useful parallelism is capped by cores, not by memory alone.

On a typical 2 vCPU / 4 GB cloud instance, four to six concurrent pages in a single browser is a realistic ceiling; push past it and page load times inflate faster than throughput improves, which shows up as p95 latency climbing while p50 stays flat. That plateau is the number you tune against, not a guessed constant. Structurally, prefer one browser with several contexts over several browsers, because contexts are nearly free and browsers are not, and reach for multiple browsers only when you need fault isolation so that one crash does not take out every in-flight job.

Bound the work with an explicit semaphore such as p-limit, or use puppeteer-cluster, which gives you CONCURRENCY_CONTEXT (one context per job, cheapest), CONCURRENCY_PAGE (one page, shares cookies) and CONCURRENCY_BROWSER (full isolation, most expensive), plus retries and per-task timeouts. Two things interviewers wait to hear. Node is single threaded, so a slow synchronous transform in your own code, for example building a large PDF buffer, serialises everything regardless of how many pages are open. And per-job timeouts are mandatory: one page that never finishes loading will otherwise hold a slot forever and quietly reduce your effective concurrency to zero.

import pLimit from 'p-limit';

const limit = pLimit(Number(process.env.RENDER_CONCURRENCY ?? 4));
const browser = await puppeteer.launch({
  args: ['--no-sandbox', '--disable-dev-shm-usage'],
});

const withTimeout = (promise, ms) =>
  Promise.race([
    promise,
    new Promise((_, rej) => setTimeout(() => rej(new Error('job timeout')), ms)),
  ]);

async function shoot(url) {
  const ctx = await browser.createBrowserContext();
  try {
    const page = await ctx.newPage();
    page.setDefaultTimeout(20_000);
    await page.goto(url, { waitUntil: 'domcontentloaded' });
    return await page.screenshot({ type: 'webp' });
  } finally {
    await ctx.close();
  }
}

const results = await Promise.allSettled(
  urls.map((u) => limit(() => withTimeout(shoot(u), 30_000))),
);
console.log(results.filter((r) => r.status === 'rejected').length, 'failed');

Answer

A plain node:22-slim image is missing most of what Chrome links against, so the first symptom is a launch that dies with 'error while loading shared libraries: libnss3.so' or the vaguer 'Failed to launch the browser process'. You either install the dependency set (libnss3, libatk-bridge2.0-0, libgbm1, libasound2, libxkbcommon0 and friends) and let Puppeteer download Chrome for Testing, or install the distro chromium and switch to puppeteer-core with PUPPETEER_EXECUTABLE_PATH. Fonts are the next surprise: without fonts-liberation and a Devanagari face such as fonts-indic or Noto, Hindi and rupee glyphs render as boxes in your PDFs, and nobody notices until an invoice reaches a customer.

On the flags: Chrome's sandbox needs kernel capabilities that a default Docker profile does not grant, so --no-sandbox is the usual workaround, but state the security cost, because it means a renderer compromise is a container compromise. The better answer is to keep the sandbox and run with the Chrome seccomp profile plus a non-root user, and only reach for --no-sandbox where the URL is fully trusted. --disable-dev-shm-usage matters because Chrome writes shared memory to /dev/shm, which Docker caps at 64 MB by default; without the flag (or docker run --shm-size=1gb) renderers crash mid-render under load, surfacing as 'Target closed'. Zombies are the last piece: Chrome forks children, and PID 1 in a container does not reap them, so defunct processes accumulate until the process table fills.

Run with --init, or use dumb-init or tini as the entrypoint. Finally, run as a non-root user and give it ownership of the browser cache directory, since a root-owned cache is the top cause of 'Could not find Chrome' at runtime.

# Dockerfile
FROM node:22-bookworm-slim

RUN apt-get update && apt-get install -y --no-install-recommends \
      chromium fonts-liberation fonts-indic fonts-noto-color-emoji \
      libnss3 libatk-bridge2.0-0 libgbm1 libasound2 libxkbcommon0 dumb-init \
 && rm -rf /var/lib/apt/lists/*

ENV PUPPETEER_SKIP_DOWNLOAD=true \
    PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium \
    NODE_ENV=production

WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .

USER node
ENTRYPOINT ["dumb-init", "--"]
CMD ["node", "server.js"]

# docker run --shm-size=1gb --init my-renderer

// server.js
const browser = await puppeteer.launch({
  executablePath: process.env.PUPPETEER_EXECUTABLE_PATH,
  args: ['--no-sandbox', '--disable-dev-shm-usage', '--disable-gpu'],
});

Answer

The first move is to stop guessing and make the headless run observable, because on CI you have no window to look at. Attach four listeners to every page in a shared fixture: 'console' to forward browser logs into the job output, 'pageerror' for uncaught exceptions inside the app, 'requestfailed' to catch the blocked CDN or the DNS failure that only happens inside the build network, and 'response' filtered to non-2xx so a silently failing API call is visible. Then capture artefacts on failure: page.screenshot({ fullPage: true }), await page.content() written to an HTML file, and page.url(), all uploaded as CI artefacts.

Those three usually identify the cause on the first failed run. For protocol-level questions, DEBUG=puppeteer:* prints every CDP message sent and received, which is verbose but decisive when you suspect the browser never received a command; narrow it to puppeteer:protocol:SEND or a single session when the volume is unusable. For interactive work, launch with headless: false, slowMo and devtools: true locally, and use page.pause() equivalents such as a long waitForFunction on a flag you set from the console.

Beyond instrumentation, the usual CI-specific causes are worth naming: a different viewport so an element sits below the fold, missing fonts changing layout, a slower machine exposing a race that a fast laptop hid, a different timezone or locale changing date formats, and headless Chrome not having a GPU so canvas or WebGL paths differ. Set viewport, timezone and locale explicitly in the fixture so the two environments actually match.

export function instrument(page, name) {
  page.on('console', (m) =>
    console.log('[browser:' + m.type() + '] ' + m.text()),
  );
  page.on('pageerror', (e) => console.error('[pageerror] ' + e.message));
  page.on('requestfailed', (r) =>
    console.warn('[failed] ' + r.url() + ' ' + r.failure()?.errorText),
  );
  page.on('response', (r) => {
    if (r.status() >= 400) console.warn('[http ' + r.status() + '] ' + r.url());
  });

  page.setDefaultTimeout(15_000);
  return async function onFailure() {
    await page.screenshot({ path: 'artifacts/' + name + '.png', fullPage: true });
    await fs.writeFile('artifacts/' + name + '.html', await page.content());
    console.error('failed at ' + page.url());
  };
}

// Make both environments identical
await page.setViewport({ width: 1440, height: 900 });
await page.emulateTimezone('Asia/Kolkata');

// Protocol-level tracing when a command seems to vanish
// DEBUG=puppeteer:protocol:SEND npm test

Answer

Jest runs each test file in its own worker process, so a browser created in beforeAll inside a spec is created once per file. With a fifteen-file suite that is fifteen Chrome launches, several seconds each, and a lot of wasted memory. The standard fix is the pattern jest-puppeteer implements: a globalSetup module launches one browser, writes its browser.wsEndpoint() to a file or to process.env.PUPPETEER_WS_ENDPOINT, every worker calls puppeteer.connect({ browserWSEndpoint }) to attach to that same process, and globalTeardown closes it.

Inside each spec you then create a BrowserContext in beforeEach and close it in afterEach, which gives you a clean cookie jar and storage per test without paying for a new browser. Two details make or break it. Workers must call browser.disconnect() rather than browser.close(), because closing kills the shared process and every other worker fails with 'Protocol error: Connection closed'.

And globalTeardown has to run even when the suite crashes, so write the endpoint and the PID to a temp file and kill the process defensively. Isolation is the other half: never share a page between tests, because leftover dialogs, listeners and scroll position leak between them and produce order-dependent failures that only appear when Jest reshuffles files. Set testEnvironment appropriately, raise testTimeout well above the Puppeteer default since a browser test easily exceeds five seconds, and cap maxWorkers so Jest's own parallelism does not multiply with your page concurrency and starve the runner of CPU.

// global-setup.js
import fs from 'node:fs/promises';
import os from 'node:os';
import path from 'node:path';
import puppeteer from 'puppeteer';

const DIR = path.join(os.tmpdir(), 'jest_puppeteer');

export default async function globalSetup() {
  const browser = await puppeteer.launch({
    args: ['--no-sandbox', '--disable-dev-shm-usage'],
  });
  globalThis.__BROWSER__ = browser;
  await fs.mkdir(DIR, { recursive: true });
  await fs.writeFile(path.join(DIR, 'ws'), browser.wsEndpoint());
}

// global-teardown.js closes globalThis.__BROWSER__ and removes DIR

// checkout.test.js
let browser, context, page;

beforeAll(async () => {
  const ws = await fs.readFile(path.join(DIR, 'ws'), 'utf8');
  browser = await puppeteer.connect({ browserWSEndpoint: ws });
});
beforeEach(async () => {
  context = await browser.createBrowserContext();
  page = await context.newPage();
});
afterEach(() => context.close());
afterAll(() => browser.disconnect()); // never close() from a worker

Answer

The naive loop scrolls to the bottom a fixed number of times and hopes. It fails in both directions: it stops early on a slow network and it spins forever on a page that recycles DOM nodes. A robust loop is state driven.

Record the current item count, trigger the load (scroll to the bottom, or click the load-more button if there is one), then wait for the count to increase using waitForFunction with the previous count passed in as an argument. If the count does not change within a short timeout, treat that as the end and break. Keep a hard cap on iterations and on elapsed time so a broken page cannot hold a worker forever.

Two structural traps. First, virtualised lists (react-window, TanStack Virtual, ag-Grid) unmount rows as they leave the viewport, so scraping only at the end gives you the last screenful; you must harvest after each step and deduplicate by a stable key such as a data-id, not by array index. Second, many feeds are backed by a paginated JSON API, and intercepting that response with page.on('response') is usually an order of magnitude cheaper and more accurate than parsing rendered DOM, because you get the raw fields and the cursor.

Check for the API before writing DOM code. Practical hygiene for Indian job and listing sites in particular: add a politeness delay between scrolls, respect robots.txt and the site's terms, block images and fonts so each step is fast, and checkpoint the harvested keys so a crash halfway through does not restart the whole crawl.

const seen = new Map();
const DEADLINE = Date.now() + 120_000;

for (let step = 0; step < 100 && Date.now() < DEADLINE; step += 1) {
  const batch = await page.$$eval('[data-job-id]', (els) =>
    els.map((el) => ({
      id: el.dataset.jobId,
      title: el.querySelector('h3')?.textContent?.trim() ?? '',
      company: el.querySelector('.company')?.textContent?.trim() ?? '',
    })),
  );
  for (const row of batch) seen.set(row.id, row); // virtualised rows unmount

  const before = batch.length;
  await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));

  try {
    await page.waitForFunction(
      (n) => document.querySelectorAll('[data-job-id]').length > n,
      { timeout: 6_000, polling: 'mutation' },
      before,
    );
  } catch {
    break; // count stopped growing: end of list
  }
  await new Promise((r) => setTimeout(r, 800)); // politeness delay
}
console.log('harvested', seen.size);

Answer

Start with the honest framing an interviewer wants: automation detection is an arms race, and no answer is permanent. The obvious signals are navigator.webdriver being true, which Chrome sets whenever it is driven by an automation protocol, and the HeadlessChrome token in the default user agent string. Beyond those, headless environments differ in ways that fingerprinting scripts read cheaply: window.chrome missing or thin, navigator.plugins and mimeTypes empty, navigator.languages not matching the Accept-Language header, WebGL vendor and renderer reporting SwiftShader or Mesa instead of a real GPU, screen dimensions that never change, missing device fonts, permissions API returning inconsistent values for Notification, and a notable absence of human input entropy since Puppeteer clicks land exactly at element centres with no mouse movement in between.

Commercial anti-bot vendors also detect the presence of an attached CDP client through timing and Runtime domain side effects, which no user-land patch can hide. puppeteer-extra with puppeteer-extra-plugin-stealth patches a long list of these properties and gets you past basic checks, but it is maintained reactively, it lags Chrome releases, and it does nothing about network-level signals such as datacentre IP ranges and TLS fingerprints, which is why residential proxies matter more than JavaScript patches in practice. The professional answer includes the boundary: on a site whose terms forbid automated access, the right path is an official API or a data partnership, and for testing your own product you should be allowlisting the automation rather than hiding it. Interviewers ask this to see whether you understand fingerprinting, not to hear a list of npm packages.

// What a detector actually reads
const fingerprint = await page.evaluate(() => ({
  webdriver: navigator.webdriver,
  ua: navigator.userAgent,
  languages: navigator.languages,
  plugins: navigator.plugins.length,
  hasChrome: typeof window.chrome !== 'undefined',
  webgl: (() => {
    const gl = document.createElement('canvas').getContext('webgl');
    const ext = gl?.getExtension('WEBGL_debug_renderer_info');
    return ext ? gl.getParameter(ext.UNMASKED_RENDERER_WEBGL) : null;
  })(),
}));
console.log(fingerprint);

// Minimum hygiene for your own load tests
await page.setUserAgent(
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +
    '(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36',
);
await page.setExtraHTTPHeaders({ 'accept-language': 'en-IN,en;q=0.9,hi;q=0.8' });
await page.evaluateOnNewDocument(() => {
  Object.defineProperty(navigator, 'webdriver', { get: () => undefined });
});

Answer

Three separate clocks exist and candidates routinely tune the wrong one. The default action timeout, thirty seconds, applies to waitForSelector, waitForFunction, locator actions and similar waits; page.setDefaultTimeout(ms) changes it for a page and passing timeout in the options changes it for one call. The navigation timeout, also thirty seconds, applies to goto, reload, goBack and waitForNavigation, and has its own setter, page.setDefaultNavigationTimeout(ms); note that setDefaultTimeout also raises the navigation timeout unless the navigation setter has been called, which is a common source of confusion.

Both accept 0 to disable, which you should almost never do outside a debugging session. The third clock is protocolTimeout, a launch and connect option rather than a page one, and it bounds how long Puppeteer waits for any single CDP command to come back. It defaults to 180000 ms in recent versions and produces the distinctive error 'Runtime.callFunctionOn timed out.

Increase the protocolTimeout setting in launch/connect calls for a higher timeout if needed.' You hit it on genuinely long single operations: printing a two hundred page PDF, screenshotting an enormous document, or an evaluate that does heavy synchronous work in the page. Raising the page timeout does nothing for it, which is the trap.

There is a fourth clock in practice, your own job timeout, and it should be the outermost bound. In production set explicit values everywhere rather than relying on defaults, keep the navigation timeout generous for slow third-party pages, keep the action timeout tight so a genuinely broken selector fails fast, and always wrap the job so no single request can hold a worker indefinitely.

const browser = await puppeteer.launch({
  protocolTimeout: 300_000, // long PDF renders exceed the 180s default
});

const page = await browser.newPage();
page.setDefaultNavigationTimeout(60_000); // slow third-party pages
page.setDefaultTimeout(10_000);           // fail fast on bad selectors

await page.goto(url, { waitUntil: 'domcontentloaded' });

// Per-call override wins over the page default
await page.waitForSelector('#report-ready', { timeout: 120_000 });

try {
  await page.pdf({ format: 'A4', printBackground: true });
} catch (err) {
  // 'Runtime.callFunctionOn timed out. Increase the protocolTimeout setting...'
  if (String(err.message).includes('protocolTimeout')) {
    console.error('single CDP command exceeded protocolTimeout, not page timeout');
  }
  throw err;
}

Answer

launch() starts a Chrome process that the Node process owns; connect() attaches to a browser that is already running somewhere else over its DevTools WebSocket. You reach for connect in four situations. First, a shared browser for a test suite, where one globalSetup launches Chrome and every Jest worker attaches, saving repeated launches.

Second, browser-as-a-service: your API pods stay small and stateless while a separate fleet (browserless, a self-hosted Chrome pool, or a vendor grid) owns the heavy processes, which lets you scale rendering independently of request handling and keeps Chrome's dependencies out of your application image. Third, attaching to a browser a human is already using, launched with --remote-debugging-port, which is invaluable for debugging a session that involved a manual login or a captcha. Fourth, cross-container setups where Chrome runs in its own container in the same pod.

Mechanically you either pass browserWSEndpoint from browser.wsEndpoint(), or pass browserURL such as http://chrome:9222 and let Puppeteer resolve it through the /json/version endpoint. Things to watch: defaultViewport is not applied to existing pages, so pass defaultViewport: null and set the viewport yourself; disconnect() detaches while close() terminates the remote browser, and calling close from a worker is how a whole suite dies at once; the WebSocket endpoint is an unauthenticated full-control channel over the machine, so never expose the debugging port beyond localhost or a private network; and the connection can drop, so listen for the browser 'disconnected' event and rebuild rather than letting every subsequent call fail with 'Protocol error: Connection closed'.

// Producer: one long-lived browser
const browser = await puppeteer.launch({ args: ['--no-sandbox'] });
console.log(browser.wsEndpoint()); // ws://127.0.0.1:PORT/devtools/browser/...

// Consumer: attach from another process
const remote = await puppeteer.connect({
  browserWSEndpoint: process.env.CHROME_WS,
  defaultViewport: null,
  protocolTimeout: 120_000,
});

// Or resolve via the HTTP endpoint of a Chrome container
const byUrl = await puppeteer.connect({ browserURL: 'http://chrome:9222' });

remote.on('disconnected', () => {
  console.error('lost the browser, reconnecting');
  scheduleReconnect();
});

const page = await remote.newPage();
await page.setViewport({ width: 1440, height: 900 });
await page.goto('https://example.com');
await page.close();

await remote.disconnect(); // NOT close(): that kills the shared browser

Answer

Puppeteer wraps a useful subset of the DevTools Protocol, and createCDPSession gives you the rest. The domains you actually reach for in production are Network for emulateNetworkConditions and setBlockedURLs, Browser for setDownloadBehavior, Performance for getMetrics when you want real paint and layout counters, Emulation for locale and precise device metrics overrides, Animation for setPlaybackRate to freeze CSS animations before a visual snapshot, Page for captureSnapshot to archive a page as MHTML, Accessibility for getFullAXTree when auditing, and Fetch when you need request control at a stage Puppeteer's interception does not expose. The session is a two-way channel: session.send(method, params) issues commands and session.on(event, handler) subscribes, after you enable the relevant domain.

Several things go wrong. Domains are stateful and shared with Puppeteer itself, so enabling Fetch while page.setRequestInterception(true) is active means two owners fight over the same paused requests and some hang forever; pick one mechanism. Sessions are bound to a target, so they die when the page closes, and using one afterwards throws 'Session closed'.

Cross-origin iframes are separate targets with their own sessions, so a command sent to the page session does not affect them. And the protocol is versioned with Chrome, not with Puppeteer, so a method that exists today can be renamed or moved to experimental in a later Chrome, which is a real upgrade risk for code that leans on it heavily. Wrap raw CDP usage in a small adapter with a feature check so an upgrade fails loudly in one place.

const cdp = await page.createCDPSession();

// Real browser-side counters, richer than page.metrics()
await cdp.send('Performance.enable');
await page.goto(url, { waitUntil: 'load' });
const { metrics } = await cdp.send('Performance.getMetrics');
console.log(Object.fromEntries(metrics.map((m) => [m.name, m.value])));

// Freeze CSS animations before a visual snapshot
await cdp.send('Animation.enable');
await cdp.send('Animation.setPlaybackRate', { playbackRate: 0 });

// Locale override, which Puppeteer does not expose directly
await cdp.send('Emulation.setLocaleOverride', { locale: 'en-IN' });

// Archive the whole page including subresources
const { data } = await cdp.send('Page.captureSnapshot', { format: 'mhtml' });
await fs.writeFile('snapshot.mhtml', data);

// Subscribe to raw events
await cdp.send('Network.enable');
cdp.on('Network.webSocketFrameReceived', (e) =>
  console.log('ws frame', e.response.payloadData.slice(0, 120)),
);

await cdp.detach();

Answer

WebDriver BiDi is the W3C standard bidirectional browser automation protocol, designed to give the event-driven capabilities CDP pioneered without being Chrome-specific. Puppeteer implements it as an alternative transport, selected with the protocol option, and it is what makes the Firefox support introduced in the v23 line possible: Chrome can be driven over either 'cdp' or 'webDriverBiDi', while Firefox is BiDi only. You choose the browser with the browser option at launch, and for Firefox Puppeteer downloads a Firefox build the same way it downloads Chrome for Testing.

The value is real cross-browser coverage from one script, which matters when a product ships to users on Firefox and you currently have no automated coverage there. What you give up is everything that maps only to CDP. Anything built on a Chrome-specific domain is unavailable or partial on BiDi: createCDPSession itself, page.metrics(), the tracing API, JS and CSS coverage, CPU throttling, some emulation overrides, and CDP-based download behaviour.

Interception exists but the semantics differ in places, so a suite full of clever request rewriting will not port unchanged. Practically, the migration test is simple: if your specs stay on navigation, locators, selectors, screenshots and basic interception, they usually run on Firefox with a browser flag and nothing else; if they reach into CDP, they do not. Puppeteer also still targets Chrome first, and BiDi feature coverage continues to fill in over releases, so the honest position in an interview is that Puppeteer plus BiDi is a credible second browser rather than a full cross-browser replacement for a tool built around it.

import puppeteer from 'puppeteer';

// Firefox, driven over WebDriver BiDi
const firefox = await puppeteer.launch({ browser: 'firefox' });

// Chrome over BiDi instead of the default CDP transport
const chromeBidi = await puppeteer.launch({
  browser: 'chrome',
  protocol: 'webDriverBiDi',
});

for (const browser of [firefox, chromeBidi]) {
  const page = await browser.newPage();
  await page.goto('https://example.com', { waitUntil: 'domcontentloaded' });
  await page.locator('::-p-text(More information)').click();
  await page.screenshot({ path: browser.browserVersion + '.png' });

  // CDP-only: throws on the Firefox session
  try {
    await page.createCDPSession();
  } catch (e) {
    console.log('no CDP on this transport:', e.message);
  }
  await browser.close();
}

// npx puppeteer browsers install firefox

Answer

Stock Chrome does not fit or run in the Lambda environment, so you use @sparticuz/chromium, a stripped Chromium build packaged for Amazon Linux, together with puppeteer-core so nothing is downloaded at install time. The build exposes chromium.args, chromium.defaultViewport, chromium.headless and an async chromium.executablePath() that extracts the binary into /tmp on first use. Size breaks first.

A zipped Lambda deployment is capped at 50 MB and 250 MB unzipped including layers, and Chromium plus your dependencies sits uncomfortably close to that, so the sane path in 2026 is a container image, which gives you 10 GB and lets you install fonts normally. Cold start breaks second: extracting and booting the browser typically costs two to five seconds on top of your own init, so hoist the browser outside the handler and reuse it across warm invocations, checking browser.connected before each use and relaunching if the sandbox was frozen and the process died. Memory is the third: give the function at least 1536 MB, partly because Lambda scales CPU with memory and rendering is CPU bound, so a larger setting is often cheaper per invocation, not more expensive.

Then the constraints that surprise people: /tmp defaults to 512 MB and holds both the extracted binary and any downloads, one Lambda container serves one request at a time so concurrency is instance count rather than pages, API Gateway caps a synchronous response at 29 seconds which a heavy PDF can exceed (use a Function URL, streaming, or an async job that writes to S3), and Indic and emoji fonts must be shipped deliberately or invoices render as boxes. Set the function timeout above your protocolTimeout expectations and always close pages in a finally block.

import chromium from '@sparticuz/chromium';
import puppeteer from 'puppeteer-core';

let browser; // hoisted: survives warm invocations

async function getBrowser() {
  if (browser?.connected) return browser;
  browser = await puppeteer.launch({
    args: [...chromium.args, '--disable-dev-shm-usage'],
    defaultViewport: chromium.defaultViewport,
    executablePath: await chromium.executablePath(),
    headless: chromium.headless,
    protocolTimeout: 60_000,
  });
  return browser;
}

export const handler = async (event) => {
  const b = await getBrowser();
  const ctx = await b.createBrowserContext();
  try {
    const page = await ctx.newPage();
    page.setDefaultTimeout(15_000);
    await page.setContent(event.html, { waitUntil: 'domcontentloaded' });
    await page.evaluate(() => document.fonts.ready);
    const pdf = await page.pdf({ format: 'A4', printBackground: true });
    return { statusCode: 200, isBase64Encoded: true,
      headers: { 'content-type': 'application/pdf' },
      body: Buffer.from(pdf).toString('base64') };
  } finally {
    await ctx.close();
  }
};

Answer

Start by refusing the obvious design: launching a browser per request. It costs hundreds of milliseconds and over a hundred megabytes, and under a spike it forks Chrome processes until the box dies. The shape that works is a warm pool.

Keep one or two browsers alive per instance, take a fresh BrowserContext per job so sessions never bleed, and bound in-flight work with a semaphore sized to where p95 latency starts climbing, typically four to six on 2 vCPU. Put a bounded queue in front and reject with 429 and a Retry-After once it is full, because unbounded queueing converts a spike into a timeout storm where every caller waits and every result is discarded. Make every job idempotent with a caller-supplied key, write output to S3 and return a URL rather than streaming megabytes through the API, and use page.setContent with a locally cached template instead of navigating to a URL, which removes a whole network dependency per document.

Then plan for the failure modes explicitly. Renderers crash: listen for the browser 'disconnected' event and for page 'error', drain and relaunch rather than serving from a dead handle. Memory creeps: recycle a browser after N jobs or an RSS threshold, and roll it out of the pool gracefully so in-flight jobs finish.

Jobs hang: enforce a hard per-job timeout outside the page timeouts and destroy the context on expiry. /dev/shm exhausts: pass --disable-dev-shm-usage or size it. Observability is what makes this operable: export queue depth, pool size, jobs per browser, render duration histograms, and browser restart count, then autoscale on queue depth rather than CPU, because CPU looks healthy right up to the moment the queue is minutes deep.

class BrowserPool {
  constructor({ size = 2, maxJobs = 400 } = {}) {
    this.size = size;
    this.maxJobs = maxJobs;
    this.slots = [];
  }

  async start() {
    for (let i = 0; i < this.size; i += 1) this.slots.push(await this.spawn());
  }

  async spawn() {
    const browser = await puppeteer.launch({
      args: ['--no-sandbox', '--disable-dev-shm-usage'],
      protocolTimeout: 120_000,
    });
    const slot = { browser, jobs: 0, retiring: false };
    browser.on('disconnected', () => {
      if (!slot.retiring) this.replace(slot);
    });
    return slot;
  }

  async replace(dead) {
    this.slots = this.slots.filter((s) => s !== dead);
    metrics.browserRestarts.inc();
    this.slots.push(await this.spawn());
  }

  async run(fn) {
    const slot = this.slots.sort((a, b) => a.jobs - b.jobs)[0];
    const ctx = await slot.browser.createBrowserContext();
    slot.jobs += 1;
    try {
      return await fn(ctx);
    } finally {
      await ctx.close();
      if (slot.jobs >= this.maxJobs && !slot.retiring) {
        slot.retiring = true;
        await this.replace(slot);
        await slot.browser.close();
      }
    }
  }
}

Answer

Visual regression fails on noise long before it catches a real bug, so the work is eliminating sources of variation one by one. Motion is first: inject a stylesheet that sets animation-duration, transition-duration and animation-delay to zero and hides the caret, and for JavaScript-driven animation send Animation.setPlaybackRate with playbackRate 0 over a CDP session. Time is second: a relative timestamp that says 'a minute ago' changes every run, so freeze the clock with evaluateOnNewDocument by overriding Date.now and the Date constructor before any page script executes, and stub the API responses that carry dates.

Fonts are third and the most common cause of a suite that only fails in Docker: wait for document.fonts.ready, and install the exact font packages in the image, including a Devanagari face if the product renders Hindi. Loading is fourth: scroll to trigger lazy images, then wait until every img has complete true and naturalWidth above zero. Then remove what you cannot control by masking, setting visibility hidden on ad slots, avatars, chart tooltips and anything driven by live data.

Two rules make the difference operationally. Generate baselines inside the same container image the CI job uses, never from a developer Mac, because font rasterisation and antialiasing genuinely differ across platforms and you will chase phantom diffs forever. And prefer element screenshots over full-page ones so a header change does not invalidate forty baselines. Compare with pixelmatch or odiff at a small threshold with antialiasing tolerance on, store baselines per viewport, and publish the diff image as a CI artefact so review is one click.

export async function stableShot(page, selector, name) {
  const cdp = await page.createCDPSession();
  await cdp.send('Animation.enable');
  await cdp.send('Animation.setPlaybackRate', { playbackRate: 0 });

  await page.addStyleTag({
    content: `*, *::before, *::after {
      animation-duration: 0s !important;
      animation-delay: 0s !important;
      transition-duration: 0s !important;
      caret-color: transparent !important;
    }`,
  });

  // Mask anything driven by live data
  await page.$$eval('[data-visual-mask]', (els) =>
    els.forEach((el) => (el.style.visibility = 'hidden')),
  );

  await page.evaluate(() => document.fonts.ready);
  await page.waitForFunction(() =>
    [...document.images].every((i) => i.complete && i.naturalWidth > 0),
  );

  const target = (await page.$(selector)) ?? page;
  return target.screenshot({ path: `shots/${name}.png` });
}

// Freeze the clock before any page script runs
await page.evaluateOnNewDocument(() => {
  const FIXED = new Date('2026-01-15T10:30:00+05:30').getTime();
  const Real = Date;
  Date = class extends Real {
    constructor(...a) { super(...(a.length ? a : [FIXED])); }
    static now() { return FIXED; }
  };
});

Answer

Four instruments, each answering a different question. page.metrics() returns browser-side counters in one call: ScriptDuration, LayoutDuration, RecalcStyleDuration, TaskDuration, LayoutCount, Nodes, JSEventListeners and JSHeapUsedSize. Sampling it before and after an interaction tells you whether a slow flow is script bound or layout bound, and a rising Nodes or JSEventListeners count across repeated interactions is a leak in the app itself. page.tracing.start({ path, screenshots: true, categories }) records a full Chrome trace you open in the Performance panel or Perfetto, which is the right tool when you need the flame chart rather than a number. For user-centric metrics, run a PerformanceObserver inside the page through evaluate to collect LCP, CLS and long tasks, and read performance.getEntriesByType('navigation') for TTFB and DOM timings.

The coverage API is the fourth: page.coverage.startJSCoverage({ resetOnNavigation: false }) and startCSSCoverage report byte ranges actually executed, which is how you justify a code-splitting project with a number instead of an opinion. Caveats matter more here than the API surface. Headless Chrome usually has no GPU, so paint-related timings are not comparable to a real device.

A single run is noise, so take the median of five or more and compare against a baseline captured on the same machine class, not against a number from someone's laptop. Pair the measurement with emulateCPUThrottling(4) and throttled network to model a mid-range Android on 4G, which is the device most Indian users actually hold. Then gate CI on a budget delta rather than an absolute threshold.

await page.emulateCPUThrottling(4);
await page.coverage.startJSCoverage({ resetOnNavigation: false });
await page.tracing.start({ path: 'trace.json', screenshots: true });

await page.evaluateOnNewDocument(() => {
  window.__lcp = 0;
  new PerformanceObserver((list) => {
    for (const e of list.getEntries()) window.__lcp = e.startTime;
  }).observe({ type: 'largest-contentful-paint', buffered: true });
});

await page.goto(url, { waitUntil: 'load' });
await page.locator('#checkout').click();

await page.tracing.stop();
const js = await page.coverage.stopJSCoverage();
const metrics = await page.metrics();

const used = js.reduce(
  (n, e) => n + e.ranges.reduce((s, r) => s + r.end - r.start, 0),
  0,
);
const total = js.reduce((n, e) => n + e.text.length, 0);

console.log({
  lcpMs: Math.round(await page.evaluate(() => window.__lcp)),
  scriptMs: Math.round(metrics.ScriptDuration * 1000),
  layoutMs: Math.round(metrics.LayoutDuration * 1000),
  nodes: metrics.Nodes,
  unusedJsPct: Math.round((1 - used / total) * 100),
});

Answer

A rendering endpoint is a browser that fetches whatever a caller tells it to, from inside your network, which makes it one of the sharpest server-side request forgery primitives you can ship. The immediate targets are the cloud metadata endpoint at 169.254.169.254, where IMDSv1 hands out instance role credentials to anything that asks, and internal services on localhost or private ranges that assume network position is authentication. Exfiltration is trivial once the page runs script: fetch the internal response, then post it to an attacker host or encode it in an image URL.

Local files are the second class of risk: an img or iframe pointing at file:///etc/passwd, made readable to script if anyone has set --allow-file-access-from-files, which you must never do. Third, --no-sandbox turns a renderer exploit into container-level code execution, and a rendering fleet is exactly where you should not be disabling the sandbox. Containment is layered.

Enforce IMDSv2 with a hop limit of 1 and, better, give the rendering task an IAM role with no permissions at all. Put egress behind an allowlisting proxy so the browser cannot reach anything you did not intend. Use request interception to reject the file, chrome, blob-from-file and data schemes for top-level loads, resolve hostnames and abort private CIDRs, and refuse redirects that leave the allowlist.

Keep the sandbox on, run as a non-root user with a read-only filesystem and dropped capabilities, and isolate the fleet in its own account or namespace. If the template does not need script, page.setJavaScriptEnabled(false) removes most of the attack surface in one line. And cap time, memory and page count so a hostile document cannot simply exhaust the host.

import dns from 'node:dns/promises';
import ipaddr from 'ipaddr.js';

const ALLOWED_SCHEMES = new Set(['http:', 'https:']);

async function isPrivate(hostname) {
  const { address } = await dns.lookup(hostname);
  const range = ipaddr.parse(address).range();
  return range !== 'unicast'; // loopback, private, linkLocal, uniqueLocal
}

await page.setJavaScriptEnabled(false); // if the template does not need it
await page.setRequestInterception(true);

page.on('request', async (req) => {
  try {
    const url = new URL(req.url());
    if (!ALLOWED_SCHEMES.has(url.protocol)) return req.abort('blockedbyclient');
    if (url.hostname === '169.254.169.254') return req.abort('blockedbyclient');
    if (await isPrivate(url.hostname)) return req.abort('blockedbyclient');
    req.continue();
  } catch {
    req.abort('blockedbyclient');
  }
});

// Keep the sandbox. Never do this on a fleet that renders untrusted input:
// args: ['--no-sandbox', '--allow-file-access-from-files']
const browser = await puppeteer.launch({
  args: ['--disable-dev-shm-usage', '--disable-extensions'],
});