By Greg Nowak. Last updated 2026-07-12.
Scraping and browser automation projects rarely fail because one selector changes. They fail because the team automates the wrong layer, overlooks operational details, or builds something nobody can diagnose after its original developer moves on.
For business owners, operations leads, and agencies, the important question is not which library is fashionable. It is how to complete the workflow reliably without creating unnecessary infrastructure and maintenance costs.
Begin with the business process, not the browser
Before choosing a tool, define what the automation must produce. Do you need a weekly data extract, a downloaded report, evidence that a customer journey works, or a complete back-office transaction? Also establish how quickly somebody must know when it fails.
Then look for the lightest dependable route. A documented API, scheduled export, partner feed, or direct HTTP request will normally be faster and easier to operate than a browser. If the required information is present in the returned HTML, parse that response directly. A real browser becomes worthwhile when JavaScript rendering, authentication, interactive controls, uploads, downloads, or screenshots are essential to the job.
| Situation | Good starting point | Why | Main caution |
|---|---|---|---|
| Data exists in HTML, an API, or an export | Direct request and parser | Fast, inexpensive, and easy to monitor | Confirm access rules and rate limits |
| Multi-step portal or cross-browser journey | Playwright | Strong locators, auto-waiting, and broad browser coverage | Browser binaries and sessions need managing |
| Focused Node.js browser utility | Puppeteer | Good fit for Chrome for Testing, PDFs, screenshots, and DevTools-based workflows | Do not assume the downloaded browser exists in every deployment |
| Analysis pipeline already lives in R | rvest, then Chromote if needed | Keeps extraction and analysis in one environment | Live browser helpers add dependencies and complexity |
When Playwright is the practical default
Playwright is a strong default for new multi-step automations. It can drive Chromium, Firefox, and WebKit, making it useful when an automation may expand into customer-journey testing or cross-browser QA.
Its locator model is particularly valuable in production. Locators based on roles, labels, text, and explicit test IDs usually communicate intent better than long CSS paths. Playwright also performs actionability checks before operations such as clicking, reducing the need for brittle timing code.
npm i -D playwright
npx playwright install chromium firefox webkitFor a Linux CI image that also needs system dependencies, install only what the job requires:
npx playwright install --with-deps chromiumPlaywright versions expect matching browser binaries, so rerun the browser installation step when upgrading the package. Treat that as part of deployment rather than an undocumented setup task.
Where Puppeteer still fits well
Puppeteer remains a sensible choice for focused Node.js utilities, particularly screenshot generation, PDF creation, or workflows built around Chrome for Testing and the DevTools protocol.
npm i puppeteerThe standard package normally downloads compatible browser binaries. Some package-manager and deployment configurations block installation scripts, however. If the browser download was skipped, the current documented fallback is:
npx puppeteer browsers installUse puppeteer-core when your infrastructure team manages the browser executable or the automation connects to a remote browser. That avoids an automatic browser download, but it also means your deployment must provide and configure the correct executable.
A pragmatic route for R teams
If the surrounding reporting or research pipeline already runs in R, begin with rvest::read_html(). The rvest documentation recommends static scraping when it works because it is faster, more robust, and does not depend on Chrome.
For content generated by JavaScript, rvest::read_html_live() provides basic scrolling, typing, and clicking through Chromote. The live interface is currently documented as experimental, so it deserves additional testing before becoming a critical operational dependency.
install.packages("chromote")
library(chromote)
b <- ChromoteSession$new()
b$go_to("https://example.com")Chromote exposes the Chrome DevTools Protocol and is appropriate when an R workflow needs more control, authentication, or screenshots. For a larger application with complex branching and extensive interactions, a dedicated Playwright service may still be easier for a development team to maintain.
Design for production conditions
- Wait for observable state. Wait for a heading, response, download, or enabled control instead of adding arbitrary sleeps.
- Separate navigation from data processing. Use the browser for login and interaction, then hand downloaded files or HTML to normal parsing code.
- Make failure visible. Record the run time, outcome, relevant URL, and error. Capture a screenshot when a browser step fails.
- Plan credential ownership. Decide who manages password rotation, session expiry, MFA, and recovery when an account is locked.
- Expect interface variation. Consent banners, regional content, feature experiments, empty states, and changed download names are normal operating conditions.
- Respect the target service. Review its terms, access permissions, robots guidance, and rate limits. Browser automation does not grant permission to collect or reuse data.
- Protect sensitive material. Keep credentials out of source code and avoid storing session cookies, customer records, or screenshots longer than necessary.
Scope one critical path first
A useful proof of concept should answer operational questions, not merely demonstrate that a page can be opened. Test one representative journey with realistic authentication, data volume, downloads, and failure reporting. You should learn how long it runs, what can interrupt it, and what evidence an operator receives.
If the browser proves unnecessary, simplify the solution. If it is essential, package the browser version, configuration, logging, and recovery process alongside the script. The best automation is not the most technically ambitious one; it is the one your team can trust and support next month.
If you are scoping a new workflow or replacing an unreliable collection of scripts, Greg can help map the process, choose an appropriate stack, and define a maintainable first release.
Related on GrN.dk
- AI agents need a browser policy before they start clicking around
- AI automations need a spend dashboard before the first runaway bill
- Background AI Tasks Need Queues, Not Just Longer API Calls
Need help with this kind of work?
Plan a maintainable automation workflow Get in touch with Greg.
