Most scraping and browser automation projects do not fail because a selector was wrong once. They fail because the team chose a full browser when a simple request would have done the job, or because nobody planned for login expiry, consent banners, downloads, and UI changes. If you are responsible for reporting, QA, partner portals, or repetitive back-office tasks, the real question is not which library is fashionable. It is which approach gives you the lowest maintenance cost while still surviving production conditions.
That distinction matters commercially. Browser automation can unlock workflows that direct requests cannot, but it also adds runtime, infrastructure, and support overhead. For business owners and operations leads, the right tool choice is often the difference between a useful recurring automation and a script that quietly breaks soon after launch.
Start With The Lightest Approach That Works
If the data is already present in the raw HTML, a feed, or a documented export, start with direct HTTP requests and parsing. That route is faster, cheaper, and much easier to support. Move to a real browser only when the site depends on JavaScript rendering, authenticated sessions, multi-step forms, infinite scroll, downloads, or user actions such as clicks, typing, and uploads.
A browser is usually justified when you need to:
- log into a portal and carry session state across multiple pages
- extract data from interfaces that render content after page load
- trigger exports, screenshots, PDFs, or browser-side downloads
- validate that an actual customer or staff journey still works after a site change
Which Tools Make Sense In 2026
Playwright: the default for new multi-step automations
For most new builds, Playwright is the safest default. It gives one API across Chromium, Firefox, and WebKit, and it is well suited to login flows, form completion, export routines, screenshots, and cross-browser QA checks. In practical terms, its locator model and auto-waiting reduce the brittle timing code that used to make older automations flaky.
npm i -D playwright
npx playwright install chromium firefox webkitPlaywright is especially useful when you expect the workflow to grow from “scrape this page” into “own this journey end to end.” If your team also wants a test runner, reporting, and retries, the wider Playwright tooling can support that. For exploratory setup work, npx playwright codegen https://example.com is a fast way to inspect a user journey, but generated code should be cleaned up before production.
Puppeteer: a good fit for Chrome-first utilities
Puppeteer remains a sensible option when the work is deliberately Chrome-centric: internal admin tooling, scripted screenshots, PDF generation, or lightweight automations that do not need broader browser coverage. Current installation is simpler than many old guides suggest because the package downloads a compatible Chrome for Testing build automatically.
npm i puppeteerIf your infrastructure team manages browser binaries centrally or you connect to a remote browser, puppeteer-core is often the better choice. I would normally reach for Puppeteer when the brief is narrow, Chrome-only, and unlikely to expand into broader QA or cross-browser work.
Chromote: the pragmatic R route
If the surrounding workflow already lives in R, keep it there as long as possible. chromote gives R teams direct access to the Chrome DevTools Protocol and also powers rvest::read_html_live() for pages that only become scrapeable after JavaScript has rendered them. That is often enough for reporting pipelines, research tasks, and analyst-led automations without introducing a separate Node service.
install.packages("chromote")
library(chromote)
b <- ChromoteSession$new()For R-heavy teams, this is often the quickest path to a workable proof of concept, especially when the browser is only needed to get through login, render a table, or trigger a download.
PhantomJS: retire it for new work
PhantomJS still shows up in old tutorials, but its own site says development is suspended. That makes it a poor foundation for new client work in 2026. If you inherit an old PhantomJS script, treat it as migration work, not a platform decision.
Questions To Answer Before You Commit
- Is there already an API, CSV export, or partner feed that avoids scraping entirely?
- Who owns the login credentials, and what is the plan for password rotation or MFA?
- How often will this run, and what is the cost if it silently fails for a day?
- Do you need a data extract, or do you also need evidence such as screenshots, PDFs, or QA pass/fail logs?
- Who will maintain the workflow when the target site changes?
These are not procurement formalities. They are the questions that decide whether an automation is genuinely useful to the business or just technically impressive.
Implementation Rules That Reduce Maintenance
- Wait on application state, not arbitrary sleeps. A script built around
sleep(5)will eventually fail when the network slows down or a modal appears. - Use stable, human-facing locators where possible. Labels, roles, visible text, and explicit test IDs usually survive redesigns better than deep CSS chains.
- Separate browser work from data work. Use the browser only for login, rendering, or exporting, then switch back to normal parsing and processing as early as you can.
- Plan for consent banners, session expiry, rate limits, and downloads from day one. Those are normal operational conditions, not edge cases.
- Keep a headed debugging path, screenshots, and basic run logs. Operations teams do not need a clever script; they need one they can diagnose quickly.
What I Would Recommend In Practice
For most client work, I would start with a very small proof of concept around one critical path. If the page turns out to be mostly static, I would simplify it down to direct requests and parsers. If the job truly depends on browser behaviour, I would usually start with Playwright. If the organisation is strongly R-based, I would first test whether chromote or rvest::read_html_live() is enough. If the requirement is tightly Chrome-centric and operationally narrow, Puppeteer may be the simpler fit.
The goal is not to pick the fanciest tool. The goal is to end up with an automation workflow your team can rerun next month without babysitting it. If you are choosing a stack, replacing a brittle scraper, or turning one-off scripts into something an operations team can actually own, Greg can help scope the workflow, choose the right stack, and reduce maintenance risk before you overbuild it.
Need help with this kind of work?
Discuss the right automation stack with Greg Get in touch with Greg.