Web scraping in JavaScript : Complete guide

To make web scraping in JavaScript, you need Node.js and a library tailored to your target audience. Use Cheerio for static pages and Puppeteer Where Playwright as soon as the content is rendered on the client side.

This guide shows you how to install, code, and run your first web scraper, step by step.

Prerequisites for web scraping in JavaScript

Before you write your first JavaScript script, two basic tools are all you need to get started.

• Node.js : a JavaScript runtime environment open-source and cross-platform. It lets you run JavaScript code outside of a browser, directly from your console. Download it from the official Node.js documentation.
• npm (Node Package Manager) : the package manager built into Node. It lets you install your libraries with a single command npm install.

Once Node.js Once you've set it up, choose a library based on the type of web page you want to scrape. Here are the most commonly used ones.

• Cheerio : perfect for the scraping static pages. It parses raw HTML using a syntax similar to jQuery. She is lightweight, fast and simple, but it does not execute the page's JavaScript. See the Cheerio documentation.
• Puppeteer and Playwright : ideal for dynamic websites. They control a true headless browser (Chrome Where Firefox) and load the page just like a real user would. See the Puppeteer documentation.

The rule is simple: Cheerio For static pages, Puppeteer Where Playwright as soon as a website loads its data using JavaScript.

Web scraping in JavaScript

The idea is simple: you take the HTML From a web page, you identify the elements that interest you, then extract the data. The choice of tool depends on a single criterion: Is the page static Where dynamic ?

Step 1: Installation and configuration

Start by downloading Node.js from the official website, then install it. Next, verify the installation in your terminal:

node -v
npm -v

If both versions appear, everything is fine. Now create your project:

mkdir my-scraping
cd my-scraping
npm init -y

This command generates a file package.json which will list your dependencies. All you have to do now is install the libraries that suit your needs.

For a static page (content already present in the HTML returned by the server), use axios and Cheerio :

npm install axios cheerio

For a dynamic page (data loaded via client-side JavaScript, such as React, Vue Where Angular), install instead Puppeteer :

npm install puppeteer

Step 2: Creating a scraping script

Before writing your JS script, identify the right ones CSS selectors. Opens the target web page in Chrome Where Firefox, right-click on the desired element, then select «Inspect.» The DOM will be displayed, allowing you to identify the tag and the class (.class) or the username (#id) to target.

Scraping a static page with Cheerio

Cheerio reads only the Raw HTML and does not execute any JavaScript scripts. On a client-side rendered site, the user will see only a blank page.

// Import libraries
const axios = require('axios');
const cheerio = require('cheerio');

// Scraper page URL
const url = "https://exemple.com";

// Main function
async function scrapePage() {
  try {
    // Download HTML content
    const { data } = await axios.get(url);

    // Load HTML with Cheerio
    const $ = cheerio.load(data);

    // Example: retrieve all h1 titles
    const titles = [];
    $("h1").each((i, elem) => {
      titles.push($(elem).text().trim());
    });

    // Display results
    console.log("Titles found:", titles);

  } catch (error) {
    console.error("Error while scraping:", error);
  }
}

// Run script
scrapePage();

Replace https://exemple.com with the URL of your choice, then adjust the selector $("h1") To focus on what interests you: $("p") For paragraphs, .class Where #id for a specific element. The method .text() returns the textContent of the element, the equivalent of jQuery on the server side.

Scraping a dynamic page with Puppeteer

Puppeteer a real driver headless browser and waits for the JavaScript to finish loading the content, where Cheerio fails.

// Import Puppeteer
const puppeteer = require("puppeteer");

// Scrape page URL
const url = "https://exemple.com";

async function scrapePage() {
  // Launch a headless browser
  const browser = await puppeteer.launch({ headless: true });
  const page = await browser.newPage();

  try {
    // Go to page
    await page.goto(url, { waitUntil: "networkidle2" });

    // Example: extract text from all h1 titles
    const titles = await page.$$eval("h1", elements =>
      elements.map(el => el.textContent.trim())
    );

    console.log("Titles found:", titles);

  } catch (error) {
    console.error("Error while scraping:", error);
  } finally {
    // Close browser
    await browser.close();
  }
}

// Run script
scrapePage();

The option waitUntil: "networkidle2" is waiting for the end of the network requests before extracting the data from the web page.

Step 3: Managing extracted data

Once the data has been retrieved, you can save it to JSON for a database, or in CSV to open them in Excel. The module fs is a native of Node.js and requires no installation. For CSV, simply add the library json2csv :

npm install json2csv

const axios = require("axios");
const cheerio = require("cheerio");
const fs = require("fs");
const { Parser } = require("json2csv");

const url = "https://exemple.com";

async function scrapeAndSave() {
  try {
    // Récupérer le HTML de la page
    const { data } = await axios.get(url);
    const $ = cheerio.load(data);

    // Extraire les titres dans un tableau d'objets
    const resultats = [];
    $("h1").each((i, elem) => {
      resultats.push({ titre: $(elem).text().trim() });
    });

    // Sauvegarde en JSON
    fs.writeFileSync("resultats.json", JSON.stringify(resultats, null, 2), "utf-8");
    console.log("Données sauvegardées dans resultats.json");

    // Sauvegarde en CSV
    const parser = new Parser();
    const csv = parser.parse(resultats);
    fs.writeFileSync("resultats.csv", csv, "utf-8");
    console.log("Données sauvegardées dans resultats.csv");

  } catch (error) {
    console.error("Erreur lors du scraping :", error);
  }
}

scrapeAndSave();

This JavaScript script does it all at once: it fetches the web page, extracts the HTML data, and then writes two files. The block try / catch handles errors to prevent a timeout or one CAPTCHA causes the entire scraping script to crash.

Best practices for web scraping in JavaScript

Before running your scripts, a few simple habits will keep your JS scraping effective, stable and complies with the terms of use of the websites in question. It's also important to make sure that your This procedure is legal depending on the context.

• Respect the robots.txt file : That's the golden rule. Before scraping a web page, open https://exemple.com/robots.txt. If the target path is in Disallow, don't scrape it. That way, you'll avoid ethical and legal issues.
• Space out your requests : Don't flood the site. Wait a while between each request to stay under the radar and avoid getting blocked.
• Customize Your User-Agent : By default, a JavaScript scraper is quickly detected. With Puppeteer, define a realistic header using page.setUserAgent() to mimic a real browser.
• Handling Errors and Deadlocks : wrap your calls in a block try / catch to prevent a timeout or a CAPTCHA from causing the entire script to crash.

What alternatives should you consider?

Web scraping isn't limited to JavaScript. Depending on your project, other languages or tools may better meet your needs.

• Python : libraries Scrapy and BeautifulSoup work wonders when it comes to extracting content from many web pages.
• PHP : This is handy if you want to integrate scraping directly into a website that's already coded in that language.
• No-code tools such as Bright Data, Octoparse, and Apify: they let you scrape web data without writing a single line of code.

Scraping API or DIY solution: Which should you choose?

The real question isn't the language, but the level of complexity you're willing to handle on your own. Your own scraping script with Puppeteer Where Cheerio remains the best option when the volume is low and the site is not well protected.

A Scraping API (such as the one offered by Bright Data) becomes more relevant as soon as you encounter advanced blocks: IP address rotation, automatic CAPTCHA solving, and large-scale JavaScript rendering.

In summary: the DIY saves you money and teaches you a lot. The’API saves you time on large or recurring projects.

FAQs

What's the best JavaScript web scraping tool?

It all depends on the type of web page you want to scrape.

• Cheerio : fast and lightweight, perfect for parse the HTML static pages.
• Puppeteer : ideal for piloting a real browser Chrome and manage the content rendered using JavaScript.
• Playwright : near Puppeteer, but with the cross-browser support (Chrome, Firefox, WebKit) and additional advanced features.

What's the best programming language for scraping?

There is no single best universal language. The choice depends on your project and your environment.

• Python : Very popular for quick projects and large volumes of web data.
• JavaScript : the right approach if you're already working in the web ecosystem with Node.js.
• PHP : Handy for integrating scraping directly into an existing website.

the web scraping in JavaScript remains an excellent choice for collect web data, that you go through Cheerio, Puppeteer Where Playwright.

To learn more without writing any code, check out our guides on Bright Data, Octoparse and Apify.