Complete guide to web scraping with PHP

<?php
require 'vendor/autoload.php';

use Goutte\Client;

$client = new Client();
$url = 'https://exemple.com/blog'; // Remplace par l'URL de ta page cible

// Faire une requête GET pour récupérer la page
$crawler = $client->request('GET', $url);

// Sélectionner les éléments <h2> dans <article> avec le sélecteur CSS
$titres = [];

$crawler->filter('article h2')->each(function ($node) use (&$titres) {
    // Récupérer le texte du titre
    $titres[] = trim($node->text());
});

// Afficher les titres extraits
print_r($titres);
?>

What this code does:

• The selector 'article h2' targets the text within the tags <h2>.
• The Method text() retrieves the text of each title.
• Each title is added to the table $itles, and then displayed with print_r().

CSS selectors are also used to extract attributes HTML elements. If each heading is a link within a D tag <a>, you retrieve the attribute href To get the article's URL:

<?php
require 'vendor/autoload.php';

use Goutte\Client;

$client = new Client();
$url = 'https://exemple.com/blog'; // Remplace par l'URL de ta page cible

// Faire une requête GET pour récupérer la page
$crawler = $client->request('GET', $url);

// Sélectionner les liens dans les titres
$titres = [];

$crawler->filter('article h2 a')->each(function ($node) use (&$titres) {
    $titre = trim($node->text());
    $lien = $node->attr('href'); // Extraire l'attribut href

    // Ajouter le titre et l'URL au tableau
    $titres[] = [
        'title' => $titre,
        'url'   => $lien,
    ];
});

// Afficher les résultats
print_r($titres);
?>

The picture $itles then contains, for each item, both the title and its link :

Array
(
    [0] => Array
        (
            [title] => Article 1 title
            [url] => /article1
        )

    [1] => Array
        (
            [title] => Article 2 title
            [url] => /article2
        )
)

Each entry is an associative array with two keys: title for the title and url for the page's URL. You get a clean structure, ready to be analyzed or stored.

Step 4: Structuring and storing data

Once the data has been extracted into the table $data, you can export in a structured format. The two most useful formats are the JSON and the CSV.

the JSON is useful for feeding data to an API or a web application:

<?php
// Exporter les données en JSON
file_put_contents('export.json', json_encode($data, JSON_PRETTY_PRINT|JSON_UNESCAPED_UNICODE));

echo "Les données ont été exportées en JSON dans 'export.json'.";
?>

The file export.json will look like this:

[
    {
        "title": "Article 1 title",
        "url": "/article1"
    },
    {
        "title": "Article 2 title",
        "url": "/article2"
    }
]

If you prefer a chart CSV, use fputcsv To write each line to a file:

<?php
// Exporter les données en CSV
$fp = fopen('export.csv', 'w');

// Ajouter l'en-tête (titres des colonnes)
fputcsv($fp, ['title', 'url']);

// Ajouter chaque ligne de données
foreach ($data as $row) {
    fputcsv($fp, [$row['title'], $row['url']]);
}

// Fermer le fichier
fclose($fp);

echo "Les données ont été exportées en CSV dans 'export.csv'.";
?>

The file export.csv will look like this:

title,url
Article title 1,/article1
Article title 2,/article2

For large amounts of data, be sure to write to the file as we go along rather than loading everything into memory. You can also save the result directly to a database, which in turn makes it easier to analyze and update the collected information.

How to deal with common web scraping problems in PHP?

Web scraping in PHP doesn't always go as planned. A server that doesn't respond, a malformed page, an IP block—these problems can arise quickly. Here's how to handle the most common ones to make your scraping program more reliable.

1. Error management

• Connection errors

Sometimes, the request doesn't even reach the server. You might run into "No network, "Invalid URL Where "Server unreachable. In that case, include a block try/catch to prevent your script from crashing.

Here is a concrete example using Guzzle :

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;
use GuzzleHttp\Exception\RequestException;

function make_request_with_guzzle() {
    $client = new Client();
    
    try {
        $response = $client->request('GET', 'https://example.com/api/data');
        
        if ($response->getStatusCode() === 200) {
            $body = $response->getBody();
            echo "Données reçues : " . $body;
        }
        
    } catch (RequestException $e) {
        if ($e->hasResponse()) {
            echo "Erreur de la requête : " . $e->getResponse()->getStatusCode();
        } else {
            echo "Erreur de connexion : " . $e->getMessage();
        }
    }
}

make_request_with_guzzle();
?>

• HTTP status codes

Even when the connection is working, the server may return an error: 404 = page not found, 500 = internal server error. You can test the returned code using getStatusCode() :

<?php
require 'vendor/autoload.php';

use GuzzleHttp\Client;
use GuzzleHttp\Exception\RequestException;

function make_request_with_guzzle() {
    $client = new Client();
    
    try {
        $response = $client->request('GET', 'https://example.com/api/data');
        $status_code = $response->getStatusCode();
        
        if ($status_code === 200) {
            $body = $response->getBody();
            echo "Réponse réussie avec le code : " . $status_code . "
";
            echo "Données reçues : " . $body;
        } elseif ($status_code === 404) {
            echo "Erreur 404 : Page non trouvée
";
        } elseif ($status_code === 500) {
            echo "Erreur 500 : Erreur interne du serveur
";
        } else {
            echo "Code de statut : " . $status_code . "
";
        }
        
    } catch (RequestException $e) {
        if ($e->hasResponse()) {
            echo "Erreur HTTP : " . $e->getResponse()->getStatusCode() . "
";
        } else {
            echo "Erreur de connexion : " . $e->getMessage();
        }
    }
}

make_request_with_guzzle();
?>

• Parsing errors

Parsing is the’HTML analysis by your scraper. If the page is malformed, DomCrawler Where Goutte may crash or return nothing. Always check that the content exists before trying to retrieve it, using conditions such as count() Where filter(), then wrap the parsing in a try/catch :

<?php
require 'vendor/autoload.php';

use Goutte\Client;
use Symfony\Component\DomCrawler\Crawler;

function scrape_website() {
    $client = new Client();
    
    try {
        $crawler = $client->request('GET', 'https://example.com');
        $elements = $crawler->filter('div.target-element');
        
        if ($elements->count() > 0) {
            $content = $elements->first()->text();
            echo "Contenu extrait : " . $content;
        } else {
            echo "L'élément cible n'a pas été trouvé sur la page.";
        }
        
    } catch (Exception $e) {
        echo "Erreur lors du parsing de la page : " . $e->getMessage();
    }
}

scrape_website();
?>

2. Bypass anti-scraping restrictions and safeguards

Many websites implement safeguards to make web scraping more difficult. Here are the main obstacles and how to handle them properly:

• IP Blocking : If you send too many requests, the site may ban your IP address. The solution is to use rotating proxies, such as those from Bright Data, to distribute your requests across multiple addresses.
• Request Interval : Space out your calls over time to stay under the radar and avoid overloading the target server.
• JavaScript Content by default, PHP cannot run JavaScript. To scrape a page with dynamically loaded content, you need a headless browser (headless browser). Tools such as Puppeteer Where Selenium render the page like a real browser before extraction.
• robots.txt file : Before running your scraper, check this file in the site's root directory. It specifies what the site does and does not allow robots to do—a key step in acting responsibly.

FAQs

Is web scraping legal?

The legality of web scraping depends on the data you collect and how you use it. In France, scraping public and non-personal data is generally tolerated, but it's not an automatic green light.

Three legal restrictions can make data extraction illegal, even on public pages:

• the RGPD : When personal data is involved, «public» does not mean free to use.
• The Website Terms of Use : If you use an account to access content, you have agreed to terms that often prohibit scraping.
• the sui generis right Regarding databases: In Europe, a database is subject to specific protection.

In practice, each case is evaluated individually. Always refer to the file robots.txt and the site's terms of use before starting your program.

What's the difference between web scraping and web crawling?

• the web scraping is to extract accurate data on a website.
• the web crawling is to crawl pages to index them, just like a search engine bot does.

How do I scrape a site that requires login?

To scrape a website that requires a login, you need to simulate authentication. With PHP, the most common solution is still Guzzle. You send your credentials via a request POST, then keep the session open to retrieve the protected pages.

How do you manage scraping of sites with dynamic AJAX-loaded pages?

PHP cannot execute client-side JavaScript code. The content loaded into AJAX so it eludes a conventional scraper.

One possible solution is to use BrowserShot, a library that relies on an actual browser running in the background (Headless Chrome) to load the page and run the JavaScript.

You can also Integrate PHP with tools based on Node.js, as Puppeteer Where Selenium, to generate the rendered HTML and then retrieve the data from your script.

Are there any alternatives to PHP for web scraping?

Yes, several programming languages are very popular for web scraping:

• Python, with its powerful libraries such as BeautifulSoup and Scrapy.
• Node.js, which is very effective for dynamic websites, using libraries such as Puppeteer Where Cheerio.

How can you program a web scraper in an ethical and responsible way?

To scrape responsibly, there are a few simple rules to follow:

1. Check the robots.txt file on the website to learn about the access rules.
2. Limit the frequency of your requests so as not to overload the server.
3. Comply with the Terms of Use of the site.
4. Does not collect personal data without authorization.

Web scraping is a powerful tool—as long as you use it methodically and with common sense. Have you ever created a PHP scraper Or with another language? Share your experience with us in the comments!