How do I web scrap with Laravel?

With Laravel, you can scrape a website in three steps: install a package such as Symfony DomCrawler via Composer, create an order Artisan, then select the elements HTML with selectors CSS.

This web scraping guide shows you exactly how to do it, using PHP code ready to use.

Prerequisites for scraping with Laravel

Laravel is a PHP framework Designed for modern web applications. Its rich ecosystem makes it an ideal foundation for building a clean, easy-to-maintain web scraping project.

Before you get started, make sure you have the right environment:

• Composer, the dependency manager for installing packages.
• A recent version of PHP compatible with your version of Laravel.
• Laravel installed via compose create-project laravel/laravel my-scraper.

As for skills, just a few basics are enough to get started:

• Understanding the basics of PHP and the structure of an application Laravel (controllers, routes, architecture MVC).
• Understanding the HTML and the game's CSS selectors to precisely target elements on a page.
• Knowing how to use Composer command-line option to add a package to the project.

As for the crawling process itself, here are the scraping tools the most useful ones:

• Symfony DomCrawler + HttpClient : the go-to duo today. It downloads a page and navigates the HTML using the filter() from DomCrawler. To initialize the client, you use HttpClient::create() with the desired options.
• HTTP Client of Laravel : based on Guzzle, it retrieves simple content using Http::get() and handles calls to a API.
• Puppeteer / Headless Chrome : a browser with no user interface, essential for scraping pages that load their content via JavaScript or via Ajax.

Tutorial for creating your first scraper with Laravel

Follow these steps to create a functional scraper with Laravel. You're going to install a scraping package and generate a command Artisan, then target the data using CSS selectors.

Step 1: Installation and configuration

Create a new project Laravel with Composer, then add Goutte, a practical integration for web scraping:

# 1) Create a new Laravel project
compose create-project laravel/laravel scraper-demo
cd scraper-demo

# 2) Add Goutte (Laravel integration)
composer require weidner/goutte

Packages weidner/goutte and fabpot/goutte are now officially obsolete. They still work, but for a new project, the recommended replacement on the Symfony is symfony/browser-kit, combined with symfony/http-client and symfony/dom-crawler. The code below still works if you use weidner/goutte, which simply encapsulates Goutte to Laravel.

Step 2: Create an Artisan Order

Generates a command that will contain your scraping logic using php artisan :

php artisan make:command ScrapeData

The file is created here : app/Console/Commands/ScrapeData.php. It is in this class that you'll write the HTTP request and extract the data. The class ScrapeData extends Command, and the method public function handle() contains all the scraping logic.

Step 3: Writing scraper code

In the generated command, you'll put together three essential components:

• A HTTP request to retrieve the HTML content of the page.
• Regarding the CSS selectors transferred to the DomCrawler to filter data using filter().
• A loop to browse items and display results.

Here is a complete code example for scraping the article titles from a blog:

info('Scraping: {$url}");

        // 1) HTTP request to retrieve the HTML
 $crawler = Goutte::request("GET', $url);

        // 2) Using CSS selectors with DomCrawler
 $nodes = $crawler->filter('h2 a');

        // 3) Iterate over the elements and display them
 $nodes->each(function (Crawler $node, $i) {
 $title = $node->text();
            $link  = $node->attr('href');
 $this->line(($i+1) . '. " . $title . " - " . $link);
 });

 return self::SUCCESS;
    }
}

Here, DomCrawler does all the HTML parsing. The method filter('h2 a') extracts every link in a title, then each() loop over it to extract the text and the attribute href. Then run the scraper from your terminal using php artisan scrape:data and you'll see the titles appear one by one. You can also run php artisan serve to test your application locally before deploying it.

Best practices for web scraping with Laravel

To scrape data properly with Laravel, a few simple habits can make all the difference. Here are the best practices to follow, whether you're managing a web scraping bot or a one-time order.

1. Task and queue management

Web scraping can take several seconds per page. Imagine having to scrape 1,000 pages of a website: your app would be blocked and unusable for quite a while. The solution can be summed up in two words: the jobs and the game's queues from Laravel.

• A job, this is a task you want to run in the background.
• A tail (queue) is where these jobs are stored so they can be executed one by one, without blocking the rest.

Here's how to encapsulate the scraping logic in a job:

// app/Jobs/ScrapePageJob.php
<?php

namespace App\Jobs;

use Goutte\Client; // Ou Guzzle/Http, selon ta stack
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;

class ScrapePageJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    protected string $url;

    public function __construct(string $url)
    {
        $this->url = $url;
    }

    public function handle(): void
    {
        $client = new Client();

        $crawler = $client-&gt;request('GET', $this-&gt;url);

        // Simple example: extract all <h1>
        $titles = $crawler-&gt;filter('h1')-&gt;each(function ($node) {
            return $node-&gt;text();
        });

        // Persistence / logs / events...
        foreach ($titles as $title) {
            \Log::info("[Scraping] {$this-&gt;url} - H1: {$title}");
        }
    }
}

// app/Http/Controllers/ScraperController.php
onQueue('scraping'); // if you want a dedicated queue
 }

 return response()->json(['status' => 'Scraping started in the background']);
    }
}

To trigger this controller, declare a road in your file routes/web.php Where routes/api.php, for example Route::get('/scrape', [ScraperController::class, 'start']). You can also generate a dedicated controller using php artisan make:controller ScraperController.

Laravel offers several ways to manage this queue. The two most commonly used are:

• File with database : Jobs are stored as rows in an SQL table and then executed one by one by a worker.
• Go ahead with Redis : Jobs are stored in memory in an ultra-fast queue, which is ideal for handling a large volume of tasks.

2. Automation with Laravel's task scheduler

Laravel integrates a task planner (scheduler) that lets you automate your scrapers without having to think about it. This way, you can schedule a command to run at regular intervals, for example every hour.

Here's how to set it up in app/Console/Kernel.php :

command('scraper:run')->hourly();

 // Useful examples:
 // $schedule->command('scraper:run')->everyFifteenMinutes();
        // $schedule->command('scraper:run')->dailyAt('02:30')->timezone('Indian/Antananarivo');
    }

 /**
 * Load commands.
     */
    protected function commands(): void
    {
 $this->load(__DIR__ . '/Commands');
    }
}

3. Bypassing anti-scraping protection

Many websites have implemented measures to protect against web scrapers. To avoid being blocked:

• Change the User-Agent : Simulates a real browser instead of the HTTP client's default header.
• Managing deadlines : inserts pauses (sleep, throttle) between requests to avoid overloading the target server.
• Using proxies : Distribute the requests across multiple IP addresses.

4. Comply with the robots.txt file and the legal framework

Bypassing a security measure doesn't mean you can do whatever you want. Web scraping is still regulated, and ignoring those rules could cost you dearly.

• Respects the robots.txt file : It specifies which pages the site allows or prohibits from being crawled. This is the first rule of courtesy to follow.
• Review the Terms of Use : A website can prohibit scraping through its Terms of Service.
• Keep the GDPR in mind : If you collect some personal data, you must comply with the European regulation governing the collection and processing of such data.

What are the alternatives to web scraping with Laravel?

Laravel remains a convenient way to integrate web scraping into a PHP application existing one. But depending on your project, other solutions may sometimes be more suitable. Here are the two main alternatives, along with their advantages and limitations.

Web Scraping with Python

Python is the most widely used language for web scraping. It relies on mature libraries such as Scrapy and BeautifulSoup.

• Advantages : a very rich ecosystem, native support for large-scale crawling, and a huge community.
• Disadvantages : You have to step outside the Laravel ecosystem. This isn't very practical if your site and API are already running on PHP.

In practical terms, you should choose Python if web scraping is at the heart of the project—not just one component of your application.

Code-free tools

More and more tools are available for web scraping without coding, sometimes with the help of the’AI. The best-known ones are Bright Data, Octoparse and Apify.

• Advantages : visual configuration, built-in JavaScript page management and error handling, no code to write.
• Disadvantages : paid subscription; less fine-grained control than a custom Laravel scraper.

FAQs

How do I scrape a login-protected website with Laravel?

The principle is simple: you simulate a connection, then reuse the session to access the protected pages. Specifically, you need to:

1. Simulate the connection with a POST request that sends the email address and password to the form.
2. Keep cookies session so that subsequent requests remain authenticated.

With the HttpBrowser of Symfony, you can chain the two steps together very neatly. It instantiates the client via HttpClient::create(), and then automatically manages cookies between each request:

use Symfony\Component\BrowserKit\HttpBrowser;
use Symfony\Component\HttpClient\HttpClient;

$browser = new HttpBrowser(HttpClient::create());

$crawler = $browser->request('GET', 'https://exemple.com/login');
$form = $crawler->selectButton('Se connecter')->form();

$browser->submit($form, [
    'email'    => 'ton@email.com',
    'password' => 'ton_mot_de_passe',
]);

// Les cookies de session sont gardés automatiquement
$page = $browser->request('GET', 'https://exemple.com/compte');
echo $page->filter('h1')->text();

Please note: the HttpBrowser It handles cookies on its own between requests. So you don't have to store anything manually.

How to manage pagination when web scraping with Laravel?

To browse through several pages in a row, the idea is to locate the “Next Page” link with a CSS selector, and then to loop as long as it exists. Here's the logic:

1. Scrape the first page and extract the relevant data.
2. Detect the “Next” link” using a CSS selector.
3. Close until there are no more pages to turn to.

$url = 'https://exemple.com/blog?page=1';

while ($url) {
    $crawler = $browser->request('GET', $url);

    $crawler->filter('.article-title')->each(function ($node) {
        echo $node->text() . PHP_EOL;
    });

    // Récupère le lien "page suivante" s'il existe
    $next = $crawler->filter('a.next');
    $url  = $next->count() ? $next->attr('href') : null;
}

This loop stops on its own as soon as the selector a.next no longer returns anything.

How do I export the scraped data (to CSV, Excel, or JSON)?

Once you've retrieved your data, you can export it in several formats using Laravel:

• fputcsv() to generate a file CSV native.
• The Bookstore Maatwebsite\Excel to create a file Excel.
• The native function json_encode() for export JSON, which is handy if you're going to expose your data via an API.

How do you handle errors and exceptions during scraping?

A website may return an error, change its HTML structure, or disconnect. To build a robust scraper, you need to anticipate these scenarios:

1. Encapsulate each request in a try/catch.
2. Check the HTTP codes (404, 429, 500…) and log the error or retry after a delay.

try {
    $crawler = $browser->request('GET', $url);
    $status  = $browser->getResponse()->getStatusCode();

    if ($status !== 200) {
        \Log::warning("Réponse inattendue: $status sur $url");
    }
} catch (\Exception $e) {
    \Log::error('Scraping échoué: ' . $e->getMessage());
}

Is web scraping legal or illegal?

The legality of web scraping depends on the target site and how you use the data. Here are three points to keep in mind:

• Sui generis Right in Databases the European Directive 96/9/EC protects the contents of a database. Extracting a significant portion of a website may be prohibited.
• Terms of use : the ruling Ryanair vs. PR Aviation (CJEU, 2015) confirmed that a website may prohibit scraping in its Terms of Service, even in the absence of copyright protection.
• RGPD : As soon as you collect some personal data (emails, names, profiles), you are subject to the European regulation. Exercise the utmost caution.

As a general rule, always respect the file robots.txt, limit the frequency of your requests and avoid collecting personal data without a legal basis. Do you have a question or a specific situation? Let us know in the comments.