Skip to main content

Crawl Framework

Introduction​

Crawl is a web scraping and crawling framework for Dart, a programming language developed by Google. It provides a convenient and efficient way to extract data from websites and automate web scraping tasks. With Crawl, developers can easily build web scrapers, crawlers, and data extraction applications.

Features​

  1. Easy to Use: Crawl provides a simple and intuitive API, making it easy for developers to get started with web scraping. It abstracts the complexity of HTTP requests and HTML parsing, allowing developers to focus on extracting the desired data.

    import 'package:crawl/crawl.dart';

    void main() async {
      final response = await Crawl.get('https://example.com');
      print(response.statusCode);
      print(response.body);
    }

    Output:

    200
    <html>...</html>

  2. Flexible HTML Parsing: Crawl leverages the powerful HTML parsing capabilities of the html package. It provides a convenient way to navigate and extract data from HTML documents using CSS selectors.

    import 'package:crawl/crawl.dart';

    void main() async {
      final response = await Crawl.get('https://example.com');
      final document = CrawlDocument.parse(response.body);
      final title = document.querySelector('title').text;
      print(title);
    }

    Output:

    Example Domain

  3. Authentication Support: Crawl supports various authentication methods, such as basic authentication and OAuth. This allows developers to scrape websites that require authentication to access certain data or resources.

    import 'package:crawl/crawl.dart';

    void main() async {
      final client = CrawlClient(auth: CrawlBasicAuth('username', 'password'));
      final response = await client.get('https://api.example.com/data');
      print(response.body);
    }

    Output:

    {"data": [...] }

  4. Concurrency and Parallelism: Crawl supports concurrent and parallel execution of web scraping tasks, which can significantly improve the performance and efficiency of data extraction. Developers can leverage Dart's async and await keywords to perform concurrent HTTP requests and data processing.

    import 'package:crawl/crawl.dart';

    void main() async {
      final urls = ['https://example.com/page1', 'https://example.com/page2'];
      final responses = await Future.wait(
          urls.map((url) => Crawl.get(url)),
          eagerError: true,
      );
      responses.forEach((response) => print(response.body));
    }

    Output:

    <html>...</html>
    <html>...</html>

  5. Middleware Support: Crawl allows developers to define middleware functions that can intercept and modify requests and responses. This provides a way to add custom headers, handle cookies, or perform additional processing before and after each request.

    import 'package:crawl/crawl.dart';

    void main() async {
      final client = CrawlClient(
        middleware: [
          (request, next) async {
            // Add custom headers
            request.headers['User-Agent'] = 'MyScraper';
            return await next(request);
          },
        ],
      );
      final response = await client.get('https://example.com');
      print(response.body);
    }

    Output:

    <html>...</html>

Examples​

  1. Scraping a Website:

    import 'package:crawl/crawl.dart';

    void main() async {
      final response = await Crawl.get('https://example.com');
      final document = CrawlDocument.parse(response.body);
      final links = document.querySelectorAll('a');
      links.forEach((link) => print(link.attributes['href']));
    }

  2. Crawling Multiple Pages:

    import 'package:crawl/crawl.dart';

    void main() async {
      final baseUrl = 'https://example.com';
      final response = await Crawl.get(baseUrl);
      final document = CrawlDocument.parse(response.body);
      final links = document.querySelectorAll('a');
      final urls = links.map((link) => '$baseUrl/${link.attributes['href']}');
      final responses = await Future.wait(
          urls.map((url) => Crawl.get(url)),
          eagerError: true,
      );
      responses.forEach((response) => print(response.body));
    }

Conclusion​

Crawl is a powerful web scraping and crawling framework for Dart that simplifies the process of extracting data from websites. Its rich features, such as easy-to-use API, flexible HTML parsing, authentication support, concurrency, and middleware, make it a valuable tool for developers working on web scraping projects. For more information and detailed documentation, please visit the official Crawl website.