crawlee

🥇Gold

Crawlee is a powerful web scraping and browser automation library for Node.js, enabling developers to create reliable crawlers in JavaScript and TypeScript. It supports data extraction for AI applications, including LLMs and GPTs, and offers features like proxy rotation and both headful and headless modes.

22,7131,1720Updated Today

Intermediate15 minutes to implementdevelopment

Saves ~45 min per use

Quick InstallView Source

claude install apify/crawlee

Works with:

ChatGPTGitHub Copilot

Overview

About This Skill

https://crawlee.dev/docs

How to Use

["1. Install Crawlee in your Node.js project using npm: `npm install crawlee`.","2. Choose the appropriate crawler type (e.g., CheerioCrawler for static pages or PuppeteerCrawler for dynamic content).","3. Define your scraping targets, including URLs and data extraction rules. Use selectors to target specific HTML elements.","4. Implement proxy rotation by configuring the `proxyConfiguration` option with a list of proxy URLs.","5. Run your crawler in headless mode by setting `headless: true` in the crawler configuration. Monitor performance and adjust settings as needed."]

Use Cases

Extracting product data from e-commerce sites for price comparison and market analysis.

Gathering competitive intelligence from industry websites to inform business strategies.

Automating data collection for research purposes, such as gathering statistics or survey results.

Downloading media files from various sources for analysis and content creation.

Best For

GrowthRevOps

Setup & Installation

Quick Install

Terminal

claude install apify/crawlee

Alternative Install (Git Clone)

git clone https://github.com/apify/crawlee

Requirements

Claude Code or compatible AI agent
Works with: ChatGPT, GitHub Copilot

Quick Start Guide

Install the Skill

Copy the install command above and run it in your terminal.

Open Your AI Agent

Launch Claude Code, Cursor, or your preferred AI coding agent.

Try It Out

Use the prompt template or examples below to test the skill.

Customize

Adapt the skill to your specific use case and workflow.

Usage Examples

Prompt Template

Using Crawlee, scrape the latest product reviews from [ECOMMERCE_SITE] for [PRODUCT_CATEGORY]. Extract key metrics like average rating, common praise points, and frequent complaints. Save the data in a structured JSON format for analysis. Use proxy rotation to avoid IP blocking and run in headless mode for efficiency.

Example Output

Successfully scraped 1,243 product reviews from Amazon for the 'Bluetooth Headphones' category. The average rating is 4.2 stars, with common praise points including 'comfortable fit' (mentioned 312 times) and 'excellent sound quality' (287 mentions). Frequent complaints include 'battery life could be better' (143 mentions) and 'connectivity issues' (98 mentions). Data saved in structured JSON format with fields for review text, rating, and date. Proxy rotation successfully avoided IP blocking, and headless mode ensured efficient scraping.