Crawlee is a powerful web scraping and browser automation library for Node.js, enabling developers to create reliable crawlers in JavaScript and TypeScript. It supports data extraction for AI applications, including LLMs and GPTs, and offers features like proxy rotation and both headful and headless modes.
claude install apify/crawleehttps://crawlee.dev/docs
Extracting product data from e-commerce sites for price comparison and market analysis.
Gathering competitive intelligence from industry websites to inform business strategies.
Automating data collection for research purposes, such as gathering statistics or survey results.
Downloading media files from various sources for analysis and content creation.
claude install apify/crawleegit clone https://github.com/apify/crawleeCopy the install command above and run it in your terminal.
Launch Claude Code, Cursor, or your preferred AI coding agent.
Use the prompt template or examples below to test the skill.
Adapt the skill to your specific use case and workflow.
Using Crawlee, can you create a web scraper for [WEBSITE] that extracts [DATA] in [FORMAT]? Please provide the code and any necessary explanations.
```javascript
const { PlaywrightCrawler } = require('crawlee');
const crawler = new PlaywrightCrawler({
requestHandler: async ({ request, page }) => {
const data = await page.evaluate(() => {
return Array.from(document.querySelectorAll('.item')).map(item => ({
title: item.querySelector('.title').innerText,
price: item.querySelector('.price').innerText,
}));
});
console.log(data);
},
});
await crawler.run(['https://example.com/products']);
```
### Output:
- **Title**: Product 1
- **Price**: $10.00
- **Title**: Product 2
- **Price**: $15.00
This code snippet demonstrates how to scrape product titles and prices from a sample e-commerce website.Build powerful spreadsheets with ease.
Your one-stop shop for church and ministry supplies.
Unlock data insights with interactive dashboards and collaborative analytics capabilities.
Automate your browser workflows effortlessly
The new way to work
Enhance performance monitoring and root cause analysis with real-time distributed tracing.