Crawlee is a powerful web scraping and browser automation library for Python, enabling the creation of reliable crawlers. It supports data extraction for AI applications, including LLMs and GPTs, and offers features like proxy rotation and compatibility with BeautifulSoup and Playwright.
claude install apify/crawlee-pythonCrawlee is a powerful web scraping and browser automation library for Python, enabling the creation of reliable crawlers. It supports data extraction for AI applications, including LLMs and GPTs, and offers features like proxy rotation and compatibility with BeautifulSoup and Playwright.
Extracting product data from e-commerce sites for competitive analysis.
Automating the monitoring of competitor websites to track changes in pricing and offerings.
Collecting customer reviews from various platforms to perform sentiment analysis.
Gathering large datasets for training AI models and enhancing machine learning algorithms.
claude install apify/crawlee-pythongit clone https://github.com/apify/crawlee-pythonCopy the install command above and run it in your terminal.
Launch Claude Code, Cursor, or your preferred AI coding agent.
Use the prompt template or examples below to test the skill.
Adapt the skill to your specific use case and workflow.
Help me create a web scraper using Crawlee in Python for [WEBSITE] to extract [DATA]. I need to ensure it handles pagination and utilizes proxy rotation. Please provide a sample code snippet.
```python
from crawlee import Scraper
# Initialize the scraper
scraper = Scraper(
start_urls=['https://example.com/products'],
proxy_rotation=True
)
# Define the data extraction function
@scraper.extract
def parse_product(response):
return {
'name': response.css('h1.product-title::text').get(),
'price': response.css('.price::text').get(),
'availability': response.css('.availability::text').get(),
}
# Handle pagination
scraper.add_pagination(
next_page_selector='.pagination .next',
)
# Run the scraper
scraper.run()
```
This code snippet initializes a web scraper for products on a sample website, extracts product names, prices, and availability, and handles pagination with proxy rotation.Your one-stop shop for church and ministry supplies.
Build powerful spreadsheets with ease.
Automate your browser workflows effortlessly
Orchestrate workloads with multi-cloud support, job scheduling, and integrated service discovery features.
Manage CI/CD processes efficiently with build configuration as code and multi-language support.
Enhance performance monitoring and root cause analysis with real-time distributed tracing.