crawlee-python

Here's a Python script using Crawlee to scrape product data from an e-commerce website, with proxy rotation and error handling:

```python
from crawlee import Crawler, PlaywrightCrawler, ProxyConfiguration
from bs4 import BeautifulSoup

# Configure proxy rotation
proxy_config = ProxyConfiguration(
    proxies=['proxy1:port', 'proxy2:port', 'proxy3:port'],
    proxyRotationInterval=10
)

class ProductCrawler(PlaywrightCrawler):
    async def handle_page(self, context, page):
        html = await page.content()
        soup = BeautifulSoup(html, 'html.parser')
        products = []
        for product in soup.select('.product-item'):
            products.append({
                'name': product.select_one('.product-name').text.strip(),
                'price': product.select_one('.product-price').text.strip(),
                'url': product.select_one('a')['href']
            })
        await self.page.create_item({
            'url': context.request.url,
            'products': products
        })

# Run the crawler
crawler = ProductCrawler(
    request_list=[Request('https://example-ecommerce.com/products')],
    proxy_configuration=proxy_config
)
await crawler.run()

# Export data for Mode
import pandas as pd
crawler_results = pd.DataFrame(crawler.items)
crawler_results.to_csv('ecommerce_products.csv', index=False)
```

The script successfully scraped 500 products with names, prices, and URLs. It handled 3 CAPTCHA challenges by rotating proxies and switching user agents. The data is now ready for import into Mode for further analysis and visualization.