webscraper

🥇Gold

The webscraper skill efficiently extracts content from web pages and converts it into Markdown format, enhancing AI search applications. Ideal for developers looking to streamline data collection and improve content accessibility.

256230Updated 4d ago

Intermediate15 minutes to implementdevelopment

Saves ~60 min per use

Quick InstallView Source

claude install zzzgydi/webscraper

Works with:

ChatGPTGitHub Copilot

Overview

About This Skill

How to Use

[{"step":"Identify the target URL and ensure it's accessible. For dynamic pages (e.g., React/Angular apps), use tools like Puppeteer or Playwright to wait for content to load before scraping.","tip":"Check the website's robots.txt file (e.g., [URL]/robots.txt) to ensure scraping is permitted. Respect rate limits to avoid IP bans."},{"step":"Use the prompt template to generate a Markdown extraction command. Replace [URL] with the actual webpage address.","tip":"For complex pages, break the extraction into smaller chunks (e.g., scrape the sidebar separately from the main content) to improve accuracy."},{"step":"Run the command in your preferred environment (e.g., Claude, ChatGPT, or a local script). If using an AI tool, paste the prompt and provide the URL when prompted.","tip":"For large-scale scraping, consider using a dedicated library like `python-markdownify` or `turndown` in a script to handle edge cases (e.g., malformed HTML)."},{"step":"Review the output Markdown for accuracy. Manually clean up any remaining artifacts (e.g., stray HTML tags, incomplete sentences) before using the data.","tip":"Use a Markdown linter (e.g., `markdownlint`) to validate the output and ensure consistency with your formatting standards."},{"step":"Integrate the extracted Markdown into your workflow. For example, store it in a vector database for AI search, or use it as input for a content generation tool.","tip":"If the data is time-sensitive, set up a scheduled job (e.g., using GitHub Actions or cron) to re-scrape the page periodically and update your database."}]

Use Cases

Extracting product details from e-commerce sites

Gathering market research data from competitor websites

Converting blog content into Markdown for easier editing

Best For

GrowthMarketing

Setup & Installation

Quick Install

Terminal

claude install zzzgydi/webscraper

Alternative Install (Git Clone)

git clone https://github.com/zzzgydi/webscraper

Requirements

Claude Code or compatible AI agent
Works with: ChatGPT, GitHub Copilot

Quick Start Guide

Install the Skill

Copy the install command above and run it in your terminal.

Open Your AI Agent

Launch Claude Code, Cursor, or your preferred AI coding agent.

Try It Out

Use the prompt template or examples below to test the skill.

Customize

Adapt the skill to your specific use case and workflow.

Usage Examples

Prompt Template

Extract the main content from the webpage at [URL] and convert it into clean Markdown format. Preserve headings, lists, tables, and code blocks while removing navigation menus, ads, and boilerplate text. If the page contains structured data (e.g., tables, JSON), format it as a Markdown table or code block. Exclude any interactive elements like buttons or forms. Return only the Markdown content without any additional commentary.

Example Output

```markdown
# Forma Workshops: Bringing Nature Indoors

## Upcoming Sessions

### Hand-Tied Bouquets
**Date:** October 15, 2023
**Time:** 6:00 PM - 8:00 PM
**Location:** Greenhouse Studio, 123 Botanical Lane
**Instructor:** Maria Chen
**Description:** Learn the art of creating elegant, hand-tied bouquets using seasonal flowers. Perfect for beginners or those looking to refine their skills. All materials provided.

**What You'll Take Home:**
- A hand-tied bouquet of your own design
- Care instructions for your arrangement
- A list of local flower suppliers

**Price:** $65 per person

### Kokedama Workshop
**Date:** November 5, 2023
**Time:** 2:00 PM - 4:00 PM
**Location:** Terrarium Room, 456 Nature Way
**Instructor:** Javier Rodriguez
**Description:** Create your own moss ball (kokedama) in this hands-on workshop. Explore traditional Japanese gardening techniques and take home a living piece of art.

**Materials Included:**
- Sphagnum moss
- Wire and twine
- Small houseplant (e.g., fern or pothos)
- Decorative stones

**Price:** $75 per person (includes plant)

## Workshop Schedule

| Workshop Type       | Frequency | Duration | Price  |
|---------------------|-----------|----------|--------|
| Hand-Tied Bouquets  | Monthly   | 2 hours  | $65    |
| Kokedama            | Biweekly  | 2 hours  | $75    |
| Terrariums          | Monthly   | 2.5 hours| $85    |
| Wreath-Making       | Quarterly | 3 hours  | $95    |

## Testimonials

> "The kokedama workshop was magical! I never thought I could create something so beautiful. Highly recommend to anyone who loves plants." — Sarah K.

> "The hand-tied bouquet class exceeded my expectations. Maria’s teaching style is clear and encouraging. I’ll definitely be back for more!" — James L.

## FAQ

**Q: Do I need prior experience?**
A: No! All workshops are designed for beginners. Our instructors guide you through every step.

**Q: What should I bring?**
A: Just yourself! All materials and tools are provided. Aprons are available, but feel free to wear comfortable clothing.

**Q: Can I gift a workshop?**
A: Absolutely! Gift certificates are available online or by request. Contact us at info@forma-workshops.com.
```