crawl4ai-mcp-server

## Setup Instructions for crawl4ai-mcp-server

To set up the crawl4ai-mcp-server, follow these steps:

1. **Install Dependencies**: Ensure you have Python 3.8 or later installed. You can install the required dependencies using pip:
   ```bash
   pip install crawl4ai-mcp-server
   ```

2. **Configure the Server**: Create a configuration file named `config.json` with the following content:
   ```json
   {
     "port": 8000,
     "allowed_domains": ["example.com"],
     "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
   }
   ```

3. **Run the Server**: Start the server by running the following command:
   ```bash
   python -m crawl4ai_mcp_server --config config.json
   ```

4. **Integrate with AI Agents**: Use the OpenAI Agents SDK or other MCP-compatible tools to integrate the crawl4ai-mcp-server into your AI workflows. Here is an example of how to use it with the OpenAI Agents SDK:
   ```python
   from openai import OpenAI
   from openai.tools import Tools
   
   client = OpenAI()
   tools = Tools()
   
   # Add the crawl4ai-mcp-server as a tool
   tools.add_tool(
       name="crawl4ai-mcp-server",
       description="A lightweight MCP server for web scraping and crawling.",
       url="http://localhost:8000"
   )
   
   # Use the tool in your AI agent
   response = client.chat.completions.create(
       model="gpt-4",
       messages=[{"role": "user", "content": "Scrape data from example.com."}],
       tools=[tools]
   )
   ```

5. **Monitor and Maintain**: Regularly monitor the server logs and maintain the server to ensure it runs smoothly.