Crawlab Lite

中文 | English

Lite version of Crawlab, golang-based web crawler management platform, supporting crawlers in any language.

Compared with Crawlab, this lite version focuses on crawler management on a single machine, it runs independent of any external database and removes a lot of non-essential features.

⚠️ This version is still in early development and some features may be unstable.

Quick Start

Docker Compose

1. Create docker-compose.yml in any directory as follows:

version: '3'
services:
  master:
    image: zkqiang/crawlab-lite:latest
    container_name: master
    ports:
      - "8080:8080"
    volumes:
      - "./data:/app/data"  # persistent volume

2. Run the command in this directory:

docker-compose up -d

3. Visit http://localhost:8080

Source Code

1. Clone repository

git clone https://github.com/crawlab-team/crawlab-lite
cd crawlab-lite

2. Run backend

cd backend
go run main.go

3. Run frontend

cd ../frontend
npm i && npm run serve

4. Visit http://localhost:8080

Screenshot

Spider List

Task List

Schedule List

Comparison with Crawlab

	Crawlab Lite	Crawlab
Cross-language spider	✅	✅
Multi node	❌	✅
Cron	✅	✅
Log output	✅	✅
Spider version manage	✅	❌
Analytics	❌	✅
Notification	❌	✅
Online code editor	❌	✅
Configurable spider	❌	✅
SDK	❌	✅