dano/websucker-pip

Daniel Hladek 52118b38d3 zz

2020-05-13 15:20:20 +02:00

522 B

Raw Blame History

Websucker

Agent for Sucking the of Web

Features

Crawling of best domains
Crawling of unvisited domains
Text mining
Evaluation of domains
Daily report
Database Summary

Requirements

Python 3
running Cassandra 3.11
optional Beanstalkd for work queue

Installation

Activate virtual environment:

python -m virtualenv ./venv
source ./venv/bin/activate

Install package:

pip install https://git.kemt.fei.tuke.sk/dano/websucker-pip/archive/master.zip

Usage

websuck --help