# Websucker

Agent for Sucking the of Web

## Features

- Crawling of best domains
- Crawling of unvisited domains
- Text mining
- Evaluation of domains
- Daily report
- Database Summary

## Requirements

- Python 3
- running Cassandra 3.11
- optional Beanstalkd for work queue

## Installation

Activate virtual environment:

    python -m virtualenv ./venv
    source ./venv/bin/activate

Install package:

    pip install https://git.kemt.fei.tuke.sk/dano/websucker-pip/archive/master.zip

### Initialize and setup database

If you have Cassandra installed, first initialize the database schema using the cqlsh command, the schema can be found in the schema.sql file


You set up the database using an environment variable (if it is on another machine):


    export CASSANDRA_HOST=localhost

    export CASSANDRA_PORT=9142


## Usage

    websuck --help
   


### Create initial domain list

Save the list of domains to a file, e.g.

    echo www.sme.sk > domains.txt


### Visit initial domains in file


    websuck --visit file domains.txt

### Visit unvisited domains

    websuck --visit unvisited 100