Installation

If you only want to install the package and are not interested in developping it the simplest way is to use pipx. The package will be installed in a temporary virtualenv and will be available on your path.

pipx install ./

If you want to install the package for development and virtualenv is installed on your system, the Makefile provides a venv target which creates a new virtual environment in the venv directory and installs the package and the dev requirements.

make venv

# Equivalent to
virtualenv -p python3.10 venv
source venv/bin/activate
pip install -U pip
pip install -r requirements-dev.txt
pip install -e .

If you want a custom python version or virtual environment location run the individual steps manually

virtualenv -p python<py-version> <venv-dest-dir>
# e.g.
virtualenv -p python3.10 venv

If virtualenv is not installed on your system create a new virtual environment using the system’s python version

python3 -m venv venv

Now install the package and its dependencies in the virtual environment

source venv/bin/activate
pip install -r requirements-dev.txt
pip install -e .

Great! The library installs the wms command, providing four entrypoints:

wms app      # run scraping and persistance services concurrently
wms scrape   # scrape sites according to the configuration's specifications
wms persist  # consume metrics from Kafka and persist them in Postgres
wms tail     # output latest records from Postgres metrics table

At this point the system is not fully functional yet, Kafka and Postgres instances need to be setup locally or in the cloud.