Configuration#

Dynaconf is used to configure the system. Its pros are:

it handles multiple environments,
it handles host-specific settings & secrets
both global and per-environment defaults can be provided
configuration can be sourced from environment variables as well
TOML is simple and straightforward to read and write

The configuration directory defaults to the conf directory in the current working directory where the command is invoked. Both the directory and the environment used (dev by default) can be overriden on the command-line with the config-dir and env options respectivelly. The site specs can be inlined in the TOML config, or provided using a separate YAML file passed with the site-specs-file option in the CLI.

The file settings.toml provided in the conf directory should work out of the box with docker compose.

Secrets can be defined in a .secrets.toml file in the same directory or passed using environment variables. Envvars have the advantage of being able to be set by external programs. Any additional configuration is merged in the global configuration, superseding it. The file .secrets.sample.toml shows an example of how to set a postgres password for the prod environment.

The configuration file has a section per environment. Each environment may define the following sub-sections: kafka, postgres, sentry, server.

Settings file#

General config#

Parameter	Default Value	Description
`certs_dir`	`certs`	Relative or absolute path to TLS certs dir. Defaults to the ‘certs’ sub-directory in working directory.
`logs_dir`	`logs`	Relative or absolute path to logging directory. Will attempt to create it if it does not exist
`log_level`	`WARNING`	Default log level.
`metrics_topic`	`<env>.site-metrics.v1`	The name of the Kafka topic where to send the metrics to.
`monitor_specs_filename`	`sites-default.yaml`	File name with monitor specs
`default_scraping_frequency_ms`	`15000`	The default scraping freqency is milliseconds (can be adjusted per-site).

Kafka#

Parameter	Default Value	Description
`bootstrap_servers`	`localhost:29092`	The URI to the Kafka servers where to publish and consume messages.
`producer_compression_type`	`snappy`	Type of compression to be used by rest proxy producers
`producer_acks`	`1`	Level of consistency desired by each producer message sent on the rest proxy. More on Kafka Producer
`producer_linger_ms`	`0`	Time to wait for grouping together requests. More on Kafka Producer
`cafile`	Empty for dev and test. `./certs/<env>/kafka/ca.pem` for staging and production	The path to the SSL CA certificate.
`certfile`	Empty for dev and test. `./certs/<env>/kafka/service.cert` for staging and production	The path to the SSL certfile.
`keyfile`	Empty for dev and test. `./certs/<env>/kafka/service.key` for staging and production	The path to the SSL keyfile.

Postgres#

Parameter	Default Value	Description
`user`	`wms_<env name>`	DB user
`dbname`	`wms_<env name>`	DB name
`password`	`wms_<env name>`, empty for staging & prod	DB password
`host`	`localhost` for dev & test, empty for staging & prod	Server hostname
`port`	`25432` for dev & test, `5432` for staging & prod	Server port

Sentry#

All keys and values will be passed to the Sentry SDK as-is. The only required value is the DSN. Setting the environment variable SENTRY_DSN will also enable sentry integration. See the configuration options in the Sentry docs.

Server#

The server is based on aiohttp.

Parameter	Default Value	Description
`host`	`127.0.0.1`	Listening host for the server. Use an empty string to listen to all available networks.
`port`	`8080`	Listening port for the server.

Sites#

The monitoring spec for each site has three keys: url (required), regexp and frequency_ms (optional). Here is an example using all three. The regular expressions are compiled by the Python standard library’s re.compile().

- url: https://status.aiven.io/
  regexp: "All Systems Operational"
- url: https://status.twilio.com/
  regexp: "INCIDENTS REPORTED"
  frequency_ms: 300000
- url: https://health.aws.amazon.com/health/status
  regexp: "No incidents reported today."

Environment variables#

Variables prefixed with WMS are parsed according to the settings files structure detailed below. For instance, WMS_kafka__bootstrap_servers sets the value for the bootstrap_servers variable of the kafka section.

See the Dynaconf documentation for more details, and scripts/export-tf-outputs.sh for a usage example.

Importantly, the environment can be set by the WMS_ENV variable. The two following commands are equivalent:

wms -e prod app

WMS_ENV=prod wms app

CLI arguments#

Some options can be passed on the CLI, overriding both configuration files and environment variables.

Options:
  -c, --config-dir DIRECTORY  Config dir (default: 'conf' sub-directory)
  -e, --env TEXT              Runtime environment (default: set in `WMS_ENV`,
                              or `dev` otherwise)
  -s, --spec-file FILE        Sites specs file path
  -h, --host TEXT             Server host (default: localhost)
  -p, --port INTEGER          Server port (default: 8080)
  --debug / --no-debug        Force debug logging