wms.utils

Utility functions.

Module Contents

Functions

build_log_filepath(→ Path | None)	Absolute path construction from some base dir and filename.
iter_random_spec_result(...)	Iterator over random specs and associated results.
send_batch_helper(→ list[asyncio.futures.Future])	Helper to build and publish payloads to some topic in batch.

wms.utils.build_log_filepath(base_dir: Path | str | None = None, filename: Path | str | None = None) → Path | None

Absolute path construction from some base dir and filename.

wms.utils.iter_random_spec_result(seed: int | float | str | bytes | None = 42, num_samples: int | None = None, status_codes: Sequence[int] | None = None) → collections.abc.Iterator[tuple[wms.config.WebsiteSpec, wms.models.WebsiteScrapeResult]]

Iterator over random specs and associated results.

async wms.utils.send_batch_helper(producer: aiokafka.producer.AIOKafkaProducer, topic: str, payloads: collections.abc.Sequence[tuple[Any, Any]], key_serializer: wms.typing.Serializer = orjson.dumps, value_serializer: wms.typing.Serializer = orjson.dumps) → list[asyncio.futures.Future]

Helper to build and publish payloads to some topic in batch.

When the return metadata is None, it means the batch is closed and the message was not added. Flush the batch and add the message to the new batch.

Reference: https://aiokafka.readthedocs.io/en/stable/examples/batch_produce.html