jztools.sqlalchemy.threaded_insert_cache#

Caches data record inserts and carries them out in batches in a separate thread. The thread is started only when necessary.

Classes

ThreadedInsertCache(data_table, engine[, ...])

Caches data record inserts and carries them out in batches in a separate thread.
class jztools.sqlalchemy.threaded_insert_cache.ThreadedInsertCache(data_table: Table, engine: Engine, max_queue_size: int = 1000, max_batch_size: int | None = None, attempts=inf)#

Bases: object

Caches data record inserts and carries them out in batches in a separate thread. The worker thread is started only when necessary and stopped if no inserts are pending. The batch size will adapt to the available records, with the thread inserting all available records (up to max_batch_size) as soon as possible.

To ensure the queue is empty and join the worker thread, call flush().

Parameters:

  • data_table – The table where inserts are done.

  • engine – The sqlalchemy engine (e.g., the output of sqlalchemy.create_engine()) used to generate connections.

  • max_queue_size – Method insert() will block after this many queued items are pending.

  • max_batch_size – Max number of elements to insert at once. Equal to max_queue_size by default.

  • attempts – Number of times to attempt a re-write upon database operational errors (e.g., file locked).

flush()#

Guarantees that at least all items inserted before flush was called are written. Blocks any insert operations until the flush finishes.