Skip to content

Enable optional event persistence #150

@andrewlangston

Description

@andrewlangston

Is your feature request related to a problem? Please describe.
Snowplow Micro appears to store events entirely in memory using two lists: one for good events and one for bad. As events are stored in memory, they are not persisted beyond the lifecycle of a Snowplow Micro container.

This can pose challenges to customers using Snowplow Micro to test schemas quickly and locally by mounting them to a Snowplow Micro container. Because the container must be stopped and restarted to pick up changes to a schema, any in-memory events are lost, and comparison of pre- and post-schema-change events becomes difficult.

Describe the solution you'd like
To improve the above experience, we propose that Snowplow Micro expose optional configuration, disabled by default, to persist events to a lightweight, embedded, and file-based database system such as MapDB or SwayDB. Storing events in this way would allow users to persist events beyond the lifecycle of a single container through the use of a volume mounting the DB file.

Describe alternatives you've considered
We briefly considered a weightier approach using an additional component/container for data storage (DB server, Redis, etc) - but we believe this to overcomplicate the Snowplow Micro development experience. Since Micro is not meant for production use, a file-based persistence system seems most transparent, portable, and potentially simplest to implement.

Additional context
N/A

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions