-
Notifications
You must be signed in to change notification settings - Fork 15
Description
Is your feature request related to a problem? Please describe.
Snowplow Micro appears to store events entirely in memory using two lists: one for good events and one for bad. As events are stored in memory, they are not persisted beyond the lifecycle of a Snowplow Micro container.
This can pose challenges to customers using Snowplow Micro to test schemas quickly and locally by mounting them to a Snowplow Micro container. Because the container must be stopped and restarted to pick up changes to a schema, any in-memory events are lost, and comparison of pre- and post-schema-change events becomes difficult.
Describe the solution you'd like
To improve the above experience, we propose that Snowplow Micro expose optional configuration, disabled by default, to persist events to a lightweight, embedded, and file-based database system such as MapDB or SwayDB. Storing events in this way would allow users to persist events beyond the lifecycle of a single container through the use of a volume mounting the DB file.
Describe alternatives you've considered
We briefly considered a weightier approach using an additional component/container for data storage (DB server, Redis, etc) - but we believe this to overcomplicate the Snowplow Micro development experience. Since Micro is not meant for production use, a file-based persistence system seems most transparent, portable, and potentially simplest to implement.
Additional context
N/A