Support mirrored ObjectStore

[shaunco]

Proposed change

The nats.go jetstream.CreateObjectStore API should support creating mirrors and sourced object stores, which means adding Mirror *StreamSource and Sources []*StreamSource to ObjectStoreConfig.

Currently it requires a call to CreateStream with all the right options:

js.CreateStream(ctx, jetstream.StreamConfig{
			Name:        "OBJ_MYBUCKET",
			Description: "Mirror of OBJ_MYBUCKET",
			Mirror: &jetstream.StreamSource{
				Name:   "OBJ_MYBUCKET",
				Domain: "myleafdomain"
			},
			Replicas:    1,
			MaxAge:      time.Hour * 24 * 14,
			MaxBytes:    -1,
			Discard:     jetstream.DiscardNew,
			AllowRollup: true,
			AllowDirect: true,
			Storage:     jetstream.FileStorage,
			Compression: jetstream.S2Compression,
		})

The manual stream creations works great since #5106 was fixed, but first class support would align with KV and streams. Also, similar request is open for nats.js: nats-io/nats.js#155

Use case

We have leafnodes that write objects to a local object store bucket, and want those objects mirrored to an object store in the cluster. Ideally, we'd like a WorkStream style policy that removes them from the leafnode once mirrored, but for now we just use a short (~12hr) TTL on the leafnode and a longer (14 day) TTL for the cluster side mirror.

Contribution

No response

[Jarema]

Unfortunately, that is not that straightforward.

To have properly working mirrors, you need to adapt subject names too and use subject transforms.
Additionally, deletes of objects will not propagete to mirrors, as mirrors/sources have different limits than the original stream, by design.

[shaunco]

If I keep the stream name the same in both domains, it seems to work just fine for copying and access.

If WorkStream style policy was supported, I would expect the objects to move from the origin to the mirror (copy, then delete from origin), but with the above mirror stream I wouldn't expect that. I simply manage cleanup on the origin through a shorter TTL than the mirror.

[ripienaar]

Same stream in multiple domains has numerous bugs and issues. You absolutely should not do that.

Until object store had a rethink I don’t think we should allow this.

[shaunco]

To be clear - not the same stream, just the same name to avoid breaking $JS.M.* hashed names that would otherwise need mappings to fix. Stream A in domain X has a mirror of Domain: Y, Stream A. The JetStream on Leaf Nodes docs specifically say:

Please be aware that each domain is an independent name space. Meaning, inside the same account it is legal to use the same stream name in different domains.

Do you have references to issue IDs I should be on the lookout for? It absolutely works for our use case currently (moving objects from the leafnode to the hub). Outright disallowing it would completely break us.

[ripienaar]

Certain kinds of consumers break and can potentially double ack between streams. Object store doesn’t surface most of them in its usage patterns but it’s certainly not good to use in the general case.

[shaunco]

A consumer unexpectedly jumping over a domain boundary to ack a stream in a totally different name space sounds like a very serious bug in nats-server that should be addressed - especially if ACLs don't allow it, and the default leaf node supporession of $JS.API.> is being bypassed. Additionally, identical stream names in different domains for KVs and generic streams are used in many nats unit tests and examples showcasing leaf nodes, and all of those should be changed or massive blinking red warnings that domains don't actually provide independent namespaces should be added everywhere.

[shaunco]

@Jarema - the requirement to adapt/map subject names seems to have been solved with nats-io/nats-server#5106 . Since you were active on that issue, any details on what I might be missing? Perhaps I'm not running into this because I just keep the stream name ("OBJ_TEST") the same on both the leaf and the hub?

[Jarema]

@shaunco yes, this is possible that you are not running in some of the issues because of that.
However, number of issue is bigger - including lack of ability to propagate deletes to mirrors, and others.

It can for sure work for your use case, and you can manually edit & create the streams, but it's not a good idea in current shape of Object Store to make it generally available, as it will cause problems and confusion to others.

When we rethink and rework Object Store (also after server supports features it requires), that should be safe for general purpose.