Support hierarchical namespaced Azure storage accounts

[matthewhudsonedb]

Is your feature request related to a problem? Please describe.
Azure allow for the provisioning of hierarchical namespaced storage accounts.
It would be good if we could support this as part of the Azure storage integration

Describe the solution you'd like
Have a provisioned Azure Storage Account with hierarchical namespaced enabled be used with Loki.

Describe alternatives you've considered
N/A

Additional context
At present when you use a hierarchical namespaced storage account you receive errors after the index compaction has run. The backend index sync attempts to download the tenant directory as it believes it is a file. There errors seen in Loki-backend are:

level=error ts=2025-11-18T04:07:17.55822398Z caller=index_set.go:308 table-name=index_20409 msg="sync failed, retrying it" err="open /var/loki/index_cache/index_20409/fake: is a directory"
open /var/loki/index_cache/index_20409/fake: is a directory
error initialising module: store
github.com/grafana/dskit/modules.(*Manager).initModule
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138
github.com/grafana/dskit/modules.(*Manager).InitModuleServices
	/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108
github.com/grafana/loki/v3/pkg/loki.(*Loki).Run
	/src/loki/pkg/loki/loki.go:531
main.main
	/src/loki/cmd/loki/main.go:129
runtime.main
	/usr/local/go/src/runtime/proc.go:283
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1700
level=info ts=2025-11-18T04:07:17.567952634Z caller=util.go:77 table-name=index_20409 file-name=fake msg="downloaded file" total_time=9.212318ms
level=error ts=2025-11-18T04:07:17.568410124Z caller=index_set.go:308 table-name=index_20409 msg="sync failed, retrying it" err="open /var/loki/index_cache/index_20409/fake: is a directory"
level=error ts=2025-11-18T04:07:17.56842428Z caller=index_set.go:104 table-name=index_20409 msg="failed to initialize table, cleaning it up" table=index_20409 err="open /var/loki/index_cache/index_20409/fake: is a directory"
level=error ts=2025-11-18T04:07:17.568892675Z caller=log.go:223 msg="error running loki" err="open /var/loki/index_cache/index_20409/fake: is a directory\nerror initialising module: store\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/v3/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:531\nmain.main\n\t/src/loki/cmd/loki/main.go:129\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:283\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1700"

Some additional details for what hierarchical namespaced Azure storage account is: https://stackoverflow.com/questions/57990850/what-is-hierarchical-namespace-in-microsoft-azure-data-lake-storage-gen2

[ringerc]

Perhaps related to https://learn.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-known-issues#blob-storage-apis

This section describes issues and limitations with using blob APIs, NFS 3.0, and Data Lake Storage APIs to operate on the same data.

When you use the List Blobs operation without specifying a delimiter, the results include both directories and blobs. If you choose to use a delimiter, use only a forward slash (/). This is the only supported delimiter.

If you use the Delete Blob API to delete a directory, that directory is deleted only if it's empty. This means that you can't use the Blob API delete directories recursively.

Also note that AFAIK this is happening with the Thanos Object Store client used for Azure storage.

Some guesswork suggests this is happening in/around

loki/pkg/storage/stores/shipper/indexshipper/storage/util.go

Line 45 in bd99fa5

func DownloadFileFromStorage(destination string, decompressFile bool, sync bool, logger log.Logger, getFileFunc GetFileFunc) error {

[ringerc]

@Anhui-tqhuang's explanation in thanos-io/objstore#239 covers some of the changed behaviour in gen2 buckets.

I suspect the actual case of this problem is picked out in their test case here:

	expected = []string{"obj_5.some", "id1/"}
	if os.Getenv("IS_AZURE_DATA_LAKE_GEN2") == "true" && bkt.Provider() == AZURE {
		expected = []string{"obj_5.some", "id1/", "id2/"} // Azure Data Lake Gen2 keeps empty dirs.
	}

What I think is happening is that the compactor expects that if fake/a is deleted, leaving fake/ "empty", then the fake directory disappears from the storage completely. This is true in gen1 because in gen1 there is no directory, there are only object-keys with delimiters in them. In gen2 the directory exists, and is implicitly created when an azblob client is used to create fake/a, but is not implicitly deleted again when the azblob client is used to delete fake/a.

If this is correct, it should be possible to work around this issue by:

• deleting empty directories in the compactor's working tree; and/or
• using a different file name for the local temp-file download that will not clash with path-component name used in the bucket