feat(databricks): allow read_only option for Databricks backend

[LegrandT]

Description of changes

Allow data access through Databricks with only READ access on the table, by skipping memory table.

Issues closed

• Resolves bug: Unable to connect to Databrick with Read only roles #11598

[cpcloud]

Is there a way we can support local data (pandas dataframes, polars dataframes, etc) that doesn't require a user to have volume writing permissions?

Lots of useful features require memtables, so it'd be nice if we can make this work without this flag.

[LegrandT]

Hi @cpcloud,

Not sure what you mean by "support local table", it would be to replace the volume created by a local table instead I guess.
If memtable is useful even when only reading tables (not writing or anything else).

Then a local path would lead to a local table validation/creation, without any CREATE VOLUME required.

I'll look into it!
My only concern is the support of "local commands" if the table is local, but let's see

[LegrandT]

To check the actual code coverage I believe that ci-run-cloud tag would be needed.

Otherwise the idea is quite simple, if the flag is given and no memtable_volume is given, it will piggyback on the Polars backend for the memtable functionalities, and keep that table local.

[LegrandT]

Hi @cpcloud , do you mind having a look to see if this is on the right way or if I need to change the approach ?