Feature Request: Add MQTT5 support including Request-Response Pattern support

[Snapstromegon]

Problem:

Building an external API interface (like a TasmotaAdapter library) currently is pretty hard when you want to use MQTT, because you can't send a command and get a response.

Possible Solution:

MQTT5 introduces support for Request-Response patterns using Response Topics and Correlation Data which allows a command issuer to know what message is a direct response to the command and avoids race conditions.

Suboptimal Alternatives:

Changing the API

Aside from supporting the "standard" MQTT5 Request/Response pattern, an alternative that implements this on top of MQTT3 is also possible, but would significantly change the way the current way of issueing commands and receiving results works and would probably either require to break the existing API (which is really bad) or to add a new MQTT API that specifically allows issueing commands with correlation data (which bloats the API surface and is also suboptimal).

Using the HTTP API

Using the HTTP API is not always possible, because most web services (e.g. when using a webapp to issue commands) are behind a HTTPS endpoint and therefore browsers won't allow access to HTTP APIs from those apps.
Also in my usecase I want to use the MQTT log to see which commands got issued when to which device. This kind of monitoring is not possible with the HTTP API (outside of fairly dirty workarounds using the MqttLog feature).

Using Berry Scripting

Scripting is disabled in default builds and is not even possible on all devices because of size constraints. This means that if someone wants to use such an adapter library, it's fairly likely that they have to reflash tasmota for it to work and to "install" the right scripts on it.

---

I'm open to other suggestions and normally I would offer to chime in with a contribution, but I don't feel like I would be able to implement a working solution here.

For reference: This started as a message in discord here: https://discord.com/channels/479389167382691863/790182550085304342/1076105810788233296

[sfromis]

As I already suggested on Discord, such features are primarily up to the MQTT library used by Tasmota. This is currently pubsubclient which can be MQTT 3.1.1 or 3.1. Not MQTT 5.

Technical detail: Berry is by default enabled on ESP32 builds, but "scripting" is an entirely different feature, not included in default builds, but also supporting ESP8266.

[Snapstromegon]

Thanks for the reply.
I searched for "MQTT" in the codebase and because of that didn't find pubsubclient.
They already have an open issue since 2018 for MQTT5 support: knolleary/pubsubclient#531

The question now is, wether Tasmota wants to wait for MQTT5 support in PubSubClient, build an own "workaround"/solution, or just postpone this idea for now.

In my opinion waiting for MQTT5 support in PubSubClient is the best aproach even though it means that this gets potentially blocked for a long time.

[barbudor]

I see moving from asynchronous to synchronous a walkback.
At first sight it makes programming easier but it's worse in term of threading, resource management.
In my business we are using Kafka, SQS, Lambda. For me, all modern computing infrastructure are based on asynchronous.

[sfromis]

The tricky thing about adding some token (to be included in responses) to the existing commands is mostly that many messages published by Tasmota are not direct responses to a particular command, even in cases where they are indirectly caused by a command being processed. This is due to Tasmota also internally being asynchronous about processing things happening, instead of simple request/response flows.

Thus, even if MQTT 5 support in pubsubclient arrived tomorrow (or getting MQTT 5 support some other way), it would not be "just so" to restructure Tasmota to have a strong correlation from commands to what is published for various reasons, not limited to commands.

[Snapstromegon]

I do not think of request/response as "synchronous", but I see where this is coming from.
In the case where I actively want to do a RPC (which executing a command in my opinion is), it's still valuable to which command a result refers to.

Right now (as I see it), it's not easily possible to actively make decision based on the state of a tasmota device without having a whole replicated state of it in store (which can move out of sync).
In general I'm also a huge fan of things like the CQRS pattern, but some things just need to associate a response to a request.

To me right now it's a little like making a DB request from a lambda function to search for a user and at some point you get a response with a user, but you don't know if the response is for your search or not.

An easy way to have a duplicated state store move out of sync is when messages get reordered on the way to or from the broker.

[sfromis]

Well, the relevant analogy instead of your db request case would be a device with 5 relays. When a relay changes state, you will get to know which relay changed state to what, but not if it was "you" causing it to change state. This is very different from not knowing if a "user" response is relevant. Thus, in the real world you generally have what you need to make a decision based on the state of the device.

This was not meant as hairsplitting about words, more that things are more straightforward with simple cases like controlling relays, instead of database requests. In other words, I think that you may be on the path of premature optimization.

[Jason2866]

There will be no support for mqtt 5.x in near future.

[s-hadinger]

Although for esp32, esp-idf supports MQTT5, it would be a major refactoring. It does not seem worth it for now.

[winksaville]

It's now going on 3 years since this particular discussion, what are the recent thoughts on MQTT5 support?

[arendst]

No change.

[s-hadinger]

Mqtt 5 would mean deep changes in Tasmota and most certainly regression. I haven’t seen any killer feature in mqtt 5 that would make the change worth the risk