Sporadic inconsistent actor state is returned through multiple ActorInterfaceMethod and ReminderMethod invocations

[shaul-apiiro]

I am encountering a sporadic issue with actor's state synchronization and would love any directions or insights

Overview
I have an actor that receives a stream of events (via pub-sub)
For each event it will update the actor's state and register a-reminder
When a-reminder triggers it reads the actor's state and starts processing, then persists the state and registers b-reminder to complete the processing
When b-reminder triggers it reads the actor's state and completes processing

Infra

• Dapr 1.15.4
• dotnet-sdk for all components
• rabbitmq as pub-sub backend
• postgresql as actor-state-store.

The issue
Sometimes I get the following flow:

1. event1 received by actor, state1 is persisted, and a-reminder is registered
2. a-reminder triggers, reads state1, unregisters itself, persists state1.1 and registers b-reminder
3. event2 received by actor, a-reminder & b-reminder are unregistered, state2 is persisted, a new a-reminder is registered
4. a-reminder triggers and reads the actor's state instead of the expected state2 it gets state1.1
   (even more puzzling to me:
5. event3 received by actor and sees state2 in state store )

Worth noting that this is not consistent, sometimes the reminder will get the expected state2 even though execution order is the same

What I looked for?

• tried reproducing with a simpler setup and could not reproduce (yet)
• tried to lookup etag errors from state-store, didn't find anything
• tried to lookup any errors in dapr-sidecar, didn't find anything
• actor logs indicate serial execution of the above flow

Expected Behavior

State store should either indicate some ETag error on StateManager.SaveStateAsync() or consistently the latest save data should return upon StateManager.GetStateAsync

Actual Behavior

Inconsistent state is returned from different reminder invocations

Steps to Reproduce the Problem

Still trying to create a minimal reproduction

Release Note

N/A

[shaul-apiiro]

I did compare my calls to StateManager.SetStateAsync followed immediately by StateManager.SaveStateAsync() vs. postgresql execution log
I see 24 calls in my app log, but only 20 INSERT INTO state calls in postgresql log

[olitomlinson]

@shaul-apiiro how many pods/replicas is your Actor app running with when you see this problem occur?

(asking incase of a split brain scenario)

[shaul-apiiro]

@olitomlinson this happens in a local docker-compose env running one container per component.

Note:
Opened this per request from @WhitWaldo on discord