Failure: clickhouse-head-x86-with-analyzer-zookeeper/rbac_1/rbac_1 - DB::Exception: Couldn't delete /var/lib/clickhouse/access/38373166-2d31-6134-6164-373733313063.sql. (FILE_DOESNT_EXIST)

[CarlosFelipeOR]

Version: 25.11.1.2598

Description

When renaming a user to an existing user name, ALTER USER ... RENAME TO ... correctly returns ACCESS_ENTITY_ALREADY_EXISTS, but the rename is still partially applied internally.

As a result:

1. user15 is renamed to user16 even though an exception is thrown.
2. Two user16 entries appear in system.users.
3. Dropping user16 and user15 removes both entries.
4. Subsequent CREATE USER OR REPLACE user15 fails with FILE_DOESNT_EXIST or ACCESS_ENTITY_ALREADY_EXISTS(on CREATE USER user15).

This indicates that the rename operation is modifying internal metadata even after the exception, leading to RBAC corruption.

Steps to reproduce the issue

CREATE USER IF NOT EXISTS user15 IDENTIFIED WITH no_password;
CREATE USER IF NOT EXISTS user16 IDENTIFIED WITH no_password;
-- Check that user15 and user16 were created:
SELECT name, storage, id FROM system.users;

-- Expected error: ACCESS_ENTITY_ALREADY_EXISTS
ALTER USER user15 RENAME TO user16;

-- Problem: even though the previous command raises an exception,
-- user15 is actually renamed and now we have two `user16` entries:
SELECT name, storage, id FROM system.users;

DROP USER IF EXISTS user16;
DROP USER IF EXISTS user15;

-- Both `user16` entries are deleted:
SELECT name, storage, id FROM system.users;

-- This is the failure Code: 107(FILE_DOESNT_EXIST) that is later caught in the regression test :
CREATE USER OR REPLACE user15;

Versions tested:

• 25.11.1.2648 - ❌ Failed
• 25.10.2.65 - ✅ Passed
• 25.9.5.21 - ✅ Passed
• 25.8.11.66 - ✅ Passed

Issue on Upstream

ALTER USER RENAME applies changes even after throwing ACCESS_ENTITY_ALREADY_EXISTS, causing RBAC corruption

---

CI Run

https://github.com/Altinity/clickhouse-regression/actions/runs/19316216971/job/55247989837

Artifacts and Reports

Artifacts will be uploaded to https://altinity-test-reports.s3.amazonaws.com/index.html#clickhouse/25.11.1.2598/19316216971/testflows/x86_64/with_analyzer/zookeeper/without_thread_fuzzer/rbac1/
Review report at https://altinity-test-reports.s3.amazonaws.com/clickhouse/25.11.1.2598/19316216971/testflows/x86_64/with_analyzer/zookeeper/without_thread_fuzzer/rbac1/report.html
Review new fails (nice) at https://altinity-test-reports.s3.amazonaws.com/clickhouse/25.11.1.2598/19316216971/testflows/x86_64/with_analyzer/zookeeper/without_thread_fuzzer/rbac1/nice-new-fails.log.txt

Failure Description

Received exception from server (version 25.11.1):
Code: 107. DB::Exception: Received from localhost:9000. DB::Exception: Couldn't delete /var/lib/clickhouse/access/38373166-2d31-6134-6164-373733313063.sql. (FILE_DOESNT_EXIST)
(query: CREATE USER OR REPLACE user15 )

[CarlosFelipeOR]

Local Retest

Executed locally using version 25.11.1.2648 (head-alpine).

• Running only the first failing test
  Command:
  python3 -u rbac/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:head-alpine --only '/rbac/part 1 syntax/alter user/I alter user default role multiple/*' --log test.log
  Result: ✅ Passed

• Running the entire suite
  Command:
  python3 -u rbac/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:head-alpine --only "/rbac/part 1/*" --log test2.log
  Result: ❌ Failed again with the same failures

• Running tests started by 'I alter user'
  python3 -u rbac/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:head-alpine --only "/rbac/part 1/syntax/alter user/I alter user*" --log test4.log
  Result: ❌ Failed again with the same failures

[CarlosFelipeOR]

CI/CD - Rerun

Rerunning: https://github.com/Altinity/clickhouse-regression/actions/runs/19316216971/job/55334292147
Result: ❌ Failed again with the same failures

[CarlosFelipeOR]

Local Retest with other versions

Version: 25.10.2.65-alpine

• Running the entire suite
  Command:
  python3 -u rbac/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:25.10.2.65-alpine --only "/rbac/part 1/*" --log test3.log
  Result: ✅ Passed

[CarlosFelipeOR]

Run pausing on fail:

Command:
python3 -u rbac/regression.py --local --clickhouse docker://clickhouse/clickhouse-server:head-alpine --pause-on-fail '/rbac/part 1/syntax/alter user/I alter user default role multiple' --only "/rbac/part 1/syntax/alter user/I alter user*" --log test6.log

After pausing the regression run with pause-on-fail and accessing the Docker container, I was able to reproduce the Code: 107 error when running:

CREATE USER OR REPLACE user15;

I also noticed that a simple:

CREATE USER user15;

fails as well, with:

Code: 493. DB::Exception: user `user15`: cannot insert because user `user15` already exists in `local_directory`. (ACCESS_ENTITY_ALREADY_EXISTS)

However, the query:

SELECT name, storage, id
FROM system.users
WHERE name = 'user15';

returns no rows, meaning ClickHouse does not show any user15 in the catalog.

I also checked the access control directories inside the container:

• /var/lib/clickhouse/access/
• /etc/clickhouse-server/

and there is no SQL file or any reference to user15 there either.

So the situation is:

• user15 does not exist in system.users
• user15 does not exist in the access directory on disk
• but ClickHouse refuses to create or replace it, because local_directory internally believes it already exists

This suggests an inconsistent RBAC state inside local_directory.

I still haven’t found the exact sequence to reproduce this manually outside the full regression run.

[CarlosFelipeOR]

Issue reproduced

I was able to reproduce the issue using only SQL commands on 25.11.1.2648.
I updated the issue description and added the steps to reproduce the problem.

[alsugiliazova]

✘ 8s 568ms  [  Fail  ] /rbac/part 1/syntax/alter user/I alter user default role multiple
Dec 08,2025 14:13:42           ⟥  Scenario I alter user default role multiple, flags:TE
                                    Requirements
                                      RQ.SRS-006.RBAC.User.Alter.DefaultRole
                                        version 1.0
Dec 08,2025 14:13:42             ⟥  Given I have a user, flags:MANDATORY|SETUP
Dec 08,2025 14:13:42               ⟥  By executing command
                                        echo -e "CREATE USER OR REPLACE user15" | clickhouse client -n 2>&1
                 4ms                    [clickhouse1] bash# echo -e "CREATE USER OR REPLACE user15" | clickhouse client -n 2>&1
            7s 934ms                    [clickhouse1] Error on processing query: Code: 32. DB::Exception: Attempt to read after eof: while receiving packet from localhost:9000, 127.0.0.1, local address: 127.0.0.1:45238. (ATTEMPT_TO_READ_AFTER_EOF) (version 25.12.1.339)
            7s 935ms                    [clickhouse1] (query: CREATE USER OR REPLACE user15
            7s 935ms                    [clickhouse1] )
             8s 59ms                    [clickhouse1] bash# echo $?
             8s 60ms                    [clickhouse1] 32
             8s 61ms                    [clickhouse1] bash#
             8s 61ms               ⟥⟤ OK executing command, /rbac/part 1/syntax/alter user/I alter user default role multiple/I have a user/executing command

That leads to server crash with address sanitizer.

Here the logs https://github.com/Altinity/clickhouse-regression/actions/runs/20030742160/job/57438996863

I took a build from last commit of this pr ClickHouse/ClickHouse#90490