[CAS] Add UnifiedOnDiskCache and OnDiskCAS #114103

cachemeifyoucan · 2024-10-29T17:37:40Z

Add a new abstraction layer UnifiedOnDiskCache that adds new functions
of disk space management and data validation that builds on top of
OnDiskGraphDB and OnDiskKeyValueDB.

Build upon UnifiedOnDiskCache, it is OnDiskCAS that implements
ObjectStore and ActionCache interface for LLVM tools to interact with
CAS storage.

Add OnDiskCAS abstraction, that implements ObjectStore and ActionCache interface using OnDiskGraphDB and OnDiskKeyValueDB. Reviewers: Pull Request: #114103

Created using spr 1.3.7

llvm/include/llvm/CAS/BuiltinUnifiedCASDatabases.h

llvm/include/llvm/CAS/ObjectStore.h

llvm/include/llvm/CAS/UnifiedOnDiskCache.h

llvm/lib/CAS/ActionCaches.cpp

llvm/lib/CAS/UnifiedOnDiskCache.cpp

bogner · 2025-10-28T17:01:36Z

llvm/lib/CAS/ActionCaches.cpp

+  Error validate() const final;
+
+private:
+  std::shared_ptr<ondisk::UnifiedOnDiskCache> UniDB;


Maybe I'm missing it, but why does this need to be a shared_ptr? I don't see obvious uses where we're really relying on the refcount here.

This is just extending the lifetime of the wrapped UnifiedOnDiskCache to make sure it is not deleted when ActionCache is still in use.

Alternatively, we can leave this as a raw pointer and left the life-time management to user, but since this is not really a ref-count that is being updated often, this safe guard is pretty safe to keep.

Created using spr 1.3.7

adrian-prantl

I appreciate how well-documented the interface is!

llvm/lib/CAS/OnDiskCAS.cpp

Created using spr 1.3.7

llvm-ci · 2025-11-04T01:02:58Z

LLVM Buildbot has detected a new failure on builder llvm-clang-x86_64-expensive-checks-win running on as-worker-93 while building llvm at step 7 "test-build-unified-tree-check-all".

Full details are available at: https://lab.llvm.org/buildbot/#/builders/14/builds/4601

Here is the relevant piece of the build log for the reference

Step 7 (test-build-unified-tree-check-all) failure: test (failure)
******************** TEST 'LLVM-Unit :: CAS/./CASTests.exe/31/36' FAILED ********************
Script(shard):
--
GTEST_OUTPUT=json:C:\a\llvm-clang-x86_64-expensive-checks-win\build\unittests\CAS\.\CASTests.exe-LLVM-Unit-14840-31-36.json GTEST_SHUFFLE=0 GTEST_TOTAL_SHARDS=36 GTEST_SHARD_INDEX=31 C:\a\llvm-clang-x86_64-expensive-checks-win\build\unittests\CAS\.\CASTests.exe
--

Script:
--
C:\a\llvm-clang-x86_64-expensive-checks-win\build\unittests\CAS\.\CASTests.exe --gtest_filter=OnDiskCAS/CASTest.BlobsBigParallel/0
--
C:\a\llvm-clang-x86_64-expensive-checks-win\llvm-project\llvm\unittests\CAS\ObjectStoreTest.cpp(315): error: Value of: llvm::detail::TakeError(CAS->getProxy(*ID).moveInto(Node))
Expected: succeeded
  Actual: failed  ('C:\Users\Buildbot\AppData\Local\Temp\lit-tmp-myr7yuce\on-disk-cas-1cec22\leaf+0.13304.v1': no such file or directory)

C:\a\llvm-clang-x86_64-expensive-checks-win\llvm-project\llvm\unittests\CAS\ObjectStoreTest.cpp(315): error: Value of: llvm::detail::TakeError(CAS->getProxy(*ID).moveInto(Node))
Expected: succeeded
  Actual: failed  ('C:\Users\Buildbot\AppData\Local\Temp\lit-tmp-myr7yuce\on-disk-cas-1cec22\leaf+0.13184.v1': no such file or directory)


C:\a\llvm-clang-x86_64-expensive-checks-win\llvm-project\llvm\unittests\CAS\ObjectStoreTest.cpp:315
Value of: llvm::detail::TakeError(CAS->getProxy(*ID).moveInto(Node))
Expected: succeeded
  Actual: failed  ('C:\Users\Buildbot\AppData\Local\Temp\lit-tmp-myr7yuce\on-disk-cas-1cec22\leaf+0.13304.v1': no such file or directory)

C:\a\llvm-clang-x86_64-expensive-checks-win\llvm-project\llvm\unittests\CAS\ObjectStoreTest.cpp:315
Value of: llvm::detail::TakeError(CAS->getProxy(*ID).moveInto(Node))
Expected: succeeded
  Actual: failed  ('C:\Users\Buildbot\AppData\Local\Temp\lit-tmp-myr7yuce\on-disk-cas-1cec22\leaf+0.13184.v1': no such file or directory)



********************

mikaelholmen · 2025-11-07T10:02:07Z

Hi @cachemeifyoucan

I saw two new test failures with this patch when I build with EXPENSIVE_CHECKS.
I used gcc 13.2.

******************** TEST 'LLVM-Unit :: CAS/./CASTests/10/39' FAILED ********************
Script(shard):
--
GTEST_OUTPUT=json:/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests-LLVM-Unit-2667138-10-39.json GTEST_SHUFFLE=0 GTEST_TOTAL_SHARDS=39 GTEST_SHARD_INDEX=10 /repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests
--

Note: This is test shard 11 of 39.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from OnDiskCASTest
[ RUN      ] OnDiskCASTest.UnifiedOnDiskCacheTest
/app/gcc/13.2.0/include/c++/13.2.0/bits/stl_algo.h:4892:
In function:
    void std::sort(_RAIter, _RAIter, _Compare) [with _RAIter =
    getAllDBDirs(llvm::StringRef, bool)::DBDir*; _Compare =
    getAllDBDirs(llvm::StringRef, bool)::<lambda(const
    getAllDBDirs(llvm::StringRef, bool)::DBDir&, const
    getAllDBDirs(llvm::StringRef, bool)::DBDir&)>]

Error: comparison doesn't meet irreflexive requirements, assert(!(a < a)).

Objects involved in the operation:
    instance "functor" @ 0x7ffc5c37012f {
    }
    iterator::value_type "ordered type"  {
    }
 #0 0x00000000004ba2ee llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x4ba2ee)
 #1 0x00000000004b70fb llvm::sys::RunSignalHandlers() (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x4b70fb)
 #2 0x00000000004b723e SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
 #3 0x00007fd6bff4b990 __restore_rt (/lib64/libpthread.so.0+0x12990)
 #4 0x00007fd6bd8e352f raise (/lib64/libc.so.6+0x4e52f)
 #5 0x00007fd6bd8b6e65 abort (/lib64/libc.so.6+0x21e65)
 #6 0x000000000040baeb (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x40baeb)
 #7 0x000000000052955b getAllDBDirs(llvm::StringRef, bool) UnifiedOnDiskCache.cpp:0:0
 #8 0x000000000052aea5 llvm::cas::ondisk::UnifiedOnDiskCache::open(llvm::StringRef, std::optional<unsigned long>, llvm::StringRef, unsigned int, llvm::cas::ondisk::OnDiskGraphDB::FaultInPolicy) (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x52aea5)
 #9 0x000000000046accb OnDiskCASTest_UnifiedOnDiskCacheTest_Test::TestBody()::'lambda'()::operator()() const UnifiedOnDiskCacheTest.cpp:0:0
#10 0x000000000046e2ff OnDiskCASTest_UnifiedOnDiskCacheTest_Test::TestBody() (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x46e2ff)
#11 0x000000000054de9f testing::Test::Run() (.part.0) gtest-all.cc:0:0
#12 0x000000000054e1e2 testing::TestInfo::Run() (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x54e1e2)
#13 0x000000000056a83c testing::TestSuite::Run() (.part.0) gtest-all.cc:0:0
#14 0x0000000000573bd8 testing::internal::UnitTestImpl::RunAllTests() (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x573bd8)
#15 0x000000000054da3c testing::UnitTest::Run() (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x54da3c)
#16 0x000000000040edb4 main (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x40edb4)
#17 0x00007fd6bd8cf7e5 __libc_start_main (/lib64/libc.so.6+0x3a7e5)
#18 0x000000000040f3ce _start (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x40f3ce)

--
exit: -6
--
shard JSON output does not exist: /repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests-LLVM-Unit-2667138-10-39.json
********************
FAIL: LLVM-Unit :: CAS/./CASTests/3/39 (84482 of 91334)
******************** TEST 'LLVM-Unit :: CAS/./CASTests/3/39' FAILED ********************
Script(shard):
--
GTEST_OUTPUT=json:/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests-LLVM-Unit-2667138-3-39.json GTEST_SHUFFLE=0 GTEST_TOTAL_SHARDS=39 GTEST_SHARD_INDEX=3 /repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests
--

Note: This is test shard 4 of 39.
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from OnDiskCASTest
[ RUN      ] OnDiskCASTest.UnifiedCASMaterializationCheckPreventsGarbageCollection
/app/gcc/13.2.0/include/c++/13.2.0/bits/stl_algo.h:4892:
In function:
    void std::sort(_RAIter, _RAIter, _Compare) [with _RAIter =
    getAllDBDirs(llvm::StringRef, bool)::DBDir*; _Compare =
    getAllDBDirs(llvm::StringRef, bool)::<lambda(const
    getAllDBDirs(llvm::StringRef, bool)::DBDir&, const
    getAllDBDirs(llvm::StringRef, bool)::DBDir&)>]

Error: comparison doesn't meet irreflexive requirements, assert(!(a < a)).

Objects involved in the operation:
    instance "functor" @ 0x7ffe670855df {
    }
    iterator::value_type "ordered type"  {
    }
 #0 0x00000000004ba2ee llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x4ba2ee)
 #1 0x00000000004b70fb llvm::sys::RunSignalHandlers() (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x4b70fb)
 #2 0x00000000004b723e SignalHandler(int, siginfo_t*, void*) Signals.cpp:0:0
 #3 0x00007f160c324990 __restore_rt (/lib64/libpthread.so.0+0x12990)
 #4 0x00007f1609cbc52f raise (/lib64/libc.so.6+0x4e52f)
 #5 0x00007f1609c8fe65 abort (/lib64/libc.so.6+0x21e65)
 #6 0x000000000040baeb (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x40baeb)
 #7 0x000000000052955b getAllDBDirs(llvm::StringRef, bool) UnifiedOnDiskCache.cpp:0:0
 #8 0x000000000052aea5 llvm::cas::ondisk::UnifiedOnDiskCache::open(llvm::StringRef, std::optional<unsigned long>, llvm::StringRef, unsigned int, llvm::cas::ondisk::OnDiskGraphDB::FaultInPolicy) (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x52aea5)
 #9 0x0000000000502e55 llvm::cas::builtin::createBuiltinUnifiedOnDiskCache(llvm::StringRef) (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x502e55)
#10 0x00000000005043de llvm::cas::createOnDiskUnifiedCASDatabases(llvm::StringRef) (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x5043de)
#11 0x000000000043c1bf OnDiskCASTest_UnifiedCASMaterializationCheckPreventsGarbageCollection_Test::TestBody() (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x43c1bf)
#12 0x000000000054de9f testing::Test::Run() (.part.0) gtest-all.cc:0:0
#13 0x000000000054e1e2 testing::TestInfo::Run() (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x54e1e2)
#14 0x000000000056a83c testing::TestSuite::Run() (.part.0) gtest-all.cc:0:0
#15 0x0000000000573bd8 testing::internal::UnitTestImpl::RunAllTests() (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x573bd8)
#16 0x000000000054da3c testing::UnitTest::Run() (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x54da3c)
#17 0x000000000040edb4 main (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x40edb4)
#18 0x00007f1609ca87e5 __libc_start_main (/lib64/libc.so.6+0x3a7e5)
#19 0x000000000040f3ce _start (/repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests+0x40f3ce)

--
exit: -6
--
shard JSON output does not exist: /repo/llvm/build-all-gcc132-expensive/unittests/CAS/./CASTests-LLVM-Unit-2667138-3-39.json

And with later compiler/header versions I saw it explicitly pointing out UnifiedOnDiskCache.cpp:176:

In function:
    void std::sort(_RandomAccessIterator, _RandomAccessIterator, _Compare)
    [_RandomAccessIterator = DBDir *, _Compare = (lambda at
    ../lib/CAS/UnifiedOnDiskCache.cpp:176:27)]

Error: comparison doesn't meet irreflexive requirements, assert(!(a < a)).

cachemeifyoucan · 2025-11-07T16:07:21Z

That is a bug in the compare function. Fix here: #166963

gkistanova · 2025-11-30T07:50:26Z

Hi @cachemeifyoucan

Looks like this commit introduced a flaky test.
Please see https://lab.llvm.org/buildbot/#/builders/14/builds/4601 along with more recent history of that bot for random failures.

cachemeifyoucan · 2025-12-01T01:12:29Z

Hi @cachemeifyoucan

Looks like this commit introduced a flaky test. Please see https://lab.llvm.org/buildbot/#/builders/14/builds/4601 along with more recent history of that bot for random failures.

Sorry I missed that. Do you know which windows version is used on that bot? The description only says windows 10, but I am very interested to know if that windows version will trigger this condition: https://github.com/llvm/llvm-project/blob/main/llvm/lib/Support/Windows/Path.inc#L966

cachemeifyoucan · 2025-12-01T01:13:30Z

@gkistanova Let me disable the test for now on windows to see if version is the problem: #170063

gkistanova · 2025-12-02T09:23:56Z

That computer is running version 22H2 (OS Build 19045.5854).

cachemeifyoucan · 2025-12-03T04:16:09Z

That computer is running version 22H2 (OS Build 19045.5854).

This is newer than my suspected version. Let me disable more versions as workaround.

akshayrdeodhar · 2025-12-03T10:15:36Z

llvm/lib/CAS/UnifiedOnDiskCache.cpp

+#include <optional>
+
+#if __has_include(<sys/sysctl.h>)
+#include <sys/sysctl.h>


I'm seeing a failure due to this with my build. The sys/sysctl.h header contains the following warning:

#warning "The <sys/sysctl.h> header is deprecated and will be removed

As a result, the compiler throws:

In file included from /localhome/local-adeodhar/llvm-solid-2/llvm/lib/CAS/UnifiedOnDiskCache.cpp:92: /usr/include/x86_64-linux-gnu/sys/sysctl.h:21:2: error: #warning "The <sys/sysctl.h> header is deprecated and will be removed." [-Werror=cpp] 21 | #warning "The <sys/sysctl.h> header is deprecated and will be removed." | ^~~~~~~ cc1plus: all warnings being treated as errors

The __has_include does not prevent the compiler from throwing an error when warnings-as-errors is enabled. I understand that this does not cause a problem for the default LLVM build (which does not have warnings-as-errors on), but might cause pain to users who have Werror enabled on a system that has sys/sysctl.h deprecated.
I might be missing something- can you help me figure out how to work around this?

Which distro are you using? Maybe the better default is to only use this on Darwin. I will do some testing to see what is the best for code portability.

I'm using Ubuntu 20.04.

Distributor ID: Ubuntu Description: Ubuntu 20.04.6 LTS Release: 20.04 Codename: focal

This was referenced Oct 29, 2024

[CAS] Add LLVMCAS library with InMemoryCAS implementation #114096

Merged

[CAS] LLVMCAS implementation #68448

Closed

[CAS] Add OnDiskCAS

4b951ef

Add OnDiskCAS abstraction, that implements ObjectStore and ActionCache interface using OnDiskGraphDB and OnDiskKeyValueDB. Reviewers: Pull Request: #114103

cachemeifyoucan changed the base branch from users/cachemeifyoucan/spr/main.cas-add-ondiskcas to main October 22, 2025 19:51

cachemeifyoucan force-pushed the users/cachemeifyoucan/spr/cas-add-ondiskcas branch from 94172a6 to 4b951ef Compare October 22, 2025 19:51

cachemeifyoucan changed the title ~~[CAS] Add OnDiskCAS~~ [CAS] Add UnifiedOnDiskCache and OnDiskCAS Oct 22, 2025

minor fixup

f7f4b68

Created using spr 1.3.7

cachemeifyoucan requested review from adrian-prantl, bogner and ilovepi October 23, 2025 16:31

Rebase and refactoring

907c73b

Created using spr 1.3.7

bogner reviewed Oct 28, 2025

View reviewed changes

Address review feedbac

d93a8fc

Created using spr 1.3.7

bogner approved these changes Oct 30, 2025

View reviewed changes

adrian-prantl approved these changes Oct 31, 2025

View reviewed changes

ilovepi approved these changes Nov 1, 2025

View reviewed changes

llvm/lib/CAS/OnDiskCAS.cpp Show resolved Hide resolved

llvm/lib/CAS/OnDiskCAS.cpp Show resolved Hide resolved

address review feedback

d2c9b3a

Created using spr 1.3.7

cachemeifyoucan merged commit 6747ea0 into main Nov 3, 2025
10 checks passed

cachemeifyoucan deleted the users/cachemeifyoucan/spr/cas-add-ondiskcas branch November 3, 2025 17:50

akshayrdeodhar reviewed Dec 3, 2025

View reviewed changes

[CAS] Add UnifiedOnDiskCache and OnDiskCAS #114103

[CAS] Add UnifiedOnDiskCache and OnDiskCAS #114103

Uh oh!

Conversation

cachemeifyoucan commented Oct 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bogner Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

cachemeifyoucan Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

adrian-prantl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

llvm-ci commented Nov 4, 2025

Uh oh!

mikaelholmen commented Nov 7, 2025

Uh oh!

cachemeifyoucan commented Nov 7, 2025

Uh oh!

gkistanova commented Nov 30, 2025

Uh oh!

cachemeifyoucan commented Dec 1, 2025

Uh oh!

cachemeifyoucan commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gkistanova commented Dec 2, 2025

Uh oh!

cachemeifyoucan commented Dec 3, 2025

Uh oh!

akshayrdeodhar Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cachemeifyoucan Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

akshayrdeodhar Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

cachemeifyoucan commented Oct 29, 2024 •

edited

Loading

cachemeifyoucan commented Dec 1, 2025 •

edited

Loading

akshayrdeodhar Dec 3, 2025 •

edited

Loading