Add memory pool for datafusion #19992
Conversation
alchemist51
force-pushed
the
memory-pool
branch
2 times, most recently
from
4a3ab2f to
668d977
Compare
|
❌ Gradle check result for 668d977: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
alchemist51
force-pushed
the
memory-pool
branch
from
668d977 to
3c82a2c
Compare
alchemist51
requested review from
a team,
Bukhtawar,
CEHENKLE,
Rishikesh1159,
anasalkouz,
andrross,
ashking94,
cwperks,
dbwiddis,
gbbafna,
jed326,
kotwanikunal,
mch2,
msfroh,
owaiskazi19,
reta,
sachinpkale,
saratvemulapalli,
shwetathareja and
sohami
as code owners
alchemist51
force-pushed
the
memory-pool
branch
from
3c82a2c to
40f669d
Compare
|
❌ Gradle check result for 40f669d: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
Signed-off-by: Arpit Bandejiya <[email protected]>
alchemist51
force-pushed
the
memory-pool
branch
from
40f669d to
c13f8b9
Compare
|
❌ Gradle check result for c13f8b9: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
| pub type Result<T, E = DataFusionError> = result::Result<T, E>; | ||
|
|
||
| #[derive(Debug)] | ||
| pub struct CustomMemoryPool { |
There was a problem hiding this comment.
Lets add comments to all the new classes please.
| .with_cache_manager( | ||
| CacheManagerConfig::default().with_list_files_cache(Some(list_file_cache.clone())), | ||
| ) | ||
| .with_memory_pool(memory_pool.get_memory_pool()) |
There was a problem hiding this comment.
why do we need memory pool per query , can we setup both cache and memory as part of the global runtime env itself ?
There was a problem hiding this comment.
I think we can setup the Memory pool as part of the global runtime, but cache will be difficult since we need the listing cache different for different queries.
Signed-off-by: Arpit Bandejiya <[email protected]>
|
❌ Gradle check result for c861096: FAILURE Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change? |
2 participants
Description
[Describe what this change achieves]
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.