WIP

Author: habedi

README.md

@@ -31,17 +31,14 @@ frequently.
 Gaggle tries to help simplify this process by hiding the complexity and letting you work with datasets directly inside
 an analytical database like DuckDB that can handle fast queries.
 
-> [!NOTE]
-> Gaggle is similar of [Hugging Face extension]() for DuckDB.
-> Although, Kaggle datasets are not directly accessible from a remote file system, like Hugging Face datasets.
-
 ### Features
 
-- Has a simple API (just a handful of SQL functions)
-- Allows you search, download, update, and delete Kaggle datasets directly from DuckDB
-- Supports datasets made of CSV, JSON, and Parquet files
+- Has a simple API to interact with Kaggle datasets from DuckDB
+- Allows you to search, download, and read datasets from Kaggle
+- Supports datasets that contain CSV, Parquet, JSON, and XLSX files
 - Configurable and has built-in caching support
-- Thread-safe and memory-efficient
+- Thread-safe, fast, and has a low memory footprint
+- Supports dataset versioning and update checks
 
 See the [ROADMAP.md](ROADMAP.md) for planned features and the [docs](docs) folder for detailed documentation.
 
@@ -94,35 +91,34 @@ make release
 #### Trying Gaggle
 
 ```sql
--- Load the Gaggle extension
-load 'build/release/extension/gaggle/gaggle.duckdb_extension';
+-- Load the Gaggle extension (only needed if you built from source)
+--load 'build/release/extension/gaggle/gaggle.duckdb_extension';
 
--- Set your Kaggle credentials (or use `~/.kaggle/kaggle.json`)
+-- Manually, set your Kaggle credentials (or use `~/.kaggle/kaggle.json`)
 select gaggle_set_credentials('your-username', 'your-api-key');
 
 -- Get extension version
 select gaggle_version();
 
--- Prime cache by downloading the dataset locally (optional, but improves first-time performance)
-select gaggle_download('habedi/flickr-8k-dataset-clean');
-
 -- List files in the downloaded dataset
+-- (Note that if the datasets is not downloaded yet, it will be downloaded and cached first)
 select *
 from gaggle_ls('habedi/flickr-8k-dataset-clean') limit 5;
 
--- Read a Parquet file from local cache using a prepared statement (no subquery in function args)
+-- Read a Parquet file from local cache using a prepared statement
+-- (Note that DuckDB doesn't support subquery in function arguments, so we use a prepared statement)
 prepare rp as select * from read_parquet(?) limit 10;
 execute rp(gaggle_file_paths('habedi/flickr-8k-dataset-clean', 'flickr8k.parquet'));
 
--- Use the replacement scan to read directly via kaggle: URL
+-- Alternatively, we can use a replacement scan to read directly via `kaggle:` prefix
 select count(*)
 from 'kaggle:habedi/flickr-8k-dataset-clean/flickr8k.parquet';
 
 -- Or glob Parquet files in a dataset directory
 select count(*)
 from 'kaggle:habedi/flickr-8k-dataset-clean/*.parquet';
 
--- Optionally, check cache info
+-- Optionally, we check cache info
 select gaggle_cache_info();
 
 -- Enforce cache size limit manually (automatic with soft limit by default)

ROADMAP.md

@@ -39,7 +39,7 @@ It outlines features to be implemented and their current status.
     * [x] CSV and TSV file reading.
     * [x] Parquet file reading.
     * [x] JSON file reading.
-    * [ ] Excel and XLSX file reading.
+    * [ ] Excel (XLSX) file reading.
 * **Querying Datasets**
     * [x] Replacement scan for `kaggle:` URLs.
     * [ ] Virtual table support for lazy loading.
@@ -49,7 +49,7 @@ It outlines features to be implemented and their current status.
 * **Concurrency Control**
     * [x] Thread-safe credential storage.
     * [x] Thread-safe cache access.
-    * [x] Concurrent dataset downloads (with per-dataset serialization to prevent race conditions).
+    * [x] Concurrent dataset downloads.
 * **Network Optimization**
     * [x] Configurable HTTP timeouts.
     * [x] Retry logic with backoff for failed requests.
@@ -80,8 +80,13 @@ It outlines features to be implemented and their current status.
 * **Testing**
     * [x] Unit tests for core modules (Rust).
     * [x] SQL integration tests (DuckDB shell).
-    * [ ] End-to-end integration tests with mocked HTTP.
+    * [x] End-to-end integration tests with mocked HTTP (basic coverage).
     * [ ] Performance benchmarks.
 * **Distribution**
     * [ ] Pre-compiled extension binaries for Linux, macOS, and Windows.
     * [ ] Submission to the DuckDB Community Extensions repository.
+
+### 7. Observability
+
+* **Logging**
+    * [x] Structured logging via `tracing` with `GAGGLE_LOG_LEVEL`.

docs/CONFIGURATION.md

@@ -88,6 +88,28 @@ Gaggle supports configuration via environment variables to customize its behavio
 
   These controls enable exponential backoff with cap across metadata/search/download requests.
 
+#### Download Coordination
+
+When multiple queries attempt to download the same dataset concurrently, Gaggle coordinates using an in-process lock.
+These settings control the wait behavior when a download is already in progress.
+
+- **GAGGLE_DOWNLOAD_WAIT_TIMEOUT_MS**
+  - **Description**: Maximum time a waiting request will block for a concurrent download to finish
+  - **Type**: Integer (milliseconds)
+  - **Default**: `30000` (30 seconds)
+  - **Example**:
+    ```bash
+    export GAGGLE_DOWNLOAD_WAIT_TIMEOUT_MS=600000 # 10 minutes
+    ```
+- **GAGGLE_DOWNLOAD_WAIT_POLL_MS**
+  - **Description**: Polling interval while waiting on another download
+  - **Type**: Integer (milliseconds)
+  - **Default**: `100`
+  - **Example**:
+    ```bash
+    export GAGGLE_DOWNLOAD_WAIT_POLL_MS=250
+    ```
+
 #### Logging Configuration
 
 ##### GAGGLE_VERBOSE
@@ -100,24 +122,36 @@ Gaggle supports configuration via environment variables to customize its behavio
   export GAGGLE_VERBOSE=1
   ```
 
-##### GAGGLE_LOG_LEVEL (planned)
+##### GAGGLE_LOG_LEVEL
 
-- **Description**: Set logging level for detailed output
-- **Type**: String (`ERROR`, `WARN`, `INFO`, `DEBUG`)
+- **Description**: Set logging level for structured logs emitted by the Rust core (via `tracing`)
+- **Type**: String (`ERROR`, `WARN`, `INFO`, `DEBUG`, `TRACE`); case-insensitive
 - **Default**: `WARN`
-- **Status**: Planned, not implemented yet
+- **Status**: ✅ Implemented
 - **Example**:
   ```bash
-  ## Show all messages including debug
-  export GAGGLE_LOG_LEVEL=DEBUG
-
-  ## Show only errors
-  export GAGGLE_LOG_LEVEL=ERROR
-
-  ## Show informational messages and above
   export GAGGLE_LOG_LEVEL=INFO
   ```
 
+  Notes:
+  - Logging is initialized lazily on first use (when the crate is loaded in-process or when `gaggle::init_logging()` is called). The environment variable is read once per process.
+  - Logs include a level prefix and optional ANSI colors if stderr is a terminal.
+
+#### Offline Mode
+
+- **GAGGLE_OFFLINE**
+  - **Description**: Disable network access. When enabled, operations that require network will fail fast unless data is already cached.
+  - **Type**: Boolean (`1`, `true`, `yes`, `on` to enable)
+  - **Default**: `false`
+  - **Effects**:
+    - gaggle_download(...) fails if the dataset isn’t cached.
+    - Version checks use cached `.downloaded` metadata when available; otherwise return "unknown".
+    - Search and metadata calls will still attempt network; consider avoiding them in offline mode.
+  - **Example**:
+    ```bash
+    export GAGGLE_OFFLINE=1
+    ```
+
 ### Usage Examples
 
 #### Example 1: Custom Cache Directory
@@ -189,6 +223,19 @@ export GAGGLE_HTTP_RETRY_MAX_DELAY_MS=60000 ## Cap at 60s
 ./build/release/duckdb
 ```
 
+#### Example 6: Offline Mode
+
+```bash
+# Enable offline mode
+export GAGGLE_OFFLINE=1
+
+# Attempt to download a dataset (will fail if not cached)
+gaggle download username/dataset-name
+
+# Querying metadata or searching will still attempt network access
+gaggle info username/dataset-name
+```
+
 ### Configuration Verification
 
 You can verify your configuration at runtime:

docs/README.md

@@ -33,9 +33,13 @@ Table function:
 
 Replacement scan (transparent table read):
 
-- Single file: `'kaggle:owner/dataset/file.parquet'`
-- Glob: `'kaggle:owner/dataset/*.parquet'`
-- Parquet paths use DuckDB’s `read_parquet`; other files default to `read_csv_auto`.
+- Single file: `'kaggle:owner/dataset/file.ext'`
+- Glob: `'kaggle:owner/dataset/*.ext'`
+- Reader is chosen by extension:
+  - `.parquet`/`.parq` -> `read_parquet`
+  - `.json`/`.jsonl`/`.ndjson` -> `read_json_auto`
+  - `.xlsx` -> `read_excel`
+  - everything else -> `read_csv_auto`
 
 ---
 

gaggle/Cargo.toml

@@ -22,6 +22,9 @@ reqwest = { version = "0.12", features = ["blocking", "rustls-tls", "json", "mul
 zip = "6.0"
 dirs = "6.0"
 urlencoding = "2.1"
+tracing = "0.1"
+tracing-subscriber = { version = "0.3", features = ["fmt", "env-filter"] }
+atty = "0.2"
 
 [dev-dependencies]
 tempfile = "3.10"

gaggle/bindings/gaggle_extension.cpp

@@ -511,6 +511,23 @@ KaggleReplacementScan(ClientContext &context, ReplacementScanInput &input,
       pattern.find('*') != string::npos || pattern.find('?') != string::npos;
   bool is_dir = pattern.empty();
 
+  auto decide_reader = [](const string &lower_ext) -> string {
+    if (StringUtil::EndsWith(lower_ext, ".parquet") ||
+        StringUtil::EndsWith(lower_ext, ".parq")) {
+      return "read_parquet";
+    }
+    if (StringUtil::EndsWith(lower_ext, ".json") ||
+        StringUtil::EndsWith(lower_ext, ".jsonl") ||
+        StringUtil::EndsWith(lower_ext, ".ndjson")) {
+      return "read_json_auto";
+    }
+    if (StringUtil::EndsWith(lower_ext, ".xlsx")) {
+      return "read_excel";
+    }
+    // Default CSV/TSV and others to DuckDB's auto CSV reader
+    return "read_csv_auto";
+  };
+
   if (is_dir || has_wildcard) {
     // Ensure dataset is downloaded and construct a glob path
     char *dir_c = gaggle_download_dataset(dataset_path.c_str());
@@ -525,10 +542,7 @@ KaggleReplacementScan(ClientContext &context, ReplacementScanInput &input,
     local_path = dir_path + tail;
 
     // Choose reader based on pattern extension if any
-    if (StringUtil::EndsWith(lower_pat, ".parquet") ||
-        StringUtil::EndsWith(lower_pat, ".parq")) {
-      func_name = "read_parquet";
-    }
+    func_name = decide_reader(lower_pat);
   } else {
     // Specific file: resolve exact path
     char *file_path_c =
@@ -551,10 +565,7 @@ KaggleReplacementScan(ClientContext &context, ReplacementScanInput &input,
 
     // Decide reader based on extension
     auto lower_name = StringUtil::Lower(pattern);
-    if (StringUtil::EndsWith(lower_name, ".parquet") ||
-        StringUtil::EndsWith(lower_name, ".parq")) {
-      func_name = "read_parquet";
-    }
+    func_name = decide_reader(lower_name);
   }
 
   // Construct a table function call: func_name(local_path)

gaggle/src/config.rs

@@ -19,6 +19,12 @@ pub struct GaggleConfig {
     /// HTTP timeout in seconds
     #[allow(dead_code)]
     pub http_timeout_secs: u64,
+    /// Download lock wait timeout in milliseconds
+    #[allow(dead_code)]
+    pub download_wait_timeout_ms: u64,
+    /// Download lock poll interval in milliseconds
+    #[allow(dead_code)]
+    pub download_wait_poll_ms: u64,
     // Future: other options
 }
 
@@ -29,6 +35,8 @@ impl GaggleConfig {
             cache_dir: Self::get_cache_dir(),
             verbose_logging: Self::get_verbose(),
             http_timeout_secs: Self::get_http_timeout(),
+            download_wait_timeout_ms: Self::get_download_wait_timeout_ms(),
+            download_wait_poll_ms: Self::get_download_wait_poll_ms(),
         }
     }
 
@@ -65,6 +73,22 @@ impl GaggleConfig {
             .and_then(|v| v.parse().ok())
             .unwrap_or(30)
     }
+
+    /// Get download wait timeout from env (default 30_000 ms)
+    fn get_download_wait_timeout_ms() -> u64 {
+        env::var("GAGGLE_DOWNLOAD_WAIT_TIMEOUT_MS")
+            .ok()
+            .and_then(|v| v.parse().ok())
+            .unwrap_or(30_000)
+    }
+
+    /// Get download wait poll interval from env (default 100 ms)
+    fn get_download_wait_poll_ms() -> u64 {
+        env::var("GAGGLE_DOWNLOAD_WAIT_POLL_MS")
+            .ok()
+            .and_then(|v| v.parse().ok())
+            .unwrap_or(100)
+    }
 }
 
 impl Default for GaggleConfig {
@@ -154,6 +178,33 @@ pub fn cache_limit_is_soft() -> bool {
         .unwrap_or(true)
 }
 
+/// Runtime-resolved download wait timeout in milliseconds
+pub fn download_wait_timeout_ms() -> u64 {
+    env::var("GAGGLE_DOWNLOAD_WAIT_TIMEOUT_MS")
+        .ok()
+        .and_then(|v| v.parse().ok())
+        .unwrap_or(CONFIG.download_wait_timeout_ms)
+}
+
+/// Runtime-resolved download wait poll interval in milliseconds
+pub fn download_wait_poll_interval_ms() -> u64 {
+    env::var("GAGGLE_DOWNLOAD_WAIT_POLL_MS")
+        .ok()
+        .and_then(|v| v.parse().ok())
+        .unwrap_or(CONFIG.download_wait_poll_ms)
+}
+
+/// Whether offline mode is enabled (disables network operations)j Controlled by GAGGLE_OFFLINE
+pub fn offline_mode() -> bool {
+    std::env::var("GAGGLE_OFFLINE")
+        .ok()
+        .map(|v| match v.to_lowercase().as_str() {
+            "1" | "true" | "yes" | "on" => true,
+            _ => false,
+        })
+        .unwrap_or(false)
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
@@ -165,6 +216,8 @@ mod tests {
         let config = GaggleConfig::default();
         assert!(!config.verbose_logging);
         assert_eq!(config.http_timeout_secs, 30);
+        assert!(config.download_wait_timeout_ms >= 1000);
+        assert!(config.download_wait_poll_ms > 0);
     }
 
     #[test]
@@ -453,4 +506,29 @@ mod tests {
         assert!(!cache_limit_is_soft());
         env::remove_var("GAGGLE_CACHE_HARD_LIMIT");
     }
+
+    #[test]
+    #[serial]
+    fn test_download_wait_runtime_overrides() {
+        env::set_var("GAGGLE_DOWNLOAD_WAIT_TIMEOUT_MS", "1234");
+        env::set_var("GAGGLE_DOWNLOAD_WAIT_POLL_MS", "17");
+        assert_eq!(download_wait_timeout_ms(), 1234);
+        assert_eq!(download_wait_poll_interval_ms(), 17);
+        env::remove_var("GAGGLE_DOWNLOAD_WAIT_TIMEOUT_MS");
+        env::remove_var("GAGGLE_DOWNLOAD_WAIT_POLL_MS");
+    }
+
+    #[test]
+    #[serial]
+    fn test_offline_mode_env_parsing() {
+        std::env::remove_var("GAGGLE_OFFLINE");
+        assert!(!offline_mode());
+        std::env::set_var("GAGGLE_OFFLINE", "1");
+        assert!(offline_mode());
+        std::env::set_var("GAGGLE_OFFLINE", "true");
+        assert!(offline_mode());
+        std::env::set_var("GAGGLE_OFFLINE", "no");
+        assert!(!offline_mode());
+        std::env::remove_var("GAGGLE_OFFLINE");
+    }
 }