Simplify the project

Author: habedi

.gitignore

@@ -95,3 +95,4 @@ _rust.h
 uv.lock
 tests/temp_models/
 *.cast
+*.proptest-regressions

Makefile

@@ -51,7 +51,7 @@ rust-format: ## Format Rust files
 .PHONY: rust-test
 rust-test: rust-format ## Run tests
 	@echo "Running the unit tests for Gaggle..."
-	@cargo test --manifest-path gaggle/Cargo.toml --all-targets --features "expose_internal" -- --nocapture
+	@cargo test --manifest-path gaggle/Cargo.toml --all-targets -- --nocapture
 
 .PHONY: rust-coverage
 rust-coverage: ## Generate code coverage report for Gaggle crate
@@ -151,8 +151,3 @@ examples: ## Run SQL examples for Gaggle extension
 		./build/release/duckdb < $$sql_file; \
 		echo "============================================================================"; \
 	done
-
-.PHONY: itest
-itest: release ## Run Python integration test against built DuckDB shell
-	@echo "Running integration tests..."
-	@uv run -q python3 test/integration/test_duckdb_integration.py

README.md

@@ -9,7 +9,7 @@
 [![Tests](https://img.shields.io/github/actions/workflow/status/CogitatorTech/gaggle/tests.yml?label=tests&style=flat&labelColor=282c34&logo=github)](https://github.com/CogitatorTech/gaggle/actions/workflows/tests.yml)
 [![Code Quality](https://img.shields.io/codefactor/grade/github/CogitatorTech/gaggle?label=quality&style=flat&labelColor=282c34&logo=codefactor)](https://www.codefactor.io/repository/github/CogitatorTech/gaggle)
 [![Examples](https://img.shields.io/badge/examples-view-green?style=flat&labelColor=282c34&logo=github)](https://github.com/CogitatorTech/gaggle/tree/main/docs/examples)
-[![Docs](https://img.shields.io/badge/docs-view-blue?style=flat&labelColor=282c34&logo=read-the-docs)](https://github.com/CogitatorTech/gaggle/tree/main/docs)
+[![Docs](https://img.shields.io/badge/docs-read-blue?style=flat&labelColor=282c34&logo=read-the-docs)](https://github.com/CogitatorTech/gaggle/tree/main/docs)
 [![License](https://img.shields.io/badge/license-MIT%2FApache--2.0-007ec6?style=flat&labelColor=282c34&logo=open-source-initiative)](https://github.com/CogitatorTech/gaggle)
 
 Kaggle Datasets for DuckDB
@@ -18,19 +18,20 @@ Kaggle Datasets for DuckDB
 
 ---
 
-Gaggle is a DuckDB extension that allows you to read and write Kaggle datasets directly in SQL queries, as if
+Gaggle is a DuckDB extension that allows you to work with Kaggle datasets directly in SQL queries, as if
 they were DuckDB tables.
 It is written in Rust and uses the official Kaggle API to search, download, and manage datasets.
 
 Kaggle hosts a large collection of very useful datasets for data science and machine learning work.
 Accessing these datasets typically involves multiple steps including manually downloading a dataset (as a ZIP file),
 extracting it, loading the files in the dataset into your data science environment, and managing storage and dataset
 updates, etc.
-This workflow can be simplified and optimized by bringing the datasets directly into DuckDB.
+This workflow can be a at time.
+Gaggle tries to help simplify this process by integrating Kaggle dataset access directly into DuckDB.
 
 ### Features
 
-- Has a simple API (just a few SQL functions)
+- Has a simple API (just a handful of SQL functions)
 - Allows you search, download, update, and delete Kaggle datasets directly from DuckDB
 - Supports datasets made of CSV, JSON, and Parquet files
 - Configurable and has built-in caching support

ROADMAP.md

@@ -10,15 +10,14 @@ It outlines features to be implemented and their current status.
 
 * **Authentication**
     * [x] Set Kaggle API credentials programmatically.
-    * [x] Support environment variables (using `KAGGLE_USERNAME` and `KAGGLE_KEY`).
-    * [x] Support `~/.kaggle/kaggle.json file`.
+    * [x] Support environment variables for authentication (`KAGGLE_USERNAME` and `KAGGLE_KEY`).
+    * [x] Support reading credentials from `~/.kaggle/kaggle.json file`.
 * **Dataset Operations**
-    * [x] Search for datasets.
+    * [x] Search for datasets on Kaggle.
     * [x] Download datasets from Kaggle.
     * [x] List files in a dataset.
     * [x] Get dataset metadata.
-    * [ ] Upload datasets to Kaggle.
-    * [ ] Delete datasets from Kaggle.
+    * [ ] Upload DuckDB tables to Kaggle.
 
 ### 2. Caching and Storage
 
@@ -28,7 +27,6 @@ It outlines features to be implemented and their current status.
     * [x] Get cache information (size and storage location).
     * [ ] Set cache size limit.
     * [ ] Cache expiration policies.
-    * [ ] Support for partial file downloads and resumes.
 * **Storage**
     * [x] Store datasets in configurable directory.
     * [ ] Support for cloud storage backends (S3, GCS, and Azure).
@@ -37,24 +35,22 @@ It outlines features to be implemented and their current status.
 
 * **File Format Support**
     * [x] CSV and TSV file reading.
-    * [x] JSON file reading.
     * [x] Parquet file reading.
+    * [x] JSON file reading.
     * [ ] Excel and XLSX file reading.
-* **Direct Query Integration**
+* **Querying Datasets**
     * [x] Replacement scan for `kaggle:` URLs.
-    * [ ] Direct SQL queries on remote datasets without full download (true streaming).
-    * [ ] Streaming data from Kaggle without caching.
     * [ ] Virtual table support for lazy loading.
 
 ### 4. Performance and Concurrency
 
 * **Concurrency Control**
     * [x] Thread-safe credential storage.
     * [x] Thread-safe cache access.
-    * [ ] Concurrent dataset downloads.
+    * [x] Concurrent dataset downloads (with per-dataset serialization to prevent race conditions).
 * **Network Optimization**
     * [x] Configurable HTTP timeouts.
-    * [ ] Retry logic with backoff (configurable attempts/delay; planned).
+    * [x] Retry logic with backoff for failed requests.
 * **Caching Strategy**
     * [ ] Incremental cache updates.
     * [ ] Background cache synchronization.
@@ -67,7 +63,7 @@ It outlines features to be implemented and their current status.
     * [x] Clear error messages for `NULL` inputs.
     * [ ] Detailed error codes for programmatic error handling.
 * **Resilience**
-    * [ ] Automatic retry on network failures (planned with backoff settings).
+    * [x] Automatic retry on network failures.
     * [ ] Graceful degradation when Kaggle API is unavailable.
     * [ ] Local-only mode for cached datasets.
 
@@ -87,4 +83,3 @@ It outlines features to be implemented and their current status.
 * **Distribution**
     * [ ] Pre-compiled extension binaries for Linux, macOS, and Windows.
     * [ ] Submission to the DuckDB Community Extensions repository.
-    * [ ] Docker image with Gaggle pre-installed.