Simplify the project

Author: habedi

.gitignore

@@ -95,3 +95,4 @@ _rust.h
 uv.lock
 tests/temp_models/
 *.cast
+*.proptest-regressions

Makefile

@@ -56,7 +56,7 @@ rust-test: rust-format ## Run tests
 .PHONY: rust-coverage
 rust-coverage: ## Generate code coverage report for Gaggle crate
 	@echo "Generating coverage report..."
-	@cargo tarpaulin --manifest-path gaggle/Cargo.toml --all-targets --out Xml
+	@cargo tarpaulin --manifest-path gaggle/Cargo.toml --all-targets --out Xml --features "expose_internal"
 
 .PHONY: rust-lint
 rust-lint: rust-format ## Run linter checks on Rust files

ROADMAP.md

@@ -10,15 +10,14 @@ It outlines features to be implemented and their current status.
 
 * **Authentication**
     * [x] Set Kaggle API credentials programmatically.
-    * [x] Support environment variables (using `KAGGLE_USERNAME` and `KAGGLE_KEY`).
-    * [x] Support `~/.kaggle/kaggle.json file`.
+    * [x] Support environment variables for authentication (`KAGGLE_USERNAME` and `KAGGLE_KEY`).
+    * [x] Support reading credentials from `~/.kaggle/kaggle.json file`.
 * **Dataset Operations**
-    * [x] Search for datasets.
+    * [x] Search for datasets on Kaggle.
     * [x] Download datasets from Kaggle.
     * [x] List files in a dataset.
     * [x] Get dataset metadata.
-    * [ ] Upload datasets to Kaggle.
-    * [ ] Delete datasets from Kaggle.
+    * [ ] Upload DuckDB tables to Kaggle.
 
 ### 2. Caching and Storage
 
@@ -28,7 +27,6 @@ It outlines features to be implemented and their current status.
     * [x] Get cache information (size and storage location).
     * [ ] Set cache size limit.
     * [ ] Cache expiration policies.
-    * [ ] Support for partial file downloads and resumes.
 * **Storage**
     * [x] Store datasets in configurable directory.
     * [ ] Support for cloud storage backends (S3, GCS, and Azure).
@@ -37,13 +35,11 @@ It outlines features to be implemented and their current status.
 
 * **File Format Support**
     * [x] CSV and TSV file reading.
-    * [x] JSON file reading.
     * [x] Parquet file reading.
+    * [x] JSON file reading.
     * [ ] Excel and XLSX file reading.
-* **Direct Query Integration**
+* **Querying Datasets**
     * [x] Replacement scan for `kaggle:` URLs.
-    * [ ] Direct SQL queries on remote datasets without full download (true streaming).
-    * [ ] Streaming data from Kaggle without caching.
     * [ ] Virtual table support for lazy loading.
 
 ### 4. Performance and Concurrency
@@ -54,7 +50,7 @@ It outlines features to be implemented and their current status.
     * [ ] Concurrent dataset downloads.
 * **Network Optimization**
     * [x] Configurable HTTP timeouts.
-    * [ ] Retry logic with backoff (configurable attempts/delay; planned).
+    * [ ] Retry logic with backoff for failed requests.
 * **Caching Strategy**
     * [ ] Incremental cache updates.
     * [ ] Background cache synchronization.
@@ -67,7 +63,7 @@ It outlines features to be implemented and their current status.
     * [x] Clear error messages for `NULL` inputs.
     * [ ] Detailed error codes for programmatic error handling.
 * **Resilience**
-    * [ ] Automatic retry on network failures (planned with backoff settings).
+    * [ ] Automatic retry on network failures.
     * [ ] Graceful degradation when Kaggle API is unavailable.
     * [ ] Local-only mode for cached datasets.