feat: Support pgvector as a custom x-sql-datatype

Signed-off-by: Edgar Ramírez-Mondragón <[email protected]>

Author: edgarrmondragon

README.md

@@ -312,11 +312,12 @@ This target supports the [`x-sql-datatype` extension](https://sdk.meltano.com/en
 
 <!-- insert a table with the mapping -->
 
-| `x-sql-datatype` | Postgres | Description                                                        |
-| :--------------- | :------- | :----------------------------------------------------------------- |
-| smallint         | smallint | small-range integer (-32768 to +32767)                             |
-| integer          | integer  | typical choice for integer (-2147483648 to +2147483647)            |
-| bigint           | bigint   | large-range integer (-9223372036854775808 to +9223372036854775807) |
+| `x-sql-datatype` | Postgres | Description                                                                                                                      |
+| :--------------- | :------- | :------------------------------------------------------------------------------------------------------------------------------- |
+| smallint         | smallint | small-range integer (-32768 to +32767)                                                                                           |
+| integer          | integer  | typical choice for integer (-2147483648 to +2147483647)                                                                          |
+| bigint           | bigint   | large-range integer (-9223372036854775808 to +9223372036854775807)                                                               |
+| pgvector         | vector   | vector similarity search (requires [pgvector](https://github.com/pgvector/pgvector) extension and the `pgvector` Python package) |
 
 ### Using the Singer catalog to narrow down the Postgres data types
 
@@ -350,6 +351,36 @@ plugins:
           x-sql-datatype: smallint
 ```
 
+For vector embeddings:
+
+```yaml
+# meltano.yml
+plugins:
+  extractors:
+  - name: tap-my-tap
+    schema:
+      some_stream_id:
+        embedding:
+          type: array
+          items:
+            type: number
+          x-sql-datatype: pgvector
+```
+
+**Important:** To use `pgvector` data types:
+1. The [pgvector extension](https://github.com/pgvector/pgvector) **MUST** be installed and enabled in your PostgreSQL database:
+   ```sql
+   CREATE EXTENSION IF NOT EXISTS vector;
+   ```
+2. The `pgvector` Python package **MUST** be installed in your environment:
+   ```bash
+   pip install pgvector
+   # or with the target
+   pip install meltanolabs-target-postgres pgvector
+   ```
+
+If the `pgvector` Python package is not installed, the target will fall back to using `ARRAY(INTEGER)` with a warning.
+
 ## Content Encoding Support
 
 Json Schema supports the [`contentEncoding` keyword](https://datatracker.ietf.org/doc/html/rfc4648#section-8), which can be used to specify the encoding of input string types.

pyproject.toml

@@ -56,13 +56,15 @@ lint = [
     "ruff>=0.1.14",
 ]
 testing = [
+    "pgvector>=0.4.1",
     "pytest>=9",
     "tap-countries",
     "tap-fundamentals",
 ]
 typing = [
+    { include-group = "testing" },
     "mypy>=1.6.1",
-    "types-paramiko>=3.3.0.0,<4",
+    "types-paramiko>=4",
     "types-simplejson>=3.19.0.2",
     "types-sqlalchemy>=1.4.53.38",
     "types-jsonschema>=4.19.0.3",
@@ -83,6 +85,12 @@ warn_redundant_casts = true
 warn_unused_configs = true
 warn_unused_ignores = true
 
+[[tool.mypy.overrides]]
+follow_untyped_imports = true
+module = [
+  "pgvector.*",
+]
+
 [build-system]
 requires = [
     "hatchling==1.27.0",
@@ -119,12 +127,6 @@ select = [
   "RUF",  # ruff
 ]
 
-[tool.ruff.lint.flake8-import-conventions]
-banned-from = ["sqlalchemy"]
-
-[tool.ruff.lint.flake8-import-conventions.extend-aliases]
-sqlalchemy = "sa"
-
 [tool.ruff.lint.pydocstyle]
 convention = "google"