Struggling to define schema overrides (on taps)

[amotl]

Hi again,

while working on MeltanoLabs/target-postgres#251, I am struggling to define schema override rules. For development purposes, I am using tap-singer-jsonl.

Within meltano.yml, I've defined a schema override rule on the tap-singer-jsonl element.

schema:
  my_table:
    value:
      type: "foo"

After running meltano invoke like:

meltano --log-level=debug invoke tap-singer-jsonl

I can see in .meltano/run/tap-singer-jsonl/tap.properties.json:

{
  "streams": [
    {
      "tap_stream_id": "array_number",
      "key_properties": [
        "id"
      ],
      "schema": {
        "properties": {
          "id": {
            "type": "integer"
          },
          "value": {
            "type": "foo"
          }
        },
        "type": "object",
        "required": [
          "id"
        ]
      },
      "metadata": {},
      "selected": true
    }
  ]
}

NB: metadata has been compressed.

However, it looks like it does not have any influence on the output of the tap at all. The schema override rule is not being taken into consideration.

With kind regards,
Andreas.

[amotl]

I should have noted the I also needed to define the capabilities attribute in the tap-singer-jsonl section of meltano.yml.

capabilities:
  - catalog
  - discover

Otherwise, invoking meltano invoke would yield such a warning, and will not propagate the schema override rule into the tap.properties.json file at all.

[warning  ] A catalog file was found, but it will be ignored as the extractor does not advertise the `catalog` or `properties` capability

This is most probably the issue here: tap-singer-jsonl just isn't ready for this kind of manipulation / behaviour override, effectively not implementing what it needs to fulfil the catalog capability contract?

NB: The original only says it provides the discover capability.

[amotl]

We converged this report into Learning Singer » Defining schema overrides on taps.