Commit f6523ee
Add support for nullable fields (closes #6)
Previously all the fields with nullable values were discarded as invalid.
However there is a wide range of fields that cannot be treated as non-nullable.
For example for enrichment failures bad row, we get following
structure (trimmed):
```
column_name | is_nullable | data_type
-------------------------------------------+-------------+-----------------------------
processor.artifact | NO | character varying
processor.version | NO | character varying
failure.messages | YES | jsonb
failure.timestamp | YES | timestamp without time zone
payload.enriched.app_id | YES | character varying
payload.enriched.base_currency | YES | character varying
...
```
Where only processor is non-nullable. Therefore enrichment failures would not be
written with evaluations like:
```
payload.enriched.br_cookies -> Some(null) -> BigInt -> Left(Invalid type
BIGINT for value Some(null))
```
This PR allows marking fields nullable by default. We might want to consider a
smarter approach at some point.1 parent 39f3eb1 commit f6523ee
File tree
2 files changed
+2
-2
lines changed- modules/common/src
- main/scala/com/snowplowanalytics/snowplow/postgres/shredding
- test/scala/com/snowplowanalytics/snowplow/postgres/streaming
2 files changed
+2
-2
lines changedLines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
185 | 185 | | |
186 | 186 | | |
187 | 187 | | |
188 | | - | |
| 188 | + | |
189 | 189 | | |
190 | 190 | | |
191 | 191 | | |
| |||
Lines changed: 1 addition & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
65 | | - | |
| 65 | + | |
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
| |||
0 commit comments