CSW harvester OutputSchema config support #258 #259

ccancellieri · 2021-10-22T12:40:40Z

This will close #258 adding support to an additional param into the csw json config:

"output_schema": "mdb"

mdb is the namespace of the schema to use (in this case it's an iso19115-3.2018)

{'mdb':'http://standards.iso.org/iso/19115/-3/mdb/2.0'}

Full Example below:

{
"user":"ckan_admin",
"cql": "dc:identifier = '0-----292--------------------------'",
"output_schema": "mdb",
"default_tags": [ ],
"default_extras": {},
"group_mapping": {},
"read_only": false
}

Doing this the CSW harvester will receive the metadata in the configured outputschema (must be supported by the target csw server).

…mespace and tag name so we can now harvest any kind of metadata (if validation is provided or ignored) ckan#209 ckan#210 ckan#219 ckan#258

ccancellieri · 2021-10-22T16:34:52Z

Can also help
#209
#210
#219

…run)

amercader

This looks good and useful @ccancellieri. I just added some minor comments

amercader · 2021-10-28T10:26:41Z

ckanext/spatial/harvesters/csw.py

-
+
+        # load config
+        self._set_source_config(harvest_object.source.config)


Can you document the new output_schema option and its default value in here so others are aware of it?

https://github.com/ckan/ckanext-spatial/blob/master/doc/harvesters.rst

added fallback to default in case the server is not supporting iso19139 -> 19115 transformation
the fallback will log and switch back to default asking for iso19139 -> iso19139.

amercader · 2021-10-28T10:28:03Z

ckanext/spatial/lib/csw_client.py

        self.sortby = SortBy([SortProperty('dc:identifier')])
+        # check capabilities
+        _cap = self.getcapabilities(endpoint)['response']
+        self.capabilities=etree.ElementTree(etree.fromstring(_cap))


Please try to follow PEP8 guidelines, specially spacing between = and , :)

sorry, I can't validate the whole project and my code editor is not helping me, good catch, I'll try to fix my bad.

amercader · 2021-10-28T10:31:05Z

ckanext/spatial/lib/csw_client.py

        csw = self._ows(**kw)

+        # fetch target csw server capabilities for requested output schema
+        output_schemas=self._get_output_schemas('GetRecords')


Can we move this call to the __init__() method to avoid duplication and multiple calls to GetCapabilities?
Something like:

def __init__(self, endpoint=None): _cap = self.getcapabilities(endpoint)['response'] self.capabilities = etree.ElementTree(etree.fromstring(_cap)) self.output_schemas = { 'GetRecords': self._get_output_schemas('GetRecords'), 'GetRecordById': self._get_output_schemas('GetRecordById'), }

ccancellieri · 2021-10-28T18:21:36Z

ckanext/spatial/lib/csw_client.py

+        # fetch target csw server capabilities for requested output schema
+        output_schemas=output_schemas = self.output_schemas['GetRecordById']
+        if not output_schemas.get(outputschema):
+            raise CswError('Output schema \'{}\' not supported by target server: '.format(output_schemas))


Probably here I should be more tolerant Logging ERROR and returning.

frafra · 2022-02-18T13:09:28Z

This is great :) Do you need any help with this PR?

frafra · 2022-03-01T15:22:08Z

I get this generic error after applying this PR (rebased on master) : Error contacting the CSW server: can only parse strings. I think there is a problem with the changes made to the __init__ function of CswService.

ccancellieri · 2022-03-01T17:19:04Z

Ciao @frafra thanks to look into this.
I think something bad could happen here:

ckanext-spatial/ckanext/spatial/lib/csw_client.py

Line 210 in b82eb89

record = self._xmd(etree.fromstring(csw.response))

Would you be able to check the response provided by the server?

I'm apologize but I'm not using this plugin anymore, I changed approach, so my help can be very limited on this.

frafra · 2022-03-04T09:22:19Z

@ccancellieri I think you are right, I will look into that.
Which approach have you taken, If I may ask? I am interested into harvesting data from GeoNetwork too.

ccancellieri · 2022-03-07T18:32:26Z

Please search for json and geospatial keywords under ckan discussions ;) Il giorno ven 4 mar 2022 alle ore 10:22 Francesco Frassinelli < ***@***.***> ha scritto:

…

@ccancellieri <https://github.com/ccancellieri> I think you are right, I will look into that. Which approach have you taken, If I may ask? I am interested into harvesting data from GeoNetwork too. — Reply to this email directly, view it on GitHub <#259 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAG5WW6GMNHSSTZJGE5UIHTU6HI5NANCNFSM5GQO7MWA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Carlo Cancellieri *Skype*: ccancellieri *Twitter*: @cancellieric *LinkedIn*: http://it.linkedin.com/in/ccancellieri/

frafra · 2022-03-08T15:43:53Z

CSW harvester OutputSchema config support ckan#258

e2349d9

fix special character formatting

33d9b70

preload config to read namespace if necessary. Generalize metadata na…

855ab7d

…mespace and tag name so we can now harvest any kind of metadata (if validation is provided or ignored) ckan#209 ckan#210 ckan#219 ckan#258

This was referenced Oct 22, 2021

Support of ISO19115-3:2016 Geospatial Standard #212

Open

Iso19115 3 #260

Open

ccancellieri added 3 commits October 23, 2021 00:08

lets reload config at each run (the config can be different for each …

bf2d1d6

…run)

improve checks and comments

fbd5d1c

namespace from getcapabilities also for getrecords

9d36378

amercader reviewed Oct 28, 2021

View reviewed changes

documentation, pep8 and improve are requested

4fb17cc

ccancellieri commented Oct 28, 2021

View reviewed changes

ccancellieri added 2 commits October 28, 2021 21:35

fallback to default schema

f91f516

fallback to default schema

b82eb89

markstuart added a commit to data-govt-nz/ckanext-spatial that referenced this pull request Jun 2, 2023

Port and improve work done in ckan#259

3b2dfa2



		# load config
		self._set_source_config(harvest_object.source.config)

CSW harvester OutputSchema config support #258 #259

Are you sure you want to change the base?

CSW harvester OutputSchema config support #258 #259

Uh oh!

Conversation

ccancellieri commented Oct 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ccancellieri commented Oct 22, 2021

Uh oh!

amercader left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

frafra commented Feb 18, 2022

Uh oh!

frafra commented Mar 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ccancellieri commented Mar 1, 2022

Uh oh!

frafra commented Mar 4, 2022

Uh oh!

ccancellieri commented Mar 7, 2022 via email

Uh oh!

frafra commented Mar 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ccancellieri commented Oct 22, 2021 •

edited

Loading

frafra commented Mar 1, 2022 •

edited

Loading

frafra commented Mar 8, 2022 •

edited

Loading