Support for multiple contact points/publishers

[hcvdwerf]

DCAT-AP 3.0 makes the inclusion of multiple contacts mandatory for datasets. Specifically, the dcat:contactPoint property must be able to reference multiple contact points. This requirement aligns with the latest DCAT-AP specification.

Code I use to test

@prefix dcterms: <http://purl.org/dc/terms/> .
@prefix dcat: <http://www.w3.org/ns/dcat#> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
@prefix ldp: <http://www.w3.org/ns/ldp#> .

<https://health-ri.sandbox.semlab-leiden.nl/dataset/d7129d28-b72a-437f-8db0-4f0258dd3c25>
  a dcat:Resource, dcat:Dataset;
  <http://www.w3.org/2000/01/rdf-schema#label> "Example";
  dcterms:title "Example";
  <https://w3id.org/fdp/fdp-o#metadataIssued> "2023-09-05T12:00:36.276171042Z"^^xsd:dateTime;
  <https://w3id.org/fdp/fdp-o#metadataModified> "2024-05-02T13:01:35.716385359Z"^^xsd:dateTime;
  dcterms:license <http://rdflicense.appspot.com/rdflicense/cc-by-nc-nd3.0>;
  dcterms:description "This is an example description.";
  dcat:contactPoint <https://orcid.org/0000-0002-9095-9201>, <https://orcid.org/0000-0003-2558-7496> .

@pytest.mark.ckan_config("ckan.plugins", "scheming_datasets")
@pytest.mark.usefixtures("with_plugins")
def test_profile_contact_point_multiple_uris():
    fdp_record_to_package = FairDataPointRecordToPackageConverter(profile="fairdatapoint_dcat_ap")
    data = Graph().parse(Path(TEST_DATA_DIRECTORY, "contact_point_multiple_urls.ttl")).serialize()
    actual = fdp_record_to_package.record_to_package(
        guid="https://health-ri.sandbox.semlab-leiden.nl/catalog/e3faf7ad-050c-475f-8ce4-da7e2faa5cd0;"
             "dataset=https://health-ri.sandbox.semlab-leiden.nl/dataset/d7129d28-b72a-437f-8db0-4f0258dd3c25",
        record=data)
    expected = {
        'extras': [],
        'title': 'Example',
        'notes': 'This is an example description.',
        'contact': [
            {
                'uri': 'https://orcid.org/0000-0002-9095-9201'
            },
            {
                'uri': 'https://orcid.org/0000-0003-2558-7496'
            }
        ],
        'license_id': '',
        'resources': [],
        'tags': [],
        'uri': 'https://health-ri.sandbox.semlab-leiden.nl/dataset/d7129d28-b72a-437f-8db0-4f0258dd3c25'
    }
    assert actual == expected

[hcvdwerf]

Same for publisher

[amercader]

@hcvdwerf The handling of multiple contact points is done via the repeating_subfields preset. This is already working in the euro_dcat_ap_3 and euro_dcat_ap_2 + euro_dcat_ap_scheming profiles. It allows you to define multiple contact points:

In your example, you would only provide the URI field.

As for publishers, DCAT AP 3 only allows one instance (0..1) but if you want to support multiple publishers you can remove the repeating_once: true bit on the field schema definition.

I'm not keen on extending support for multiple instances to the legacy harvesters using contact_* extras as this would add complexity and we want to encourage users to migrate to scheming-based profiles.

Let me know if this makes sense

[hcvdwerf]

I think the issue arises during harvesting, not when entering data. If you refer to this pull request, you'll notice that the _publisher function does not return a list.

I’ve run multiple tests with various contact points, but only one is returned. Also, the repeating_once: true property is not defined in the DCAT extension itself.

For reference, you can see a class full of tests involving multiple contact points here: test_profiles.py.

[Markus92]

There's defintely a bug in the harvester: even though there is a for-loop, it only returns a single dict. The dict gets overwritten on every iteration. The changes in this PR make it such that a list is returned of dicts, which we understood to be the intended be intended behavior.

@amercader If you'd like the legacy fields to remain as is, we can modify the PR such that it will only ever return one element in that case(publisher/contact). The only thing is you'd have no control whatsoever which publisher/contact it would be in the case of multi-valued input, as graphs are inherently out-of-order (it'd be pretty much random, or down to the used memory allocator, to determine the order they are looped over).
We use the scheming ourselves too, no legacy stuff.

Mark,
SW Engineer at Health-RI

[amercader]

@Markus92 @hcvdwerf let me have a closer look next week, but I agree that it seems like a bug in the harvester. It shouldn't matter whether a dataset was harvested, created via UI or API, all use the same DCAT -> CKAN parser
We probably need to review the harvester to make sure scheming-based datasets store all fields properly

Also, the repeating_once: true property is not defined in the DCAT extension itself.

This is just used here to prevent showing the "Add more" button in the form for this particular field, but yeah, datasets created via API or harvester can have (or should have after we fix this bug) multiple publishers

[hcvdwerf]

@amercader any update from your side? Tnx in advance!

[amercader]

@hcvdwerf Sorry I finally got time to work on this. I encountered the same issue while working on #314 and I'm hoping to have a generic fix for this today. I'll ping you when it's ready so you can test if it fixes your issue.

[amercader]

@hcvdwerf @Markus92 let me know if #317 works for your issue

[hcvdwerf]

@amercader I have looked to the code and did some tests, but until now it is working. For example

[hcvdwerf]

@amercader When are you planning to do a release?

[amercader]

This week or the next one, once the DCAT US and multilingual PRs are merged