I don't know which is harvesting server link

[Thalia-Uranga]

When I enable harvesting server it does not give me a link to perform interoperability tests. How do I see this part?

[pdurbin]

@thalia25 and I have been discussing this at http://irclog.iq.harvard.edu/dataverse/2017-10-13#i_58686 and it sounds like there are two problems:

• There is a usability issue wherein people who have just set up the Dataverse installation as a harvesting server by following http://guides.dataverse.org/en/4.8/admin/harvestserver.html gets no indication in the user interface of what URL they should share with the people who run the harvesting client. The goal is for the harvesting client (DSpace, for example) to start harvesting data from the Dataverse installation.
• The docs above should be improved to explain this.

[pdurbin]

Yeah, the UI doesn't tell you which URL to share. Maybe the URL could be included where it says "Successfully created harvesting set". Here's a screenshot:

[pdurbin]

I'm looking at http://guides.dataverse.org/en/4.8/admin/harvestserver.html but I'm not sure what URL to send for the set I just created. The only OAI-PMH verb I know is "Identify".

https://dev1.dataverse.org/oai?verb=Identify shows this:

[pdurbin]

@thalia25 does this help?

https://dev1.dataverse.org/oai?verb=ListSets

https://dev1.dataverse.org/oai?verb=ListRecords&metadataPrefix=oai_dc&set=goose

I found these at https://memory.loc.gov/ammem/oamh/oai_request.html

[jggautier]

Hope this helps!: I think the server URL for @pdurbin's example would be https://dev1.dataverse.org/oai. For Harvard Dataverse, it would be https://dataverse.harvard.edu/oai. That's what I would include as the Server URL when I'm adding harvesting client info in the Dataverse dashboard.

So I think if you followed that pattern with your installation (repositoryURL.xxx/oai?verb=ListSets) and got that page of xml of your list of oai sets, you'd know that the server URL is repositoryURL.xxx/oai.

We've noted that the guides, and I agree with @pdurbin that the application, should show what the server URL is.

[Thalia-Uranga]

@jggautier Hi, So the link that I must share to interoperate with another repository is with the link that you mention adding oai?

[jggautier]

@thalia25 Yes, that's what I would think. I'm only familiar with how to set up harvesting between Dataverse repositories, but I looked through DSpace's latest docs (version 6.x) about importing metadata, and it has some info about using an "OAIPMHDataLoader".

The OAIPMHDataLoader has the following properties:
a) fieldMap: Same as above, the mapping between the input keys holding the metadata and the ones that we want to have internal in BTE.
b) serverAddress: The base address of the OAI provider (server). Base address can be specified also in the "-s" option of the command prompt. If is specified in both places, the one specified from the command line is preferred.
c) prefix: The metadata prefix to be used in OAI requests.

As you said, I think that the serverAddress for your Dataverse installation would be the address of your installation plus /oai at the end.

Let me know if that makes sense and is helpful (or doesn't and isn't!).

(Eventually I think we'll need to get touch with someone who can say authoritatively what oai-pmh info Dataverse admins need, and how to get it, in order to share the metadata in their installation.with other systems.)

[Thalia-Uranga]

@jggautier and @pdurbin What is the difference between OAI server and OAI sets?

[jggautier]

The server URL points to the OAI server. When you enable harvesting, your Dataverse installation is a server. The Dataverse guides have a section on what OAI sets are that I think should help:

Once the service is enabled, you define collections of local datasets that will be available to remote harvesters as OAI Sets.

Have you seen this page on DSpace's guides about harvesting non-DSpace content into a DSpace installation?: https://wiki.duraspace.org/display/DSDOC6x/XMLUI+Configuration+and+Customization#XMLUIConfigurationandCustomization-HarvestingItemsfromXMLUIviaOAI-OREorOAI-PMH

[jggautier]

Would it be helpful if in the "Managing Harvesting Server and Sets" section of the Dataverse guides we add that

the default OAI-PMH service base address for Dataverses that enable harvesting is https://example.dataverse.org/oai

?

Also, I came across DataCite's OAI-PMH guide recently. (Wish I'd seen it a year ago!) I think it very clearly answers a lot of the types of questions Thalia was asking. Might be nice to borrow from it as well :)

[pdurbin]

@jggautier yes, I think an example like that would be very helpful and easy to add.

[Thalia-Uranga]

@pdurbin the one who did not know the OAI-PMH address was in the installation of dataverse version 4.7, it was thanks to you that I learned about the URL. I think it would be good to mention it in the versions that don't show the OAI-PMH address on the dataverse page.

[jggautier]

Thanks @thalia25! I just want to clarify that the OAI-PMH address is not in the UI of any Dataverse installation version on and after 4.0. (I'm not sure about Dataverse versions before 4.0.)

[pdurbin]

Today at http://irclog.iq.harvard.edu/dataverse/2018-12-18#i_83107 @poikilotherm said, "The docs could be more specific about the endpoint URL" and he said he'd roll a small doc change for this issue into pull request #5371 since he's already editing that part of the guides.

[poikilotherm]

Hey @thalia25,
do you think my recent docs addition in 0258d19 is a sufficient solution to this issue?
Cheers,
Oliver

[pdurbin]

@poikilotherm it looks fine to me. I haven't seen @thalia25 in IRC lately. (We miss you! 😄 ) Can you please go ahead and make a pull request? Thanks!

[poikilotherm]

This is already part of #5371 as discussed on IRC (sry, cannot find it right now in the logs)

[pdurbin]

@poikilotherm oh! I forgot. Please remind us to move this issue to QA once that pull request is moved to QA.

[pdurbin]

@poikilotherm pull request #5371 currently has merge conflicts. Do you feel like cherry-picking 0258d19 into a new branch and pull request?

[poikilotherm]

@pdurbin sorry, I missed this/forgot. Done, see #5834