lubitchv · lubitchv · Feb 18, 2020 · Jan 17, 2020 · Jan 17, 2020 · Jan 22, 2020
diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md
@@ -0,0 +1,76 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+ advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+ address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+ professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at support at dataverse dot org. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq
diff --git a/conf/docker-aio/0prep_deps.sh b/conf/docker-aio/0prep_deps.sh
@@ -17,12 +17,12 @@ if [ ! -e dv/deps/glassfish4dv.tgz ]; then
 	# assuming that folks usually have /tmp auto-clean as needed
 fi
 
-if [ ! -e dv/deps/solr-7.3.1dv.tgz ]; then
+if [ ! -e dv/deps/solr-7.7.2dv.tgz ]; then
 	echo "solr dependency prep"
 	# schema changes *should* be the only ones...
 	cd dv/deps/
 	#wget https://archive.apache.org/dist/lucene/solr/7.3.0/solr-7.3.0.tgz -O solr-7.3.0dv.tgz
-	wget https://archive.apache.org/dist/lucene/solr/7.3.1/solr-7.3.1.tgz -O solr-7.3.1dv.tgz
+	wget https://archive.apache.org/dist/lucene/solr/7.7.2/solr-7.7.2.tgz -O solr-7.7.2dv.tgz
 	cd ../../
 fi
 
diff --git a/conf/docker-aio/1prep.sh b/conf/docker-aio/1prep.sh
@@ -4,9 +4,9 @@
 # this was based off the phoenix deployment; and is likely uglier and bulkier than necessary in a perfect world
 
 mkdir -p testdata/doc/sphinx-guides/source/_static/util/
-cp ../solr/7.3.1/schema*.xml testdata/
-cp ../solr/7.3.1/solrconfig.xml testdata/
-cp ../solr/7.3.1/updateSchemaMDB.sh testdata/
+cp ../solr/7.7.2/schema*.xml testdata/
+cp ../solr/7.7.2/solrconfig.xml testdata/
+cp ../solr/7.7.2/updateSchemaMDB.sh testdata/
 cp ../jhove/jhove.conf testdata/
 cp ../jhove/jhoveConfig.xsd testdata/
 cd ../../

diff --git a/conf/docker-aio/c7.dockerfile b/conf/docker-aio/c7.dockerfile
@@ -17,7 +17,7 @@ COPY testdata/sushi_sample_logs.json /tmp/
 COPY disableipv6.conf /etc/sysctl.d/
 RUN rm /etc/httpd/conf/*
 COPY httpd.conf /etc/httpd/conf 
-RUN cd /opt ; tar zxf /tmp/dv/deps/solr-7.3.1dv.tgz 
+RUN cd /opt ; tar zxf /tmp/dv/deps/solr-7.7.2dv.tgz 
 RUN cd /opt ; tar zxf /tmp/dv/deps/glassfish4dv.tgz
 
 # this copy of domain.xml is the result of running `asadmin set server.monitoring-service.module-monitoring-levels.jvm=LOW` on a default glassfish installation (aka - enable the glassfish REST monitir endpoint for the jvm`
@@ -28,9 +28,9 @@ RUN sudo -u postgres /usr/pgsql-9.6/bin/initdb -D /var/lib/pgsql/data
 
 # copy configuration related files
 RUN cp /tmp/dv/pg_hba.conf /var/lib/pgsql/data/
-RUN cp -r /opt/solr-7.3.1/server/solr/configsets/_default /opt/solr-7.3.1/server/solr/collection1
-RUN cp /tmp/dv/schema*.xml /opt/solr-7.3.1/server/solr/collection1/conf/
-RUN cp /tmp/dv/solrconfig.xml /opt/solr-7.3.1/server/solr/collection1/conf/solrconfig.xml
+RUN cp -r /opt/solr-7.7.2/server/solr/configsets/_default /opt/solr-7.7.2/server/solr/collection1
+RUN cp /tmp/dv/schema*.xml /opt/solr-7.7.2/server/solr/collection1/conf/
+RUN cp /tmp/dv/solrconfig.xml /opt/solr-7.7.2/server/solr/collection1/conf/solrconfig.xml
 
 # skipping glassfish user and solr user (run both as root)
 

diff --git a/conf/docker-aio/entrypoint.bash b/conf/docker-aio/entrypoint.bash
@@ -2,7 +2,7 @@
 export LANG=en_US.UTF-8
 #sudo -u postgres /usr/bin/postgres -D /var/lib/pgsql/data &
 sudo -u postgres /usr/pgsql-9.6/bin/postgres -D /var/lib/pgsql/data &
-cd /opt/solr-7.3.1/
+cd /opt/solr-7.7.2/
 # TODO: Run Solr as non-root and remove "-force".
 bin/solr start -force
 bin/solr create_core -c collection1 -d server/solr/collection1/conf -force

diff --git a/conf/docker-aio/testscripts/install b/conf/docker-aio/testscripts/install
@@ -15,7 +15,7 @@ export SMTP_SERVER=localhost
 export MEM_HEAP_SIZE=2048
 export GLASSFISH_DOMAIN=domain1
 cd scripts/installer
-cp pgdriver/postgresql-42.2.2.jar $GLASSFISH_ROOT/glassfish/lib
+cp pgdriver/postgresql-42.2.9.jar $GLASSFISH_ROOT/glassfish/lib
 #cp ../../conf/jhove/jhove.conf $GLASSFISH_ROOT/glassfish/domains/$GLASSFISH_DOMAIN/config/jhove.conf
 cp /opt/dv/testdata/jhove.conf $GLASSFISH_ROOT/glassfish/domains/$GLASSFISH_DOMAIN/config/jhove.conf
 cp /opt/dv/testdata/jhoveConfig.xsd $GLASSFISH_ROOT/glassfish/domains/$GLASSFISH_DOMAIN/config/jhoveConfig.xsd

diff --git a/conf/docker/dataverse-glassfish/Dockerfile b/conf/docker/dataverse-glassfish/Dockerfile
@@ -70,7 +70,7 @@ RUN /tmp/dvinstall/glassfish-setup.sh
 ###glassfish-setup will handle everything in Dockerbuild
 
 ##install jdbc driver 
-RUN cp /tmp/dvinstall/pgdriver/postgresql-42.2.2.jar /usr/local/glassfish4/glassfish/domains/domain1/lib
+RUN cp /tmp/dvinstall/pgdriver/postgresql-42.2.9.jar /usr/local/glassfish4/glassfish/domains/domain1/lib
 
 # Customized persistence xml to avoid database recreation
 #RUN mkdir -p /tmp/WEB-INF/classes/META-INF/ 

diff --git a/conf/solr/7.3.1/readme.md → conf/solr/7.7.2/readme.md b/conf/solr/7.3.1/readme.md → conf/solr/7.7.2/readme.md
diff --git a/conf/solr/7.3.1/schema.xml → conf/solr/7.7.2/schema.xml b/conf/solr/7.3.1/schema.xml → conf/solr/7.7.2/schema.xml
diff --git a/conf/solr/7.3.1/schema_dv_mdb_copies.xml → conf/solr/7.7.2/schema_dv_mdb_copies.xml b/conf/solr/7.3.1/schema_dv_mdb_copies.xml → conf/solr/7.7.2/schema_dv_mdb_copies.xml
diff --git a/conf/solr/7.3.1/schema_dv_mdb_fields.xml → conf/solr/7.7.2/schema_dv_mdb_fields.xml b/conf/solr/7.3.1/schema_dv_mdb_fields.xml → conf/solr/7.7.2/schema_dv_mdb_fields.xml
diff --git a/conf/solr/7.3.1/solrconfig.xml → conf/solr/7.7.2/solrconfig.xml b/conf/solr/7.3.1/solrconfig.xml → conf/solr/7.7.2/solrconfig.xml
diff --git a/conf/solr/7.3.1/updateSchemaMDB.sh → conf/solr/7.7.2/updateSchemaMDB.sh b/conf/solr/7.3.1/updateSchemaMDB.sh → conf/solr/7.7.2/updateSchemaMDB.sh
diff --git a/doc/release-notes/6262-release-notes.md b/doc/release-notes/6262-release-notes.md
@@ -0,0 +1,4 @@
+### Dataverse Linking Fix
+
+The fix implemented for ticket 6262 will display the datasets contained in linked dataverses in the linking dataverse. Going forward this will happen automatically whenever a dataverse is linked. 
+In order for datasets belonging to dataverses previously linked to be displayed properly in the linking dataverse you must re-index those dataverses that had been linked. It's probably easiest to simply re-index all.
diff --git a/doc/release-notes/6522-datafile-duplicates.md b/doc/release-notes/6522-datafile-duplicates.md
@@ -0,0 +1,27 @@
+In this Dataverse release, we are adding a database constraint to
+prevent duplicate DataFile objects pointing to the same physical file
+from being created.
+
+Before this release can be deployed, your database must be checked
+for any such duplicates that may already exist. If present,
+the duplicates will need to be deleted, and the integrity of the 
+stored physical files verified.
+
+(We have notified the community about this issue ahead of the release,
+so you may have already addressed it. In this case, please disregard
+this release note)
+
+Please run the diagnostic script provided at
+https://github.com/IQSS/dataverse/raw/develop/scripts/issues/6522/find_duplicates.sh.
+The script relies on the PostgreSQL utility `psql` to access the
+database. You will need to edit the credentials at the top of the script
+to match your database configuration.
+
+If this issue is not present in your database, you will see a message
+`... no duplicate dvObjects in your database. Your installation is
+ready to be upgraded to Dataverse 4.20`.
+
+If duplicates are detected, it will provide further instructions. We
+will need you to send us the produced output. We will then assist you
+in resolving this problem in your database.
+
diff --git a/doc/release-notes/6534-new-guestbook-columns.md b/doc/release-notes/6534-new-guestbook-columns.md
@@ -0,0 +1,10 @@
+Users of downloaded guestbooks should note that two new columns have been added:
+
+- Dataset PID
+- File PID
+
+If you are expecting column in the CVS file to be in a particular order, you will need to make adjustments. Please see below for details:
+
+Old columns: Guestbook, Dataset, Date, Type, File Name, File Id, User Name, Email, Institution, Position, Custom Questions
+
+New columns: Guestbook, Dataset, Dataset PID, Date, Type, File Name, File Id, File PID, User Name, Email, Institution, Position, Custom Questions
diff --git a/doc/release-notes/6570-search-api-contact-affiliation.md b/doc/release-notes/6570-search-api-contact-affiliation.md
@@ -0,0 +1 @@
+As reported in https://github.com/IQSS/dataverse/issues/6570 the affiliation for dataset contacts has been wrapped in parentheses in the JSON output from the Search API. These parentheses have now been removed. This is a backward incompatible change but it is hoped that it won't cause any problems for integrations and API users.
diff --git a/doc/release-notes/6590-citation-reload.md b/doc/release-notes/6590-citation-reload.md
@@ -0,0 +1,4 @@
+5. Update Citation Metadata Block
+
+- `wget https://github.com/IQSS/dataverse/releases/download/$RELEASE_NUMBER/citation.tsv`
+- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"`
diff --git a/doc/release-notes/6599-update-solr-772.md b/doc/release-notes/6599-update-solr-772.md
@@ -0,0 +1,9 @@
+With this release we upgrade to the latest available stable release in the
+Solr 7.x branch.
+
+We recommend a fresh installation of Solr 7.2.2 (the index will be empty)
+followed by an "index all".
+
+Before you start the "index all", Dataverse will appear to be empty because
+the search results come from Solr. As indexing progresses, results will appear
+until indexing is complete.
diff --git a/doc/sphinx-guides/source/_static/admin/dataverse-external-tools.tsv b/doc/sphinx-guides/source/_static/admin/dataverse-external-tools.tsv
@@ -1,5 +1,5 @@
 TwoRavens	explore	file	A system of interlocking statistical tools for data exploration, analysis, and meta-analysis: http://2ra.vn. See the :doc:`/user/data-exploration/tworavens` section of the User Guide for more information on TwoRavens from the user perspective and the :doc:`/installation/r-rapache-tworavens` section of the Installation Guide. 
 Data Explorer	explore	file	A GUI which lists the variables in a tabular data file allowing searching, charting and cross tabulation analysis. See the README.md file at https://github.com/scholarsportal/Dataverse-Data-Explorer for the instructions on adding Data Explorer to your Dataverse; and the :doc:`/installation/prerequisites` section of the Installation Guide for the instructions on how to set up **basic R configuration required** (specifically, Dataverse uses R to generate .prep metadata files that are needed to run Data Explorer). 
 Whole Tale	explore	dataset	A platform for the creation of reproducible research packages that allows users to launch containerized interactive analysis environments based on popular tools such as Jupyter and RStudio. Using this integration, Dataverse users can launch Jupyter and RStudio environments to analyze published datasets. For more information, see the `Whole Tale User Guide <https://wholetale.readthedocs.io/en/stable/users_guide/integration.html>`_.
-File Previewers	explore	file	A set of tools that display the content of files - including audio, html, `Hypothes.is <https://hypothes.is/>`_ annotations, images, PDF, text, video, tabular data, and spreadsheets - allowing them to be viewed without downloading. The previewers can be run directly from github.io, so the only required step is using the Dataverse API to register the ones you want to use. Documentation, including how to optionally brand the previewers, and an invitation to contribute through github are in the README.md file. Initial development was led by the Qualitative Data Repository and the spreasdheet previewer was added by the Social Sciences and Humanities Open Cloud (SSHOC) project. https://github.com/QualitativeDataRepository/dataverse-previewers
+File Previewers	explore	file	A set of tools that display the content of files - including audio, html, `Hypothes.is <https://hypothes.is/>`_ annotations, images, PDF, text, video, tabular data, and spreadsheets - allowing them to be viewed without downloading. The previewers can be run directly from github.io, so the only required step is using the Dataverse API to register the ones you want to use. Documentation, including how to optionally brand the previewers, and an invitation to contribute through github are in the README.md file. Initial development was led by the Qualitative Data Repository and the spreasdheet previewer was added by the Social Sciences and Humanities Open Cloud (SSHOC) project. https://github.com/GlobalDataverseCommunityConsortium/dataverse-previewers
 Data Curation Tool	configure	file	A GUI for curating data by adding labels, groups, weights and other details to assist with informed reuse. See the README.md file at https://github.com/scholarsportal/Dataverse-Data-Curation-Tool for the installation instructions.
diff --git a/doc/sphinx-guides/source/_static/installation/files/etc/init.d/solr b/doc/sphinx-guides/source/_static/installation/files/etc/init.d/solr
@@ -5,7 +5,7 @@
 # chkconfig: 35 92 08
 # description: Starts and stops Apache Solr
 
-SOLR_DIR="/usr/local/solr/solr-7.3.1"
+SOLR_DIR="/usr/local/solr/solr-7.7.2"
 SOLR_COMMAND="bin/solr"
 SOLR_ARGS="-m 1g -j jetty.host=127.0.0.1"
 SOLR_USER=solr

diff --git a/doc/sphinx-guides/source/_static/installation/files/etc/systemd/solr.service b/doc/sphinx-guides/source/_static/installation/files/etc/systemd/solr.service
@@ -5,9 +5,9 @@ After = syslog.target network.target remote-fs.target nss-lookup.target
 [Service]
 User = solr
 Type = forking
-WorkingDirectory = /usr/local/solr/solr-7.3.1
-ExecStart = /usr/local/solr/solr-7.3.1/bin/solr start -m 1g -j "jetty.host=127.0.0.1"
-ExecStop = /usr/local/solr/solr-7.3.1/bin/solr stop
+WorkingDirectory = /usr/local/solr/solr-7.7.2
+ExecStart = /usr/local/solr/solr-7.7.2/bin/solr start -m 1g -j "jetty.host=127.0.0.1"
+ExecStop = /usr/local/solr/solr-7.7.2/bin/solr stop
 LimitNOFILE=65000
 LimitNPROC=65000
 Restart=on-failure

diff --git a/doc/sphinx-guides/source/admin/integrations.rst b/doc/sphinx-guides/source/admin/integrations.rst
@@ -19,9 +19,9 @@ If your researchers have data on Dropbox, you can make it easier for them to get
 Open Science Framework (OSF)
 ++++++++++++++++++++++++++++
 
-The Center for Open Science's Open Science Framework (OSF) is an open source software project that facilitates open collaboration in science research across the lifespan of a scientific project. 
+The Center for Open Science's Open Science Framework (OSF) is an open source software project that facilitates open collaboration in science research across the lifespan of a scientific project.
 
-For instructions on depositing data from OSF to your installation of Dataverse, your researchers can visit http://help.osf.io/m/addons/l/863978-connect-dataverse-to-a-project 
+For instructions on depositing data from OSF to your installation of Dataverse, your researchers can visit http://help.osf.io/m/addons/l/863978-connect-dataverse-to-a-project
 
 RSpace
 ++++++
@@ -41,6 +41,22 @@ As of this writing only OJS 2.x is supported and instructions for getting starte
 
 If you are interested in OJS 3.x supporting deposit from Dataverse, please leave a comment on https://github.com/pkp/pkp-lib/issues/1822
 
+Renku
++++++
+
+Renku is a platform that enables collaborative, reproducible and reusable
+(data)science. It allows researchers to automatically record the provenance of
+their research results and retain links to imported and exported data. Users
+can organize their data in "Datasets", which can be exported to Dataverse via
+the command-line interface (CLI).
+
+Renku dataset documentation: https://renku-python.readthedocs.io/en/latest/commands.html#module-renku.cli.dataset
+
+Flagship deployment of the Renku platform: https://renkulab.io
+
+Renku discourse: https://renku.discourse.group/
+
+
 Embedding Data on Websites
 --------------------------
 
@@ -58,7 +74,7 @@ Analysis and Computation
 Data Explorer
 +++++++++++++
 
-Data Explorer is a GUI which lists the variables in a tabular data file allowing searching, charting and cross tabulation analysis. 
+Data Explorer is a GUI which lists the variables in a tabular data file allowing searching, charting and cross tabulation analysis.
 
 For installation instructions, see the :doc:`external-tools` section.
 
@@ -95,6 +111,20 @@ Researchers can launch Jupyter Notebooks, RStudio, and other computational envir
 
 Institutions can self host BinderHub. Dataverse is one of the supported `repository providers <https://binderhub.readthedocs.io/en/latest/developer/repoproviders.html#supported-repoproviders>`_.
 
+Renku
++++++
+
+Researchers can import Dataverse datasets into their Renku projects via the
+command-line interface (CLI) by using the Dataverse DOI. See the `renku Dataset
+documentation
+<https://renku-python.readthedocs.io/en/latest/commands.html#module-renku.cli.dataset>`_
+for details. Currently Dataverse ``>=4.8.x`` is required for the import to work. If you need
+support for an earlier version of Dataverse, please get in touch with the Renku team at
+`Discourse <https://renku.discourse.group>`_ or `GitHub <https://github.com/SwissDataScienceCenter/renku>`_.
+The UI implementation of the import is in progress and will be
+completed in Q12020.
+
+
 Discoverability
 ---------------
 
@@ -116,7 +146,7 @@ Research Data Preservation
 Archivematica
 +++++++++++++
 
-`Archivematica <https://www.archivematica.org>`_ is an integrated suite of open-source tools for processing digital objects for long-term preservation, developed and maintained by Artefactual Systems Inc. Its configurable workflow is designed to produce system-independent, standards-based Archival Information Packages (AIPs) suitable for long-term storage and management. 
+`Archivematica <https://www.archivematica.org>`_ is an integrated suite of open-source tools for processing digital objects for long-term preservation, developed and maintained by Artefactual Systems Inc. Its configurable workflow is designed to produce system-independent, standards-based Archival Information Packages (AIPs) suitable for long-term storage and management.
 
 Sponsored by the `Ontario Council of University Libraries (OCUL) <https://ocul.on.ca/>`_, this technical integration enables users of Archivematica to select datasets from connected Dataverse instances and process them for long-term access and digital preservation. For more information and list of known issues, please refer to Artefactual's `release notes <https://wiki.archivematica.org/Archivematica_1.8_and_Storage_Service_0.13_release_notes>`_, `integration documentation <https://www.archivematica.org/en/docs/archivematica-1.8/user-manual/transfer/dataverse/>`_, and the `project wiki <https://wiki.archivematica.org/Dataverse>`_.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		As reported in https://github.com/IQSS/dataverse/issues/6570 the affiliation for dataset contacts has been wrapped in parentheses in the JSON output from the Search API. These parentheses have now been removed. This is a backward incompatible change but it is hoped that it won't cause any problems for integrations and API users.