README.rst

FindRD
======

Purpose
-------

FindRD crawls through an (institutional) text publication repository and visits
the respective journal’s website to find the corresponding data for each text
publication.  Then, this data is entered to the (institutional) research data
repository.


Architecture
------------

Database
........

At its heart, there is a MongoDB “FindRD” with the following collections:

findings
  This contains metadata of research data publications.  The only mandatory
  field is the PID.  It contains publications from external sources,
  i.e. Zenodo, Pangaea, datapub, and all text journals.  It is ever growing.

items
  This contains metadata of research data publications.  The only mandatory
  field is the PID.  It contains publications from our institutional Dataverse.
  It is ever growing.

aliases
  This maps PID aliases to their prime PID.  In “findings” and “items” are only
  prime PIDs in the MongoDB documents.

crawlings
  This contains the timestamp of last crawling for each crawler.


JuSER crawler
.............

This is a cron job.

This crawler fetches all eligible DOIs from the text publication repository.

At the same time, it creates a “text DOI” → “data PID” mapping from the data
repository.

It then contacts a scraper (see below) which returns the bibliographic metadata
for the accompanying research data.  In particular, it returns a PID for the
research data if it can be found.  Finally, it checks whether the research data
should be added to the data repository and if it should, it does so.


Scraper
,,,,,,,

This is a deployment.

The scraper works stateless as “cattle”.  Each instance listens to HTTP GET
requests which have a DOI as the parameter.  Then, the DOI is resolved and the
landing page searched for data about the research data.  This is then returned
in the same (possibly quite long-lasting) HTTP request.

The scraper has to resolve the DOI to a landing page.  For this, a Redis is
used as a cache.


Other crawlers
..............

These are cron jobs.

Other crawlers are more straightforward.  They harvest repositories and add
them to the “findings” collection.  They store their latest run in the
“crawling” collection *after* the run, to make sure that nothing is lost, while
not every time the whole repository is harvested.


Dataverse crawler
.................

This is a cron job.

This crawler is special because it adds records to the “items” collection.


Steward
.......

This is a cron job.

The steward takes all PIDs from “findings” that are not in “items” and does
something about it, e.g.:

- adds it to Dataverse
- sends an email to the authoring scientist
- sends an email to the FDM team


Documents
---------

The documents in the collections “findings” and “items” must have a “``pid``”
field.  This is the prime PID.  The rest is optional:

- ``institute``
- ``pof4_topic``
- ``aliases`` (alias PIDs)
- ``email`` (of contact person)
- …

..  LocalWords:  FindRD