doi2dataset

doi2dataset is a Python tool designed to process DOIs and generate metadata for Dataverse.org datasets. It retrieves metadata from external APIs (such as OpenAlex and CrossRef), maps metadata fields, and can optionally upload the generated metadata to a Dataverse.org instance.

Features

• DOI Validation and Normalization: Validates DOIs and converts them into a standardized format.
• Metadata Retrieval: Fetches metadata such as title, abstract, license, and author information from external sources.
• Metadata Mapping: Automatically maps and generates metadata fields (e.g., title, description, keywords) including support for controlled vocabularies and compound fields.
• Optional Upload: Allows uploading of metadata directly to a Dataverse.org server.
• Progress Tracking: Uses the Rich library for user-friendly progress tracking and error handling.

Installation

Clone the repository from GitHub:

git clone https://git.athemis.de/Athemis/doi2dataset
cd doi2dataset

Configuration

Configuration

Before running the tool, configure the necessary settings in the config.yaml file located in the project root. This file contains configuration details such as:

• Connection details (URL, API token, authentication credentials)
• Mapping of project phases
• Principal Investigator (PI) information
• Default grant configurations

Usage

Run doi2dataset from the command line by providing one or more DOIs:

python doi2dataset.py [options] DOI1 DOI2 ...

Command Line Options

• -f, --file Specify a file containing DOIs (one per line).

• -o, --output-dir Directory where metadata files will be saved.

• -d, --depositor Name of the depositor.

• -s, --subject Default subject for the metadata.

• -m, --contact-mail Contact email address.

• -u, --upload Upload metadata to a Dataverse.org server.

• -r, --use-ror Use Research Organization Registry (ROR) identifiers for institutions when available.

Documentation

Documentation is generated using Sphinx. See the docs/ directory for detailed API references and usage examples.

Testing

Testing

Tests are implemented with pytest. The test suite provides comprehensive coverage of core functionalities. To run the tests, execute:

pytest

Or using the Python module syntax:

python -m pytest

Code Coverage

The project includes code coverage analysis using pytest-cov. Current coverage is approximately 61% of the codebase, with key utilities and test infrastructure at 99-100% coverage.

To run tests with code coverage analysis:

pytest --cov=.

Generate a detailed HTML coverage report:

pytest --cov=. --cov-report=html

This creates a htmlcov directory. Open htmlcov/index.html in a browser to view the detailed coverage report.

A .coveragerc configuration file is provided that:

• Excludes test files, documentation, and boilerplate code from coverage analysis
• Configures reporting to ignore common non-testable lines (like defensive imports)
• Sets the output directory for HTML reports

Recent improvements have increased coverage from 48% to 61% by adding focused tests for:

• Citation building functionality
• License processing and validation
• Metadata field extraction
• OpenAlex integration
• Publication data parsing and validation

Areas that could benefit from additional testing:

• More edge cases in the MetadataProcessor class workflow
• Additional CitationBuilder scenarios with diverse inputs
• Complex network interactions and error handling

Test Structure

The test suite is organized into six main files:

1. test_doi2dataset.py: Basic tests for core functions like phase checking, name splitting and DOI validation.
2. test_fetch_doi_mock.py: Tests API interactions using a mock OpenAlex response stored in srep45389.json.
3. test_citation_builder.py: Tests for building citation metadata from API responses.
4. test_metadata_processor.py: Tests for the metadata processing workflow.
5. test_license_processor.py: Tests for license processing and validation.
6. test_publication_utils.py: Tests for publication year extraction and date handling.

Test Categories

The test suite covers the following categories of functionality:

Core Functionality Tests

• DOI Validation and Processing: Parameterized tests for DOI normalization, validation, and filename sanitization with various inputs.
• Phase Management: Tests for checking publication year against defined project phases, including boundary cases.
• Name Processing: Extensive tests for parsing and splitting author names in different formats (with/without commas, middle initials, etc.).
• Email Validation: Tests for proper validation of email addresses with various domain configurations.

API Integration Tests

• Mock API Responses: Tests that use a saved OpenAlex API response (srep45389.json) to simulate API interactions without making actual network requests.
• Data Fetching: Tests for retrieving and parsing data from the OpenAlex API.
• Abstract Extraction: Tests for extracting and cleaning abstracts from OpenAlex's inverted index format, including handling of empty or malformed abstracts.
• Subject Mapping: Tests for mapping OpenAlex topics to controlled vocabulary subject terms.

Metadata Processing Tests

• Citation Building: Tests for properly building citation metadata from API responses.
• License Processing: Tests for correctly identifying and formatting license information from various license IDs.
• Principal Investigator Matching: Tests for finding project PIs based on ORCID identifiers.
• Configuration Loading: Tests for properly loading and validating configuration from files.
• Metadata Workflow: Tests for the complete metadata processing workflow.

These tests ensure that all components work correctly in isolation and together as a system, with special attention to edge cases and error handling.

Contributing

Contributions are welcome! Please fork the repository and submit a pull request with your improvements.

License

This project is licensed under the MIT License. See the LICENSE.md file for details.