Dug: digging up dark data
Dug applies semantic web and knowledge graph techniques to biomedical research metadata, letting a researcher search underused datasets such as dbGaP by concept rather than exact keyword, based on curated ontologies.
Dug is built around the problem of finding relevant study variables in large biomedical metadata collections, such as dbGaP records derived from clinical research like the TOPMed datasets, where the concepts a researcher wants are rarely phrased the way the metadata is written. It annotates study metadata with terms from curated, peer-reviewed biomedical ontologies, places those terms in a shared upper ontology so studies from different sources can be connected, and builds a full-text search index over the resulting knowledge graph.
A command-line tool crawls source metadata into the graph and then supports searching by concept or by variable, and a REST API exposes the same search. Backend services, including Elasticsearch and Redis, run through Docker Compose.
It is a Python project released under the MIT license, with unit, integration and doctest suites.