Presentation Type: Poster Session
Methods: A manual review of 6,000 archived papers by six raters began with two pilot phases, where decisions regarding inclusion or exclusion from the DIOC were assessed. The pilot phases allowed raters to evaluate 400 papers, resolve areas of disagreement, and calculate Cohen's kappas to establish inter-rater reliability. During the production phase of the project, assessors evaluated both singly rated papers along with common papers to monitor consistent inter-rater reliability.
Results: We have completed the two part pilot phase along with the first half of the production phase. These assessments have allowed us to establish a relative distribution of dental informatics and IT in dentistry papers and identify effective as well as problematic MeSH headings for optimizing the MEDLINE retrieval and for classifier development. Results from the machine learning algorithm indicate that the classifier is reproducing the human ratings of papers.
Conclusion: This work demonstrates how a specialized MEDLINE publication filter, when used in conjunction with a classifier, can selectively define papers of interest and assist in future decisions regarding publication specificity. This approach can be implemented across disciplines using similar techniques. Funded by NLM 3-T15-LM007059-23S1, NLM 1-G08-LM-8667-1A1 and NIDCR K08DE018957.
Keywords: Assessment, Decision-making, Dental informatics, Education research and Methodology