2117 Developing Focused Literature Collections: A Pilot Study from Dental Informatics

Friday, March 18, 2011: 2 p.m. - 3:15 p.m.
Location: Hall C (San Diego Convention Center)
Presentation Type: Poster Session
R. REED1, H. SPALLEK2, J. JIANG3, D. HE3, A. ACHARYA4, J. CLOSE5, M. SONG5, T. SUCKOW2, T. THYVALIKAKATH2, and T. SCHLEYER2, 1Department of Biomedical Informatics, University of Pittsburgh, Pittsburgh, PA, 2School of Dental Medicine, University of Pittsburgh, Pittsburgh, PA, 3School of Information Science, University of Pittsburgh, Pittsburgh, PA, 4Marshfield Clinic, Marshfield Clinic Research Foundation, Marshfield, WI, 5Dental Public Hlth & Info Mgt, University of Pittsburgh, Pittsburgh, PA
Objectives: Since 2003 the Publication Archive of the Dental Informatics Online Community (DIOC) has served as a valuable resource for people interested in dental informatics by providing a comprehensive collection of papers in dental informatics and information technology in dentistry. The MEDLINE search in place for this retrieval captures many targeted papers but also some outside of the scope of the DIOC. To increase the number of relevant papers deposited in the Archive, we designed a process that combines human review with a na´ve Bayes classifier to filter the literature. The goal of this project was to develop a methodology that can be implemented when designing specific literature collections.

Methods: A manual review of 6,000 archived papers by six raters began with two pilot phases, where decisions regarding inclusion or exclusion from the DIOC were assessed. The pilot phases allowed raters to evaluate 400 papers, resolve areas of disagreement, and calculate Cohen's kappas to establish inter-rater reliability. During the production phase of the project, assessors evaluated both singly rated papers along with common papers to monitor consistent inter-rater reliability.

Results: We have completed the two part pilot phase along with the first half of the production phase. These assessments have allowed us to establish a relative distribution of dental informatics and IT in dentistry papers and identify effective as well as problematic MeSH headings for optimizing the MEDLINE retrieval and for classifier development. Results from the machine learning algorithm indicate that the classifier is reproducing the human ratings of papers.

Conclusion: This work demonstrates how a specialized MEDLINE publication filter, when used in conjunction with a classifier, can selectively define papers of interest and assist in future decisions regarding publication specificity. This approach can be implemented across disciplines using similar techniques. Funded by NLM 3-T15-LM007059-23S1, NLM 1-G08-LM-8667-1A1 and NIDCR K08DE018957.

This abstract is based on research that was funded entirely or partially by an outside source:

Keywords: Assessment, Decision-making, Dental informatics, Education research and Methodology