|1953 Evaluating NLP Tools for Classifying Head and Neck Radiology Reports|
M. TORRES-URQUIDY1, T.P. THYVALIKAKATH2, P. HERNANDEZ2, K. LIU2, and W. CHAPMAN2, 1University of Pittsburgh, School of Dental Medicine, PA, USA, 2University of Pittsburgh, PA, USA|
Natural language processing (NLP) systems combined with data mining allow us to extract clinical or epidemiological information from pathology or radiology reports or from large data repositories such as the Practice-based Research Networks. NLP as a subfield of artificial intelligence can be defined as “the ability of a computer to process the same language that humans use in normal discourse”. However, NLP tools are usually domain specific and their effectiveness depends on the area of interest. NLP tools developed for the medical domain, although effective, may require adjustments if used specifically for dentistry. Objective: evaluate the effectiveness of the Identify Patient Sets (IPS) system and a custom-made string-matching software for retrieving Head and Neck Radiology Reports positive for oral surgery-related findings. Methods: six questions related to oral surgery and a Gold Standard (GS) of records that addressed those questions were created (by two domain experts). An example of the questions is “show all radiology reports where there are possible mandible fractures”. We compared the IPS system and a string-matching application against the GS and calculated sensitivity, specificity and Positive Predictive Value (PPV) in identifying relevant reports. The systems were “trained” using a subset of the GS (training set, n=206) and later tested using the reminder of the reports (n=338). Results: the average observed agreement between the two experts was 0.944. The highest PPV for the IPS Automatic Model Constructor (AMC) was 0.68 for the question “show the reports present a fracture involving floor of the orbit”. IPS' AMC models showed a better performance in retrieving relevant head and neck radiology reports than the string-matching application. Conclusion: The IPS system showed promise at retrieving patient records relevant to the dental investigator. Support for Dr. Torres-Urquidy was provided by CONACYT grant #167967.
|Seq #221 - Diagnostic Techniques - Mostly Digital|
2:00 PM-3:00 PM, Saturday, 11 March 2006 Dolphin Hotel Pacific Hall