J Am Med Inform Assoc - Evaluating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named entity recognition in clinical trial announcements.

Tópicos

{ extract(1171) text(1153) clinic(932) }
{ cost(1906) reduc(1198) effect(832) }
{ intervent(3218) particip(2042) group(1664) }
{ case(1353) use(1143) diagnosi(1136) }
{ time(1939) patient(1703) rate(768) }
{ monitor(1329) mobil(1314) devic(1160) }
{ sampl(1606) size(1419) use(1276) }
{ estim(2440) model(1874) function(577) }
{ model(3404) distribut(989) bayesian(671) }
{ system(1976) rule(880) can(841) }
{ studi(2440) review(1878) systemat(933) }
{ care(1570) inform(1187) nurs(1089) }
{ data(3963) clinic(1234) research(1004) }
{ perform(999) metric(946) measur(919) }
{ research(1085) discuss(1038) issu(1018) }
{ model(2656) set(1616) predict(1553) }
{ use(976) code(926) identifi(902) }
{ result(1111) use(1088) new(759) }
{ detect(2391) sensit(1101) algorithm(908) }
{ can(774) often(719) complex(702) }
{ measur(2081) correl(1212) valu(896) }
{ treatment(1704) effect(941) patient(846) }
{ framework(1458) process(801) describ(734) }
{ algorithm(1844) comput(1787) effici(935) }
{ model(2341) predict(2261) use(1141) }
{ ehr(2073) health(1662) electron(1139) }
{ research(1218) medic(880) student(794) }
{ drug(1928) target(777) effect(648) }
{ imag(1947) propos(1133) code(1026) }
{ data(1737) use(1416) pattern(1282) }
{ inform(2794) health(2639) internet(1427) }
{ imag(1057) registr(996) error(939) }
{ bind(1733) structur(1185) ligand(1036) }
{ sequenc(1873) structur(1644) protein(1328) }
{ method(1219) similar(1157) match(930) }
{ featur(3375) classif(2383) classifi(1994) }
{ imag(2830) propos(1344) filter(1198) }
{ network(2748) neural(1063) input(814) }
{ imag(2675) segment(2577) method(1081) }
{ patient(2315) diseas(1263) diabet(1191) }
{ take(945) account(800) differ(722) }
{ motion(1329) object(1292) video(1091) }
{ assess(1506) score(1403) qualiti(1306) }
{ surgeri(1148) surgic(1085) robot(1054) }
{ problem(2511) optim(1539) algorithm(950) }
{ error(1145) method(1030) estim(1020) }
{ chang(1828) time(1643) increas(1301) }
{ learn(2355) train(1041) set(1003) }
{ concept(1167) ontolog(924) domain(897) }
{ clinic(1479) use(1117) guidelin(835) }
{ method(1557) propos(1049) approach(1037) }
{ data(1714) softwar(1251) tool(1186) }
{ design(1359) user(1324) use(1319) }
{ control(1307) perform(991) simul(935) }
{ model(2220) cell(1177) simul(1124) }
{ general(901) number(790) one(736) }
{ method(984) reconstruct(947) comput(926) }
{ search(2224) databas(1162) retriev(909) }
{ featur(1941) imag(1645) propos(1176) }
{ howev(809) still(633) remain(590) }
{ studi(1410) differ(1259) use(1210) }
{ risk(3053) factor(974) diseas(938) }
{ system(1050) medic(1026) inform(1018) }
{ import(1318) role(1303) understand(862) }
{ visual(1396) interact(850) tool(830) }
{ compound(1573) activ(1297) structur(1058) }
{ perform(1367) use(1326) method(1137) }
{ studi(1119) effect(1106) posit(819) }
{ blood(1257) pressur(1144) flow(957) }
{ spatial(1525) area(1432) region(1030) }
{ record(1888) medic(1808) patient(1693) }
{ health(3367) inform(1360) care(1135) }
{ model(3480) simul(1196) paramet(876) }
{ state(1844) use(1261) util(961) }
{ patient(2837) hospit(1953) medic(668) }
{ data(2317) use(1299) case(1017) }
{ age(1611) year(1155) adult(843) }
{ medic(1828) order(1363) alert(1069) }
{ signal(2180) analysi(812) frequenc(800) }
{ group(2977) signific(1463) compar(1072) }
{ gene(2352) biolog(1181) express(1162) }
{ data(3008) multipl(1320) sourc(1022) }
{ first(2504) two(1366) second(1323) }
{ activ(1138) subject(705) human(624) }
{ patient(1821) servic(1111) care(1106) }
{ use(2086) technolog(871) perceiv(783) }
{ can(981) present(881) function(850) }
{ analysi(2126) use(1163) compon(1037) }
{ health(1844) social(1437) communiti(874) }
{ structur(1116) can(940) graph(676) }
{ high(1669) rate(1365) level(1280) }
{ cancer(2502) breast(956) screen(824) }
{ use(1733) differ(960) four(931) }
{ implement(1333) system(1263) develop(1122) }
{ survey(1388) particip(1329) question(1065) }
{ decis(3086) make(1611) patient(1517) }
{ process(1125) use(805) approach(778) }
{ activ(1452) weight(1219) physic(1104) }
{ method(1969) cluster(1462) data(1082) }
{ method(2212) result(1239) propos(1039) }

Resumo

JECTIVE: To present a series of experiments: (1) to evaluate the impact of pre-annotation on the speed of manual annotation of clinical trial announcements; and (2) to test for potential bias, if pre-annotation is utilized.METHODS: To build the gold standard, 1400 clinical trial announcements from the clinicaltrials.gov website were randomly selected and double annotated for diagnoses, signs, symptoms, Unified Medical Language System (UMLS) Concept Unique Identifiers, and SNOMED CT codes. We used two dictionary-based methods to pre-annotate the text. We evaluated the annotation time and potential bias through F-measures and ANOVA tests and implemented Bonferroni correction.RESULTS: Time savings ranged from 13.85% to 21.5% per entity. Inter-annotator agreement (IAA) ranged from 93.4% to 95.5%. There was no statistically significant difference for IAA and annotator performance in pre-annotations.CONCLUSIONS: On every experiment pair, the annotator with the pre-annotated text needed less time to annotate than the annotator with non-labeled text. The time savings were statistically significant. Moreover, the pre-annotation did not reduce the IAA or annotator performance. Dictionary-based pre-annotation is a feasible and practical method to reduce the cost of annotation of clinical named entity recognition in the eligibility sections of clinical trial announcements without introducing bias in the annotation process.

Resumo Limpo

jectiv present seri experi evalu impact preannot speed manual annot clinic trial announc test potenti bias preannot utilizedmethod build gold standard clinic trial announc clinicaltrialsgov websit random select doubl annot diagnos sign symptom unifi medic languag system uml concept uniqu identifi snome ct code use two dictionarybas method preannot text evalu annot time potenti bias fmeasur anova test implement bonferroni correctionresult time save rang per entiti interannot agreement iaa rang statist signific differ iaa annot perform preannotationsconclus everi experi pair annot preannot text need less time annot annot nonlabel text time save statist signific moreov preannot reduc iaa annot perform dictionarybas preannot feasibl practic method reduc cost annot clinic name entiti recognit elig section clinic trial announc without introduc bias annot process

Resumos Similares

AMIA Annu Symp Proc - Inter-annotator reliability of medical events, coreferences and temporal relations in clinical narratives by annotators with varying levels of clinical expertise. ( 0,880536682800786 )
J. Med. Internet Res. - Web 2.0-based crowdsourcing for high-quality gold standard development in clinical natural language processing. ( 0,875959138727967 )
J Am Med Inform Assoc - Combining rules and machine learning for extraction of temporal expressions and events from clinical narratives. ( 0,871069991121248 )
J Am Med Inform Assoc - An end-to-end system to identify temporal relation in discharge summaries: 2012 i2b2 challenge. ( 0,869295884395608 )
J Biomed Inform - MedTime: a temporal information extraction system for clinical narratives. ( 0,863057682198964 )
J Am Med Inform Assoc - Eventual situations for timeline extraction from clinical reports. ( 0,859844014825021 )
J Biomed Inform - Semi-automatic semantic annotation of PubMed queries: a study on quality, efficiency, satisfaction. ( 0,855329708507876 )
J Am Med Inform Assoc - A hybrid system for temporal information extraction from clinical text. ( 0,854273608173465 )
Appl Clin Inform - Representation of information about family relatives as structured data in electronic health records. ( 0,853905379401634 )
J Biomed Inform - Anaphoric reference in clinical reports: characteristics of an annotated corpus. ( 0,852811502664914 )
J Am Med Inform Assoc - Evaluating temporal relations in clinical text: 2012 i2b2 Challenge. ( 0,843468937721135 )
J Biomed Inform - Development and evaluation of RapTAT: a machine learning system for concept mapping of phrases from medical narratives. ( 0,842255938041144 )
J Am Med Inform Assoc - Assessing the role of a medication-indication resource in the treatment relation extraction from clinical text. ( 0,841528875317986 )
Artif Intell Med - Biomedical events extraction using the hidden vector state model. ( 0,840827337412042 )
J Am Med Inform Assoc - A sense inventory for clinical abbreviations and acronyms created using clinical notes and medical dictionary resources. ( 0,840566494137967 )
J Biomed Inform - A concept-driven biomedical knowledge extraction and visualization framework for conceptualization of text corpora. ( 0,836793902679227 )
J Biomed Inform - Towards generating a patient's timeline: extracting temporal relationships from clinical notes. ( 0,836093757054876 )
J Am Med Inform Assoc - Knowledge-based biomedical word sense disambiguation: an evaluation and application to clinical document classification. ( 0,833933772640195 )
J Am Med Inform Assoc - MedXN: an open source medication extraction and normalization tool for clinical text. ( 0,833674702090321 )
AMIA Annu Symp Proc - Natural language processing for lines and devices in portable chest x-rays. ( 0,831394282944055 )
J Am Med Inform Assoc - Automatic discourse connective detection in biomedical text. ( 0,831121614049076 )
AMIA Annu Symp Proc - Throw the bath water out, keep the baby: keeping medically-relevant terms for text mining. ( 0,829161256949138 )
J Am Med Inform Assoc - Anaphoric relations in the clinical narrative: corpus creation. ( 0,827467273758915 )
AMIA Annu Symp Proc - Building gold standard corpora for medical natural language processing tasks. ( 0,826732141578071 )
Int J Med Inform - Detection of infectious symptoms from VA emergency department and primary care clinical documentation. ( 0,824452905451127 )
Int J Med Inform - Detecting temporal expressions in medical narratives. ( 0,820554448384289 )
AMIA Annu Symp Proc - Semantic processing to identify adverse drug event information from black box warnings. ( 0,818723561164134 )
J Biomed Inform - Evaluating the effects of machine pre-annotation and an interactive annotation interface on manual de-identification of clinical text. ( 0,818628215969382 )
J Biomed Inform - NCBI disease corpus: a resource for disease name recognition and concept normalization. ( 0,817135178298986 )
AMIA Annu Symp Proc - Automatically pairing measured findings across narrative abdomen CT reports. ( 0,814498942502121 )
Comput Math Methods Med - Ranking biomedical annotations with annotator's semantic relevancy. ( 0,812126999116094 )
AMIA Annu Symp Proc - Mapping annotations with textual evidence using an scLDA model. ( 0,810742965353382 )
J Med Syst - Redactable signatures for signed CDA Documents. ( 0,808926340143065 )
J Biomed Inform - Identifying non-elliptical entity mentions in a coordinated NP with ellipses. ( 0,808544149856838 )
J Biomed Inform - UMLS content views appropriate for NLP processing of the biomedical literature vs. clinical text. ( 0,807543621275956 )
J Biomed Inform - Lessons learnt from the DDIExtraction-2013 Shared Task. ( 0,804339201544868 )
J Am Med Inform Assoc - A flexible framework for recognizing events, temporal expressions, and temporal relations in clinical text. ( 0,803014875659863 )
AMIA Annu Symp Proc - Natural language processing to extract follow-up provider information from hospital discharge summaries. ( 0,801034023095225 )
J Biomed Inform - A new clustering method for detecting rare senses of abbreviations in clinical notes. ( 0,800675344268081 )
J Am Med Inform Assoc - Assisted annotation of medical free text using RapTAT. ( 0,798969103533282 )
J Biomed Inform - Text summarization in the biomedical domain: a systematic review of recent research. ( 0,797536285993917 )
J Am Med Inform Assoc - 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. ( 0,795654830468744 )
J Integr Bioinform - Automatic extraction of microorganisms and their habitats from free text using text mining workflows. ( 0,79558658494462 )
AMIA Annu Symp Proc - Detecting abbreviations in discharge summaries using machine learning methods. ( 0,793346200186619 )
AMIA Annu Symp Proc - Extracting Concepts Related to Homelessness from the Free Text of VA Electronic Medical Records. ( 0,793243547488736 )
Int J Med Inform - Bootstrapping a de-identification system for narrative patient records: cost-performance tradeoffs. ( 0,792927879535152 )
Perspect Health Inf Manag - A comparison of two approaches to text processing: facilitating chart reviews of radiology reports in electronic medical records. ( 0,792699305658333 )
J Am Med Inform Assoc - A study of machine-learning-based approaches to extract clinical entities and their assertions from discharge summaries. ( 0,790912863565051 )
BMC Med Inform Decis Mak - Text summarization as a decision support aid. ( 0,789895324861399 )
J Biomed Inform - Extraction of events and temporal expressions from clinical narratives. ( 0,789692869083456 )
AMIA Annu Symp Proc - Pharmacovigilance on twitter? Mining tweets for adverse drug reactions. ( 0,788692275116286 )
AMIA Annu Symp Proc - A comparative study of current Clinical Natural Language Processing systems on handling abbreviations in discharge summaries. ( 0,787306650098823 )
J Biomed Inform - Lexical patterns, features and knowledge resources for coreference resolution in clinical notes. ( 0,78711832479515 )
AMIA Annu Symp Proc - Towards a semantic lexicon for clinical natural language processing. ( 0,786367005975813 )
Comput Methods Programs Biomed - BioAnnote: a software platform for annotating biomedical documents with application in medical learning environments. ( 0,785216816373112 )
J Biomed Inform - Degree centrality for semantic abstraction summarization of therapeutic studies. ( 0,781775062540937 )
J Am Med Inform Assoc - The effect of word familiarity on actual and perceived text difficulty. ( 0,778830957425756 )
Sci Data - Building the graph of medicine from millions of clinical narratives. ( 0,776655076538945 )
J Am Med Inform Assoc - MITRE system for clinical assertion status classification. ( 0,774744194581881 )
J Biomed Inform - Text de-identification for privacy protection: a study of its impact on clinical text information content. ( 0,772355074143178 )
J Biomed Inform - Automatically extracting information needs from complex clinical questions. ( 0,772075993021546 )
AMIA Annu Symp Proc - Combining corpus-derived sense profiles with estimated frequency information to disambiguate clinical abbreviations. ( 0,770878639440297 )
Brief. Bioinformatics - A survey on annotation tools for the biomedical literature. ( 0,767325402888381 )
AMIA Annu Symp Proc - Automated illustration of patients instructions. ( 0,766177727297738 )
J Biomed Inform - Enhancing clinical concept extraction with distributional semantics. ( 0,765931507877851 )
J Am Med Inform Assoc - Syntactic parsing of clinical text: guideline and corpus development with handling ill-formed sentences. ( 0,765929812922021 )
J Biomed Inform - The DDI corpus: an annotated corpus with pharmacological substances and drug-drug interactions. ( 0,765735267039804 )
J Biomed Inform - Annotating temporal information in clinical narratives. ( 0,765691815439627 )
Int J Med Inform - A methodology to enhance spatial understanding of disease outbreak events reported in news articles. ( 0,763795721515865 )
J Am Med Inform Assoc - Extracting drug indication information from structured product labels using natural language processing. ( 0,761871716625577 )
AMIA Annu Symp Proc - Application of a temporal reasoning framework tool in analysis of medical device adverse events. ( 0,758902103243311 )
J Am Med Inform Assoc - Large-scale evaluation of automated clinical note de-identification and its impact on information extraction. ( 0,758392899362317 )
J Biomed Inform - Desiderata for ontologies to be used in semantic annotation of biomedical documents. ( 0,758002625635227 )
AMIA Annu Symp Proc - Semantic processing to identify adverse drug event information from black box warnings. ( 0,757164586127943 )
J Am Med Inform Assoc - Towards comprehensive syntactic and semantic annotations of the clinical narrative. ( 0,756905739930518 )
J Am Med Inform Assoc - Hybrid methods for improving information access in clinical documents: concept, assertion, and relation identification. ( 0,756355536318449 )
AMIA Annu Symp Proc - Critical finding capture in the impression section of radiology reports. ( 0,752303429490278 )
J Am Med Inform Assoc - Automatic abstraction of imaging observations with their characteristics from mammography reports. ( 0,752004422987003 )
J Am Med Inform Assoc - Automated clinical trial eligibility prescreening: increasing the efficiency of patient identification for clinical trials in the emergency department. ( 0,751185331946834 )
J Am Med Inform Assoc - Using rule-based natural language processing to improve disease normalization in biomedical text. ( 0,750679665256276 )
AMIA Annu Symp Proc - TagLine: Information Extraction for Semi-Structured Text in Medical Progress Notes. ( 0,750535614047549 )
AMIA Annu Symp Proc - Voice-dictated versus typed-in clinician notes: linguistic properties and the potential implications on natural language processing. ( 0,749832313375976 )
AMIA Annu Symp Proc - Developing a section labeler for clinical documents. ( 0,74838471367337 )
J Am Med Inform Assoc - Comparison of a semi-automatic annotation tool and a natural language processing application for the generation of clinical statement entries. ( 0,747013192294703 )
AMIA Annu Symp Proc - Extracting patient demographics and personal medical information from online health forums. ( 0,746939899803772 )
BMC Med Inform Decis Mak - The freetext matching algorithm: a computer program to extract diagnoses and causes of death from unstructured text in electronic health records. ( 0,746258893894102 )
J. Med. Internet Res. - Evaluating a web-based clinical decision support system for language disorders screening in a nursery school. ( 0,744436910135739 )
AMIA Annu Symp Proc - It's about this and that: a description of anaphoric expressions in clinical text. ( 0,74357233962344 )
AMIA Annu Symp Proc - Extracting semantic lexicons from discharge summaries using machine learning and the C-Value method. ( 0,741181726768935 )
AMIA Annu Symp Proc - Sophia: A Expedient UMLS Concept Extraction Annotator. ( 0,740591920073717 )
AMIA Annu Symp Proc - Extracting temporal constraints from clinical research eligibility criteria using conditional random fields. ( 0,737512398370239 )
AMIA Annu Symp Proc - A Knowledge Intensive Approach to Mapping Clinical Narrative to LOINC. ( 0,73634062397158 )
J Biomed Inform - Evaluating measures of semantic similarity and relatedness to disambiguate terms in biomedical text. ( 0,736251784202355 )
J Biomed Inform - Semantator: semantic annotator for converting biomedical text to linked data. ( 0,733268458443002 )
J Biomed Inform - Ontology modularization to improve semantic medical image annotation. ( 0,730030204141608 )
AMIA Annu Symp Proc - Discovering peripheral arterial disease cases from radiology notes using natural language processing. ( 0,729484092762439 )
J Biomed Inform - The EU-ADR corpus: annotated drugs, diseases, targets, and their relationships. ( 0,726820050342347 )
J Biomed Inform - Approaches to verb subcategorization for biomedicine. ( 0,724943784955127 )
J Am Med Inform Assoc - Temporal reasoning over clinical text: the state of the art. ( 0,722673499994823 )
J Biomed Inform - Automatic recognition of disorders, findings, pharmaceuticals and body structures from clinical text: an annotation and machine learning study. ( 0,721527186880548 )