J Chem Inf Model - Real external predictivity of QSAR models: how to evaluate it? Comparison of different validation criteria and proposal of using the concordance correlation coefficient.


{ measur(2081) correl(1212) valu(896) }
{ model(2656) set(1616) predict(1553) }
{ method(1557) propos(1049) approach(1037) }
{ error(1145) method(1030) estim(1020) }
{ sampl(1606) size(1419) use(1276) }
{ state(1844) use(1261) util(961) }
{ studi(2440) review(1878) systemat(933) }
{ model(3404) distribut(989) bayesian(671) }
{ network(2748) neural(1063) input(814) }
{ model(2220) cell(1177) simul(1124) }
{ howev(809) still(633) remain(590) }
{ decis(3086) make(1611) patient(1517) }
{ data(1737) use(1416) pattern(1282) }
{ clinic(1479) use(1117) guidelin(835) }
{ research(1085) discuss(1038) issu(1018) }
{ model(3480) simul(1196) paramet(876) }
{ cost(1906) reduc(1198) effect(832) }
{ first(2504) two(1366) second(1323) }
{ result(1111) use(1088) new(759) }
{ surgeri(1148) surgic(1085) robot(1054) }
{ framework(1458) process(801) describ(734) }
{ problem(2511) optim(1539) algorithm(950) }
{ chang(1828) time(1643) increas(1301) }
{ general(901) number(790) one(736) }
{ perform(999) metric(946) measur(919) }
{ compound(1573) activ(1297) structur(1058) }
{ studi(1119) effect(1106) posit(819) }
{ use(2086) technolog(871) perceiv(783) }
{ health(1844) social(1437) communiti(874) }
{ high(1669) rate(1365) level(1280) }
{ use(976) code(926) identifi(902) }
{ activ(1452) weight(1219) physic(1104) }
{ can(774) often(719) complex(702) }
{ imag(1947) propos(1133) code(1026) }
{ inform(2794) health(2639) internet(1427) }
{ system(1976) rule(880) can(841) }
{ imag(1057) registr(996) error(939) }
{ bind(1733) structur(1185) ligand(1036) }
{ sequenc(1873) structur(1644) protein(1328) }
{ method(1219) similar(1157) match(930) }
{ featur(3375) classif(2383) classifi(1994) }
{ imag(2830) propos(1344) filter(1198) }
{ imag(2675) segment(2577) method(1081) }
{ patient(2315) diseas(1263) diabet(1191) }
{ take(945) account(800) differ(722) }
{ motion(1329) object(1292) video(1091) }
{ assess(1506) score(1403) qualiti(1306) }
{ treatment(1704) effect(941) patient(846) }
{ learn(2355) train(1041) set(1003) }
{ concept(1167) ontolog(924) domain(897) }
{ algorithm(1844) comput(1787) effici(935) }
{ extract(1171) text(1153) clinic(932) }
{ data(1714) softwar(1251) tool(1186) }
{ design(1359) user(1324) use(1319) }
{ control(1307) perform(991) simul(935) }
{ care(1570) inform(1187) nurs(1089) }
{ method(984) reconstruct(947) comput(926) }
{ search(2224) databas(1162) retriev(909) }
{ featur(1941) imag(1645) propos(1176) }
{ case(1353) use(1143) diagnosi(1136) }
{ data(3963) clinic(1234) research(1004) }
{ studi(1410) differ(1259) use(1210) }
{ risk(3053) factor(974) diseas(938) }
{ system(1050) medic(1026) inform(1018) }
{ import(1318) role(1303) understand(862) }
{ model(2341) predict(2261) use(1141) }
{ visual(1396) interact(850) tool(830) }
{ perform(1367) use(1326) method(1137) }
{ blood(1257) pressur(1144) flow(957) }
{ spatial(1525) area(1432) region(1030) }
{ record(1888) medic(1808) patient(1693) }
{ health(3367) inform(1360) care(1135) }
{ monitor(1329) mobil(1314) devic(1160) }
{ ehr(2073) health(1662) electron(1139) }
{ research(1218) medic(880) student(794) }
{ patient(2837) hospit(1953) medic(668) }
{ data(2317) use(1299) case(1017) }
{ age(1611) year(1155) adult(843) }
{ medic(1828) order(1363) alert(1069) }
{ signal(2180) analysi(812) frequenc(800) }
{ group(2977) signific(1463) compar(1072) }
{ gene(2352) biolog(1181) express(1162) }
{ data(3008) multipl(1320) sourc(1022) }
{ intervent(3218) particip(2042) group(1664) }
{ activ(1138) subject(705) human(624) }
{ time(1939) patient(1703) rate(768) }
{ patient(1821) servic(1111) care(1106) }
{ can(981) present(881) function(850) }
{ analysi(2126) use(1163) compon(1037) }
{ structur(1116) can(940) graph(676) }
{ cancer(2502) breast(956) screen(824) }
{ use(1733) differ(960) four(931) }
{ drug(1928) target(777) effect(648) }
{ implement(1333) system(1263) develop(1122) }
{ survey(1388) particip(1329) question(1065) }
{ estim(2440) model(1874) function(577) }
{ process(1125) use(805) approach(778) }
{ method(1969) cluster(1462) data(1082) }
{ method(2212) result(1239) propos(1039) }
{ detect(2391) sensit(1101) algorithm(908) }


The main utility of QSAR models is their ability to predict activities/properties for new chemicals, and this external prediction ability is evaluated by means of various validation criteria. As a measure for such evaluation the OECD guidelines have proposed the predictive squared correlation coefficient Q(2)(F1) (Shi et al.). However, other validation criteria have been proposed by other authors: the Golbraikh-Tropsha method, r(2)(m) (Roy), Q(2)(F2) (Schu?u?rmann et al.), Q(2)(F3) (Consonni et al.). In QSAR studies these measures are usually in accordance, though this is not always the case, thus doubts can arise when contradictory results are obtained. It is likely that none of the aforementioned criteria is the best in every situation, so a comparative study using simulated data sets is proposed here, using threshold values suggested by the proponents or those widely used in QSAR modeling. In addition, a different and simple external validation measure, the concordance correlation coefficient (CCC), is proposed and compared with other criteria. Huge data sets were used to study the general behavior of validation measures, and the concordance correlation coefficient was shown to be the most restrictive. On using simulated data sets of a more realistic size, it was found that CCC was broadly in agreement, about 96% of the time, with other validation measures in accepting models as predictive, and in almost all the examples it was the most precautionary. The proposed concordance correlation coefficient also works well on real data sets, where it seems to be more stable, and helps in making decisions when the validation measures are in conflict. Since it is conceptually simple, and given its stability and restrictiveness, we propose the concordance correlation coefficient as a complementary, or alternative, more prudent measure of a QSAR model to be externally predictive.

Resumo Limpo

main util qsar model abil predict activitiesproperti new chemic extern predict abil evalu mean various valid criteria measur evalu oecd guidelin propos predict squar correl coeffici qf shi et al howev valid criteria propos author golbraikhtropsha method rm roy qf schuurmann et al qf consonni et al qsar studi measur usual accord though alway case thus doubt can aris contradictori result obtain like none aforement criteria best everi situat compar studi use simul data set propos use threshold valu suggest propon wide use qsar model addit differ simpl extern valid measur concord correl coeffici ccc propos compar criteria huge data set use studi general behavior valid measur concord correl coeffici shown restrict use simul data set realist size found ccc broad agreement time valid measur accept model predict almost exampl precautionari propos concord correl coeffici also work well real data set seem stabl help make decis valid measur conflict sinc conceptu simpl given stabil restrict propos concord correl coeffici complementari altern prudent measur qsar model extern predict

Resumos Similares

Int J Health Geogr - Incorporating geographical factors with artificial neural networks to predict reference values of erythrocyte sedimentation rate. ( 0,800604388062683 )
J Chem Inf Model - Best of both worlds: combining pharma data and state of the art modeling technology to improve in Silico pKa prediction. ( 0,784092746229628 )
J Clin Monit Comput - Comparison of two different generations of NIRS devices and transducers in healthy volunteers and ICU patients. ( 0,688205960175181 )
BMC Med Inform Decis Mak - Concordance and predictive value of two adverse drug event data sets. ( 0,687458174490555 )
J Chem Inf Model - Real external predictivity of QSAR models. Part 2. New intercomparable thresholds for different validation criteria and the need for scatter plot inspection. ( 0,680586449641865 )
J Clin Monit Comput - Masseter muscle oxygen saturation is associated with central venous oxygen saturation in patients with severe sepsis. ( 0,678860725969247 )
Int J Health Geogr - Comparative analysis of remotely-sensed data products via ecological niche modeling of avian influenza case occurrences in Middle Eastern poultry. ( 0,668135664172584 )
Comput. Aided Surg. - Evaluation of a computational model to predict elbow range of motion. ( 0,666761155273486 )
J Chem Inf Model - Time-split cross-validation as a method for estimating the goodness of prospective prediction. ( 0,666160622007754 )
Med Decis Making - Prediction of health preference values from CD4 counts in individuals with HIV. ( 0,661427896938234 )
J Chem Inf Model - GRID-based three-dimensional pharmacophores II: PharmBench, a benchmark data set for evaluating pharmacophore elucidation methods. ( 0,660928541111923 )
AMIA Annu Symp Proc - Predicting the dengue incidence in Singapore using univariate time series models. ( 0,660501952178004 )
J Clin Monit Comput - Validation of indirect calorimetry for measurement of energy expenditure in healthy volunteers undergoing pressure controlled non-invasive ventilation support. ( 0,654323255172025 )
J Chem Inf Model - Calculation of aqueous solubility of crystalline un-ionized organic chemicals and drugs based on structural similarity and physicochemical descriptors. ( 0,654293404959583 )
J. Comput. Biol. - The complexity of the dirichlet model for multiple alignment data. ( 0,647112178786154 )
J Clin Monit Comput - Comparison of SNAP? II and BIS Vista indices during normothermic cardiopulmonary bypass under isoflurane anesthesia. ( 0,643398250069025 )
Comput. Biol. Med. - Gait variability and stability measures: minimum number of strides and within-session reliability. ( 0,637819065889767 )
J Chem Inf Model - Beyond the scope of Free-Wilson analysis: building interpretable QSAR models with machine learning algorithms. ( 0,630025291444815 )
Artif Intell Med - Training artificial neural networks directly on the concordance index for censored data using genetic algorithms. ( 0,62813233451485 )
Comput Math Methods Med - Long-term prediction of emergency department revenue and visitor volume using autoregressive integrated moving average model. ( 0,62714093641804 )
Comput Methods Programs Biomed - Kinetic modelling of haemodialysis removal of myoglobin in rhabdomyolysis patients. ( 0,624134569830649 )
J Clin Monit Comput - Comparison of ear and chest probes in transcutaneous carbon dioxide pressure measurements during general anesthesia in adults. ( 0,622226226917706 )
J Clin Monit Comput - Evaluation of the estimated continuous cardiac output monitoring system in adults and children undergoing kidney transplant surgery: a pilot study. ( 0,614601506837905 )
J Clin Monit Comput - Cardiac index measurements by transcutaneous Doppler ultrasound and transthoracic echocardiography in adult and pediatric emergency patients. ( 0,605485552406059 )
Comput. Biol. Med. - The dominant morphology of fractionated atrial electrograms has greater temporal stability in persistent as compared with paroxysmal atrial fibrillation. ( 0,605448237383862 )
J Clin Monit Comput - Peripheral tissue oximetry: comparing three commercial near-infrared spectroscopy oximeters on the forearm. ( 0,602418477419228 )
Comput Methods Programs Biomed - Improving reliability of live/dead cell counting through automated image mosaicing. ( 0,599698925178394 )
J Chem Inf Model - CSAR data set release 2012: ligands, affinities, complexes, and docking decoys. ( 0,598878279414137 )
AMIA Annu Symp Proc - Effect of data combination on predictive modeling: a study using gene expression data. ( 0,597977407295958 )
J Chem Inf Model - iLOGP: a simple, robust, and efficient description of n-octanol/water partition coefficient for drug design using the GB/SA approach. ( 0,595570266629354 )
Med Biol Eng Comput - Intra-protocol repeatability and inter-protocol agreement for the analysis of scapulo-humeral coordination. ( 0,595456655143728 )
J Chem Inf Model - Does rational selection of training and test sets improve the outcome of QSAR modeling? ( 0,593013976041555 )
BMC Med Inform Decis Mak - Statistical process control for data without inherent order. ( 0,588766818758034 )
Int J Med Robot - Intraoperative measurement of femoral antetorsion using the anterior cortical angle method: a novel use for smartphones. ( 0,587565744266192 )
J Chem Inf Model - Comparative studies on some metrics for external validation of QSPR models. ( 0,586364699835911 )
Comput. Aided Surg. - Non-invasive quantification of lower limb mechanical alignment in flexion. ( 0,585277207508874 )
J Chem Inf Model - Study of chromatographic retention of natural terpenoids by chemoinformatic tools. ( 0,584526953853398 )
J Clin Monit Comput - Non-invasive measurement of cardiac output in obese children and adolescents: comparison of electrical cardiometry and transthoracic Doppler echocardiography. ( 0,583945567929564 )
J Chem Inf Model - Pharmacophore assessment through 3-D QSAR: evaluation of the predictive ability on new derivatives by the application on a series of antitubercular agents. ( 0,58286261417369 )
Methods Inf Med - Personal adaptive method to assess mental tension during daily life using heart rate variability. ( 0,58143809081103 )
Comput Methods Programs Biomed - Interstitial insulin kinetic parameters for a 2-compartment insulin model with saturable clearance. ( 0,581353639340538 )
J Chem Inf Model - Combined 3D-QSAR, molecular docking, and molecular dynamics study on piperazinyl-glutamate-pyridines/pyrimidines as potent P2Y12 antagonists for inhibition of platelet aggregation. ( 0,581147839205699 )
J Chem Inf Model - Molecular dynamics simulation and binding energy calculation for estimation of oligonucleotide duplex thermostability in RNA-based therapeutics. ( 0,57869564653003 )
Comput. Biol. Med. - Inter- and intra-observer variability analysis of completely automated cIMT measurement software (AtheroEdge?) and its benchmarking against commercial ultrasound scanner and expert Readers. ( 0,577709996614789 )
J Am Med Inform Assoc - Choosing blindly but wisely: differentially private solicitation of DNA datasets for disease marker discovery. ( 0,577686813134172 )
Methods Inf Med - Measuring inter-observer agreement in contour delineation of medical imaging in a dummy run using Fleiss' kappa. ( 0,577068769806772 )
AMIA Annu Symp Proc - Identifying Deviations from Usual Medical Care using a Statistical Approach. ( 0,575717240983756 )
Med Biol Eng Comput - Development of a comprehensive musculoskeletal model of the shoulder and elbow. ( 0,570421753971655 )
Artif Intell Med - Fuzzy model identification of dengue epidemic in Colombia based on multiresolution analysis. ( 0,566730017967705 )
J Med Syst - Automatic quantification of spinal curvature in scoliotic radiograph using image processing. ( 0,563751428420323 )
J Chem Inf Model - Predicting pK(a) values of substituted phenols from atomic charges: comparison of different quantum mechanical methods and charge distribution schemes. ( 0,561189965688489 )
Comput. Biol. Med. - Approach for streamlining measurement of complex physiological phenotypes of upper airway collapsibility. ( 0,558171851545542 )
Int J Health Geogr - Comparing the accuracy of two secondary food environment data sources in the UK across socio-economic and urban/rural divides. ( 0,557579843272579 )
AMIA Annu Symp Proc - Advanced proficiency EHR training: effect on physicians' EHR efficiency, EHR satisfaction and job satisfaction. ( 0,554858675338154 )
J Chem Inf Model - Three useful dimensions for domain applicability in QSAR models using random forest. ( 0,552963867619414 )
Res Synth Methods - Practicalities of using a modified version of the Cochrane Collaboration risk of bias tool for randomised and non-randomised study designs applied in a health technology assessment setting. ( 0,551251288279203 )
Comput Math Methods Med - Paraxial ocular measurements and entries in spectral and modal matrices: analogy and application. ( 0,551205569427127 )
J Chem Inf Model - Rank order entropy: why one metric is not enough. ( 0,550850868137655 )
Neural Comput - Distinguishing the causes of firing with the membrane potential slope. ( 0,550174205546645 )
AMIA Annu Symp Proc - Motivating the additional use of external validity: examining transportability in a model of glioblastoma multiforme. ( 0,546737558025794 )
J Clin Monit Comput - Reliability of the volatile agent consumption display in the Draeger Primus? anesthesia machine. ( 0,54640621991413 )
Perspect Health Inf Manag - Projected impact of the ICD-10-CM/PCS conversion on longitudinal data and the Joint Commission Core Measures. ( 0,545712029125219 )
AMIA Annu Symp Proc - Metrics for assessing the quality of value sets in clinical quality measures. ( 0,545307984324417 )
Comput Methods Programs Biomed - Ultrasound IMT measurement on a multi-ethnic and multi-institutional database: our review and experience using four fully automated and one semi-automated methods. ( 0,544900107577823 )
J Clin Monit Comput - The ability of the Vigileo-FloTrac system to measure cardiac output and track cardiac output changes during one-lung ventilation. ( 0,544884971493753 )
Brief. Bioinformatics - A comparative study of statistical methods used to identify dependencies between gene expression signals. ( 0,542149458992504 )
Int J Comput Assist Radiol Surg - Correlation between external and internal respiratory motion: a validation study. ( 0,541855572917968 )
Comput Methods Programs Biomed - Extracting more information from EEG recordings for a better description of sleep. ( 0,541436492869387 )
BMC Med Inform Decis Mak - Influence of data quality on computed Dutch hospital quality indicators: a case study in colorectal cancer surgery. ( 0,538218416858378 )
Int J Comput Assist Radiol Surg - A method for segmentation of dental implants and crestal bone. ( 0,538107480188172 )
J Clin Monit Comput - Evaluation of a computer program for non-invasive determination of pulmonary shunt and ventilation-perfusion mismatch. ( 0,536138744916445 )
Med Biol Eng Comput - Dynamic cerebral autoregulation: different signal processing methods without influence on results and reproducibility. ( 0,534522761927867 )
J Biomed Inform - MysiRNA: improving siRNA efficacy prediction using a machine-learning model combining multi-tools and whole stacking energy (G). ( 0,533623944625715 )
Artif Intell Med - Support vector methods for survival analysis: a comparison between ranking and regression approaches. ( 0,532329403797394 )
Brief. Bioinformatics - Data construction for phosphorylation site prediction. ( 0,531683043248909 )
J Chem Inf Model - Statistical analysis and compound selection of combinatorial libraries for soluble epoxide hydrolase. ( 0,531319540924329 )
Med Decis Making - Standard error of measurement of 5 health utility indexes across the range of health for use in estimating reliability and responsiveness. ( 0,531227528026993 )
Comput. Biol. Med. - A correction method using a support vector machine to minimize hematocrit interference in blood glucose measurements. ( 0,530916389828226 )
Int J Comput Assist Radiol Surg - New method for internal anal sphincter measurements: feasibility study. ( 0,529370351960383 )
J Chem Inf Model - RS-Predictor models augmented with SMARTCyp reactivities: robust metabolic regioselectivity predictions for nine CYP isozymes. ( 0,528377712023475 )
Neural Comput - Molecular diffusion model of neurotransmitter homeostasis around synapses supporting gradients. ( 0,527139190950477 )
Neural Comput - Impact of correlated neural activity on decision-making performance. ( 0,526502272718526 )
J. Med. Internet Res. - Estimation of physical activity levels using cell phone questionnaires: a comparison with accelerometry for evaluation of between-subject and within-subject variations. ( 0,525931712976574 )
J Chem Inf Model - Acetylcholinesterase inhibitors: structure based design, synthesis, pharmacophore modeling, and virtual screening. ( 0,522811063599191 )
Comput. Biol. Med. - An effective measure for assessing the quality of biclusters. ( 0,522743818567657 )
J Telemed Telecare - Comparison of two motion sensors for use in cardiac telerehabilitation. ( 0,52185475940803 )
Med Decis Making - Developing a tuberculosis transmission model that accounts for changes in population health. ( 0,521755312113507 )
Comput. Biol. Med. - Development and evaluation of an algorithm for computer analysis of maternal heart rate during labor. ( 0,521364372822458 )
Comput Math Methods Med - Prediction of breeding values for dairy cattle using artificial neural networks and neuro-fuzzy systems. ( 0,521207614350355 )
Res Synth Methods - Inaccuracy of regression results in replacing bivariate correlations. ( 0,513394794288507 )
Comput Methods Programs Biomed - Indirect measurement of the vascular endothelial glycocalyx layer thickness in human submucosal capillaries with a plug-in for ImageJ. ( 0,513270904911358 )
J Clin Monit Comput - The effect of desflurane versus propofol on regional cerebral oxygenation in the sitting position for shoulder arthroscopy. ( 0,512585692838214 )
J Clin Monit Comput - Validation of continuous and noninvasive hemoglobin monitoring by pulse CO-oximetry in Japanese surgical patients. ( 0,51169762767587 )
AMIA Annu Symp Proc - Selecting cases for whom additional tests can improve prognostication. ( 0,511089517936537 )
BMC Med Inform Decis Mak - Regression tree construction by bootstrap: model search for DRG-systems applied to Austrian health-data. ( 0,510779587078296 )
J Chem Inf Model - A comparison of different QSAR approaches to modeling CYP450 1A2 inhibition. ( 0,510259752080478 )
Comput Math Methods Med - Statistical evaluation of a fully automated mammographic breast density algorithm. ( 0,509536561222879 )
J Chem Inf Model - Applicability domains for classification problems: Benchmarking of distance to models for Ames mutagenicity set. ( 0,508378250592302 )
Neural Comput - Input statistics and Hebbian cross-talk effects. ( 0,502981292197175 )
Comput Methods Programs Biomed - Estimation of the concordance correlation coefficient for repeated measures using SAS and R. ( 0,502698578802945 )