AMIA Annu Symp Proc - Testing the calibration of classification models from first principles.

Tópicos

{ model(3404) distribut(989) bayesian(671) }
{ can(981) present(881) function(850) }
{ learn(2355) train(1041) set(1003) }
{ result(1111) use(1088) new(759) }
{ assess(1506) score(1403) qualiti(1306) }
{ surgeri(1148) surgic(1085) robot(1054) }
{ group(2977) signific(1463) compar(1072) }
{ data(3008) multipl(1320) sourc(1022) }
{ imag(1057) registr(996) error(939) }
{ network(2748) neural(1063) input(814) }
{ howev(809) still(633) remain(590) }
{ can(774) often(719) complex(702) }
{ imag(1947) propos(1133) code(1026) }
{ system(1976) rule(880) can(841) }
{ sequenc(1873) structur(1644) protein(1328) }
{ featur(3375) classif(2383) classifi(1994) }
{ take(945) account(800) differ(722) }
{ framework(1458) process(801) describ(734) }
{ error(1145) method(1030) estim(1020) }
{ featur(1941) imag(1645) propos(1176) }
{ case(1353) use(1143) diagnosi(1136) }
{ spatial(1525) area(1432) region(1030) }
{ first(2504) two(1366) second(1323) }
{ intervent(3218) particip(2042) group(1664) }
{ drug(1928) target(777) effect(648) }
{ process(1125) use(805) approach(778) }
{ method(2212) result(1239) propos(1039) }
{ detect(2391) sensit(1101) algorithm(908) }
{ data(1737) use(1416) pattern(1282) }
{ inform(2794) health(2639) internet(1427) }
{ measur(2081) correl(1212) valu(896) }
{ bind(1733) structur(1185) ligand(1036) }
{ method(1219) similar(1157) match(930) }
{ imag(2830) propos(1344) filter(1198) }
{ imag(2675) segment(2577) method(1081) }
{ patient(2315) diseas(1263) diabet(1191) }
{ studi(2440) review(1878) systemat(933) }
{ motion(1329) object(1292) video(1091) }
{ treatment(1704) effect(941) patient(846) }
{ problem(2511) optim(1539) algorithm(950) }
{ chang(1828) time(1643) increas(1301) }
{ concept(1167) ontolog(924) domain(897) }
{ clinic(1479) use(1117) guidelin(835) }
{ algorithm(1844) comput(1787) effici(935) }
{ extract(1171) text(1153) clinic(932) }
{ method(1557) propos(1049) approach(1037) }
{ data(1714) softwar(1251) tool(1186) }
{ design(1359) user(1324) use(1319) }
{ control(1307) perform(991) simul(935) }
{ model(2220) cell(1177) simul(1124) }
{ care(1570) inform(1187) nurs(1089) }
{ general(901) number(790) one(736) }
{ method(984) reconstruct(947) comput(926) }
{ search(2224) databas(1162) retriev(909) }
{ data(3963) clinic(1234) research(1004) }
{ studi(1410) differ(1259) use(1210) }
{ risk(3053) factor(974) diseas(938) }
{ perform(999) metric(946) measur(919) }
{ research(1085) discuss(1038) issu(1018) }
{ system(1050) medic(1026) inform(1018) }
{ import(1318) role(1303) understand(862) }
{ model(2341) predict(2261) use(1141) }
{ visual(1396) interact(850) tool(830) }
{ compound(1573) activ(1297) structur(1058) }
{ perform(1367) use(1326) method(1137) }
{ studi(1119) effect(1106) posit(819) }
{ blood(1257) pressur(1144) flow(957) }
{ record(1888) medic(1808) patient(1693) }
{ health(3367) inform(1360) care(1135) }
{ model(3480) simul(1196) paramet(876) }
{ monitor(1329) mobil(1314) devic(1160) }
{ ehr(2073) health(1662) electron(1139) }
{ state(1844) use(1261) util(961) }
{ research(1218) medic(880) student(794) }
{ patient(2837) hospit(1953) medic(668) }
{ model(2656) set(1616) predict(1553) }
{ data(2317) use(1299) case(1017) }
{ age(1611) year(1155) adult(843) }
{ medic(1828) order(1363) alert(1069) }
{ signal(2180) analysi(812) frequenc(800) }
{ cost(1906) reduc(1198) effect(832) }
{ sampl(1606) size(1419) use(1276) }
{ gene(2352) biolog(1181) express(1162) }
{ activ(1138) subject(705) human(624) }
{ time(1939) patient(1703) rate(768) }
{ patient(1821) servic(1111) care(1106) }
{ use(2086) technolog(871) perceiv(783) }
{ analysi(2126) use(1163) compon(1037) }
{ health(1844) social(1437) communiti(874) }
{ structur(1116) can(940) graph(676) }
{ high(1669) rate(1365) level(1280) }
{ cancer(2502) breast(956) screen(824) }
{ use(976) code(926) identifi(902) }
{ use(1733) differ(960) four(931) }
{ implement(1333) system(1263) develop(1122) }
{ survey(1388) particip(1329) question(1065) }
{ estim(2440) model(1874) function(577) }
{ decis(3086) make(1611) patient(1517) }
{ activ(1452) weight(1219) physic(1104) }
{ method(1969) cluster(1462) data(1082) }

Resumo

The accurate assessment of the calibration of classification models is severely limited by the fact that there is no easily available gold standard against which to compare a model's outputs. The usual procedures group expected and observed probabilities, and then perform a (2) goodness-of-fit test. We propose an entirely new approach to calibration testing that can be derived directly from the first principles of statistical hypothesis testing. The null hypothesis is that the model outputs are correct, i.e., that they are good estimates of the true unknown class membership probabilities. Our test calculates a p-value by checking how (im)probable the observed class labels are under the null hypothesis. We demonstrate by experiments that our proposed test performs comparable to, and sometimes even better than, the Hosmer-Lemeshow goodness-of-fit test, the de facto standard in calibration assessment.

Resumo Limpo

accur assess calibr classif model sever limit fact easili avail gold standard compar model output usual procedur group expect observ probabl perform goodnessoffit test propos entir new approach calibr test can deriv direct first principl statist hypothesi test null hypothesi model output correct ie good estim true unknown class membership probabl test calcul pvalu check improb observ class label null hypothesi demonstr experi propos test perform compar sometim even better hosmerlemeshow goodnessoffit test de facto standard calibr assess

Resumos Similares

IEEE Trans Pattern Anal Mach Intell - Modeling Natural Images Using Gated MRFs. ( 0,833945543259809 )
Spat Spatiotemporal Epidemiol - Goodness-of-fit measures for individual-level models of infectious disease in a Bayesian framework. ( 0,832466930099919 )
J. Comput. Biol. - A spatial haplotype copying model with applications to genotype imputation. ( 0,827467694522244 )
IEEE Trans Pattern Anal Mach Intell - Are Gibbs-Type Priors the Most Natural Generalization of the Dirichlet Process? ( 0,82061903614522 )
Neural Comput - A semiparametric Bayesian model for detecting synchrony among multiple neurons. ( 0,819937930490374 )
IEEE Trans Image Process - A Bayesian framework for image segmentation with spatially varying mixtures. ( 0,810927334091709 )
Lifetime Data Anal - Bayesian nonparametric models for ranked set sampling. ( 0,797923041816562 )
IEEE Trans Image Process - Bayesian robust principal component analysis. ( 0,784982616813095 )
Res Synth Methods - A Bayesian nonparametric meta-analysis model. ( 0,782880962726144 )
Med Decis Making - Calibration of complex models through Bayesian evidence synthesis: a demonstration and tutorial. ( 0,773798233774891 )
Comput Methods Programs Biomed - Identification of an integrated mathematical model of standard oral glucose tolerance test for characterization of insulin potentiation in health. ( 0,770850619710724 )
Comput Math Methods Med - Inference for ecological dynamical systems: a case study of two endemic diseases. ( 0,764814353624698 )
Comput Math Methods Med - Bayesian inference of the Weibull model based on interval-censored survival data. ( 0,755260254429222 )
Artif Intell Med - On the interplay of machine learning and background knowledge in image interpretation by Bayesian networks. ( 0,754332699728665 )
IEEE Trans Neural Netw Learn Syst - Variational Bayesian Inference Algorithms for Infinite Relational Model of Network Data. ( 0,75375073921694 )
IEEE Trans Image Process - Variational Bayesian method for Retinex. ( 0,752952488436876 )
J. Comput. Biol. - On the inference of dirichlet mixture priors for protein sequence comparison. ( 0,751666633595312 )
Comput Math Methods Med - An empirical Bayes optimal discovery procedure based on semiparametric hierarchical mixture models. ( 0,749112970685049 )
Lifetime Data Anal - Bayesian semiparametric modeling for stochastic precedence, with applications in epidemiology and survival analysis. ( 0,745223883370315 )
J. Comput. Biol. - Computational methods for a class of network models. ( 0,744706976894228 )
Neural Comput - Bayesian sparse partial least squares. ( 0,739475428779693 )
IEEE Trans Image Process - Computationally tractable stochastic image modeling based on symmetric Markov mesh random fields. ( 0,739394067359234 )
Lifetime Data Anal - Bayesian local influence for survival models. ( 0,738887746504652 )
IEEE Trans Image Process - Blind image quality assessment: a natural scene statistics approach in the DCT domain. ( 0,73848288070434 )
Res Synth Methods - Random-effects meta-analysis of time-to-event data using the expectation-maximisation algorithm and shrinkage estimators. ( 0,734567993173729 )
Neural Comput - Multiple tests based on a gaussian approximation of the unitary events method with delayed coincidence count. ( 0,734127289592273 )
Spat Spatiotemporal Epidemiol - A Bayesian space-time model for discrete spread processes on a lattice. ( 0,73385309667974 )
Med Decis Making - Bayesian calibration of a natural history model with application to a population model for colorectal cancer. ( 0,731464393598653 )
Res Synth Methods - Critical interpretation of Cochran's Q test depends on power and prior assumptions about heterogeneity. ( 0,727690044952189 )
Neural Comput - Determination and the no-free-lunch paradox. ( 0,726943423822032 )
IEEE Trans Image Process - Probabilistic image modeling with an extended chain graph for human activity recognition and image segmentation. ( 0,723793534194048 )
Comput Math Methods Med - Bayesian hierarchical modeling for categorical longitudinal data from sedation measurements. ( 0,722544420468546 )
Comput Methods Programs Biomed - The exponentiated exponential mixture and non-mixture cure rate model in the presence of covariates. ( 0,721502930556125 )
Neural Comput - Efficient Markov chain Monte Carlo methods for decoding neural spike trains. ( 0,719872271870092 )
IEEE Trans Pattern Anal Mach Intell - Negative Binomial Process Count and Mixture Modeling. ( 0,715082381272479 )
Med Biol Eng Comput - A poisson process model for hip fracture risk. ( 0,711166773241084 )
Neural Comput - Learning coefficient of generalization error in Bayesian estimation and vandermonde matrix-type singularity. ( 0,709702239192583 )
Med Decis Making - Assessing uncertainties surrounding combined endpoints for use in economic models. ( 0,70890768647579 )
Res Synth Methods - Bayesian model selection for meta-analysis of diagnostic test accuracy data: Application to Ddimer for deep vein thrombosis. ( 0,708386969986943 )
Med Decis Making - Not simply more of the same: distinguishing between patient heterogeneity and parameter uncertainty. ( 0,706290907440799 )
Spat Spatiotemporal Epidemiol - Bayesian hierarchical modeling of the dynamics of spatio-temporal influenza season outbreaks. ( 0,705848956091291 )
Comput. Biol. Med. - Expectation-maximization technique for fibro-glandular discs detection in mammography images. ( 0,701759629123299 )
Spat Spatiotemporal Epidemiol - Inference from ecological models: estimating the relative risk of stroke from air pollution exposure using small area data. ( 0,701431478580281 )
J. Comput. Biol. - The 5'-3' distance of RNA secondary structures. ( 0,700981032256179 )
J Biomed Inform - Link-topic model for biomedical abbreviation disambiguation. ( 0,699505638777063 )
Neural Comput - Universal approximation depth and errors of narrow belief networks with discrete units. ( 0,698754816683837 )
IEEE Trans Image Process - Generative Bayesian image super resolution with natural image prior. ( 0,694820745661352 )
IEEE Trans Neural Netw Learn Syst - Generalized multiple kernel learning with data-dependent priors. ( 0,690951129223176 )
IEEE Trans Image Process - Blind separation of time/position varying mixtures. ( 0,690940298427948 )
IEEE Trans Image Process - A study of multiplicative watermark detection in the contourlet domain using alpha-stable distributions. ( 0,689796336467425 )
J Biomed Inform - Learning patient-specific predictive models from clinical data. ( 0,688884979090142 )
Med Decis Making - Accounting for methodological, structural, and parameter uncertainty in decision-analytic models: a practical guide. ( 0,68713118147069 )
Neural Comput - Attention as reward-driven optimization of sensory processing. ( 0,685520894527472 )
IEEE Trans Neural Netw Learn Syst - Incorporating Wind Power Forecast Uncertainties Into Stochastic Unit Commitment Using Neural Network-Based Prediction Intervals. ( 0,680105608675796 )
Spat Spatiotemporal Epidemiol - The detection of spatially localised outbreaks in campylobacteriosis notification data. ( 0,67998109678826 )
Comput Methods Programs Biomed - NIMROD: a program for inference via a normal approximation of the posterior in models with random effects based on ordinary differential equations. ( 0,675116235548329 )
Comput Methods Programs Biomed - Multivariate Bayesian modeling of known and unknown causes of events--an application to biosurveillance. ( 0,673969358095133 )
Res Synth Methods - A basic introduction to fixed-effect and random-effects models for meta-analysis. ( 0,673283370451454 )
Comput Methods Programs Biomed - Life prediction of different commercial dental implants as influence by uncertainties in their fatigue material properties and loading conditions. ( 0,666491739632209 )
Artif Intell Med - Impact of precision of Bayesian network parameters on accuracy of medical diagnostic systems. ( 0,665700337244018 )
Artif Intell Med - Improving Bayesian credibility intervals for classifier error rates using maximum entropy empirical priors. ( 0,661114506490646 )
J. Comput. Biol. - Expectation-maximization algorithm for determining natural selection of Y-linked genes through two-sex branching processes. ( 0,657583609750816 )
Neural Comput - Efficient sensory encoding and Bayesian inference with heterogeneous neural populations. ( 0,654660953822468 )
Comput. Biol. Med. - Vector autoregression, structural equation modeling, and their synthesis in neuroimaging data analysis. ( 0,652229625249247 )
IEEE Trans Image Process - Shape-based normalized cuts using spectral relaxation for biomedical segmentation. ( 0,651587734770622 )
Spat Spatiotemporal Epidemiol - A space-time point process model for analyzing and predicting case patterns of diarrheal disease in northwestern Ecuador. ( 0,650439193683402 )
Neural Comput - The neural representation of time: an information-theoretic perspective. ( 0,650013709312961 )
IEEE Trans Pattern Anal Mach Intell - Bayesian Nonparametric Methods for Partially-Observable Reinforcement Learning. ( 0,648106286173993 )
IEEE Trans Image Process - Bayesian inference of models and hyperparameters for robust optical-flow estimation. ( 0,64198679382151 )
Comput Math Methods Med - A generalized gamma mixture model for ultrasonic tissue characterization. ( 0,640417745544168 )
IEEE Trans Image Process - SAR-based terrain classification using weakly supervised hierarchical Markov aspect models. ( 0,640236861283856 )
Spat Spatiotemporal Epidemiol - Mapping gender variation in the spatial pattern of alcohol-related mortality: a Bayesian analysis using data from South Yorkshire, United Kingdom. ( 0,639929958826668 )
IEEE Trans Pattern Anal Mach Intell - Temporal Analysis of Motif Mixtures using Dirichlet Processes. ( 0,639634533821819 )
Med Decis Making - Identifying best-fitting inputs in health-economic model calibration: a Pareto frontier approach. ( 0,637419045288586 )
BMC Med Inform Decis Mak - A simulation model of colorectal cancer surveillance and recurrence. ( 0,634776980247372 )
Brief. Bioinformatics - CaliBayes and BASIS: integrated tools for the calibration, simulation and storage of biological simulation models. ( 0,634212446716851 )
Comput. Biol. Med. - Parameterization of the distribution of white and grey matter in MRI using the a-stable distribution. ( 0,634164330139496 )
Int J Health Geogr - Gumbel based p-value approximations for spatial scan statistics. ( 0,631565285823804 )
Neural Comput - Bayesian community detection. ( 0,629554975700705 )
IEEE Trans Pattern Anal Mach Intell - Articulated Human Detection with Flexible Mixtures-of-Parts. ( 0,629278210590511 )
Comput Methods Programs Biomed - A Bayesian multilevel model for fMRI data analysis. ( 0,629099324759295 )
IEEE Trans Image Process - Studentized dynamical system for robust object tracking. ( 0,628301598690097 )
IEEE Trans Image Process - Cross-camera knowledge transfer for multiview people counting. ( 0,62671618875385 )
IEEE Trans Image Process - Posterior-mean super-resolution with a causal Gaussian Markov random field prior. ( 0,625763549721682 )
IEEE Trans Pattern Anal Mach Intell - Causal Inference on Discrete Data using Additive Noise Models. ( 0,625542054405707 )
AMIA Annu Symp Proc - Learning to predict post-hospitalization VTE risk from EHR data. ( 0,625110494150041 )
IEEE Trans Image Process - Bayesian estimation of linear mixtures using the normal compositional model. Application to hyperspectral imagery. ( 0,6233984218917 )
J. Comput. Biol. - Characterizing the empirical distribution of prokaryotic genome n-mers in the presence of nullomers. ( 0,622973317745248 )
Med Decis Making - Linear regression metamodeling as a tool to summarize and present simulation model results. ( 0,622961231089549 )
IEEE Trans Neural Netw Learn Syst - Robust Novelty Detection via Worst Case CVaR Minimization. ( 0,620166093270646 )
IEEE Trans Image Process - A Bayesian hierarchical factorization model for vector fields. ( 0,618695188758298 )
Med Decis Making - Estimating multiparameter partial expected value of perfect information from a probabilistic sensitivity analysis sample: a nonparametric regression approach. ( 0,616657149529068 )
IEEE Trans Vis Comput Graph - Anisotropic Sampling of Planar and Two-Manifold Domains for Texture Generation and Glyph Distribution. ( 0,616618678957557 )
Artif Intell Med - Clinical time series prediction: Toward a hierarchical dynamical system framework. ( 0,616057058895978 )
Lifetime Data Anal - Diagnostic tools for bivariate accelerated life regression models. ( 0,615774061116227 )
J Biomed Inform - Error-correction learning for artificial neural networks using the Bayesian paradigm. Application to automated medical diagnosis. ( 0,61562200963151 )
Med Decis Making - Exploring model uncertainty in economic evaluation of health interventions: the example of rotavirus vaccination in Vietnam. ( 0,614584331982953 )
Med Decis Making - Comparing Bayesian and frequentist approaches for multiple outcome mixed treatment comparisons. ( 0,612906136493615 )
IEEE Trans Pattern Anal Mach Intell - Gaussian Process-Mixture Conditional Heteroscedasticity. ( 0,612733267744868 )
IEEE Trans Image Process - Bayesian nonparametric dictionary learning for compressed sensing MRI. ( 0,611680334476532 )