J. Med. Internet Res. - Crowdsourcing a normative natural language dataset: a comparison of Amazon Mechanical Turk and in-lab data collection.

Tópicos

{ survey(1388) particip(1329) question(1065) }
{ extract(1171) text(1153) clinic(932) }
{ age(1611) year(1155) adult(843) }
{ data(2317) use(1299) case(1017) }
{ time(1939) patient(1703) rate(768) }
{ health(1844) social(1437) communiti(874) }
{ perform(999) metric(946) measur(919) }
{ data(3008) multipl(1320) sourc(1022) }
{ learn(2355) train(1041) set(1003) }
{ general(901) number(790) one(736) }
{ take(945) account(800) differ(722) }
{ error(1145) method(1030) estim(1020) }
{ data(1737) use(1416) pattern(1282) }
{ studi(2440) review(1878) systemat(933) }
{ data(1714) softwar(1251) tool(1186) }
{ howev(809) still(633) remain(590) }
{ spatial(1525) area(1432) region(1030) }
{ research(1218) medic(880) student(794) }
{ cost(1906) reduc(1198) effect(832) }
{ sampl(1606) size(1419) use(1276) }
{ use(976) code(926) identifi(902) }
{ imag(1947) propos(1133) code(1026) }
{ inform(2794) health(2639) internet(1427) }
{ measur(2081) correl(1212) valu(896) }
{ imag(2830) propos(1344) filter(1198) }
{ clinic(1479) use(1117) guidelin(835) }
{ algorithm(1844) comput(1787) effici(935) }
{ compound(1573) activ(1297) structur(1058) }
{ studi(1119) effect(1106) posit(819) }
{ group(2977) signific(1463) compar(1072) }
{ intervent(3218) particip(2042) group(1664) }
{ activ(1138) subject(705) human(624) }
{ can(981) present(881) function(850) }
{ high(1669) rate(1365) level(1280) }
{ detect(2391) sensit(1101) algorithm(908) }
{ model(3404) distribut(989) bayesian(671) }
{ can(774) often(719) complex(702) }
{ system(1976) rule(880) can(841) }
{ imag(1057) registr(996) error(939) }
{ bind(1733) structur(1185) ligand(1036) }
{ sequenc(1873) structur(1644) protein(1328) }
{ method(1219) similar(1157) match(930) }
{ featur(3375) classif(2383) classifi(1994) }
{ network(2748) neural(1063) input(814) }
{ imag(2675) segment(2577) method(1081) }
{ patient(2315) diseas(1263) diabet(1191) }
{ motion(1329) object(1292) video(1091) }
{ assess(1506) score(1403) qualiti(1306) }
{ treatment(1704) effect(941) patient(846) }
{ surgeri(1148) surgic(1085) robot(1054) }
{ framework(1458) process(801) describ(734) }
{ problem(2511) optim(1539) algorithm(950) }
{ chang(1828) time(1643) increas(1301) }
{ concept(1167) ontolog(924) domain(897) }
{ method(1557) propos(1049) approach(1037) }
{ design(1359) user(1324) use(1319) }
{ control(1307) perform(991) simul(935) }
{ model(2220) cell(1177) simul(1124) }
{ care(1570) inform(1187) nurs(1089) }
{ method(984) reconstruct(947) comput(926) }
{ search(2224) databas(1162) retriev(909) }
{ featur(1941) imag(1645) propos(1176) }
{ case(1353) use(1143) diagnosi(1136) }
{ data(3963) clinic(1234) research(1004) }
{ studi(1410) differ(1259) use(1210) }
{ risk(3053) factor(974) diseas(938) }
{ research(1085) discuss(1038) issu(1018) }
{ system(1050) medic(1026) inform(1018) }
{ import(1318) role(1303) understand(862) }
{ model(2341) predict(2261) use(1141) }
{ visual(1396) interact(850) tool(830) }
{ perform(1367) use(1326) method(1137) }
{ blood(1257) pressur(1144) flow(957) }
{ record(1888) medic(1808) patient(1693) }
{ health(3367) inform(1360) care(1135) }
{ model(3480) simul(1196) paramet(876) }
{ monitor(1329) mobil(1314) devic(1160) }
{ ehr(2073) health(1662) electron(1139) }
{ state(1844) use(1261) util(961) }
{ patient(2837) hospit(1953) medic(668) }
{ model(2656) set(1616) predict(1553) }
{ medic(1828) order(1363) alert(1069) }
{ signal(2180) analysi(812) frequenc(800) }
{ gene(2352) biolog(1181) express(1162) }
{ first(2504) two(1366) second(1323) }
{ patient(1821) servic(1111) care(1106) }
{ use(2086) technolog(871) perceiv(783) }
{ analysi(2126) use(1163) compon(1037) }
{ structur(1116) can(940) graph(676) }
{ cancer(2502) breast(956) screen(824) }
{ use(1733) differ(960) four(931) }
{ drug(1928) target(777) effect(648) }
{ result(1111) use(1088) new(759) }
{ implement(1333) system(1263) develop(1122) }
{ estim(2440) model(1874) function(577) }
{ decis(3086) make(1611) patient(1517) }
{ process(1125) use(805) approach(778) }
{ activ(1452) weight(1219) physic(1104) }
{ method(1969) cluster(1462) data(1082) }
{ method(2212) result(1239) propos(1039) }

Resumo

CKGROUND: Crowdsourcing has become a valuable method for collecting medical research data. This approach, recruiting through open calls on the Web, is particularly useful for assembling large normative datasets. However, it is not known how natural language datasets collected over the Web differ from those collected under controlled laboratory conditions.OBJECTIVE: To compare the natural language responses obtained from a crowdsourced sample of participants with responses collected in a conventional laboratory setting from participants recruited according to specific age and gender criteria.METHODS: We collected natural language descriptions of 200 half-minute movie clips, from Amazon Mechanical Turk workers (crowdsourced) and 60 participants recruited from the community (lab-sourced). Crowdsourced participants responded to as many clips as they wanted and typed their responses, whereas lab-sourced participants gave spoken responses to 40 clips, and their responses were transcribed. The content of the responses was evaluated using a take-one-out procedure, which compared responses to other responses to the same clip and to other clips, with a comparison of the average number of shared words.RESULTS: In contrast to the 13 months of recruiting that was required to collect normative data from 60 lab-sourced participants (with specific demographic characteristics), only 34 days were needed to collect normative data from 99 crowdsourced participants (contributing a median of 22 responses). The majority of crowdsourced workers were female, and the median age was 35 years, lower than the lab-sourced median of 62 years but similar to the median age of the US population. The responses contributed by the crowdsourced participants were longer on average, that is, 33 words compared to 28 words (P<.001), and they used a less varied vocabulary. However, there was strong similarity in the words used to describe a particular clip between the two datasets, as a cross-dataset count of shared words showed (P<.001). Within both datasets, responses contained substantial relevant content, with more words in common with responses to the same clip than to other clips (P<.001). There was evidence that responses from female and older crowdsourced participants had more shared words (P=.004 and .01 respectively), whereas younger participants had higher numbers of shared words in the lab-sourced population (P=.01).CONCLUSIONS: Crowdsourcing is an effective approach to quickly and economically collect a large reliable dataset of normative natural language responses.

Resumo Limpo

ckground crowdsourc becom valuabl method collect medic research data approach recruit open call web particular use assembl larg normat dataset howev known natur languag dataset collect web differ collect control laboratori conditionsobject compar natur languag respons obtain crowdsourc sampl particip respons collect convent laboratori set particip recruit accord specif age gender criteriamethod collect natur languag descript halfminut movi clip amazon mechan turk worker crowdsourc particip recruit communiti labsourc crowdsourc particip respond mani clip want type respons wherea labsourc particip gave spoken respons clip respons transcrib content respons evalu use takeoneout procedur compar respons respons clip clip comparison averag number share wordsresult contrast month recruit requir collect normat data labsourc particip specif demograph characterist day need collect normat data crowdsourc particip contribut median respons major crowdsourc worker femal median age year lower labsourc median year similar median age us popul respons contribut crowdsourc particip longer averag word compar word p use less vari vocabulari howev strong similar word use describ particular clip two dataset crossdataset count share word show p within dataset respons contain substanti relev content word common respons clip clip p evid respons femal older crowdsourc particip share word p respect wherea younger particip higher number share word labsourc popul pconclus crowdsourc effect approach quick econom collect larg reliabl dataset normat natur languag respons

Resumos Similares

J. Med. Internet Res. - Web-based versus traditional paper questionnaires: a mixed-mode survey with a Nordic perspective. ( 0,750347965123828 )
J. Med. Internet Res. - Comparison of US panel vendors for online surveys. ( 0,746564625047782 )
Telemed J E Health - Physical activity behavior, barriers to activity, and opinions about a smartphone-based physical activity intervention among rural residents. ( 0,743637786014863 )
J. Med. Internet Res. - The influence of response mode on study results: offering cigarette smokers a choice of postal or online completion of a survey. ( 0,734500049538498 )
J. Med. Internet Res. - Motives for participating in a web-based nutrition cohort according to sociodemographic, lifestyle, and health characteristics: the NutriNet-Sant? cohort study. ( 0,731970469059988 )
Int J Health Geogr - Evaluation of the role of location and distance in recruitment in respondent-driven sampling. ( 0,726700429080815 )
J. Med. Internet Res. - Web-based recruiting for health research using a social networking site: an exploratory study. ( 0,709647049915292 )
J. Med. Internet Res. - Bias in online recruitment and retention of racial and ethnic minority men who have sex with men. ( 0,709598326878625 )
J. Med. Internet Res. - Feasibility of internet-based health-related quality of life data collection in a large patient cohort. ( 0,702088383286648 )
J Am Med Inform Assoc - Survey non-response in an internet-mediated, longitudinal autism research study. ( 0,699142818011564 )
J. Med. Internet Res. - Challenges for researchers investigating contraceptive use and pregnancy intentions of young women living in urban and rural areas of Australia: face-to-face discussions to increase participation in a web-based survey. ( 0,698856355009896 )
J. Med. Internet Res. - Considerations for conducting Web-based survey research with people living with human immunodeficiency virus using a community-based participatory approach. ( 0,698424815565264 )
J. Med. Internet Res. - Use of a text message-based pharmacovigilance tool in Cambodia: pilot study. ( 0,696910240951579 )
J. Med. Internet Res. - The good, bad, and ugly of online recruitment of parents for health-related focus groups: lessons learned. ( 0,696239519545487 )
J. Med. Internet Res. - Development of a web-based survey for monitoring daily health and its application in an epidemiological survey. ( 0,695817753466959 )
J. Med. Internet Res. - Consumer participation in quality improvements for chronic disease care: development and evaluation of an interactive patient-centered survey to identify preferred service initiatives. ( 0,695334595183271 )
BMC Med Inform Decis Mak - An online expert network for high quality information on occupational safety and health: cross-sectional study of user satisfaction and impact. ( 0,690321757585506 )
Telemed J E Health - Connecting to the future: telepsychiatry in postgraduate medical education. ( 0,686520032189584 )
J. Med. Internet Res. - A comparison of a postal survey and mixed-mode survey using a questionnaire on patients' experiences with breast care. ( 0,679635386441592 )
Int J Med Inform - The feasibility of using SMS as a health survey tool: an exploratory study in patients with rheumatoid arthritis. ( 0,676785081812699 )
J. Med. Internet Res. - Recruitment via the Internet and social networking sites: the 1989-1995 cohort of the Australian Longitudinal Study on Women's Health. ( 0,672808371924688 )
J. Med. Internet Res. - An assessment of incentive versus survey length trade-offs in a Web survey of radiologists. ( 0,669069620850735 )
J. Med. Internet Res. - Use of web 2.0 to recruit Australian gay men to an online HIV/AIDS survey. ( 0,659723111840819 )
Telemed J E Health - Asynchronous web-based learning, a practical method to enhance teaching in emergency medicine. ( 0,659255976060425 )
J. Med. Internet Res. - Using online social media for recruitment of human immunodeficiency virus-positive participants: a cross-sectional survey. ( 0,657659122017426 )
J. Med. Internet Res. - The use of physician-patient email: a follow-up examination of adoption and best-practice adherence 2005-2008. ( 0,657555802960193 )
Appl Clin Inform - How online crowds influence the way individual consumers answer health questions: an online prospective study. ( 0,649700659766071 )
J. Med. Internet Res. - Determinants of participation in a web-based health risk assessment and consequences for health promotion programs. ( 0,648733591574197 )
J. Med. Internet Res. - Tablet, web-based, or paper questionnaires for measuring anxiety in patients suspected of breast cancer: patients' preferences and quality of collected data. ( 0,647690225207188 )
J. Med. Internet Res. - Nonprobability Web surveys to measure sexual behaviors and attitudes in the general population: a comparison with a probability sample interview survey. ( 0,644780857456922 )
J. Med. Internet Res. - Validation of an informant-reported web-based data collection to assess dementia symptoms. ( 0,63486471906969 )
J. Med. Internet Res. - Active-Q: validation of the web-based physical activity questionnaire using doubly labeled water. ( 0,63425253119349 )
J Med Syst - Physicians' views and assessments on picture archiving and communication systems (PACS) in two Turkish public hospitals. ( 0,633652671860657 )
BMC Med Inform Decis Mak - A cross-sectional survey assessing the acceptability and feasibility of self-report electronic data collection about health risks from patients attending an Aboriginal Community Controlled Health Service. ( 0,631970211509572 )
J. Med. Internet Res. - Effects of a financial incentive on health researchers' response to an online survey: a randomized controlled trial. ( 0,629379832166488 )
J. Med. Internet Res. - Parent opinions about use of text messaging for immunization reminders. ( 0,62753088960257 )
J Telemed Telecare - A survey of telehealth coordinators in Canada. ( 0,626463043730633 )
J. Med. Internet Res. - Physician response time when communicating with patients over the Internet. ( 0,626452339034287 )
J. Med. Internet Res. - Evaluation of end-user satisfaction among employees participating in a web-based health risk assessment with tailored feedback. ( 0,624328326244248 )
J. Med. Internet Res. - Using web-based and paper-based questionnaires for collecting data on fertility issues among female childhood cancer survivors: differences in response characteristics. ( 0,622366126017048 )
Telemed J E Health - Pilot assessment and survey of syrian refugees' psychological stress and openness to referral for telepsychiatry (PASSPORT Study). ( 0,618364959240414 )
J. Med. Internet Res. - Impact of the presence of medical equipment in images on viewers' perceptions of the trustworthiness of an individual on-screen. ( 0,616801572008668 )
J. Med. Internet Res. - Analysis of 4999 online physician ratings indicates that most patients give physicians a favorable rating. ( 0,61580937133599 )
J Telemed Telecare - The utility of providing automated medication dose reminders to young children on chronic medication. ( 0,615118897889908 )
Telemed J E Health - Streamlining HIV testing in the emergency department-leveraging kiosks to provide true universal screening: a usability study. ( 0,61450365866197 )
BMC Med Inform Decis Mak - The anxious wait: assessing the impact of patient accessible EHRs for breast cancer patients. ( 0,613343501179651 )
J. Med. Internet Res. - Beyond traditional advertisements: leveraging Facebook's social structures for research recruitment. ( 0,612635250631666 )
BMC Med Inform Decis Mak - Acceptability and adoption of handheld computer data collection for public health research in China: a case study. ( 0,612092896271888 )
J. Med. Internet Res. - Perceptions of family physician trainees and trainers regarding the usefulness of a virtual community of practice. ( 0,611918766245808 )
Comput Math Methods Med - Antibiotic use as a tragedy of the commons: a cross-sectional survey. ( 0,611503511613106 )
J Am Med Inform Assoc - Design and implementation of an automated email notification system for results of tests pending at discharge. ( 0,611293994290485 )
J Telemed Telecare - Caller satisfaction with the Cannabis Information and Helpline. ( 0,611180786752227 )
J. Med. Internet Res. - Expectations of iPad use in an internal medicine residency program: is it worth the hype? ( 0,609643697035251 )
J. Med. Internet Res. - Relative validity of micronutrient and fiber intake assessed with two new interactive meal- and Web-based food frequency questionnaires. ( 0,603089867085792 )
Int J Med Inform - The use of tags and tag clouds to discern credible content in online health message forums. ( 0,601990240577043 )
Int J Med Robot - Teaching robotic surgery skills: comparing the methods of generalists and subspecialists. ( 0,601425866648317 )
J. Med. Internet Res. - Participant profiles according to recruitment source in a large Web-based prospective study: experience from the Nutrinet-Sant? study. ( 0,600716013193245 )
Telemed J E Health - TXT@WORK: pediatric hospitalists and text messaging. ( 0,599289914168063 )
J Clin Monit Comput - A pilot study quantifying the shape of tidal breathing waveforms using centroids in health and COPD. ( 0,598784700101216 )
J Telemed Telecare - Robot-assisted home hazard assessment for fall prevention: a feasibility study. ( 0,598726289578298 )
J. Med. Internet Res. - Estimation of geographic variation in human papillomavirus vaccine uptake in men and women: an online survey using facebook recruitment. ( 0,597876770622145 )
J Telemed Telecare - Teleconsultation in vascular surgery: a 13 year single centre experience. ( 0,597612424782816 )
J. Med. Internet Res. - Comparison of Web-based and paper-based administration of ADHD questionnaires for adults. ( 0,597210617246041 )
J. Med. Internet Res. - Two new meal- and web-based interactive food frequency questionnaires: validation of energy and macronutrient intake. ( 0,596885131124529 )
Telemed J E Health - Ophthalmic patients' utilization of technology and social media: an assessment to improve quality of care. ( 0,595147194594372 )
AMIA Annu Symp Proc - Push and pull: physician usage of and satisfaction with health information exchange. ( 0,594993219665488 )
J. Med. Internet Res. - 3D visualization as a communicative aid in pharmaceutical advice-giving over distance. ( 0,592133954199464 )
Healthc (Amst) - A potential new data source for assessing the impacts of health reform: Evaluating the Gallup-Healthways Well-Being Index. ( 0,591320492554256 )
Telemed J E Health - A Holistic approach to assess older adults' wellness using e-health technologies. ( 0,591161756265668 )
Appl Clin Inform - Increasing patient engagement: patients' responses to viewing problem lists online. ( 0,590726764013508 )
AMIA Annu Symp Proc - Comparing the effectiveness of a clinical registry and a clinical data warehouse for supporting clinical trial recruitment: a case study. ( 0,587242278462496 )
Telemed J E Health - Information extraction for tracking liver cancer patients' statuses: from mixture of clinical narrative report types. ( 0,586619323505854 )
J Biomed Inform - Automatically extracting information needs from complex clinical questions. ( 0,584182700704479 )
BMC Med Inform Decis Mak - Development and evaluation of a new survey instrument to measure the quality of colorectal cancer screening decisions. ( 0,583682316736802 )
J. Med. Internet Res. - Communicating the experience of chronic pain and illness through blogging. ( 0,583336732741448 )
Telemed J E Health - Young women's experience with using videoconferencing for the assessment of sexual behavior and microbicide use. ( 0,58209887798015 )
Appl Clin Inform - Attitudes and perceptions of pediatric residents on transitioning to CPOE. ( 0,579991789817054 )
J Telemed Telecare - Spanish-speaking patients' engagement in interactive voice response (IVR) support calls for chronic disease self-management: data from three countries. ( 0,579415072697977 )
J. Med. Internet Res. - Emotional approach coping and the effects of online peer-led support group participation among patients with breast cancer: a longitudinal study. ( 0,578346456779702 )
J Telemed Telecare - User satisfaction with maritime telemedicine. ( 0,577747821259972 )
J. Med. Internet Res. - A field test of Web-based screening for dry eye disease to enhance awareness of eye problems among general Internet users: a latent strategy to promote health. ( 0,577281471984739 )
J. Med. Internet Res. - Broad reach and targeted recruitment using Facebook for an online survey of young adult substance use. ( 0,575994761964436 )
J Am Med Inform Assoc - An assessment of pharmacists' readiness for paperless labeling: a national survey. ( 0,574499629282103 )
J. Med. Internet Res. - Common cold symptoms in children: results of an Internet-based surveillance program. ( 0,573102056801118 )
Int J Med Inform - Reporting systems, reporting rates and completeness of data reported from primary healthcare to a Swedish quality register--the National Diabetes Register. ( 0,572863308754931 )
Perspect Health Inf Manag - The role of information and communication technology in community outreach, academic and research collaboration, and education and support services (IT-CARES). ( 0,572702518679943 )
J. Med. Internet Res. - Smartphone versus pen-and-paper data collection of infant feeding practices in rural China. ( 0,572571845754393 )
J. Med. Internet Res. - The Sexunzipped trial: young people's views of participating in an online randomized controlled trial. ( 0,572520926546714 )
J. Med. Internet Res. - Web-based counseling for problem gambling: exploring motivations and recommendations. ( 0,571003938355743 )
Health Informatics J - Family perceptions of the usability and value of chronic disease web-based patient portals. ( 0,570613824425193 )
J Am Med Inform Assoc - A survey of SNOMED CT direct users, 2010: impressions and preferences regarding content and quality. ( 0,569757463392351 )
Med Decis Making - Qualitative and Quantitative Analysis of Interviewer Help Answering the Time Tradeoff. ( 0,569637832929155 )
Telemed J E Health - Application of international videoconferences for continuing medical education programs related to laparoscopic surgery. ( 0,568859393240588 )
J. Med. Internet Res. - Pediatric caregiver attitudes toward email communication: survey in an urban primary care setting. ( 0,564700306333857 )
J Am Med Inform Assoc - Impact of an automated email notification system for results of tests pending at discharge: a cluster-randomized controlled trial. ( 0,564382421786886 )
J Am Med Inform Assoc - Resident physicians as human information systems: sources yet seekers. ( 0,563521023841001 )
Spat Spatiotemporal Epidemiol - Geographic access to healthy and unhealthy food sources for children in neighbourhoods and from elementary schools in a mid-sized Canadian city. ( 0,561420321548552 )
J Telemed Telecare - Delivery of Internet-based cancer genetic counselling services to patients' homes: a feasibility study. ( 0,560847850872588 )
Telemed J E Health - A web-based versus paper questionnaire on alcohol and tobacco in adolescents. ( 0,559318136992311 )
J. Med. Internet Res. - Learning in a virtual world: experience with using second life for medical education. ( 0,558706024290644 )