Spreadsheet: 12cd3da6-ba8b-444e-977a-75bee8f9a0e2
Sheet: Sheet1

Timestamp | Your name | Primary Affiliation (short), City, Country | Your e-mail address | Do you want to be acknowledged (with your name and affilitation) in the CONSORT-EHEALTH publication? | Your role/experience with ehealth-trials | Other guidelines | 1a-i) Identify the mode of delivery in the title | Comment on subitem 1a-i) | 1a-ii) Non-web-based components or important co-interventions in title | Comment on subitem 1a-ii) | 1a-iii) Primary condition or target group in the title | Comment on subitem 1a-iii) | Add a subitem under CONSORT item 1a | 1b-i) Key features/functionalities/components of the intervention and comparator in the abstract | Comment on subitem 1b-i) | 1b-ii) Level of human involvement in the abstract | Comment on subitem 1b-ii) | 1b-iii) Open vs. closed, web-based (self-assessment) vs. face-to-face assessments in abstract | Comment on subitem 1b-iii) | 1b-iv) Results in abstract must contain use data | Comment on subitem 1b-iv) | 1b-v) Conclusions/Discussions in abstract for negative trials | Comment on subitem 1b-v) | Add a subitem under CONSORT item 1b | 2a-i) [NAME] and the type of system/solution | Comment on subitem 2a-i) | 2a-ii) Scientific background, rationale | Comment on subitem 2a-ii) | Add a subitem under CONSORT item 2a | (no EHEALTH-specific subitems under CONSORT item 2b) | (no EHEALTH-specific subitems under CONSORT item 3a) | 3b-i) Bug fixes, Downtimes, Content Changes | Comment on subitem 3b-i) | Add a subitem under CONSORT item 3b | 4a-i) Computer / Internet literacy | Comment on subitem 4a-i) | 4a-ii) Open vs. closed, web-based vs. face-to-face assessments: | Comment on subitem 4a-ii) | 4a-iii) Information giving durnig recruitment | Comment on subitem 4a-iii) | Add a subitem under CONSORT item 4a | 4b-i) Report if outcomes were (self-)assessed through online questionnaires | Comment on subitem 4b-i) | 4b-ii) Report how institutional affiliations are displayed | Comment on subitem 4b-ii) | Add a subitem under CONSORT item 4b | 5-i) Mention names, credential, affiliations of the developers, sponsors, and owners | Comment on subitem 5-i) | 5-ii) Describe the history/development process | Comment on subitem 5-ii) | 5-iii) Revisions and updating | Comment on subitem 5-iii) | 5-iv) Quality assurance methods | Comment on subitem 5-iv) | 5-v) Ensure replicability by publishing the source code, and/or providing screenshots/screen-capture video, and/or providing flowcharts of the algorithms used | Comment on subitem 5-v) | 5-vi) Digital preservation | Comment on subitem 5-vi) | 5-vii) Access | Comment on subitem 5-vii) | 5-viii) [NAME] of delivery, features/functionalities/components of the intervention and comparator, and the theoretical framework | Comment on subitem 5-viii) | 5-ix) Describe use parameters | Comment on subitem 5-ix) | 5-x) Clarify the level of human involvement | Comment on subitem 5-x) | 5-xi) Report any prompts/reminders used | Comment on subitem 5-xi) | 5-xii) Describe any co-interventions (incl. training/support) | Comment on subitem 5-xii) | Add a subitem under CONSORT item 5 | 6a-i) Online questionnaires: describe if they were validated for online use [6] and apply CHERRIES items to describe how the questionnaires were designed/deployed | Comment on subitem 6a-i) | 6a-ii) Describe whether and how “use” (including intensity of use/dosage) was defined/measured/monitored | Comment on subitem 6a-ii) | 6a-iii) Describe whether, how, and when qualitative feedback from participants was obtained | Comment on subitem 6a-iii) | Add a subitem under CONSORT item 6a | (no EHEALTH-specific subitems under CONSORT item 6b) | 7a-i) Describe whether and how expected attrition was taken into account when calculating the sample size | Comment on subitem 7a-i) | Add a subitem under CONSORT item 7a | (no EHEALTH-specific subitems under CONSORT item 7b) | (no EHEALTH-specific subitems under CONSORT item 8a) | (no EHEALTH-specific subitems under CONSORT item 8b) | (no EHEALTH-specific subitems under CONSORT item 9) | (no EHEALTH-specific subitems under CONSORT item 10) | 11a-i) Specify who was blinded, and who wasn’t | Comment on subitem 11a-i) | 11a-ii) Discuss e.g., whether participants knew which intervention was the “intervention of interest” and which one was the “comparator” | Comment on subitem 11a-ii) | Add a subitem under CONSORT item 11a | (no EHEALTH-specific subitems under CONSORT item 11b) | 12a-i) Imputation techniques to deal with attrition / missing values | Comment on subitem 12a-i) | Add a subitem under CONSORT item 12a | (no EHEALTH-specific subitems under CONSORT item 12b) | [NAME]) Comment on ethics committee approval | Comment on subitem X26-i) | x26-ii) Outline informed consent procedures | Comment on subitem X26-ii) | [NAME]) Safety and security procedures | Comment on subitem X26-iii) | Add a subitem under item [NAME] | (no EHEALTH-specific subitems under CONSORT item 13a) | 13b-i) Attrition diagram | Comment on subitem 13b-i) | Add a subitem under CONSORT item 13b | 14a-i) Indicate if critical “secular events” [1] fell into the study period | Comment on subitem 14a-i) | Add a subitem under CONSORT item 14a | (no EHEALTH-specific subitems under CONSORT item 14b) | 15-i) Report demographics associated with digital divide issues | Comment on subitem 15-i) | Add a subitem under CONSORT item 15 | 16-i) Report multiple “denominators” and provide definitions | Comment on subitem 16-i) | 16-ii) Primary analysis should be intent-to-treat | Comment on subitem 16-ii) | Add a subitem under CONSORT item 16 | 17a-i) Presentation of process outcomes such as metrics of use and intensity of use | Comment on subitem 17a-i) | Add a subitem under CONSORT item 17a | (no EHEALTH-specific subitems under CONSORT item 17b) | 18-i) Subgroup analysis of comparing only users | Comment on subitem 18-i) | Add a subitem under CONSORT item 18 | 19-i) Include privacy breaches, technical problems | Comment on subitem 19-i) | 19-ii) Include qualitative feedback from participants or observations from staff/researchers | Comment on subitem 19-ii) | Add a subitem under CONSORT item 19 | 22-i) Restate study questions and summarize the answers suggested by the data [2], starting with primary outcomes and process outcomes (use) | Comment on subitem 22-i) | 22-ii) Highlight unanswered new questions, suggest future research [2] | Comment on subitem 22-ii) | Add a subitem under CONSORT item 22 | 20-i) Typical limitations in ehealth trials | Comment on subitem 20-i) | Add a subitem under CONSORT item 20 | 21-i) Generalizability to other populations | Comment on subitem 21-i) | 21-ii) Discuss if there were elements in the RCT that would be different in a routine application setting | Comment on subitem 21-ii) | Add a subitem under CONSORT item 21 | (no EHEALTH-specific subitems under CONSORT item 23) | (no EHEALTH-specific subitems under CONSORT item 24) | (no EHEALTH-specific subitems under CONSORT item 25) | X27-i) State the “relation of the study team towards the system being evaluated” | Comment on subitem X27-i) | Add a subitem under item X27 | Do you want to become involved in the writing committee working on the elaboration document? If yes, please provide the subitems you wish to elaborate on
2011-04-04 20:31:13 | [NAME] | Talaria, Inc., Seattle, WA USA | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports, I have experience mainly from a policy/implementation/decision-maker point of view |  | 5 | Standardization of terminology is essential. I have favored "Internet-based" over "web-based" because web pages are not necessarily accessed over the Internet, but the latter term seems more precise if we are defining "web-based" as accessed over the World-Wide Web. Perhaps HTML-based "web pages" that are not accessed over the World-Wide Web should simply be called "HTML-based." | 4 | To the extent that titles are limited in length, I see this as less essential than identifying the primary modality and target group. | 5 |  |  | 5 | Standardizing terminology (as in 1a-i) also seems essential for key features and functionalities. Here, should we also complement or substitute product names with broader terms for the class of products (such as “multimedia” instead of “Flash”)? | 5 |  | 5 |  | 3 | It would be challenging to do an adequate job of describing use metrics in an abstract.  Without important details, use metrics can be misleading.  Identifying primary and secondary outcomes in the abstract, on the other hand, is essential. | 3 | Again, it seems like it would be challenging to do an adequate job of "discussing" possible reasons for a negative result in the abstract. Could this be simplified to recommend that authors cite the supposed primary reason for a negative result in the abstract and discuss that and possible alternatives in the discussion? |  | 5 |  | 5 |  |  |  |  | 5 |  |  | 4 |  | 5 |  | 5 |  |  | 5 |  | 4 |  |  | 4 |  | 4 | Essential unless these were published previously elsewhere. | 5 |  | 4 | There should be examples of what counts as a quality assurance method. | 1 | I am a psychologist and Research Scientist for a small business. Publishing our source code is not something the owners would allow.  Replicability should hinge on the theory guiding the development of the intervention and the hypothesized active ingredients.  It should be up to the replicators to develop something comparable, sufficient detail should be provided to enable this, but not the original source code.  Source code should be proprietary. | 3 | As with publishing the source code, it is not always possible to digitally archive an intervention.  Again, sufficient detail should be provided to enable readers to understand all the components of the intervention to replicate or build upon prior work. | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  |  | 4 |  | 5 |  | 5 |  |  |  | 2 | This seems primarily important in the context of a negative result. If a significant effect was not found, was it because there was not sufficient power to detect an effect.  If a significant effect was found, it seems like a moot point. |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  |  | 4 |  | 4 |  | 4 |  |  |  | 4 |  |  | 4 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 4 | Usage metrics are critically important, but "average session length" is problematic when users access online interventions outside the laboratory.  Even when timeouts are used, how do we know that users are attending to the intervention when the clock is ticking? |  |  | 5 |  |  | 4 |  | 4 |  |  | 5 |  | 5 |  |  | 5 |  |  | 5 |  | 5 |  |  |  |  |  | 5
2011-04-05 04:03:00 | [NAME] | University of Oslo, Oslo, Norway | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports |  | 5 |  | 5 |  | 4 | If primary condition/target group is not mentioned in the title, it is absolutely essential to mention in the abstract. |  | 2 | It is more important to have a more elaborate description of intervention features in the body to help identify what makes an intervention work/not work. | 5 |  | 4 |  | 1 |  | 5 |  | Primary condition or target group in the abstract (see 1a-iii above) | 3 |  | 5 |  |  |  |  | 5 |  |  | 2 | Only to be stated and assessed if there is reason to suspect that a target group may be illiterate in the world of technology. For example, studying a group of ADHD patients where dyslexia is a highly prevalent condition or older participants with age >60 years. | 4 | An important point is that it may be extremely difficult or even impossible for researchers to have a complete overview of the entire recruitment process due to the fact that, at least, the Internet is a medium which is hard to control. One such example which serves to identify many of the problems I mention is viral recruitment where a researcher uses his or others' social networks to recruit participants. The researcher can control his 1st degree contacts, but this is hardly possible for 2nd, 3rd... degree contacts. That is simply the nature of online viral recruitment. This is similar to referral networks often used in clinical research, however, referral networks are much more controllable because of the face-to-face contact and because they are much smaller in number. So, yes, I believe this is an imporatnt point to include, but reviewers should not dismiss papers that use viral approaches to recruitment. | 1 | I see publishing informed consenst as appendix as a bit redundant, because many informed consents will have to be translated to English  which obviously is problematic and in a way biased because of challenges with translation.  However, I would highly appreciate that researchers state in what sequence information was given, briefly state what information was given, how many contact points there were with researcher, etc. |  | 5 |  | 3 |  |  | 4 |  | 5 | Including a logical/process model of the intervention, for example in a flow chart, would really enhance the publications. It may be necessary to add this in an appendix.  There should preferrably be some guidelines on how to develop a logical/process model.  This is very important to for designers, practitioners, and academics to learn more about the success/pitfalls of intervention design and development. | 5 |  |  | Need more information on this point to assess its importance. Do you mean whether the information was created by health personnel, verified by experts, etc.? | 1 | In the name of science and research, I completely understand why we would want researchers to publish source code and algorithms.  If it does not preclude or interfere with e.g. company and commercial interests, intellectual property rights, and any other conflicts of interest, then of course, we would like to see source code and algorithms published.  However, I'm afraid this may be completely unrealistic in some cases and it would be such a shame to see otherwise high-quality research papers rejected because researchs may not be able to publish source code. This would introduce a severe publication bias in the scientific literature we do not wish.  This is particularly true in cases with industry-oriented research.  However, journal editors and reviewers should encourage the primary researcher(s) to help and invite other researchers to evaluate their interventions and products although they may not be in the position of disclosing source code.  Providing screenshots is more than welcome and should not be problematic. | 4 |  | 5 |  | 5 | See my comments on 5-ii) Describe the history/development process | 5 |  | 5 |  | 5 |  | 5 |  | See my comments on 5-ii) Describe the history/development process on adding a subitem on logical/process models. | 5 | Although important, it should not be a panacea whether a measure or scale has been validated for online use, as many measures and scales have not. | 4 | Although it is considered important and should be published, it should not be applied uncritically. A large body of literature in information systems research (see for example [NAME] et al. (1993) "measuring system usage") which shows how problematic measures of usage actually can be. | 2 |  |  |  | 5 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  | In the case of clustered randomized trials and repeated measurements, reserachers should be required to conduct multilevel analysis. It is a too common statistical mistake to conduct single-level analysis in cases where there are two or more levels of data. For example, time-level data which are repeatedly measured and person-level data which are measured on one occasion (typically baseline) in a repeated measurements study design. Otherwise, one runs the risk of getting entirely wrong results and drawing wrong conclusions. |  | 5 |  | 3 | An outline of the informed consent is welcome, however, translating and publishing the informed consent is not very important. | 5 |  |  |  | 5 |  |  | 5 |  |  |  | 5 |  | See my comments under item 12a. In case of including multilevel analysis as a necessary statistical procedure as required by the study design, separate correlation matrices for variables at the different levels should be included - at least in multilevel regression analysis, latent growth curves and most probably multilevel structural equation models. | 5 |  | 2 | Important to distinguish between effectiveness analyses (i.e. analysis of the whole sample) and efficacy analyses (i.e. sub-group analyses) which in the latter case is no longer a randomized sample. This should be made explicit in all papers.  However, the intent-to-treat principle has its advantages, but also disadvantages. It seems to me that the statistical and methodological scientific community is becoming more and more pro imputation techniques because of the disadvantages associated with the ITT principle. Thus, I don't think that the primary analysis should or must be intent-to-treat. It could just as much, or even better, apply imputations. |  | 4 | Separate analyses should be run on self-reported and objective usage data as these are differently correlated with e.g. self-reported questionnaires. Self-reported usage data correlate more highly with other self-reported questionnaire data than more objective usage data. |  |  | 5 |  |  | 5 |  | 4 |  |  | 5 | I would also like to stress the importance of utilizing RCTs to conduct process research which includes far more than just usage. Process research could for example include data on mediation and moderation effects. How do participatns develop on theoretically relevant variables during the intervention? And how do these variables relate to primary outcomes? Are there any participant characteristics that interact with intervention assignment? Etc. | 4 |  |  | 5 |  |  | 5 |  | 4 |  |  |  |  |  | 5 |  |  | 5-ii - developing a framework for a logical/process model
2011-04-05 06:19:06 | [NAME] | Academic Medical Hospital, University of Amsterdam, Amsterdam, The Netherlands | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports |  | 4 |  | 2 |  | 3 |  |  | 4 |  | 5 |  | 4 |  | 2 |  | 3 |  | Response rates of assessment / data collection should be disclosed in the abstract (eg. At 3 months, the follow-up response rate was 74%.). | 4 |  | 4 |  |  |  |  | 4 |  |  | 3 |  | 4 |  | 4 |  |  | 4 |  | 3 |  |  | 5 |  | 4 |  | 4 | This may be difficult, especially for "social" networking-based interventions with a lot of user generated content. Such interventions will change continuously as a direct consequence of the use of these interventions by the participants - something uncontrolled by the researcher / project leader, and probably even difficult to track / disclose in a publication. | 5 |  | 1 | This may be too extensive for a research article. Maybe as an appendix / online materials? May also be difficult in case the intervention is not fully open-source. | 4 |  | 4 |  | 3 |  | 5 |  | 5 |  | 4 |  | 4 |  |  | 5 | CHERRIES contains some 30 items, and although it is a truly valuable checklist for online surveys, the inclusion of 30 more items in the current, already extensive CONSORT-EHEALTH checklist may become conflicting with the maximum length of research papers for a large proportion of Journals. Maybe a selection of the most relevant CHERRIES items is possible? | 5 |  | 3 |  |  |  | 5 |  |  |  |  |  |  |  | 4 |  | 3 |  |  |  | 5 |  |  |  | 2 |  | 2 |  | 5 |  | Considering the fact that most ehealth trial collect participant data online, and different database systems / programming languages are available - all with their unique merits and possible security problems - it may be desirable to inform on which database systems / programming languages have been used (both for webbased interventions and webbased datacollection), and how the security measures for the data(bases) were implemented. |  | 4 |  |  | 2 |  |  |  | 4 |  |  | 3 |  | 5 |  |  | 4 |  |  |  | 2 |  |  | 4 |  | 3 |  |  | 3 |  | 4 |  |  | 4 |  |  | 4 |  | 2 |  |  |  |  |  | 3 |  |  | 6a-i, 7a-i, 12a-i, 16-ii, [NAME]
2011-04-05 07:01:22 | [NAME] | Maastricht University/CAPHRI, Maastricht, the Netherlands | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports |  | 4 | I would recommend not using Internet-based as a separate term, but rather specify this; e.g., web-based with e-mail support (instead of Internet-based). | 4 |  | 4 | I would recommend requiring both primary condition AND the target group. |  | 5 | I would recommend requiring the description of theories used. | 4 |  | 4 | What about mixed designs using both open and closed recruitment methods? | 5 |  | 4 | I would recommend to also describe the impact of intervention use if the primary outcome did change. |  | 4 |  | 4 |  |  |  |  | 4 | Especially the reasons are important to elaborate on. |  | 2 | I consider this part of the needs assessment and this should only be mentioned in the discussion section if this appears to be a problem. | 4 | Please leave open the option for mixed methods. | 3 |  |  | 4 |  | 2 |  | If not self-assessed, it is needed to describe (a) how participants were assessed and (b) whether this was problematic given the setting of a web-based intervention. | 3 |  | 5 | Also describe how formative evaluations shaped the intervention. | 3 |  | 3 |  | 5 | Essential but difficult in strongly tailored interventions. Maybe provide the possibility for researchers to have access to the intervention? | 5 |  | 5 |  | 5 |  | 5 |  | 4 |  | 5 | I would recommend requiring to describe in general (not only prompts/reminders) which methods/strategies are used to facilitate use: first visit, staying and revisiting. | 4 |  |  | 4 |  | 5 |  | 4 |  |  |  | 5 |  |  |  |  |  |  |  | 4 |  | 4 |  |  |  | 5 |  |  |  | 4 |  | 4 |  | 4 |  |  |  | 4 |  | Describe whether there was differential attrition (i.e., a differences in attrition between the two groups) and how was dealt with this. | 4 |  |  |  | 4 |  |  | 5 |  | 5 |  |  | 5 |  |  |  | 5 |  |  |  |  | 3 |  |  | 5 |  | 5 |  |  | 5 |  | Use and differential attrition deserve extra attention in eHealth trials. | 4 |  | 4 |  |  |  |  |  | 4 |  |  | use: 5-ix, 5-xi, 6a-ii; differential attrition: 12a, 13b
2011-04-12 03:02:59 | [NAME] | Australian National University ACT Australia | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have read many ehealth RCT/evaluation reports | I have conducted RCT drug trials under CONSORT guidelines.  Reported using STROBE guidelines for cohort studies. | 5 | The inclusion of your clarification of "web-based" "online" etc as done above is essential. Standardisation of these terms would be of great assistance to future systematic reviews. | 2 | This will vary with the importance of the non-web based component - if a 24 hour help line number is listed but no one uses it? | 4 |  |  | 4 | key features/functionalities/components of the intervention and comparator: this is essential  theories and principles used for designing the site: this is not essential (score 1 or 2) | 4 | “fully automated” vs. “therapist/nurse/care provider/physician-assisted” : essential   mention number and expertise of providers involved: low priority for abstract (score 1) | 3 | I have concerns that this may be too limiting in terms of "self-assessment" eg if all participants were provided with remote telemetry or if the study uses data from GPS is this still 'self-assessment. Can this be clarified here?  Open v closed recruitment:  essential web-based v face-to-face components: essential | 5 | participants enrolled/assessed in each group: essential 5  the use/uptake of the intervention (e.g., attrition/adherence metrics, use over time, number of logins etc. - Essential for the key endpoint eg at/for 12 months (score 5)  primary/secondary outcomes: essential 5 | 4 |  | Clinical Trial Registration number e.g. http://www.anzctr.org.au   In all aspects relating to the abstract I am aware of the very tight word limits in some journals (versus the needs of readers potential systematic reviewers) | 5 |  | 5 |  |  |  |  | 3 | Conventional trials may have 'down time' etc which I would expect to be reported if they were significant, so major breaks should be reported.   However, major changes to functionality or content or design would be essetial to report (score 5) |  | 2 | Low importance in open trials e.g. users have found the site. However, in closed trials this may be an important factor e.g. if they are referred to the site for treatment.  In both types of trial, this may also impact on attrition. | 4 | See comment on "self-assessment" in the abstract. Does remote data collection (telemetry need to be defined) | 1 | While this information may be useful/interesting, until all journals allow on-line supplements it can not be an essential requirement. |  | 5 | Data collection instruments and methods should be described | 2 | This is common to face-to-face interventions where universities / hospitals may be regarded as more trust worthy than drug companies. |  | 1 | CoI section essential (score 5) but this bias exists in all intervention types. | 2 | If these are not previously published, I'm not sure that saying "focus groups were conducted" adds to my ability to interpret the outcomes. | 5 |  | 3 |  |  | Whilst replicability is essential, I see objections to revealing commercially valuable material |  |  | 5 | “backdoor” login account or demo mode for reviewers/readers to explore the application (also important for archiving purposes..)" This would be absolutely wonderful.  Participant access needs to be described - essentail (score 5) | 5 |  | 5 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | (Note at ISRII several people reported interim analyses - none memtion if this was pre-specied in the protocol or what statistical adjustemnt was made for these 'additional' analayses). |  |  |  |  | 5 |  | 5 | This is the same as a face-to-face trial. |  | Surely it is relevant - eg. were the intervention and comparitor of similar intensity? did they require the same level of engagement (reading just information versus writing replies)? | 5 |  |  |  | 5 | Most (all?) journals require a statement that for human research IRB approaval was required. It seems appropriate that this is included as a fundamental aspect of research. | 4 |  | 4 |  |  |  | 4 | An interesting contrast to drug trials - where self-report or pill count is all that is often done to evaluate 'adherence' with treatment. |  | 4 |  |  |  | 3 |  |  | 3 | While these multiple outcomes are of interest surely the primary measure for an RCT should be ITT? |  |  |  | 3 |  |  |  | 5 |  |  | 2 | I would put technical issue / privacy breaches in the IRB field - but report unitended consequences. | 1 | It is time to slice the salami! A good quality qualitative paper is justified but a paragraph in the primary paper seems unlikely to adequately address the issues. |  | 4 |  | 3 |  |  | 5 |  |  | 3 |  | 5 |  |  |  |  |  | 3 | As per the ISRII discussion - this is generally not explicity stated in face-to-face trials where it is a similar problem. |  | Possibly
2011-04-12 10:50:06 | [NAME] | Northwestern University | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports |  | 4 | I think it is extremely important to bring some definition to the field.  My hesitancy in saying this is essential is only that it seems this is being subsumed under the title, but it is a larger issue.  While we could discuss it in the text, it might be useful to have this stand out in some way - a table or box? | 5 |  | 5 |  |  | 4 | The "theories and principles used for designing the site" is vague.  This could refer to the intervention theory (CBT), design theory (some form of instructional design), or theory at any level on a spectrum.  Right now there is attention to CBT versus interpersonal theory, etc.  My sense is these are theoretical structures that have been ported over from FtF intervention that they are becoming increasingly less relevant.  The "key features/functionalities..." are what are critical. | 5 |  | 5 | Interviewer based assessments are commonly performed via telephone. | 5 | I would make clear the distinction between adherence to treatment from lost-to-follow-up rate (assessment).  This gets confused/confounded in too many papers.  Again, the semantic ambiguity probably contributes to this problem.  My preference is to call loss of data "lost-to-follow-up" but whatever the term, this would be an opportunity to nail down that distinction. | 5 | The comment on what to discuss if the trial is negative seems overly specific. There are a multitude of ways that discussions can be misinterpretations.  Maybe make more general. Eg. "consider the effects of tx adherence on outcome, the impact of methodological problems encountered in the trial, and place outcomes in the context of larger literature." |  | 5 |  | 5 |  |  | I assume there is, however, a requirement for specific objectives. |  | 5 |  |  | 3 | I think this important generally.  But this is essentially getting at the issue of barriers to treatment, which are not typically a requirement.  For example, in RCTs of injectable medications, there is no requirement to examine injection anxiety, ability to self-inject etc.  So it is setting a higher standard for e-health than for traditional trials. | 5 | Just to help the reader, I'd include the word "elibibility" before assessment in the title here.  There seem to be several issues embedded here.  1) Sample definition: issues related to recruitment, identification of patients who repeatedly try to enroll (e.g. if they don't make a screening cutoff they keep coming back until the get in), how patients were screened, etc.; 2) medium through which assessment was conducted (which could be an issue for follow-up), and 3) masking: the level of anonymity to the trial/trial personnel.  I would recommend making these issues explicit so that the intent is clear.  That will help authors and editors when situations arise that have unique issues that may not fit the specific criteria. | 5 | Yes! |  | 5 | I would ask for clarification on 1) the assessment medium (online, mail, telephone interview, etc.) and 2) perspective (self-report, interviewer, etc.) | 3 | This is the issue of the credibility of website or mobile intervention.  It is not unique to ehealth (a pharmacotherapy trial by Harvard may be perceived differently than one run out of a private practice).  This seems to be adding a new criterion category. |  | 5 | I'm not clear how this differs from the conflict of interest issue. | 4 |  | 5 |  |  | I'm not clear what is being asked here.  Is it the quality of the static content? or the quality of feedback provided (e.g. when an intervention performs assessments and feedback as part of the intervention, if validated assessment tools are used)?  Or implementation? | 5 | 5-v - 5-viii appear to reflect the disclosure of precisely what the intervention is, which is critical. These seem geared towards web-based treatments. I've seen everything from sites that simply allow people to download pdfs (essentially delivery of text-based information) to sophisticated mobile interventions that rely primarily on interactive technologies with little text-based information.  The questions seem to be what should be disclosed, and how it should be disclosed. I'd recommend guidelines for what is described in the methods (since most people will not go beyond that - assisting systematic reviewers), and what is provided through other channels For what:   Delivery medium/media   What is delivered (text, and in what form, video, animation, interactive tools) and descriptions of each.  For how to disclose additional information, the range of options seems good, although I'd give a nod to mhealth - screenshots of phone interventions, examples of automated SMS messages, etc. | 5 |  | 5 | This item seems to confound participant access and reviewer access.  I think separating out the requirements of reporting about the intervention from the methods of disclosure would clarify. | 5 | Yes!  I would also include 1) descriptions of interactive tools (data input from users, how data were manipulated, and output to patient). 2) automated monitoring/feedback (visual, quantified or qualitative feedback). | 5 | Yes! | 5 | Also, identify any manuals, procedures or principals used in guiding the human support. | 5 |  | 5 | It sounds like you mean training/support to the patient, not a coach. I'd just clarify in the title. | Many interventions use progression rules. Some information on how people move through a staged intervention (e.g. are they allowed to move one only upon completion of a set of material [activity based rules], or after a certain time period [time based rules], or based on acheivement of their own goals, or upon a coach/therapist determination, or was the entire intervention open? This could be embedded above rather than creating a new item | 5 |  | 5 | This is a critical item.  My sense is a lot of this gets buried.  For example, people often use "module completion" as a criterion.  But that is not defined. In some cases it just means they downloaded a pdf, but it is not clear that the module was read. That is different from seeing that someone went through all pages of an online module. Some info on how data were cleaned is important (e.g. if a browser is left open, it can appear that someone spent hours on a module). For interventions that include multiple types of intervention components (didactic, interactive tools, discussion boards), it is important to break out the use data by component. This will also be increasingly important for systematic reviews, as such reviews begin to try to understand what components of interventions are active. | 3 |  |  |  | 3 | This is somewhat dependent on the statistical analysis method used. Many account for loss-to-follow-up. |  |  |  |  |  |  | 5 | Yes. FYI - I've seen growing numbers of journals ask for the work "masking" since blinding is purportedly pejorative.  I would also ask that if an assessment of the effectiveness of blinding was performed, that it be provided. (patients often mention their tx assigment to evaluators - it is important to know if that was monitored and if so, how it was managed when it occured). | 4 | This is an important issue. But it is setting a higher level for ehealth than other areas. |  |  | 5 |  |  |  | 5 |  | 5 |  | 5 |  |  |  | 5 | I would distinguish attrition from intervention from lost-to-follow-up from assessment. | It is covered above under methods, but I would have use data here.  It is really important to report a clear metric for use, separate from attrition.  Someone who logs into a site every day over 12 weeks has a different dose from someone who logs in once every week. And I suspect this will be important to systematic reviewers in the future. | 5 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 5 | Yes - seems to address one of my comments above. |  |  | 5 | I'm not sure subgroup analysis is always "self-selected" and they are sometimes the primary hypothesis. |  | 5 | Might clarify level of harm - sounds like adding privacy breaches to adverse events reporting, which is important. | 4 |  |  | 3 |  | 4 |  |  | 5 |  |  | 5 |  | 5 |  |  |  |  |  | 5 |  |  | 4a-ii, 5viii, 5x, insertion of addition item in 5, 6a-ii
2011-04-14 14:08:28 | [NAME] | The George Washington University | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports |  | 5 |  | 3 |  | 5 |  |  | 4 |  | 5 |  | 5 |  | 3 |  | 5 |  |  | 5 |  | 4 |  |  |  |  | 5 |  |  | 4 |  | 5 |  | 4 |  |  | 5 |  | 4 |  |  | 5 |  | 4 |  | 4 |  | 4 |  | 2 |  | 5 |  | 5 |  | 5 |  | 4 |  | 4 |  | 5 |  | 5 |  |  | 5 |  | 4 |  | 3 |  |  |  | 5 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 4 |  |  |  | 3 |  | 3 |  | 4 |  |  |  | 3 |  |  | 5 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  |  |  | 4 |  |  | 5 |  | 5 |  |  | 4 |  | 5 |  |  | 5 |  |  | 5 |  | 5 |  |  |  |  |  | 5
2011-04-14 15:28:35 | [NAME] | VU University Amsterdam, the Netherlands | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports |  | 5 |  | 2 |  | 5 |  | Preferably some sort of intervention description. May only apply to E-mental health (e.g. "Randomized trial of web-based CBT for elderly academics with depression") | 5 |  | 4 |  | 2 |  | 5 | Absolutely vital to assess an e-health trial at a glance. | 4 |  |  | 4 |  | 4 |  | Possibility of wider implementation of the system. |  |  | 3 |  | Brief description of website layout and complexity ("After a brief welcome screen, participants enter their weight and length and proceed to the food diary in the taskbar etc.") | 3 | Moot point - how do we measure literacy, short from asking everyone to fill in the eHEALS? | 5 |  | 4 |  | If available, clarify which/why eligible participants declined to enter the trial (demographics, etc.) | 4 | If possible, comment on how successful the adaptation of an existing pen-and-paper to an online questionnaire was (e.g., visual analogue scale required with pen-and-paper test but was not available in online questionnaire software, etc.) | 3 |  | Are online questionnaire metrics reported, and how were "unlikely" questionnaire metrics handled (e.g., were questionnaires that were answered unusually quickly included (indicative of participant inattention or laziness)) | 4 |  | 4 |  | 4 |  | 4 |  | 2 |  | 3 |  | 4 |  | 5 | Trying to grasp the workings of a website with the minimum of information provided by most articles is almost impossible. Vital!!! | 5 |  | 4 |  | 5 | Also, report what the prompts/reminders were meant to accomplish | 3 |  |  | 4 |  |  | Describe whether Google Analytics was used to track visitors and clarify any dips / peaks in participant traffic | 4 |  |  |  | 5 | Also, cite relevant sources of previous trials for a well-founded estimate of attrition |  |  |  |  |  |  | 3 |  | 4 |  |  |  | 5 |  |  |  | 4 |  | 4 |  | 4 |  |  |  | 5 | Specify both drop-out and non-use attrition separately |  | 4 |  | Would this include events like Hurricane Katrina or 9/11, in which cases people would rather watch television that use a web-based intervention. |  | 4 |  |  | 5 |  | 5 |  |  | 5 |  |  |  | 4 |  |  | 4 |  | 3 |  |  | 5 |  | 4 |  |  | 5 |  |  | 5 | Implementation of successful E-health interventions will be crucial in the following years | 4 |  |  |  |  |  |  |  |  | 1b-iv, 6a-ii, 5-xiii
2011-04-14 16:04:06 | REMOVED BY EDITOR | University of Utah | REMOVED BY EDITOR | no | I have experience with conducting ehealth studies myself, but no RCTs |  | 4 |  | 2 |  |  |  |  | 5 |  | 4 |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 5 |  |  |  |  | 4 |  |  | 3 |  | 5 |  | 4 |  |  | 5 |  | 4 |  |  | 4 |  | 3 |  | 4 |  | 5 |  | 5 |  | 3 |  | 4 |  | 5 |  | 5 |  | 4 |  | 5 |  | 5 |  |  | 4 |  | 5 |  | 5 |  |  |  | 5 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  |  | 4 |  | 3 |  | 4 |  |  |  | 5 |  |  | 3 |  |  |  | 4 |  |  | 5 |  | 5 |  |  | 5 |  |  |  | 5 |  |  | 4 |  | 5 |  |  | 5 |  | 5 |  |  | 4 |  |  | 4 |  | 5 |  |  |  |  |  | 5
2011-04-14 16:43:39 | REMOVED BY EDITOR | [NAME] Health System | REMOVED BY EDITOR | no | I have conducted ehealth RCTs |  | 5 |  | 4 | If the title becomes too long by including the non-web-based components, these should be included in the abstract and described in the manuscript. | 4 | If the title becomes too long with the target group(my RCT was directed to generally healthy adults), this should be clearly described in the abstract and manuscript. |  | 4 | My RCT included a combination of 3 behavior change theories and did not use a single theoretical model. With limitation on word count in the abstract, listing the models there would mean leaving out other important details. | 5 |  | 5 |  | 5 |  | 4 |  |  | 5 |  | 3 |  |  |  |  | 3 |  |  | 3 | We recruited 2500+ adults for an online-only intervention, including sign-on, enrollment, all surveys, and the intervention. We asked how comfortable are you with using the Internet and got a broad range of answers. I don't know how you are defining computer literacy - is there a definition? | 4 |  | 3 |  |  | 5 |  | 3 |  |  | 2 | This should be done through contact with the first or corresponding author, not through the manuscript. | 2 | With word limits, there is not space to provide much of the development story if you are presenting results. We used a separate manuscript to describe our formative focus groups. There's too much to say, other than "based on focus group feedback" in an outcomes paper. | 3 |  | 3 | Quality assurance is up to the PI, who should be following ethical standards. I can't imagine how this could be reported, unless there is standard language somewhere to follow, so that this report would be understood or valuable. | 1 |  | 2 | If this is a funded research project, the URL will be inactive by the time the manuscript is published. This digital preservation is not likely or even possible in many cases. It would be a "demo" only - which is not very helpful. | 5 |  | 4 |  | 5 | If there is paradata available, also report the extent to which intended doses compared with actual use. | 3 |  | 5 |  | 4 |  |  | 5 |  | 5 |  | 4 |  |  |  | 4 |  |  |  |  |  |  |  | 4 |  | 4 |  |  |  | 5 |  |  |  | 3 |  | 5 |  | 4 |  |  |  | 4 | We did not have any way to collect reasons for discontinuing participation - as we had no contact with our participants other than email/online intervention. We had no IRB permission to request details of discontinuing participation. There are many cases in which reasons for attrition would be available. |  | 5 |  |  |  | 5 |  |  | 4 |  | 4 | With the clear possibility of attrition in these "hands-off" interventions, there should be a way to analyse "what worked" for the people who participated through the end of the study, so the Intent To Treat should not be the only accepted outcome measure. This shift or a "both-and" approach should be strongly considered for this new technology that has such accessibility and clearly autonomous, voluntary participation. |  | 5 |  |  |  | 4 |  |  | 4 |  | 4 | With space limitations, this report would be limited in scope to 1 or 2 sentences. I do not think it should be required. |  | 5 |  | 5 |  |  | 4 |  |  | 4 |  | 3 |  |  |  |  |  | 4
2011-04-14 21:26:08 | [NAME] | Mahidol University, BKK, Thailand | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have experience mainly from a consumer/patient point of view |  | 5 |  | 5 |  | 4 |  |  | 5 |  | 3 |  | 4 |  | 5 |  | 5 |  |  | 5 |  | 5 |  |  |  |  | 4 |  |  | 4 |  | 5 |  | 5 |  |  | 4 |  | 4 |  |  | 5 |  | 3 |  | 4 |  | 5 |  | 5 |  | 4 |  | 5 |  | 4 |  | 4 |  | 4 |  | 5 |  | 4 |  |  | 4 |  | 4 |  | 5 |  |  |  | 5 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  |  | 3 |  | 3 |  | 5 |  |  |  | 5 |  |  | 5 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  | 4 |  |  | 5 |  |  | 5 |  | 5 |  |  |  |  |  | 5
2011-04-14 23:16:18 | Pongsak Kooprasertying | Drug information service, Bumrungrad international hospital, Thailand | REMOVED BY EDITOR | yes | I have read many ehealth RCT/evaluation reports, I have experience mainly from a consumer/patient point of view |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 5 |  |  |  |  | 5 |  |  | 5 |  | 5 |  | 5 |  |  | 3 |  | 4 |  |  | 5 |  | 3 |  | 4 |  | 5 |  | 5 |  | 3 |  | 3 |  | 4 |  | 4 |  | 5 |  | 4 |  | 5 |  |  | 4 |  | 4 |  | 4 |  |  |  | 3 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  |  | 2 |  | 4 |  | 4 |  |  |  | 5 |  |  | 5 |  |  |  | 5 |  |  | 4 |  | 5 |  |  | 5 |  |  |  | 4 |  |  | 4 |  | 4 |  |  | 5 |  | 5 |  |  | 5 |  |  | 5 |  | 5 |  |  |  |  |  | 3 |  |  | 1a-i,1a-ii,1a-iii
2011-04-15 01:38:32 | [NAME] | The University of New South Wales, Sydney, Australia | REMOVED BY EDITOR | yes | I have read many ehealth RCT/evaluation reports |  | 5 |  | 4 |  | 5 |  |  | 5 |  | 5 |  | 4 |  | 5 |  | 4 |  |  | 5 |  | 4 |  |  |  |  | 4 |  |  | 3 |  | 3 |  | 3 |  |  | 4 |  | 3 |  |  | 4 |  | 4 |  | 4 |  | 4 |  | 5 |  | 3 |  | 3 |  | 4 |  | 4 |  | 4 |  | 4 |  | 4 |  |  | 3 |  | 4 |  | 3 |  |  |  | 4 |  |  |  |  |  |  |  | 4 |  | 4 |  |  |  | 3 |  |  |  | 4 |  | 4 |  | 5 |  |  |  | 3 |  |  | 3 |  |  |  | 4 |  |  | 3 |  | 4 |  |  | 3 |  |  |  | 4 |  |  | 5 |  | 4 |  |  | 4 |  | 5 |  |  | 4 |  |  | 5 |  | 4 |  |  |  |  |  | 4
2011-04-15 02:49:19 | [NAME] | University of Nairobi, Kenya | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have read many ehealth RCT/evaluation reports, I have experience mainly from a consumer/patient point of view |  | 5 | A common terms dictionary will give all a common understanding of terms. | 3 | To allow others who are not e-halth proffesional to understand our results. | 5 |  |  | 4 |  | 5 |  | 5 |  | 5 |  | 5 |  |  | 4 |  | 5 |  |  |  |  | 4 |  |  | 5 |  | 5 |  | 4 |  |  | 5 |  | 4 |  |  | 4 |  | 5 |  | 4 |  | 5 |  | 4 |  | 5 |  | 5 |  | 4 |  | 5 |  | 5 |  | 4 |  | 5 |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  |  |  |  |  |  | 4 |  | 5 |  |  |  | 5 |  |  |  | 5 |  | 4 |  | 5 |  |  |  | 5 |  |  | 4 |  |  |  | 5 |  |  | 4 |  |  |  |  | 5 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 4 |  | 5 |  |  | 5 |  |  | 5 |  | 4 |  |  |  |  |  | 4 |  |  | 1a-i
2011-04-15 03:44:20 | [NAME] | University of Kent, Medway, UK | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have read many ehealth RCT/evaluation reports |  | 4 |  | 4 |  | 5 | I think it is important to phrase in terms of the WHO ICF. |  | 3 |  | 2 |  | 4 |  | 4 | Some of this information is difficult to obtain where a study is open web-based. | 3 |  |  | 4 |  | 5 |  |  |  |  | 4 |  |  | 4 |  | 5 |  | 5 | I think this information should be provided but doesn't necessarily require every informed consent documentation to be put in an appendix. |  | 5 |  | 3 |  |  | 4 |  | 3 |  | 4 |  | 3 | Only if appropriate. | 2 |  | 3 |  | 5 |  | 3 | Elements of this section as appropriate to the particular study. | 5 |  | 3 |  | 4 |  | 4 |  |  | 5 |  | 4 |  | 3 | Only if appropriate for the study. |  |  | 3 |  |  |  |  |  |  |  | 5 |  | 4 |  |  |  | 3 |  |  |  | 5 |  | 5 |  | 5 |  |  |  | 3 |  |  | 5 |  |  |  | 4 | If known |  | 3 |  | 2 |  |  | 3 |  |  |  | 4 |  |  | 3 |  | 3 |  |  | 5 |  | 5 |  |  | 4 |  |  | 5 |  | 3 |  |  |  |  |  | 4
2011-04-15 04:54:39 | [NAME] | Radboud University Nijmegen Medical Center, the Netherlands | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have read many ehealth RCT/evaluation reports, I have experience mainly from a policy/implementation/decision-maker point of view |  | 4 |  | 3 |  | 5 |  |  | 5 |  | 4 |  | 5 | This is really important. I conducted a review on eHealth applications myself and it was hard sometimes to compare results, because recruitment and setting data were lacking. |  | attrition rate is very important I would also like to see the exposure time or the amount of exposure of the intervention to participants mentioned. This is often lacking in ehealth trials | 4 |  |  | 5 | In this sense it is very important that is becomes clear that developing a web-based interventions was goal-driven instead of tool-driven. I find often that studies focus too much on the technology, while the goal behind it should be the drive for developing the intervention. | 4 |  |  |  | I agree, this is often too complex to perform in ehealth trials | 4 |  |  | 5 |  | 5 | see also my comment under 1 | 5 |  | As many definitions or synonyms appear in literature for ehealth interventions (such as web-based, health technology, health information technology, internet-based, ehealth etc), it must be made clear what definition the authors use and why. This makes it easier in reviews to compare studies to one another. | 4 |  | 3 |  | I am not sure if this is the right heading for this item. However, to my opinion it is important to state how participants got to know about the intervention. How was it spread/promoted to potential users? Often this is one of the reasons why interventions are not used, because potential users are not reached. | 4 |  | 5 | This is very important and often lacking in trials at present. I would also like to see whether a user-centered design is adopted. I think this is crucial, so should be made explicit. | 5 | this is absolutely important when talking about internet interventions. They are dynamic. It should be stated clearly how components could have changed. | 3 |  | 3 | screen shots are illustrative, but not crucial to my opinion, when a website is available. However, often the website mentioned i nstudies is not available anymore. in that case, screenshots would be useful. | 5 | see my comment above. | 5 |  |  | A taxonomy for the different 'ingredients' of the interventions would be useful. However, ehealth interventions often have a 'innovative' or 'disruptive' character. The value is that it is something new. So I would recommend to avoid being rigid on this one. New 'ingredients' which are not justified by some sort of model or taxonomy are welcome. As long as there is some sort of rationale to do so. | 5 | exposure is very important!! | 4 | this might especially provide important information for implementation strategies of ehealth interventions. It might be that in the beginning more human involvement is needed. | 4 |  | 5 |  |  | 4 |  | 5 |  | 5 |  | It might be that also some unexpected side effects could be detected. researchers should acknowledge the value of this for the young ehealth research field. These should also be stated when encountered. | It might be that also some unexpected side effects could be detected. researchers should acknowledge the value of this for the young ehealth research field. These should also be stated when encountered.  It could provide important clues for future trials what we could expect to be outcomes of an intervention. | 5 |  |  |  |  |  |  |  | 4 |  | 4 |  |  | however, it could ofcourse be the case, that patients use other existing forums/bulletin boards/information/wikis on the Internet if they are allocated to the control group etc. This could mimic components of the intervention and consequently could bias your results. Participants should be questioned on this in order to take this into account when interpreting/analysing the results. | 5 |  |  |  | 5 |  | 4 |  | 5 | Important to know how privacy was guaranteed, because patients have concerns about this. |  |  | 5 |  |  | 5 |  |  | this could also be the case in ehealth trials. So I would not omit this item from this checklist. | 4 |  |  | 5 |  | 4 |  |  | 5 |  |  |  | 5 |  |  | 4 |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  |  | 5 |  | 5 | Important!!! Useful for readers to know how to implement this intervention in daily practice. |  |  |  |  | 4 |  |  | 2a, 4b, 5-ii, 5-viii, 5-x, 6a-iii/6b, 11b, 21-iii
2011-04-15 06:30:40 | [NAME] | University Medical Centre St Radboud, Nijmegen, The Netherlands | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports |  | 5 |  | 5 |  | 5 |  | Mention if the intervention is for individuals or groups. Mention if the intervenion is educational (information only) or therapeutical. | 3 |  | 5 |  | 5 |  | 5 |  | 5 |  |  |  | mention the traditional interventions for the particular patient population and the advantages of the new system | 5 |  |  |  |  | 5 | Authors should mention if new content was added to system during the trial. In my opinion this is often the case. |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 4 |  | Report if supporting patient organisations or sponsors are displayed, since these may also influence potential participants. | 5 |  | 5 | Mention if the application was tested in a pilot study before it was in the RCT | 5 |  | 5 |  | 4 | Replicability is a problem in non-pharmacological trials, since treatment protocols are often not available. Key elements in the application should be available as well as details concerning the 'dose' (detailed description of the intervention) and timing. | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 | The content of the prompts is also essential. Prompts may be extremely reinforcing the use of the intervention. | 4 |  |  | 4 |  | 5 |  | 5 |  |  |  | 5 |  |  |  |  |  |  |  | 5 |  | 3 |  |  |  | 5 |  |  |  | 5 |  | 3 |  | 5 |  |  |  | 5 |  |  | 4 |  |  |  | 4 |  |  | 5 |  | 5 |  |  | 5 |  |  |  | 4 |  |  | 5 |  | 4 |  |  | 3 |  | 5 |  |  | 5 |  |  | 5 |  | 5 |  |  |  |  |  | 5 |  |  | 3b-i, 4a-ii
2011-04-15 10:46:11 | Isabel de la Torre | University of Valladolid, Valladolid, Spain | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs |  | 3 | Electronic and mobile health record systems | 4 | with tablet PC support | 4 | An mobile EHR with tablet PC support for patients with mental disabilities |  | 4 | Security management module Presentation interface management logic EHR management logic Database access logic | 4 | therapist/physician assisted | 4 | from an open access website | 3 | use over time | 4 | primary otucome not changed |  | 4 | A significant number of people with mental diseases remain unnoticed due to the incorrect identification of the symptomatology, the resistance to seek either help or information regarding these services, among others. Mobile technologies can offer a full potential for helping people with cognitive problems and their supporting staff. The system presented has as its main goal the remote access to the EHR of any patient with a cognitive disorder, through mobile devices (smartphone, PDA, tablet PC, etc.). | 4 | Accessing to EHRs through mobile devices provides a number of advantages both for health centers and clinical staff, and for patients. Among these advantages are: accessing to patients’ information in real time (from wherever and whenever), resource savings, improving the information management, and reducing the delay in health care. In the field of mental health, there are important epidemiological studies releasing relevant information about types and rates of more frequent disorders. However, a significant number of people with mental diseases remain unnoticed due to the incorrect identification of the symptomatology, the resistance to seek either help or information regarding these services, among others.  Shyu et al. (2006) performed a mobile EHR system for the family medicine department in the NTUH (National Taiwan University Hospital) from Taiwan. [NAME] (2000) examines the important need to implement mobile health management system that is able to provide continual health care delivery even while an individual is on the move. In the senSAVE project authors developed a mobile system to monitor vital parameters. The user interface and the interaction were specifically adapted to the needs of the elderly |  |  |  | 4 |  |  | 4 |  | 4 |  |  |  |  | 4 |  | 4 |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 1a
2011-04-15 12:51:19 | [NAME] | ISA Associates | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports |  | 4 |  | 2 |  | 4 |  |  | 4 |  | 4 |  | 3 |  | 2 |  | 3 |  |  | 5 |  | 3 |  |  |  |  | 4 |  |  | 3 |  | 4 |  | 5 |  |  | 5 |  | 3 |  |  | 4 |  | 3 |  | 4 |  | 3 |  | 2 | This is a tough one.  In most non-web-based trials, information necessary for replication is important, if not essential, and if not in the body of the text, available somewhere.   But web-based interventions are typically so complex, multi-faceted, and idiosyncratic that providing this kind of information in the article is nearly impossible. | 4 |  | 4 |  | 3 | Another tough one.  Ideally, all this information would be present; however, such elements as "in-depth description of the content," may not be feasible in the case of multi-module, large-volume web-based interventions. | 4 |  | 4 |  | 4 |  | 5 |  |  |  | I do not know what CHERRIES items are.  You should have defined/explained the term! | 4 |  | 4 |  |  |  | 2 | It's important, perhaps essential, that power analysis be conducted by the investigator when the trial is designed, but I don't believe the description of how that was done is essential to the reporting of the study. |  |  |  |  |  |  | 3 |  | 4 |  |  |  | 5 |  |  |  | 5 |  | 5 |  | 3 |  |  |  | 4 |  |  | 4 |  |  |  | 4 |  |  | 4 |  | 4 |  |  | 4 |  |  |  | 5 |  |  | 4 |  | 4 |  |  | 4 |  | 4 |  |  | 4 |  |  | 5 |  | 5 |  |  |  |  |  | 5 |  |  | No (sorry!)
2011-04-15 13:55:43 | [NAME] | Connecticut, US | REMOVED BY EDITOR | yes | I have read many ehealth RCT/evaluation reports, I have experience mainly from a policy/implementation/decision-maker point of view | No, these proposed guidelines that I have seen dealing with eHealth studies. | 5 | I like the explanation, it is detailed enough to provide guidance on the wording that would allow for consistency and allow for comparison across eHealth studies. | 5 | Similar comment, essential to remove possible confounding factors. | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | More specifics regarding the software used during the interventions and whatever was used for analysis. | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses  Issues regarding access to software would be important to include. If participants must use a particular program, are participants provided with the equipment, etc. | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses  Sponorship should always be clearly disclosed as it can impact results and intepretation of findings | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses  Good idea about the demo pages. I can see how proprietary software may be problematic. | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | Also provide details about how the co-interventions have been used in other settings or with other populations, if applicable | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  |  | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | Yes, something should be included about this. Maybe this should be included as part of the methodology since interim analyses would be useful and probably hold more weight than, say, post-hoc analyses. Stopping guidelines would be useful as well if researchers can anticipate what negative outcomes might occur as a result of conducting the study | Yes, this should be included. Any method that was used to determine how participants were recruited would be essential | Yes, this should be included. A good methodology would allow for evaluation throughout | Yes, this should be included | Yes, this should be included | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | If more studies are reported using these guidelines, then future studies should really include a literature review of interventions are done and reported in the literature | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | Include statistical procedures used and why they were chosen | Yes, definitely | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | Details of why people dropped out would be helpful to determine what didn't work and how this may have impacted on the final results | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | Yes, this is would be important to include in case anyone else may want to try and replicate the study | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | Maybe some additional guidelines on some standard ways of tabulating the demographic data would be very useful | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | Maybe some guidelines about how to quantify qualitative feedback to make it more useful to others who may want to replicate the study | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | 5 |  | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses |  | Yes, this would be important to include. If all this is in a database then interested parties may be able to search and find similar studies, etc. | An online database would be good. | Yes, definitely, as part of sponsorships and conflict of interest disclosures | 5 | Essential to allow for future comparisons across studies and allow the possibility of being part of meta-analyses | Maybe some more detail about what is meant by "system being evaluated." A list of key words would be good for purposes of future online database searching. | Possibly. Depends of the outcome of this survey
2011-04-15 14:34:02 | [NAME] | San Jose State University, CA, USA | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs |  | 5 | The disambiguation proposed here is essential. | 5 | Possible to add:  Personal, face-to-face meetings | 5 | I favor as much clarity and specificity as possible here. For example. . .   . . . children with Type 1 DM as defined in . . . or as diagnosed by . . .  Similar specificity for psychological categories |  | 5 | Key features/functionalities/components = Essential in abstract  Key theories and principles = May not be able to fit in abstract! | 5 | Overall level of human involvement = Essential in abstract  Number and expertise of providers = Perhaps no space in abstract! | 3 | As with previous items, subject to limits of abstract length | 3 | Participant numbers = Important  Other items = Too detailed for abstract | 4 | Primary outcome = Important  Other items = May be too detailed for abstract |  | 5 | Essentail | 5 | Essential |  |  |  | 4 | Such change events can be represented in a timeline fashion. |  | 4 | Important - If known | 5 | Essential | 5 | Essential |  | 5 | Essential | 4 | A very nice idea |  | 5 | Essential | 3 | Possibly important in some circumstances - Especially if previous evaluations were publicized or have potential to be communicated by word of mouth within, say, a specific, narrow demographic segment | 4 | Wow! This brings up all kinds of possibilities. . . needs careful consideration | 5 | Essential! | 1 | May be proprietary | 5 | Demo pages accessible without login makes sense here. | 5 | Essential | 5 | Essential | 5 | Essential | 5 | Essential | 4 | Very important | 4 | Very important |  | 5 | Essential | 5 | Essential | 4 | Important if analyzed - Statistical summary |  |  | 3 | Possible |  |  | Yes - Randomization approach should be described. |  |  |  | 5 | Essential | 5 | Essential !!! |  |  | 4 | Important and routine |  |  |  |  | 3 |  | 3 | If applicable |  |  | 3 | A very useful idea - Can be reported in a table elsewhere, though. . . |  | 3 | Japan is a recent example. |  |  | 5 | As available |  | 5 | Essential ! | 3 | Not necessarily ITT |  | 4 | As available . . . |  |  | 4 | Important, if such analyses were performed |  | 3 | If applicable | 3 | Hmm . . . This really depends on the extent to which the qual data affected analysis and interpretation. |  | 5 | Essential | 5 | Essential |  | 5 | Essential |  | 5 | Essential | 3 | Only of applicable |  |  |  |  | 3 |  |  | 5ii, 5iii, 13b-i, 19-ii
2011-04-15 14:56:07 | [NAME] | Autonomous University of Barcelona, Barcelona, Spain | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have read many ehealth RCT/evaluation reports, I have experience mainly from a consumer/patient point of view, I have experience mainly from a policy/implementation/decision-maker point of view |  | 5 |  | 2 |  | 5 |  | Unlike the traditional randomized clinical trials and due to psychosocial factors and cultural differences between countries. I believe that it would be important to include the place/country where the interventions is made in title (eg, "in Spain") | 5 |  | 5 |  | 3 |  | 2 |  | 5 |  |  | 5 |  | 5 |  |  |  |  | 5 |  |  | 3 |  | 4 |  | 4 |  |  | 4 |  | 5 |  |  | 5 |  | 2 |  | 3 |  | 3 |  | 2 |  | 5 |  | 3 |  | 2 |  | 4 |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 2 |  | 2 |  |  |  | 5 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  |  | 5 |  | 5 |  | 4 |  |  |  | 5 |  |  | 5 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  |  |  | 4 |  |  | 5 |  | 2 |  |  | 5 |  | 5 |  |  | 4 |  |  | 4 |  | 5 |  |  |  |  |  | 5
2011-04-15 14:59:15 | [NAME] | Autonomous University of Barcelona, Barcelona, Spain | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have read many ehealth RCT/evaluation reports, I have experience mainly from a consumer/patient point of view, I have experience mainly from a policy/implementation/decision-maker point of view |  | 4 |  | 2 |  | 5 |  |  | 4 |  | 3 |  | 3 |  | 3 |  | 3 |  |  | 4 |  | 5 |  |  |  |  | 5 |  |  | 3 |  | 3 |  | 3 | Only include if information has been provided (informed consent) |  | 4 |  | 4 |  |  | 5 |  | 2 |  | 2 |  | 3 |  | 2 |  | 3 |  | 4 |  | 4 |  | 4 |  | 3 |  | 3 |  | 5 |  |  | 5 |  | 2 |  | 4 |  |  |  | 5 |  |  |  |  |  |  |  | 5 |  | 4 |  |  |  | 5 |  |  |  | 5 |  | 5 |  | 4 |  |  |  | 5 |  |  | 4 |  |  |  | 5 |  |  | 4 |  | 4 |  |  | 4 |  |  |  | 4 |  |  | 3 |  | 3 |  |  | 4 |  | 5 |  |  | 5 |  |  | 4 |  | 4 |  |  |  |  |  | 5 |  |  | 6a-i
2011-04-15 16:20:09 | [NAME] | Tehran University of Medical Sciences, Tehran, Iran | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have read many ehealth RCT/evaluation reports, I have experience mainly from a policy/implementation/decision-maker point of view |  | 5 | I think that the Operating system or running platforms that have been used or compared by researchers should be stated in the title if appropriate; especially for the "mobile" class of products (e.g. [NAME], iOS, Microsoft Windows Mobile, etc.)  I think that even words such as “smart phone” are somehow ambiguous. | 5 | I think that the exmample can be written in a more complete way like this: “with telephone support”; or "by SMS"; or by "MMS", etc. | 5 | The word "mobile" has the ambiguity I mentioned in 1-ai). | I suggest using acronyms for the RCTs that use mHealth or eHealth platforms; acronyms like eRCT ot mRCT. | 5 | I think the sponsored trials should mention the supporting body(ies) in this section. | 4 | The example can be completed in this way: “therapist/nurse/care provider/physician/patient-assisted” I don't think that mentioning number of providers involved will be necessary; just the required expertise will be enough. | 4 | The mentioned sentences shall be re-written for better understanding. They are somehow vague. I suggest putting them in 3 separate paragraphs: one for Open vs. closed; one for web-based (self-assessment) vs. face-to-face assessments; and the third for clearly saying if outcomes were self-assessed or not. | 5 | As a complement to reporting just the numbers, one can think about the usage/compliance/attrition/adherence indices that can be reported instead of just telling the raw numbers of this and that.  The "information and computer literacy" of the participants (the investigators and the patients) and methods to assess its level across the study, and also its effects on study results (primary and/or secondary outcomes) should be discussed briefly in the abstract and with details in the article. | 5 |  | The minimum system/infrastructure requirements for planning and conducting such an intervention in other contexts should be written in an itemized format; like the instances we want to install software and we should read and be aware of the minimum requirements for the installation. | 5 |  | 5 | I think that brief justification of other (known) solutions, platforms, and/or interventions seems necessary. The process by which an intervention can be adopted, installed and maintained can also be discussed. Of course, it can be transferred to the part on methods. |  |  |  | 5 | I think that "updates" shall be a fixed part in this section; as many eHealth or mHealth solutions need regular updating in thier course of development. The process by which these [NAME] fixes, Downtimes, Content Changes are proposed and managed should be discussed (for example "decisions on Bug fixes, Downtimes, Content Changes were made by a special IT committee" under the condition of X, Y, and Z). | After completion of trials, there may be a "lag" time for researchers to prepare and publish the results of their study. It may happen that the version(s) of the solutions used at the time of study design and conduct, change during this "lag" time. The discussion on these kinds of "version" changes seems necessary. | 5 | Any standardized measure to assess Computer / Internet literacy, should be mentioned briefly (or in particular instances, in detail). There seems to be a need to develop such instruments and questionnaires for assessing and measuring participants' Computer / Internet literacy. | 5 | I think a discussion on "multi-login" or "verification" of the participants is necessary. The investigators should ensure that every participant (himself or herself) has passed throough the intervention. | 5 | I think a few sample SNAPSHOTS from what has been used/designed as the graphical user interfaces (GUIs) for participants should be added to the supplementary materials if appropriate.  Any educational sessions for ensuring consistency in information and computer literacy of the participants should be discussed. |  | 5 | Details of pilot-testing of the designed questionnaires and the pros and cons of the self-assessment questionnaires seem to be complementary. | 2 |  |  | 5 | I think the cost(s) of developing (and/or acquiring) the software (especially the direct costs) and also software availability, access and intellectual properties and licenses for use in other countries should be reported. | 5 | Any special phase(s) in the development process that is (are) unique for the reported RCT should be discussed.  Quality checking and assurance of the solution should be mentioned briefly.  I think one can consider at least Phase II and III trials for RCTs of mHealth solutions; conducting and RCT in a small-scale and large-scale form, respectively. | 5 | For "static" solutions, and specific software "package(s)" and/or version(s) should be mentioned. | 5 |  | 5 |  | 4 |  | 5 | Any specific constraint in access should be discussed; for example the need for internet access, wireless service subscription, a special bandwidth, mobile phone plan, etc. | 5 | Any pilot-testing of the communication delivery channels and assessment of their usability should be discussed.  Any specific skills necessary for the establishment of intervention and/or any changes that should be made in usual managerial routines of the organization will be complementary to the replicability of the intervention. | 5 | I think a few measures of COMPLIANCE to the intervention should be mentioned in this part as an example. | 3 |  | 4 | These ways of reminding have different effects in different contexts and for different interventions. How one can assure the reader those different prompts/reminders may have the same effect(s) in other situations and contexts? | 5 | Number and duration of the training sessions are important issues to consider. The trainers' skills should be part of the discussion in this part.  A few challenges still remain here: if an intervention is defined as using a "drug", we have biochemical tests to assess its therapeutic dosage in the participants; is there any such test(s) that can tell us a training has increased the effects of an e-intervention to its "therapeutic dosage(s)"? |  | 5 | Software requirements/limitations for browsing the online questionnaires (for example need for using IE, [NAME], [NAME], etc.) should be discussed. | 5 |  | 4 |  |  |  | 5 | Any pilot-testing for sample size determination and attrition estimation should be mentioned. | Sampling method (cluster, stratified, etc.) and sampling frame should be discussed. |  |  |  |  |  | 5 | Any effects of "non-blinding" on the possible outcomes of the study, should be discussed or at least acknowledged.  A discussion on "blinding protection" throughout the intervention seems necessary; as the participants or care providers may gain access to internet and search for the possibility of association(s) between the intervention(s) and outcome(s). | 4 | Challenges on developing "sham" e-interventions, "placebo" e-intervention or the "comparator" e-intervention (if appropriate) should be discussed. |  |  | 4 |  |  |  | 5 | Issues such as Licenses, [NAME] properties, Costs and/or Price(s), Trademarks, [NAME], Open- or closed-access of the solution(s) etc. should be provided. | 4 | Methods for authentication of the person(s) who signs the informed consent should be discussed.  A part in the informed consent should be dedicated to the rights and intellectual properties of the solution developers that should be met by participants. | 5 | Any specific precautions for participants in special cases should be mentioned; for example the children-rating of the materials and/or contents, the orientation to adults, offensiveness of the material etc. |  | Any specific method for analysis should be mentioned. For example "intention to treat" etc. | 5 | I am strongly in favor of developing and using attrition statistics in this part. |  | 4 |  |  |  | 5 |  |  | 5 |  | 3 | It depends on the objective of study; but generally speaking, the intention-to-treat seems more appropriate. |  | 2 | It can be a supplementary part to the report; not a necessary part.  But standardized measures should be developed and used in order to become unique and specific currencies to be used in future systematic reviews and meta-analyses. Meta-analyses of e-Health intervention should be able to summarize usage/attrition statistics in addition to the effect size summarization for primary and/or secondary outcomes. |  |  | 4 |  |  | 5 |  | 4 |  |  | 5 |  | 5 | I think mentioning the reproducibility of the intervention in other platforms is an appropriate suggestion for future research. |  | 5 | Measures taken by investigators to lessen the amount and limit the sources of biases should be discussed. |  |  | Any assumption and pre-requisite made by the investigators that seems to have a positive/negative effects on generalizability of findings should be discussed in this part. | 5 |  |  | Any registration number of the developed solutions can be mentioned too. |  | Sources of funding and other support from businesses are so important in EHEALTH-specific RCTs. | 5 |  |  | 1a-ii, 1a-iii, 1b, 4a-I, 4a-ii, 4a-iii, 5-i, 5-viii, 5-ix, 5-xi, 6a-i, 6a-ii, [NAME], 13b-i, 17a-i
2011-04-15 22:08:52 | [NAME] | McMaster University | REMOVED BY EDITOR | yes | I have read many ehealth RCT/evaluation reports |  | 4 | Efforts to reduce ambiguity in titles (and inconsistency throughout a manuscript) regarding the mode of delivery are needed. | 3 |  | 5 |  |  | 5 | Avoiding ambiguous or overly general terminology may be useful here (as it would throughout the abstract & manuscript). | 4 |  | 2 |  | 3 |  | 4 |  |  | 5 | Specifically the problem and goals of the intervention and the evaluation are essential. Points regarding the nature of the intervention may be less essential, since they may be described in detail in Methods. | 2 |  |  |  |  | 3 |  |  | 3 |  | 5 |  | 2 |  |  | 5 |  | 2 |  |  | 4 |  | 5 | Important to declare previous evaluations so reviewers can check them, and to avoid duplicate publication. | 4 | Differences from versions featured in previous evaluations need to be clear. | 3 |  | 2 | How much of this is provided should be up to discretion of the journal, according to space considerations. | 4 |  | 4 |  | 4 | Might theoretical framework of intervention sometimes be covered in Introduction? Can item be simplified (although much of it is useful)? Perhaps another round of consensus is needed re points to include in this item. | 2 | I imagine this would be reported as needed, and doesn't need to be a guideline. | 4 |  | 3 |  | 5 |  |  | 4 |  | 3 |  | 5 |  |  |  | 3 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 2 | If it's not recommended for other types of RCTs (even if less common) I don't know if eHealth authors should be the first to be singled out or burdened with this as a requirement. |  |  | 4 |  | 4 |  | 4 |  |  |  | 3 |  |  | 4 |  |  |  | 4 | What rapid computer literacy assessment tool can be recommended? |  | 3 |  | 1 | Depends on purpose of evaluation. Also, "intention-to-treat" needs to be defined in all cases (eg, loss to follow-up is often considered differently). |  | 3 |  |  |  | 4 |  |  | 4 |  | 3 |  |  | 5 |  | 4 |  |  | 4 |  |  | 4 | And avoid over-generalizing the implications of the study (a danger with eHealth trials). | 3 |  |  |  |  |  | 4
2011-04-16 00:14:16 | [NAME] | UniMelb & MCRI, Melbourne, AUSn | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports | Just the ones mentioned at the IRSII CONSORT workshop | 4 | I work with mobile phones and there is a lot of ambiguity about whether phone calls, SMS, internet-based programs or stand-alone applications are used. I'd suggest including these as well as the class of product as suggested. | 4 |  | 4 |  |  | 4 |  | 4 |  | 4 | Levels of intervention might be useful such as universal prevention, selected and indicated preventions, self-, doctor-indicated treatment or clinical treatment | 5 |  | 5 |  |  | 4 |  | 3 |  |  |  |  | 5 |  |  | 4 |  | 3 |  | 2 | This is important for all studies,  not just eHealth, but I think it's important. I don't think adding the documentation as an appendix is necessary though, this may add considerable bulk to the journal. |  | 4 |  | 1 | This is an interesting aspect, however, I don't think it's an essential item. Would make an interesting study. |  | 3 |  | 3 | The history may already be published elsewhere. These publications should be referred to with a brief description of this item. | 3 |  | 3 |  | 3 | May have implications about intellectual property and commercialisation. Including source code may make the article very long. | 4 |  | 4 |  | 3 |  | 3 |  | 3 | High-risk alerts of participants that may harm themselves etc. and how these were dealt with (automated, human-intervention etc). | 4 |  |  |  |  | 3 |  | 4 |  | 3 |  |  |  | 3 |  |  |  |  |  |  |  |  |  |  |  |  |  | 3 |  |  |  | 3 | Where and when obtained, any adverse events. | 3 |  | 4 | Detection of harm - if human intervention is require or occurs, how quickly it occurs and the outcome of the harm. |  |  | 4 |  |  | 3 |  |  |  | 4 |  |  | 4 |  | 4 |  |  | 4 |  |  |  | 4 |  |  | 4 |  | 2 |  |  | 4 |  | 3 |  |  | 4 |  |  | 3 |  | 3 |  |  |  |  |  | 3 |  |  | Yes please! 1a-i, 1b-iii, any others needed.
2011-04-16 05:30:11 | REMOVED BY EDITOR | SRI, Menlo Park, California | REMOVED BY EDITOR | no | I have experience with conducting ehealth studies myself, but no RCTs | I have a suggestion.  This approach to "writing" is so routinized, why not just have your authors fill out an extensive checklist and not bother to actually write anything at all?  Make it like the IRS tax forms, just boxes to fill in.  I frankly don't like this approach AT ALL and believe it will simply generate mechanical writing, uncreative thinking, a checklist mentality, and dull science. I should add, I served as one of the lead editors of a new journal through its first decade, so I am not saying this lightly or casually. I simply think it's a bad approach. |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | NO
2011-04-16 05:57:37 | [NAME] | Oslo University College, Oslo, Norway | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports |  | 5 |  | 4 |  | 5 |  |  | 5 |  | 5 |  | 5 |  | 4 |  | 5 |  |  | 5 |  | 5 |  |  |  |  | 5 |  |  | 4 |  | 5 |  | 4 |  |  | 5 |  | 5 |  |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 4 |  | 4 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 4 |  | 5 |  |  |  | 4 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  |  | 4 |  | 5 |  | 4 |  |  |  | 4 |  |  | 4 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 4 |  |  |  | 5 |  |  | 4 |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  |  | 5 |  | 4 |  |  |  |  |  | 5
2011-04-16 06:57:08 | Rajneesh Choubisa | Indian Institute of Technology, Delhi, INDIA | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports | No, I came across the 1996, 2001 and recent 2010 guidelines of CONSORT (Consolidated Standards of Reporting Trials) group.  Reference:  http://www.ncbi.nlm.nih.gov/pubmed/20346624 | 5 | Yes, This gives a clear distinction between the various delivery modalities and help one categorize the e Health intervention.But the problem with most of the people is that the term computer based or internet based or web based bears a similar literal connotation for a layman or a new reader. This classification would enable both the authors and the readers to look precisely into the matter they have been interested. | 2 | It will be self explanatory to have written the heading in such a manner that sub heading is not required to support it. After all, the aptness of the title lies in the topic and it must as a general rule of thumb convey the research work described. | 4 | Yes, I particularly think that the interventions designs vary for the cultural, ethnic and participant groups and since there is plethora of research segmentation in terms of participant groups, mentioning the participants in abstract phrases such as college students, older adults, children, learning disabled, etc. would make the title more precise. | Abstract:  Since the technique is a sophisticated one and cannot be explained briefly, the abstract length should be between 200-300 word instead of 150-200 words. | 2 | Although, as is obvious the methodology varies from one RCT to other in terms of conceptual and theoretical formulation, it is not easy to cover it in abstract. Only superficial information about the components is revealed and the details can be referred in the methodology section. | 5 | Introducing the relevant phrase will be sufficient in abstract section. | 2 | This sub item will be useless as it has already been differentiated in the beginning. | 3 | Now, This cannot be demonstrated if one is using an intervention module consisting of 3-5 components and the delivery design is a longitudinal i.e ranging over a span of time. This is because there would be many statistical values which makes the job difficult to put everything in the abstract section. | 5 | Yes, if in case or by chance the intervention didn't work and the baseline and post intervention data is not significant leading to negative results then it must be mentioned in the abstract. This will help future researchers to not include the constructs which have been insignificant as intervention components.Put simply, they will come to know that the road ahead is a dead end and they will improvise. | A specific format as to how the abstract can be written may be incorporated for a clear and unambiguous understanding. | 5 | Sometimes the components are based on more than two theoretical foundations, means that they can be approached from two different ways leading to a converging path. In order to deal with such issues the segmentation of introduction is advisable.This will give a clearer picture of the problem and the ways in which it has been dealt with.On the other hand segmentation would add more complexity to the overall comprehension. | 3 | This is the portion where the soul of the research lies. It must include every piece of important information. | No Additions. | No comments. | That will be mentioned except for the allocation ratio because the allocation was coded randomized during the programming process (and normally it is not a rule to provide the algorithm) mere mention of the design and allocation ratio would be ideal. | 3 | Yes, This is also a very important problem. But for that the efforts done by the maintenance team cannot be mentioned in the paper as such it is the testing and retesting which makes a website finally functional. | Technical Problem Section can be added which will specify the type of problems encountered and the time duration/expertise to deal with it. Alternatively, A paragraph of 3-5 lines would be sufficient to let the reader know the error typology and the most feasible solution to that particular error. | 1 | It is obvious that only interested and motivated participants who normally are internet savvy and having access to internet will recruit themselves to such studies.In some institutions the concept of free internet is exists which aides in the whole process as most people in an economy like India for example won't go to the cybercafe to use internet, pay for its usage and get themselves enrolled for the program. | 5 | Yes, this is very important. Every minute detail should be provided because the internet behavior of people vary even across a small area. | 4 | The detailed information provided by the participant during recruitment (mostly termed a demographic information) can be tabulated. | Nothing specific remains. | 5 | The psychometric properties and the rationale of choosing the questionnaires should be mentioned. | 3 | Again a distinction between self directed and institutionally mediated trials should be there. | It is well justified and again obvious that a website can be accessed from any part of the world with requite technical support. But suppose if a country or region specific intervention is developed then it will be interesting and useful only for the participants of that locality,community, region, state, or country. | 4 | Of course, this things will go in the acknowledgment section. | 2 | Again , this is the algorithm which is the code that is generally not presented. So a general introduction in the section will be good enough. | 4 | Iterations till the final module become operational will also provide the amount of effort put in by the researcher. But, Ideally what matters is the end product so mentioning the number of rounds/ rectifications/ modifications and the time required for that would be sufficient. | 2 | This is somewhat skeptical as you make the website highly encrypted (means everything will be tested and validated for every user) the attrition will increase owing to the sophistication involved in the technology. This must utilize a lenient criterion which can me made more stringent and participant focused with due course of time. Trust is the virtue which people lack in an online environment. | 5 | Screen shots must be provided (if not the code) as they will help the reader understand how does it look like when it was used by the participants. | 2 | This should remain the sole right of the website administrator.Normally preservation info is deleted over a period of time so the author/s can be contacted. | 5 | Very useful information, It must be tabbed in the beginning during the recruitment process and should be tabulated with maximum details. | 2 | Yes the process of tailoring and other intervention specific details may be discussed in the methodology section. Separate segment would complicate the matters.Each and every technicality related ti interventions should be discussed in the intervention section. | 4 | In the intervention section only, can be showcased with a pictorial representation or a flow diagram. | 5 | In the intervention section. | 4 | In the intervention methodology section. | 4 | May be introduced in a separate sub section or experiment. | - | 4 | This is important and must be mentioned during the discussion on measures. | 4 | if that is applicable to a particular RCT it must need a mention. | 4 | It has to be described and the results of the qualitative analyses can be tabulated to show the trend. | - | Will be obviously mentioned. | 3 | These factors are beyond the control of the researcher.In general, a separate study should be done on this issue. | - | - | Discussed already in the methodology section. | if applicable, should be included in the methodology section only. | Should be mentioned in the Methodology subsection. | It must be specified in the methodology section as to the participants were recruited to different group randomly through a programming protocol or any other protocol. | 4 | Depends as most of the interventions are meant not to be double blind. Mention in methodology section. | 3 | Now it involves skills as to generate a program which wont let the participants guess. mention in the methodology section. | - | Through references in text mainly in discussion part. | 5 | Very important and must be properly tabulated in the results section. | - | If Applicable. | 1 | Between Acknowledgement and Conflict of Interest section.Not needed generally. | 4 | Can be represented as a screen shot. | 3 | Appropriate Section. May be after the discussion. | - | Via a Flow Chart or a pictorial representation. | 2 | Inclusive in the flow chart and the information can be interpreted indirectly.No special mention except of an uncanny significance. | - | 2 | Experiment wise, in the methodology section. only month and year would be enough. | - | Reason should be quoted in the discussion. | 5 | Always beneficial and illustrative. | - | 4 | Already outlined in the Flow chart. Separately mentioning these things would increase the length of the paper. Somethings should be indirectly inferred and interpreted from the tables and figures which should be briefly described in the discussion section. | 2 | Normally, the intent to treat analysis (Gross & Frogg, 2004)  is used in clinical trails, but if there are e health intervention where the foundation is positive psychology or other happiness theory. In that case the treatment intention would be a fallacy.So it should be a point of later thought in some exceptional cases. | - | 3 | Should be clearly mentioned in the Table of results and discussion. | - | Have to be incorporated as per the case.Most useful in case of exact replication otherwise absolute effect sizes would be sufficient. | 3 | Appropriately done. | - | 4 | If they are of magnitude higher than the one fixed by the regulating agency. | 4 | In order to improve the website and make it more user friendly. | - | 4 | Discussion part should be in a detailed manner and must clear all the inherent doubts. | 5 | This will provide direction and the the researcher's wisdom and perspective to the potential new researcher. | - | 5 | Last section should focus on the limitations and delimitation's. | - | 3 | depends from a study to other. Generalization again will be difficult. or ex If a particular study is designed keeping in mind the study habits and achievement of Indian students it cannot be generalized beyond the territories of the the nation in question. | 4 | Yes, the alternative mechanisms should be rightfully mentioned. | - | Optional, depend upon the researcher. Mostly a name is provided like STUDENT WELL BEING ENHANCEMENT PROGRAM, Depression prevention program, Happier program etc. | Contact Authors.Best person to tell about the protocols used. | Whatever the case may be (in acknowledgements) | 4 | As applicable. Vary from journal to journal. | - | 8, 10, 16, 19
2011-04-16 09:13:03 | [NAME] | University of Cape Coast | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs |  | 2 |  | 2 |  | 4 |  |  | 3 |  | 3 |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 4 |  |  |  |  | 3 |  |  | 4 |  | 4 |  | 4 |  |  | 4 |  | 3 |  |  | 3 |  | 3 |  | 4 |  | 5 |  | 3 |  | 3 |  | 5 |  | 4 |  | 3 |  | 4 |  | 3 |  | 3 |  |  | 3 |  | 3 |  | 3 |  |  |  | 5 |  |  |  |  |  |  |  | 3 |  |  |  |  |  | 4 |  |  |  | 3 |  | 4 |  | 3 |  |  |  | 3 |  |  | 3 |  |  |  | 3 |  |  | 3 |  |  |  |  | 3 |  |  |  |  |  |  | 3 |  |  |  |  | 4 |  | 3 |  |  | 4 |  |  |  |  | 3 |  |  |  |  |  | 2
2011-04-16 13:38:35 | [NAME] | Center for Primary Health Care Research, Malmö, Sweden | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs |  | 5 |  | 4 |  | 5 |  |  | 5 |  | 5 |  | 5 |  | 5 |  | 3 |  |  | 5 |  | 5 |  |  |  |  | 5 |  |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 4 |  |  | 4 |  | 4 |  | 5 |  | 5 |  | 5 |  | 4 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 5 |  | 5 |  |  |  | 5 |  |  |  |  |  |  |  | 4 |  | 4 |  |  |  | 5 |  |  |  | 5 |  | 5 |  | 4 |  |  |  | 5 |  |  | 4 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  |  |  | 4 |  |  | 4 |  | 4 |  |  | 4 |  | 5 |  |  | 5 |  |  | 5 |  | 5 |  |  |  |  |  | 5
2011-04-17 11:08:42 | [NAME] | Université Laval, Queébec, Canada | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have read many ehealth RCT/evaluation reports |  | 5 |  | 3 |  | 4 |  |  | 5 |  | 3 |  | 5 |  | 4 |  | 5 |  |  | 5 |  | 5 |  |  | I agree that IRB approval should appear in Methods only |  | 4 |  |  | 4 |  | 5 |  | 5 |  |  | 5 |  | 3 |  |  | 5 |  | 4 |  | 4 |  | 5 |  | 4 |  | 4 |  | 5 |  | 5 |  | 5 |  | 5 |  | 4 |  | 5 |  |  | 5 |  | 5 |  | 4 |  |  |  | 5 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  |  | 4 |  | 4 |  | 5 |  |  |  | 5 |  |  | 5 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  |  |  | 5 |  |  | 4 |  | 4 |  |  | 5 |  | 5 |  |  | 5 |  |  | 5 |  | 5 |  |  |  |  |  | 5 |  |  | No
2011-04-17 12:23:23 | [NAME] | Bezmialem Vakif University, Istanbul, Turkey | REMOVED BY EDITOR | yes | I have experience mainly from a consumer/patient point of view |  | 4 |  | 4 |  | 4 |  |  | 2 |  | 4 |  | 4 |  | 1 |  | 2 |  |  | 5 |  | 4 |  |  |  |  | 5 |  |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  |  | 4 |  | 5 |  | 5 |  |  |  | 5 |  |  |  |  |  |  |  | 3 |  | 4 |  |  |  | 5 |  |  |  | 5 |  | 5 |  | 5 |  |  |  | 5 |  |  | 5 |  |  |  | 4 |  |  | 5 |  | 4 |  |  | 4 |  |  |  | 4 |  |  | 4 |  | 4 |  |  | 5 |  | 4 |  |  | 5 |  |  | 4 |  | 4 |  |  |  |  |  | 4
2011-04-17 18:52:05 | [NAME] | University of Maryland | REMOVED BY EDITOR | yes | I have experience mainly from a policy/implementation/decision-maker point of view |  | 5 | Using standard language facilitates comparison. First part of third sentence is still unclear. | 3 |  | 3 |  | Sample size mentioned in the title. Suggest standardization of terms, such as small (<=100), medium (101-1000), large (1001-10000), very large (10,001-100k), and massive (>100k), for example. | 5 |  | 4 |  | 5 |  | 4 |  | 4 |  |  | 4 |  | 5 |  |  |  |  | 5 |  |  | 2 |  | 5 |  | 5 |  |  | 4 |  | 5 |  |  | 5 |  | 4 |  | 5 |  | 5 |  | 5 | Very important | 5 |  | 4 |  | 4 |  | 5 |  | 3 |  | 5 |  | 5 |  |  | 3 |  | 5 |  | 4 |  |  |  | 4 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 4 |  |  |  | 4 |  | 4 |  | 5 |  |  |  | 3 |  |  | 4 |  |  |  | 5 |  |  | 5 |  | 4 |  |  | 5 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 4 |  | 4 |  |  | 4 |  |  | 5 |  | 4 |  |  |  |  |  | 5
2011-04-17 20:48:19 | [NAME] | University of Sydney, Sydney, Australia | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports |  | 4 |  | 5 |  | 5 |  |  | 4 |  | 5 |  | 3 |  | 1 |  | 1 | I am not sure whether authors have the ability to make this conclusion based on results from an RCT. While we would like to think this is the reason that the intervention did not affect the outcome, this may not be justified. |  | 4 |  | 3 |  |  |  |  | 4 |  |  | 1 |  | 5 |  | 4 |  |  | 5 |  | 1 |  |  | 1 | This should be in the acknowledgement section - not the body of the paper. | 2 | Whilst important, this is often done in a pilot study paper. If no pilot paper can be referenced, then this becomes more important | 5 |  | 4 |  | 1 | I've been involved in several web-based RCTs - and I have no idea what information this subitem is asking me to provide. | 4 |  | 5 |  | 4 |  | 1 |  | 5 |  | 5 |  | 4 |  |  | 3 |  | 1 |  | 5 |  |  |  | 5 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  |  | 4 |  | 5 |  | 3 |  |  |  | 5 |  |  | 4 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 3 |  |  |  | 1 | pragmatic RCTs should reflect real world situations - 'only users' analysis is of limited use and does not reflect how the intervention would perform outside the trial. |  | 3 |  | 3 |  |  | 5 |  | 4 |  |  | 4 |  |  | 5 |  | 4 |  |  |  |  |  | 4
2011-04-17 22:28:28 | [NAME] Labrecque | Université Laval, Québec, Canada | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports |  | 4 | definitions will need to be clearly decribed | 3 |  | 5 |  |  | 5 |  | 3 |  | 4 | this item confounds intervention and outcome assessment. All aspect of describing intervention should be in  1bi | 3 |  | 2 | i would suggest that usually reasons should be discussed in the manuscript and not in the abstract. For some study however a word about reasons many be indicated but i do not think it should be mandatory |  | 5 |  | 5 |  |  |  |  | 5 |  |  | 2 |  |  | This items should be restricted to recruitment : I suggest  Open vs. closed, web-based vs. face-to-face recruitement: Mention how participants were recruited (online vs. offline), e.g., from an open access website or from a clinic with phone or  face-to-face contacts i.e., to what degree got the study team to know the participant. In online-only trials, clarify if participants were quasi-anonymous and whether having multiple identities was possible or whether technical or logistical measures (e.g., cookies, email confirmation, phone calls) were used to detect/prevent these.  Elements about intervention and outcome assessment should be in other items | 5 | essential for external validity |  | 5 |  | 4 |  |  | 5 |  | 4 |  | 5 |  | 5 |  | 2 | source code of software should be available if an open source application was used but what about commercial softwares? | 5 |  | 5 |  | 5 | This is all relevant to a good description of the intervention. | 5 | idem | 5 | idem | 5 | idem | 5 | idem |  | 5 |  | 5 |  | 5 |  |  |  | 5 |  |  |  | within the  trial of some e-health applications randomization and allocation can be done at the same time by computer just at the moment of use of the application (exp.or control) without pre generating an allocation sequence. This should be mentioned specifically because allocation concealement is of course fully respected in that case. |  | see above 8a | see above 8 a | 5 | essentiall for assessors but as blinding of participants to what the are viewing is impossible non blinding of participants is not at all important to mention. | 5 | This could be associated to a form of "blinding" |  |  | 5 |  |  |  | 1 | not sure I understand. Why comment on ethics committee approval apart mentionning it? | 4 |  | 4 |  |  |  | 4 |  |  | 5 | it's a co-intervention |  |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 3 |  | 3 |  |  | 4 |  |  | 5 |  | 5 |  |  |  |  |  | 5
2011-04-18 03:23:57 | [NAME] | University of Southern Denmark | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports |  | 5 |  | 3 |  | 4 |  |  | 5 |  | 4 |  | 4 |  | 5 |  | 5 |  |  | 5 |  | 5 |  |  |  |  | 5 |  |  | 3 |  | 5 |  | 5 |  |  | 5 |  | 2 |  |  | 3 |  | 3 |  | 3 |  | 4 |  | 4 |  | 3 |  | 5 |  | 5 |  | 3 |  | 4 |  | 5 |  | 5 |  |  | 5 |  | 5 |  | 3 |  |  |  | 4 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  |  | 3 |  | 3 |  | 3 |  |  |  | 5 |  |  | 3 |  |  |  | 4 |  |  | 2 |  | 4 |  |  | 3 |  |  |  | 5 |  |  | 3 |  | 4 |  |  | 5 |  | 5 |  |  | 5 |  |  | 5 |  | 5 |  |  |  |  |  | 4
2011-04-18 07:39:48 | REMOVED BY EDITOR | University of Pittsburgh, Pittsburgh, United States | REMOVED BY EDITOR | no | I have read many ehealth RCT/evaluation reports | The committee might consider the RE-AIM framework (http://www.re-aim.org/what-we-do/framework-overview.aspx):  RE-AIM was originally developed as a framework for consistent reporting of research results and later used to organize reviews of the existing literature on health promotion and disease management in different settings.  The acronym stands for Reach, Effectiveness, Adoption, Implementation, and [NAME] which together determine public health impact. Since the original paper in 1999, there have been approximately 100 publications on RE-AIM by a variety of authors in fields as diverse as aging, cancer screening, dietary change, physical activity, medication adherence, health policy, environmental change, chronic illness self-management, well-child care, eHealth, worksite health promotion, women's health, smoking cessation, quality improvement, weight loss, diabetes prevention, and practice-based research.  More recently, RE-AIM has been used to translate research into practice and to help plan programs and improve their chances of working in "real-world" settings. The framework has also been used to understand the relative strengths and weaknesses of different approaches to health promotion and chronic disease self-management-such as in-person counseling, group education classes, telephone counseling, and internet resources. The overall goal of the RE-AIM framework is to encourage program planners, evaluators, readers of journal articles, funders, and policy-makers to pay more attention to essential program elements including external validity that can improve the sustainable adoption and implementation of effective, generalizable, evidence-based interventions. | 5 |  | 3 |  | 5 |  |  | 5 |  | 4 |  | 4 |  | 3 |  | 4 |  |  | 5 |  | 4 |  |  |  |  | 5 |  |  | 3 |  | 4 |  | 4 |  |  | 4 |  | 3 |  |  | 3 |  | 5 |  | 4 |  | 4 |  | 5 |  | 5 |  | 5 |  | 5 |  | 4 |  | 4 |  | 4 |  | 4
2011-04-18 08:33:07 | Tuomas Lehto | University of Oulu, Oulu, Finland | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have read many ehealth RCT/evaluation reports |  | 5 | This subitem addresses a very important issue. The terminology used in the eHealth RCTs and other studies has been rather inconsistent and even misleading. | 4 | The more substantial the role of the non-web-based component the more essential it is to mention it in the title. | 5 | Presenting the user context is highly relevant. |  | 5 | As large amount of the related articles fail to explicate the key functionalities, components and underlying theories, and even the intended change type (e.g. attitude change/behavior change/compliance), too many questions remain unanswered. This is an adverse “black box” approach. Thus, it is highly essential to explicate these concisely already in the abstract. | 4 |  | 4 |  | 4 | Participant enrollment/assessment data is highly important. Primary/secondary outcomes are equally important. | 3 |  |  | 5 | Revealing the intent and the overall goal of the intervention is crucial. | 5 | This is a basic requirement. How does the study contribute to the current body of knowledge? |  |  |  | 4 | Major bug fixes and changes in the functionality and/or content should be obviously taken into account when reporting. |  | 4 |  | 4 |  | 4 |  |  | 5 |  | 4 |  |  | 5 |  | 4 | The history of the development process should be concise. | 4 | This subitem is closely related to 3b-i. | 3 |  | 5 | Screenshots and flowcharts would be the minimum requirement. Screen-capture video would be a definite plus. However, it may be unrealistic to request for the source code to be publicly shared.   Also, the screenshots and videos might not be very useful if they are in foreign language. | 5 | This is more crucial than sharing the source code; providing an access to the implemented intervention. The intervention sites tend to disappear very quickly. | 5 | Describing the use/usage context is a necessity. Providing "Backdoor" login for evaluation purposes would be highly useful. | 5 | There is considerable variation in ways of reporting Web-based interventions. A major weakness in the published articles is that the actual implemented information systems have not been described in sufficient detail. For instance, neglecting to describe the actual use/user/technology context is a foible in Web-based intervention research thus far. A high abstraction level makes it strenuous for readers to grasp what kind of interaction under what circumstances really takes place through the system, and to what extent the potential change may be due to the intervention. | 3 |  | 3 |  | 5 | Reminders/prompts are persuasive elements, potentially affecting the use of the intervention. Thus, reporting the use of such reminders would be advantageous. | 4 |  |  | 4 |  | 4 |  | 4 |  |  |  | 3 |  |  |  |  |  |  |  | 3 |  | 4 |  |  |  | 4 |  |  |  | 3 |  | 3 |  | 3 |  |  |  | 5 |  |  | 4 | related to 3b-i and 5iii |  |  | 4 |  |  | 3 |  | 3 |  |  | 4 |  |  |  | 3 |  |  | 4 | related to 3b-i; 5iii and 14a-i | 3 |  |  | 5 |  | 4 |  |  | 4 |  |  | 4 |  | 3 |  |  |  |  |  |  |  |  | 1b-i, 5
2011-04-18 08:37:36 | [NAME] | Democritus University of Thrace, Alexandroupoli, Greece | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have read many ehealth RCT/evaluation reports | STARE-HI guidelines Reference: Talmon, J., Ammenwerth, E., Brender, [NAME], [NAME], [NAME], Nykanen, P., Rigby, M. (2009).  STARE-HI – Statement on reporting of evaluation studies in Health Informatics, International Journal of Medical Informatics, vol. 78,1-9 Endorsed by: European Federation of Medical Informatics (EFMI), International Medical Informatics Association (IMIA) and American Medical Informatics Association (AMIA)  Description: The objectives of STARE-HI (STAtement on the Reporting of Evaluation studies in Health Informatics) are to provide guidelines for writing and assessing evaluation reports in Health Informatics; and by doing this to improve the quality of published evaluation studies in Health Informatics; and thus to improve the evidence-base of Health Informatics. The STARE-HI guidelines include a comprehensive list of principles relevant for properly describing  Health Informatics evaluations, and they recommend a structured list of items that should be included in Health Informatics evaluation reports. These items are organized in various categories: (1) title; (2) abstract; (3) keywords; (4) introduction; (5) study context; (6) methods; (7) results; (8) discussion; (9) conclusion; (10) authors’ contribution;  (11) competing interests; (12) acknowledgments ; (13) references; and (14) appendices. These primary categories include more detailed items. | 4 | A list of all suggested "reserved" terms should be available and clearly defined (if possible in different languages). Provision for new "reserved"well-defined terms should be made, to account for technology evolution and innovation. Names of products/brands should be mentioned in the case an application is exclusively designed for a specific product/brand. | 3 | Innovative co-interventions should be mentioned - not sure if one needs to list all implicit conventional aspects of the intervention in the title. | 5 | Porbably use well-accepted controlled vocabulary/ontology for condition (e.g. as an API to the NCBO BioPortal collection), and preset the desired degree of detail in indetifying the condition and/or target group. This last one might be tricky... |  | 5 |  | 2 | I find the degree of human involvement a very subjective and qualitative issue. It may create noise. | 4 |  | 5 |  | 5 | Probably a few subitems to coherently/objectively describe nature of negative results and reasons. |  | 5 |  | 5 | Could further structure the type of the study along different axes: e.g. (a) ability-efficiency-effectiveness-understanding, (b) reason of system deployment (mandatory change, automation, added-value, decision support, infrastucture upgrading, strategic intervention, business transformation/re-engineering, etc |  |  |  | 3 | Very importnat, but rather subjective, thus it may add to noise. |  | 5 | Computer/internet literacy (reported in detail) is very important. Most interesting results/outcomes may be due to involving non-literate participants, or an equal mix of various degrees of literacy etc, and this should be explicitly stated. | 5 |  | 5 |  |  | 5 |  | 4 |  |  | 5 |  | 5 |  | 5 |  | 4 |  | 3 | Publishing the source code may be tricky and it depends on IPRs. Providing access to executables or services, together with detailed screenshots etc. is more appropriate. | 4 | Important, but again one has to resolve and accommodate IPR and 'software plagiarism' issues. | 5 |  | 3 |  |  |  | 4 | Clear explanation is required, as this item can be subjective. | 3 |  | 5 |  |  | 3 | Needs futher explanation, as there is an overlap between CHERRIES items and other items and the proposed work. | 5 |  | 5 |  |  |  | 3 |  |  |  |  |  |  |  | 5 |  | 5 | Unlike medication, when technology intervention is concerned, it may not be very easy to hide the comparator.... |  |  | 4 |  |  |  | 5 |  | 5 |  | 4 |  |  |  | 4 |  |  | 4 |  |  |  | 5 |  |  | 5 |  | 3 | Intent-to-treat implies using the technological intervetion as a drug. However, an other interesting approach is to see the intervention as a technical/managerial intervention. And a most interesting approach would be to see it as a social system directly influencing and being influenced by its placement in a social group. |  | 4 |  |  |  | 4 |  |  | 4 |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  |  | 5 |  | 5 |  |  |  |  |  | 5
2011-04-18 14:54:57 | REMOVED BY EDITOR | University of California, Berkeley | REMOVED BY EDITOR | no | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 4 | Include numbers in the denominator, so can clearly see whether it was a true intent to treat. | 5 | If trial is negative, include power to detect the outcome, given the numbers obtained. |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 5 | I suggest this should contain an explicit admonition: "Do not use the term 'intent to treat' unless all initially-enrolled subjects are included in the analysis". (I've seen JMIR papers that failed this.) And then specify how the missing outcome variables were imputed -- e.g., impute zero change? |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | no
2011-04-18 17:12:17 | [NAME] | University of Oslo, Oslo, Norway | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 4 |  | 5 |  | 5 | . | 4 |  |  | 5 |  | 5 |  |  |  |  | 3 |  |  | 3 |  | 4 |  | 5 |  |  | 5 |  | 3 |  |  | 5 |  | 4 |  | 2 |  | 4 |  | 4 |  | 5 |  | 5 |  | 4 |  | 4 |  | 4 |  | 2 |  | 4 |  |  | 5 |  | 3 |  | 5 |  |  |  | 5 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  |  | 5 |  | 5 |  | 5 |  |  |  | 4 |  |  | 4 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  |  |  | 3 |  |  | 5 |  | 4 |  |  | 5 |  | 3 |  |  | 4 |  |  | 3 |  | 4 |  |  |  |  |  | 5 |  |  | 21, and sample side calculations
2011-04-18 18:06:23 | Ming-Yuan Chih | University of Wisconsin Madison, [NAME], WI, USA | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have experience mainly from a consumer/patient point of view | No. | 5 | It is important for reader to easily identify the mode of delivery in order to find the most relevant RCT.    It might be a good idea to have more formal and detailed definitions about the different modes of delivery. As in the current descriptions, it can be the combination of modes, mobile vs. non-mobile, web vs. non-web, electronic/computer vs. internet, and/or game vs. other applications. However, the definition needs to be not only specific but also general enough to adopt new technology advancements. This is a hard one. | 5 | This is very important to include other components or important co-interventions as those may have have impacts on how participants use the eHealth interventions. For example, the telephone support may related to higher and longer use of eHealth intervention because the participants may less likely to withdraw from the study because of the difficulties encountered in using the intervention. | 5 | It is important to know the target population and disease. | No. | 5 | It is important to include the key features of the intervention in abstract. However, as some interventions may offer a variety of "key features", it may be too much to describe even the simplified or summarized version of it. My suggestion would be to keep this point as abstract as possible and leave the more detailed description in the text. | 5 | This points seems to somewhat overlap the next point. It is important to mention a few words to highlight human involvement in the intervention. | 3 | It may be better to clearly identify the different modes (self-assessment vs. face to face) in different stage of the trials, such as in recruitment or trial, because of different assessment purposes and influences in the trial.   However, this point seems to be not as important to be included in the abstract as the other points above. The self-assessment vs. face to face assessment may have implications in terms of validity and reliability. This information seems to be more applicable and feasible to leave in the text rather than must-to-be-included in the abstract. | 5 | This is a very interesting and important point. The use data would be very helpful for readers to get a quick sense of how the intervention was actually implemented/used. As eHealth intervention often suffer from the law of attrition, it would be a good idea to include some general use data. However, keep the use data reporting more general, such as attrition or overall use, than specific, the use of particular components. | 5 | This may be the most important point so far in the abstract section. The eHealth intervention trial is different from traditional medical or pharmaceutical RCT in its changing nature in adoption, adherence, use behavior, impacts on outcomes, and inability to control influences from other factors. It would be important lessons learned even if the results are not as desired. | No. | 5 | Good point. | 5 | Good point. | No. | No. | No. | 5 | Good point. |  | 5 | Without reporting a formal evaluation of computer/internet literacy, it could be a pretty general statement. Reporting whether the system usability and content reading level would match the participants' computer/internet literacy are probably more useful. | 5 | Good point. | 5 | Good point. | No. | 5 | Good point. | 5 | Good point. | No. | 5 | Good point. | 5 | Good point. | 5 | Good point. | 5 | Good point. | 5 | Good point. | 5 | Good point. | 5 | Good point. | 5 | Good point. | 5 | This is an important point but may very likely to be hard to do especially for a new technological intervention where we don't have much previous data to rely on. However, an estimated "intended dose" was important as the baseline of comparison. | 5 | Here we talk only the human involvement in intervention phase.  Good point. | 5 | The prompts/reminders should be part of the intervention but sometimes, it can be an after intervention efforts to overcome low use situation. Therefore if these prompts are related to implementation issue, it is important to report the frequency and the starting points of prompts/reminders. If these prompts are integral part of the intervention, such as high symptom alerts to physicians, it is important to distinguish this intervention prompts from the previous implementation prompts. | 5 | Good point. | No. | 5 | Good point. | 5 | Good point. | 5 | Good point. | No. | No. | 5 | It is not clear to me what the expected attrition means here. Does it mean participants' drop out before reaching outcome time points, having missing surveys or not using the intervention?  It would be great if the authors can define what the expected attribution are in the study and then report according to this point. | No. | No. | No. | No. | No. | No. | 5 | Good point. | 5 | Good point. | No. | No. | 5 | Good point. | No. | No. | 5 | No. | 5 | Good point. | 5 | Good point. | No. | No. | 5 | Adding use component is important. However, if this diagram or information could be related to the expected intended dose somehow, like a threshold line or number of users who reach the intended dose could be very helpful for readers to better interpret the results. | No. | 5 | Good point. However, if these events have been described in the "Method" section, it is probably a redundant to mention it again in the "Results" section here. | No. | No. | 5 | Good point. | No. | 5 | Good point. | 5 | Good point. | No. | 5 | Good point. | No. | No. | 5 | Good point. If a subgroup analysis of comparing only users is reported, proper analytic procedures should be included in "Method" section too, | No. | 5 | Any adverse events should be reported. It is important especially when people using communication technology may be in a greater risks of any assaults from interacting with others through the intervention. | 5 | It is very important as descriptive feedback may be the important improving points. | No. | 5 | Good point. | 5 | Good point. | It would be very useful if researchers could emphasize and elaborate on their lessons learned in the "design" and "implementation" of the intervention based on both primary and process outcome findings as well as unintended outcome findings. This information would be not only important for readers to see the actual picture of the intervention effects but also very beneficial to future researchers and developers so that the same lessons won't need to be learned again. | 5 | Good point. | No. | 5 | Good point. | 5 | Good point. |  | No. | No. | No. | 5 | Good point. | No. | 22, 17a-i (use measure), 5-x,5-xi,19, or 4a-i.
2011-04-19 16:04:53 | [NAME] | NST, Tromsø, Norway | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs |  | 5 |  | 5 |  | 5 |  | If it is a controlled trial, it would be useful to report the condition of the controlled arm of the study. | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  |  | 4 |  | 5 |  |  |  |  | 5 |  |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 5 |  | 5 |  |  |  | 5 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  |  | 5 |  | 5 |  | 4 |  |  |  | 5 |  |  | 5 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  |  |  | 5 |  |  | 5 |  | 3 |  |  | 4 |  | 3 |  |  | 5 |  |  | 3 |  | 3 |  |  |  |  |  | 5 |  |  | 2021-05-11 00:00:00
2011-04-21 05:26:57 | Hilde Eide | Buskerud University College, Drammen, Norway | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports | I think it would be usefull to refer also to development of complex interventions.  However this CONSORT statement relates to RCT-trials, many of the aspects sugegsted here can also be used for presenting open studies and pre-trial studies  [NAME], Dieppe P, [NAME], [NAME], [NAME] I, [NAME] and evaluating complex interventions: the new Medical Research Council guidance.  BMJ. 2008 Sep 29;337:a1655. doi: 10.1136/bmj.a1655. www.mrc.ac.uk/complexinterventionguidance | 5 | I think this is very important. We have struggled to find the right term(s) son this is very helpfull | 2 | The title can be to long - many journals want short titles. | 4 |  |  | 5 |  | 5 |  | 4 |  | 3 | It depends on the research question | 4 |  |  | 5 |  | 5 |  |  | I agree - ethics belong in the method section ift  not a part of the research question(s) |  | 4 |  |  | 2 | It can also be an improtant part of the research question - | 5 |  | 3 | This is a radical suggestion - interesting - editors could object to this |  | 4 |  | 2 |  |  | 5 |  | 5 |  | 5 |  | 4 |  | 3 | This depends on the research question. | 3 | Good suggestion - but perhaps difficult to do? | 5 |  | 3 | Depending on what is the actual outcome. Can be a very space consuming demand | 4 |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 4 |  | 4 |  |  |  | 4 |  |  |  |  |  |  |  | 1 | I think the notion of blinding should be questioned.  In small research groups it s difficult to have other persons analyzing the data blind. If correct methods are used - blinding skould have no influence on the data. The transaprency of the research - and a possibilty to reanalyse the data if in doubt and furthermore the obligation of truth skould be enough. | 4 |  |  |  | 5 |  |  |  | 3 | depends if it is relevant | 3 | see above | 3 | see above |  |  | 5 |  |  | 5 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  |  |  | 4 |  |  | 5 |  | 3 | Depends on the research question. These aspects can also be reported in separate publications - as this kind of studies need a separate methodology. |  | 3 | In some journals the research question(s) is(are) not restated in the discussion as it is regarded as known | 3 | Is this not redundant? |  | 3 | Also redundant? However nive to have in a list as many reporting  e-health stuies do not have a substantial social science research methods education |  | 3 | same as above | 3 | same as above |  |  |  |  | 5
2011-04-22 18:59:09 | [NAME] | University of Newcastle, Newcastle, Australia | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have experience mainly from a policy/implementation/decision-maker point of view |  | 4 | The sentence beginning with Use "Internet-based" is confusing. | 4 |  | 5 |  |  | 4 |  | 3 |  | 5 |  | 2 | Use/uptake data can be quite complicated to report, especially for a multi-component trial, and is not essential in the Abstract. | 3 |  |  | 4 |  | 4 |  |  |  |  | 3 |  |  | 3 | Trials often have multiple sources of recruitment (which should be clearly specified) - so, it may also be useful to encourage authors to report on the steps taken to reduce dependence on a particular modality - and to be less influenced by literacy issues. | 5 |  | 4 | I would suggest that the authors are encouraged to make the informed consent documentation available (but not required to publish it as an appendix). |  | 4 |  | 4 |  |  | 3 |  | 3 |  | 3 |  | 3 |  | 3 | Flowcharts of the algorithms used may be useful, but publication of the source code should not be a requirement - and may not be possible - depending on who owns rights to the code, etc. | 3 | Digital preservation is sensible, but the precise methods may be difficult to specify. | 4 |  | 4 | A good idea - but these requirements need to be simplified. | 3 |  | 3 |  | 3 |  | 3 |  |  | 3 |  | 3 |  | 3 |  |  |  | 3 |  |  |  |  |  |  |  | 3 |  | 2 | It is useful to know the "real world framework" that was used to "sell" the study, whether or not the participants were aware of the various conditions. |  | I agree - drop it from the checklist. | 4 |  |  |  | 3 |  | 3 |  | 3 |  |  |  | 3 |  |  | 3 |  |  |  | 3 |  |  | 4 |  | 3 | I feel that too much emphasis is placed on the primacy of intent-to-treat analyses. A range of approaches should be considered by each researcher, within which intent-to-treat should also be examined, but with a greater overall focus on the robustness of the findings to variations in analytical methods. |  | 3 |  |  | Once again, the real issue is the robustness of the findings to variations in analytical methods. | 3 |  |  | 3 |  | 3 |  |  | 3 | Clear linkages between study outcomes and study questions is the real issue - not "restatement" of questions. | 3 |  |  | 4 |  |  | 3 |  | 3 | Plus information about program availability, restrictions and costs. |  |  |  |  | 3
2011-04-24 23:17:40 | [NAME] | Chinese University of Hong Kong, Hong Kong, China | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs |  | 5 |  | 4 |  | 5 |  |  | 5 |  | 5 |  | 5 |  | 4 |  | 5 |  |  | 5 |  | 5 |  |  |  |  | 3 |  |  | 5 |  | 5 |  | 4 |  |  | 5 |  | 5 |  |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 4 |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 4 |  | 5 |  |  |  | 5 |  |  |  |  |  |  |  | 5 |  | 3 |  |  |  | 5 |  |  |  | 2 |  | 4 |  | 4 |  |  |  | 5 |  |  | 5 |  |  |  | 5 |  |  | 3 |  | 4 |  |  | 3 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  |  | 5 |  | 5 |  |  |  |  |  | 5
2011-04-25 09:49:02 | [NAME] | Weill Cornell Medical College | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports, I have experience mainly from a policy/implementation/decision-maker point of view | CONSORT is the primary one. | 5 |  | 3 | Could reasonably be relegated to the abstract in interest of brevity | 5 |  |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 5 |  |  | I concur that IRB/ethical issues should appear in Methods, not Intro. |  | 5 |  |  | 4 |  | 5 |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  | 5 | Can be accomplished very briefly - deep detail not needed. | 5 |  | 5 |  | 5 | I support the publication of screen shots, video, and algorithm flowcharts, but do not think the publication of full source code is essential. | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  |  | 4 | Depends upon importance of the online questionnaire. Short questionnaires to assess acceptability and usability may not require full reporting/validation. | 5 |  | 5 |  |  |  | 5 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  |  | 5 | I find this wording confusing. It is essential to report whether IRB approval was obtained, but I don't think that "comment on ethics committee approval" is needed in addition. | 5 |  | 5 |  |  |  | 5 | Strongly support this. Can be published as electronic supplement if necessary. |  | 5 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  |  |  | 4 |  |  | 5 |  | 4 |  |  | 5 |  | 4 |  |  | 4 |  |  | 4 |  | 5 |  |  |  |  |  | 5
2011-04-25 23:39:35 | [NAME] | Monash University | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports, I have experience mainly from a consumer/patient point of view, I have experience mainly from a policy/implementation/decision-maker point of view |  | 5 |  | 5 |  | 4 |  |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 5 |  |  |  |  | 5 |  |  | 5 |  | 5 |  | 5 | Briefly! |  | 5 |  | 4 |  |  | 5 |  | 5 | Brief is fine | 5 |  | 5 |  | 5 | May be an appendix | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  |  | 5 |  | 5 |  | 5 |  |  |  | 5 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  |  | 5 |  | 3 |  | 5 |  |  |  | 4 |  |  | 5 |  |  | EHEALTH-specific additions here: Secular events, political regime change in the instance of countries choosing to withdraw from the Internet. | 5 |  |  | 5 |  | 4 |  |  | 5 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  | 4 |  |  | 5 |  |  | 4 |  | 5 |  |  |  |  |  | 5 |  |  | 12a, x26
2011-05-04 00:06:14 | [NAME] | Université Laval | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs |  | 5 | The title should also contain the term "social media" if a web-based intervention is delivered using a social media application. | 5 |  | 5 |  |  | 5 | I think we should specify what the word site means. Web "site"? | 5 |  | 5 |  | 5 | Total population from which participants are sampled (i.e. all eligible participants) is important to include as well. | 5 |  |  | 4 |  | 5 |  |  |  |  | 5 |  |  | 5 |  | 5 |  | 4 |  |  | 5 |  | 3 | I don't think this would threaten the validity of a study. |  | 5 |  | 5 | The cost of developing a system would be an important factor to include. | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 |  | 5 | The same description of human involvement should be described in the comparison intervention. | 5 |  | 5 |  |  | 3 | A better description of what the CHERRIES items are should be included. | 5 | A better definition of the behaviour of "using" a web-based intervention is an essential component as with wikis for example, people can use wiki content, contribute content to a wiki, edit existing content within a wiki. I think we should find a better way of defining each specific behaviour related to "using" different e-health interventions as the work "using" can contain many different smaller behaviours essential to highlight and understand. | 4 |  |  |  | 5 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  |  | 5 |  | 3 |  | 4 |  |  |  | 5 |  |  | 5 |  |  |  | 5 |  |  | 5 | N eligible to be exposed to intervention should also be indicated. | 5 |  |  | 5 |  |  |  | 5 |  |  | 5 |  | 5 | This will give an insight into how an intervention worked or did not work. Ideally, these comments should be guided by a behaviour change theory that would allow to understand the determinants of a successful or unsuccessful intervention. |  | 5 |  | 4 |  |  | 5 |  |  | 5 | Discuss elements of sustainability as well. | 5 |  |  |  |  |  | 5 |  |  | 6a-ii
2011-05-05 01:01:29 | [NAME] | Department and Institute of Psychiatry, São Paulo University Medical School | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have experience mainly from a policy/implementation/decision-maker point of view |  | 5 | All acronyms should be defined by the consort also | 5 |  | 5 | All acronyms should be defined by the consort |  | 5 | Theories and principles of design could be cited in the abstract - but any kind of description or comparisons should be made at the introduction, methods or discussion. | 4 |  | 5 |  | 4 | Secondary outcome should be described at result section only | 4 |  |  | 3 | The key focus should explore the context/condition where the problem or type system/solution, descriptions just when necessary | 4 |  |  |  |  | 5 |  |  | 5 |  | 5 |  | 4 |  |  | 5 |  | 4 |  |  | 5 |  | 4 | That item could be an appendix also My fear is the paper could become too big | 5 |  | 4 |  | 4 |  | 3 | Is is possible to create an space in the ISRII especially for that? Once the the intervention change, they could moved to the ISRII, as library  Every time that someone wants a reference, they have a place to start. | 5 |  | 3 | Another item that could be included in an appendix | 4 |  | 5 |  | 4 |  | 5 |  |  | 4 |  | 4 |  | 5 |  |  |  | 3 |  |  |  |  |  |  |  | 5 |  | 5 |  |  |  | 5 |  |  | Subgroup analysis should be carefully evaluated, specially when originated from post-hoc analysis. In general they should be avoided. | 3 |  | 5 |  | 5 |  |  |  | 5 |  |  | 4 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 5 |  |  | Emphasis to use NNT and NNH- number needed to treat and number needed to harm whem possible | 5 | They should beadressed as secondary outcome and not be described in the abstract |  | 4 |  | 4 | Another item that could part of an appendix when it is not the main outcome. |  | 5 |  | 4 |  |  | 5 |  |  | 4 |  | 4 |  |  |  |  |  | 4
2011-05-07 22:37:48 | [NAME] | University of New South Wales, Sydney, Australia | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports, I have experience mainly from a consumer/patient point of view | 2010-CONSORT | 5 |  | 3 |  | 3 |  | Need to mention primary outcomes in title. | 4 |  | 4 | To achieve this, we need clear definitations on what each term means (e.g. open, closed, automated, human- assisted etc) | 4 | To achieve this, we need clear definitations on what each term means (e.g. open, closed, automated, human- assisted etc) | 4 | Also need to mention drop-out rates, no. of participants recruited/met inclusion criteria. | 4 |  | Need to add setting and for how long the trial was conducted, how many arms in the trial, whether randomisation achieved balance of participants between the arms | 4 | perhaps also mention the rationale for this sytem, and its intended use; e.g. is it intended for participants to use this on a daily basis or once off. | 4 | also need to say how common is the comparator being used in usual care; e.g is the comparator a gold standard, does it have active ingredients/evidence-based material to influence the primary outcome, or a typical control intervention that field uses | need to clearly state what gap this study addresses, and how it contributes to knowledge |  | need to mention whether there's follow-up, and for how long | 4 | also need to mention whether participants reported adverse events | perhaps outcome strategy on maintaining adherence rate? | 3 | how would one define levels of internet/computer literacy though?  are there standards to say that?  how about health literacy or general language literacy? | 4 | pls refer to comment from earlier related item. | 4 |  | Would be good to highline strategy for adherence | 4 | Need to mention whether self-reports are validated with other sources, e.g. clinical audits etc | 3 |  |  | 4 |  | 3 |  | 3 |  | 3 | But need to provide examples of quality assurance methods | 3 |  | 3 |  | 4 |  | 4 |  | 3 |  | 4 |  | 4 |  | 4 |  |  | 4 |  | 5 |  | 4 |  |  | esp. if participants reported adverse events | 4 |  |  |  |  |  |  |  | 3 |  | 3 |  |  |  | 3 |  |  |  | 5 |  | 4 |  | 3 |  |  |  | 3 |  |  | 3 |  |  |  | 4 | table on demographics is important but unsure about digitcal divide |  | 4 |  | 4 |  |  | 3 |  |  |  | 3 |  |  | 3 |  | 4 |  |  | 4 |  | 4 |  |  | 3 |  |  | 3 |  | 3 |  |  |  |  |  | 3 |  |  | any item, but probably prefer those that I've placed comments
2011-05-17 10:09:59 | [NAME] | Joanneum Research, Graz, Austria | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports | The Cochrane EPOC (Effective Practice and Organisation of Care) Group has created many reviews and meta-analyses which are supported by IT / eHealth. (e.g. The effects of on-screen, point of care computer reminders ...)  I am not directly involved but have read many of them.  The group for example has created a taxonomy of quality improvement strategies which is a good resource for finding a common terminology. | 5 |  | 3 |  | 4 |  | The type of study,  (e.g. ... - a cluster-randomized trial) (That's already mentioned in the 1a title, but not present in the subitems...) | 5 |  | 3 |  | 4 |  | 4 | Use data have to contain - how many used the system once - how many completed the relevant steps - how many were eligible for data analysis - how many were lost for follow-up There is huge potential for selection bias otherwise! | 3 |  | Integration into existing care practice? - Was the system integrated into an existing care model (e.g. diabetes care centres) - What were the most important changes to the previously existing care practice? | 5 |  | 5 | In "what is known about the (type of) system" and "what is the motivation for the study" distinguish effects on - process improvement (e.g. better therapy adherence) - short/intermediate term outcome improvement (effects on HbA1c, Blood pressure) - long-term outcome improvement (reduced mortality, long-term complications such as like stroke...) - patient outcomes: satisfaction, quality of life - cost-effectiveness |  |  |  | 4 |  |  | 4 |  | 5 |  | 4 |  |  | 5 |  | 4 |  |  | 4 |  | 3 |  | 3 |  | 4 |  | 4 |  | 3 |  | 4 |  | 5 |  | 4 |  | 4 |  | 4 |  | 5 |  |  | 4 |  | 4 |  | 4 |  |  |  | 4 |  |  |  |  |  |  |  | 3 |  | 3 |  |  |  | 5 |  |  |  | 3 |  | 3 |  | 3 |  |  |  | 5 |  |  | 4 |  |  |  | 5 |  |  | 5 |  | 5 |  |  | 4 |  |  |  | 3 |  |  | 4 |  | 4 |  |  | 5 |  | 3 |  |  | 4 |  |  | 5 | Discuss "compatibility" of the healthcare system - which changes to the healthcare system would improve use/effectiveness of the eHealth intervention? | 5 |  |  |  |  |  | 4
2011-06-08 04:42:19 | [NAME] | University of Oxford | REMOVED BY EDITOR | yes | I have experience with conducting ehealth studies myself, but no RCTs, I have read many ehealth RCT/evaluation reports | We are currently working on guidelines for psychological and social interventions and have a list of relevant studies that we would be happy to share. | 3 | The general idea is sound (i.e. the type of programme should be clear in the title) but I don't think this is specific to e-health.  I agree that each part of a PICOT question should be clear and that RCTs titles should resemble titles of Cochrane reviews.  The particular terms mentioned haven't been important in my work; I don't feel 'web' is less ambiguous than 'internet'.  Perhaps it's more important that the abstract says something about the programme components?  For reviewers, it's easy to search for 'internet' or 'web'.  Perhaps these suggestions could be mentioned in the text but not included in the checklist? | 2 | I support use of PICOT format, but perhaps not all co-interventions need to appear in the title.  Imagining how this might look in my field, I think the following would be a realistic but excessively long title.    A randomised controlled trial of web-based cognitive behavioural therapy with email support and an online forum for generalised anxiety disorders in adults: a six month follow-up | 4 |  | As above, use of PICOT format makes sense.  If you're going this route, consider adding the comparison and outcomes (if not the time point). | 4 | I like the modification.  The wording will require some discussion.  For web-based cognitive therapy, you might want to know that the treatment includes cognitive restructuring, behavioural experiments, etc. and also that the programme is interactive, provides feedback about progress, etc.  That is, general treatment components and technical features are both important.  Our group has done some work about reporting the implementation of interventions in systematic reviews, which I would be happy to share. | 5 | Excellent item.  This requires standardised reporting and is a key problem in interpreting and synthesising studies in this field.  If the purpose of this Extension is to tailor items to e-health rather than to improve CONSORT more generally, this sort of item is key. | 3 | Perhaps it needs to be in the text but not the abstract? | 5 | To avoid reporting bias in abstracts, I think this is particularly important for negative/non-significant findings.  I have just been asked by a major medical journal to remove non-significant results from an abstract of a systematic review; it may be necessary to convince people that this is important. | 2 | This seems important in a discussion section, but not in all abstracts. |  | 2 | Background sections are often highly selective. I'm don't think this item solves that problem. It would be helpful for authors of RCTs to reference systematic reviews, etc.  This seems to be moving in the direction of Cochrane structure for systematic reviews (description of the condition, description of the intervention, how the intervention might work), which I find useful, so perhaps the idea of more uniform structure is a good one.  More particularly, authors should be discouraged from reporting 'goals' and encouraged to report what they actually did.  As a reader, I'm less interested to learn that 'email contact aimed to...' and more interested to learn that 'on average, there were 8 reciprocal emails contacts'.  This information should appear in the methods and results (bits often appear in introductions and discussions) and should not be confused with what was intended but not delivered. | 2 | As above |  |  | How will this guideline relate to extensions for non-inferiority, cluster, etc.? | 2 | The broader point about implementation details is not unique to e-health, and could be expanded.  I agree that these specific issues should be considered, but I think the current wording covers these sorts of changes. | I'm not sure where to put this, but I often read a few trials about the same 'intervention' (e.g. Beating the Blues).   Whist it's likely that such programmes evolve over time, differences between interventions across trials are rarely mentioned. | 3 | If participants are recruited online, trialists may apply no restriction (and the results will generalise to people on the internet/particular websites).  If participants are recruited by newspaper advertisements or through a GP office, the trialists may or may not exclude people based on reported computer literacy, and the results will have corresponding external validity.  This is obviously important and should be reported, but I think the issue could be clarified in text - I'm not sure this item adds much. | 5 | This is an important item as CONSORT does not include enough information about external validity. | 4 | I don't think this is the right wording, but I agree that more information is required about recruitment processes. | I suggest extending the flowchart to begin with 'participants approached' so the recruitment process is clear. | 5 | Entirely agree.  Methods of data collection are very often inadequately reported. | 2 | I think this might be important.  I don't think there's sufficient evidence to add this as an item, but I'm willing to be convinced.  (Also perhaps not specific to e-health?) | Where should authors include a link to their website, informed consent process, etc.? | 3 | If people are in contact with participants, this is important because it tells us something about external validity/how to reproduce the intervention.  If the product is a stand-alone website, the characteristics of the developers seem less relevant. | 3 | I broadly support more information about the design, delivery and uptake of interventions.  This item seems relevant, though I would still prefer more information about *what it is* (i.e. using language that would allow reproduction) rather than vague statements about a process that cannot be reproduced. | 4 | There are too many concepts in this item. | 2 | I don't understand this item.  Is this data validation or oversight of people running the site? | 5 | Yes!  Many online and computerised interventions are commercially owned, so this will be difficult, but I think greater transparency is much needed. | 5 | Could this be linked with 5-v as a single item? | 4 | Perhaps mention if they had to be on a particular network (e.g. university). | 5 | This is all good, but far too much for a single item. | 3 | And any restrictions (e.g. participants could not access module 2 until week 2). | 5 | Absolutely essential, and badly reported in the vast majority of studies about web-based treatments for mental health problems. | 5 | Again, absolutely essential. | 4 | I'm not sure I understand the item and how it differs from some of the other items (e.g. human support).  Perhaps there are part of a broader concept? | Security arrangements for data storage and access might affect disclosure.  Technical requirements (e.g. need for broadband) might limit some home users (e.g. forcing people to use the intervention at the office or not at all).  Not necessary to add these as items, but perhaps think about the problems in the wording of items above. | 3 | As in all trials, authors should reference relevant validation studies.  This guideline might suffer if it tried to cover questionnaire development as well as trial conduct. | 5 | I agree that passive use statistics (number of logins, etc.) should be collected.  Information that users have to enter raises different problems. | 2 | This is unclear and, I think, not essential.  I'm not sure if this means information about acceptability of the trial or the development of the site?  In either case, I think qualitative research should be reported following appropriate methods and standards for reporting.  This doesn't seem to be the right place. |  | Agreed | 1 | I worry that item 7 encourages power calculations that I often find misleading. |  | Agreed | Agreed | Agreed, though the relationship between this and other guidelines should be clarified. | Agreed, though perhaps use examples appropriate to e-health |  | 1 | I don't think this wording adds to the current statement. | 2 | I agree with the general point, though I'm not familiar with cases where this is both important and unclear.  I'm not convinced it's essential. |  | More broadly, we need clear descriptions of control conditions with the same sorts of information provided about treatment conditions. | 2 | I think this is essential, but I'm not sure what this adds to the current wording.  Is this unique to e-health or a broader point? |  |  | 2 | Journals will usually require this, so perhaps not essential unless a noted problem in e-health (it isn't in my particular area, anxiety and depression)? | 4 | This could be combined with a more complete description of recruitment processes above. | 3 | I think privacy and other security precautions are separate issues.  There are good reasons for implementing/not implementing hotlines, etc.  For example, suicide monitoring for a site about depression limits the external validity of the trial unless the methods for monitoring could be scaled-up alongside rollout.  I would see hotlines, etc. as co-interventions that could be covered above. |  | Number of people *in each analysis*.  This is frequently unclear and the number of people in each analysis is essential for calculating an effect for meta-analysis.  However, this isn't unique to e-health. | 3 | I agree that attrition is often badly reported, but I'm not sure what sort of diagram is being suggested and how this would add to a flowchart. |  | 2 | I can see how this is important, but I'm not sure it's essential. |  |  | 3 | Clearly relevant, but perhaps not in this guideline.  Seems to be related to  standardised outcome sets? |  | 3 | This is a good principle, but too vaguely worded. | 1 | This strays from a reporting guideline into dictating the conduct and analysis of a trial, and problematically so.  Firstly, 'intent-to-treat' is inconsistently used.  Does it mean analysing those who provided data in their original groups, LOCF, multiple imputation, or all of these methods?  There are perfectly good reasons to analyse survivors in some studies (e.g. I might want to know how many people died and the quality of life of survivors) and imputation should not be required in all cases. |  | 3 | This seems to be related to items above.  Perhaps there needs to be a discussion about where information about uptake/compliance should be reported and that information could be consolidated in a section. |  |  | 1 | I don't think this adds to current guidelines. |  | 4 | This is the sort of information that is particular to e-health and that this guideline could usefully elucidate for authors/editors. | 1 | I don't think RCTs should include token qualitative information.  This is often selective/misleading/inappropriately analysed.  More qualitative studies are needed, and these should be conducted/analysed/reported properly. |  | 2 | This doesn't add much to the current version of CONSORT. | 2 | This doesn't add much to the current version of CONSORT. |  | 3 | More thought is needed about how this would add to the current version of CONSORT. |  | 3 | Information about generalisability needs to be greatly expanded, but I'd rather see more relevant data in the methods and results than read speculation in the discussion. | 3 | I think this is very relevant, but procedures should be specified in the methods section, not the discussion. |  | Agreed. | Agreed. | Agreed. | 5 | Absolutely right. |  | 3b, 4a, 4b, 5, x26-ii, 21
2011-06-17 03:22:28 | [NAME] | University of Leeds, Leeds, UK | REMOVED BY EDITOR | yes | I have conducted ehealth RCTs, I have read many ehealth RCT/evaluation reports |  | 4 |  | 4 |  | 4 |  |  | 3 |  | 4 |  | 4 |  | 4 |  | 4 |  |  | 2 |  | 2 |  |  |  |  | 4 |  |  | 2 |  | 4 |  | 3 |  |  | 4 |  | 3 |  |  | 4 |  | 3 |  | 3 |  | 3 |  | 2 |  | 2 |  | 3 |  | 4 |  | 3 |  | 4 |  | 4 |  | 4 |  |  | 3 |  | 4 |  | 4 |  |  |  | 4 |  |  |  |  |  |  |  | 4 |  | 4 |  |  |  | 4 |  |  |  | 5 |  | 3 |  | 4 |  |  |  | 4 |  |  | 4 |  |  |  | 4 |  |  | 4 |  | 4 |  |  | 4 |  |  |  | 4 |  |  | 4 |  | 3 |  |  | 4 |  | 4 |  |  | 4 |  |  | 3 |  | 4 |  |  |  |  |  | 3