untitled h1
principal investigator

Carolyn Heckman, PhD
Co-Investigator(s):
Melissa Auerbach, PhD, Fox Chase Cancer Center Elliot Coups, PhD, Rutgers Cancer Institute of New Jersey Karen Glanz, PhD, MPH, University of Pennsylvania Sharon Manne, PhD, Rutgers Cancer Institute of New Jersey Lee Ritterband, PhD, University of Virginia/BeHealth Solutions
untitled h1
collaborating institutions

University of Virginia/BeHealth Solutions Statistician:
Elizabeth Handorf, Ph.D
Funding Source:
NCI
Initial Version Date: 07/20/2016 Amendment \#
N/A
Version date:
07/20/2016
untitled h1
table of contents

Section  
| 1.0   | Introduction                                                 | 3   |
|-------|--------------------------------------------------------------|-----|
| 2.0   | Objectives                                                   | 4   |
| 2.1   | Primary Aims                                                 | 4   |
| 2.2   | Secondary Aims                                               | 4   |
| 2.3   | Hypotheses                                                   | 4   |
| 3.0   | Background/Rationale                                         | 5   |
| 4.0   | Study Design                                                 | 6   |
| 4.1   | Recruitment                                                  | 6   |
| 4.2   | Inclusion and Exclusion Criteria6                            |     |
| 4.3   | Procedures                                                   | 7   |
| 4.3.1 | Overview of Intervention Conditions                          | 7   |
| 4.4   | Measures                                                     | 9   |
| 4.4.1 | Measures for Primary Aims                                    | 9   |
| 4.4.2 | Measures for Secondary Aims                                  | 10  |
| 4.5   | Compensation                                                 | 10  |
| 5.0   | Risks to Participants                                        | 11  |
| 6.0   | Potential Benefits to Participants                           | 11  |
| 7.0   | Provisions to Maintain the Confidentiality of Data           | 11  |
| 8.0   | Costs to Participants                                        | 11  |
| 9.0   | Consent Process                                              | 11  |
| 10.0  | Off-Study Criteria                                           | 12  |
| 11.0  | Drugs and Devices                                            | 12  |
| 12.0  | Multi-Site Research Study                                    | 12  |
| 13.0  | Statisical Analysis                                          | 12  |
| 13.1  | Statistical Analyses for Primary Aims                        | 12  |
| 13.2  | Statistical Analyses for Secondary Aims 13                   |     |
| 14.0  | Data Safety Monitoring Plan                                  | 14  |
| 15.0  | Adverse Events 14                                            |     |
| 16.0  | Quality Assurance Procedures and Participant Confidentiality | 14  |
| 17.0  | Participant Informed Consent                                 | 15  |
| 18.0  | References                                                   | 15  |  
Skin cancer is the most common cancer and can be deadly, debilitating, damaging, and disfiguring, yet is highly preventable. In 2014, the US Surgeon General made a call to action about the "major public health problem" of skin cancer, noting potential contributions of behavioral science and education, and a need for investments in such efforts 1. Almost five million Americans are treated for skin cancer annually, and incidence is rising 1-5. If current trends continue, melanoma will be the only Healthy People 2020 cancer objective to not meet death reduction goals1. Additionally, non-melanoma skin cancer (NMSC) can be a chronic disease for some, requiring ongoing costly treatments and decreases of quality of life similar to some other cancers6. Risk factors for melanoma and NMSC include personal or family history of melanoma or NMSC, certain phenotypic (e.g., fair skin) and other physical characteristics (e.g., numerous moles) 7-19, as well as excessive ultraviolet (UV) radiation exposure 18,20-25. Most skin cancers are preventable with skin protection such as minimizing UV exposure and wearing protective clothing and sunscreen 26,27. US adolescents have the lowest skin protection rates of all age groups26 and also increase exposure to natural and artificial UV radiation as they progress into adulthood27. Though childhood is a particularly high-risk period for UV exposure and skin damage, research suggests that only 25% of lifetime UV is accumulated by age 18 28. Our work shows that skin cancer risk behaviors, including sunburns, indoor tanning, and lack of protection peak at age 25 29,30. Thus, young adulthood is an important window for skin cancer risk reduction interventions. However, young adults tend to be resistant to public health recommendations because, as a group, they perceive themselves as having more immediate priorities than disease prevention, that the consequences of their current health behaviors are in the distant future, and they also tend to be experimenters and risk-takers highly influenced by peers 31-33. RE-AIM Framework. One of the major problems with health behavior research is that many interventions that demonstrate initial effects are never tested in effectiveness or dissemination trials, nor are they distributed to populations who most need them, thus, having little impact on population health 34. To address these concerns, the RE-AIM Framework was developed by Glasgow and colleagues 35. "REAIM" refers to Reach, Effectiveness, Adoption, Implementation, and Maintenance. Reach refers to the number, proportion, and representativeness of individuals willing to participate in a given initiative, indicating potential generalizabiity 36. Effectiveness refers to the impact of an intervention on important outcomes, including potential negative effects, quality of life, and economic outcomes. Adoption refers to the absolute number, proportion, and representativeness of settings and intervention agents who are willing to initiate a program (not a main focus of the proposed project given the self-administered and automated intervention). Implementation refers to the individual's use of the intervention. Maintenance refers to the long-term effects of a program on outcomes. Although widely used since 1999, most reviews have found that the components of the RE-AIM framework are not assessed and/or reported comprehensively or well 37-41. Additionally, most dissemination and implementation (D&I) studies focus on dissemination of guidelines, policies, practices, or interventions to healthcare agencies or practitioners rather than D&I of online self-administered interventions directly to consumers. We selected the RE-AIM framework over many others because we believe it is the most appropriate to investigate the dissemination, implementation, and effectiveness of an innovative self-administered and automated online intervention. We could identify no relevant studies that addressed D&I issues related to self-administered materials online. The PI developed a web-based intervention (UV4.me) that was found to significantly decrease UV exposure and increase skin protection behaviors among young adults in a randomized controlled trial (RCT) of nearly 1000 participants recruited from a consumer research panel. The intervention (UV4.me)
is individually-tailored, interactive, and multimedia in nature, and based on the Integrative Model of Behavioral Prediction. Similar to other online trials, 73% of eligible individuals completed the baseline questionnaire, 70% who were randomized to the intervention accessed it, and 68% accessed it and completed at least one module. However, we have an opportunity to increase the engagement, implementation, and ultimately, the impact of UV4.me. We will do this by adding several key interactive features/strategies suggested by participants, our data, and supported by the literature (i.e., by creating a mobile version, adding incentives embedded in the intervention, a behavior tracking and feedback feature, peer interaction component, and ongoing news updates). This Hybrid Type 2 dissemination-effectiveness project's purpose is to implement the enhanced intervention (UV4.me2) with adults aged 18-25 years at moderate to high risk of developing skin cancer and evaluate the intervention's effectiveness in a sample recruited online through national dissemination to the general population. This project will use the REAIM framework to determine the reach, effectiveness, implementation, maintenance, and cost of UV4.me2.
untitled h1
2.0 objectives

The objective of this project is to investigate the reach, effectiveness, implementation, maintenance, and cost of the enhanced UV4.me2 intervention in a large national randomized controlled trial. The ultimate goal is to improve the skin cancer protection behaviors (and potentially decrease skin cancer incidence)
among a national sample of young adults at moderate to high risk of developing skin cancer.
untitled h1
2.1 primary aims

Primary Aim 1. To enhance and determine intervention reach (i.e., enrollment, representativeness). Primary Aim 2. To determine the effectiveness of the enhanced intervention. 2.2 Secondary Aims Secondary Aim 1. To determine maintenance of the UV4.m4 and UV4.me2 interventions through evaluation at 6 and 12-month follow-up. Secondary Aim 2. To determine intervention implementation by young adults.
untitled h1
2.3 hypotheses

Hypothesis 1. We hypothesize that the study enrollment rate will be greater than in the original UV4.me efficacy study and that our sample will be representative of national demographics of young adults other than race/ethnicity (associated with skin cancer risk), indicating potential generalizability. Hypothesis 2. We hypothesize that the UV4.me2 will be the most effective intervention, followed by UV4.me.  
Hypothesis 3. We hypothesize that maintenance outcomes will be best in the UV4.me2 group, next best for UV4.me, and least for the e-pamphlet group. Hypothesis 4. We hypothesize that greater intervention utilization (e.g., logins, module completion) and satisfaction will be associated with better behavioral outcomes Hypothesis 5. We hypothesize that UV4.me2 and UV4.me will have higher incremental costs than the epamphlet. However, because UV4.me2 and UV4.me are also expected to result in better behavioral outcomes, we will assess cost-effectiveness to address whether these interventions are worth the additional costs.
untitled h1
3.0 background/rationale

Large skin cancer prevention campaigns have the potential to reduce incidence, mortality, and morbidity as well as be cost-effective. For example, a long-term comprehensive multi-modality skin cancer prevention campaign in Australia returned US$3.27 per dollar invested in terms of life-years saved and disability-adjusted life-years 42. A recent study reported that a comprehensive US skin cancer prevention program similar to Australia's would be estimated to prevent 20% of US melanomas, or 21,000 melanoma cases annually, with an average annual reduction of $250 million spent on new melanomas 43. The US spends several hundred thousand dollars more per year than Australia on skin cancer 44, and the annual cost of treating new melanomas in the US is projected to increase threefold from 2011 to 2030 43. US and Australian (one randomized controlled trial) studies have found that NMSC can also be reduced by 14-40% with regular sunscreen use 45-47. A few prior interventions targeted to US young adults have shown increased skin protection and/or decreased UV exposure in research settings; however, their reach and sustainability have been limited because most have been conducted in person and had brief followups 48-54. Innovative, age-appropriate interventions are important to reduce skin cancer risk among young adults. Ninety-three percent of US young adults use the web 55, and web interventions can be disseminated widely and be cost-effective to maintain 56,57. Often websites are designed to provide information rather than as interventions to improve health behaviors and/or disease risk. Despite this purpose, a review of existing melanoma websites found that the majority failed to include complete information on important topics such as risk factors, diagnosis, and prevention, with a significant minority containing inaccuracies 58. However, web interventions designed to improve health behaviors (e.g., exercise and weight loss) have been found to produce medium effect sizes and consistently outperform similar non-web interventions 57.  
We developed a web intervention that included many of the components found in successful internet interventions. We found significantly improved skin cancer risk behaviors among young adults in an RCT that enrolled almost 1000 participants from a consumer research panel 59. To our knowledge, this is the only empirically-tested internet intervention focused on skin cancer risk behaviors targeting young adults. The intervention (UV4.me) is individually-tailored, interactive, and multimedia based on Ritterband and colleagues' 60 model for online behavior change interventions and the Integrative Model of Behavioral Prediction 61, which includes background variables such as demographics; cognitive variables such as beliefs, attitudes, norms, and self-efficacy; intentions; and behavior. We also emphasized appearance concerns, which is a major factor associated with young adult tanning 62-64. In addition to successful outcomes, approximately two thirds of eligible participants enrolled in the study and completed the baseline survey, two thirds randomized to UV4.me accessed the intervention, and two thirds completed the 3-month follow-up. Yet, we still have an opportunity to increase UV4.me implementation and impact. We propose to do this by enhancing recruitment and enrollment strategies as well as incorporating additional interactive features into the intervention (i.e., creating a mobile version, embedding incentives into the intervention, adding a behavior tracking and feedback feature, peer interaction component, and ongoing news updates). Features/strategies were chosen based on participant feedback, our data (e.g., attempted use of mobile devices), and reviews/models of effective e-Health interventions including for tobacco cessation, physical activity, weight loss, and nutrition 60,65-68 and online implementation strategies 69-72. Several enhancements are expected to improve reach, implementation, and behavioral outcomes simultaneously. To our knowledge, only 15 papers on dissemination and/or implementation of skin cancer-related interventions exist, and most were authored by Dr. Glanz about her Pool Cool Program, which improved skin cancer risk and protection policies and behaviors among aquatic staff and children at Hawaii and Massachusetts pools 73,74. The D&I trial assessed effects of an enhanced vs. basic D&I strategy on adoption, implementation, and maintenance of protection promotion policies and practices by aquatic staff at >400 pools across the country based on Diffusion of Innovation theory 75-80. Program adoption was high (86.6%), the enhanced intervention produced greater adoption, implementation was similar across conditions, and core skin protection elements were maintained for two years. Dr. Glanz has published several papers describing the correlates of successful adoption and implementation 81-86. She has also published other D&I papers related to other health behaviors such as dietary choices and intake.  
4.1 Recruitment A small number of human subjects will participate in acceptability testing (20-25 participants) and usability testing (12-15 participants). After acceptability and usability testing, a national sample of 1500 human subjects will be recruited to participate in the Randomized Control Trial (RCT). Therefore, we have a lower accrual goal of 1,532 and an upper accrual goal of 1,540.  
Five types of online sources will be used to reach young adults: 1) national non-profit and for-profit skincare organizations (e.g., the Melanoma Research Foundation and Blue Lizard Sunscreen), 2) Google Adwords, 3) paid Facebook ads, 4) a consumer research panel, and 5) word of mouth (e.g., unpaid study Facebook page, earned social and traditional media [free publicity from a third party]). Google Adwords and Facebook are common online recruitment sources for young adults that can produce cost-effective, representative, and quality samples/data 87-95. Facebook is the largest social networking website and the second most popular website in the US after Google 96. Because some participants may know one another (e.g., from Facebook), we will inquire about/control for contamination between intervention conditions. Consumer research panels possess the qualities of Google Adwords and Facebook and can also produce guaranteed enrollment/data collection within a specified time-frame and budget 97-103, as occurred in the original UV4.me study. The team will refine web banner ads and create a Facebook study webpage to recruit individuals from the five online sources. Enrolled participants will be informed about the Facebook page, and non-enrolled individuals will be able to find the publicly available page organically by being referred by friends or searching Facebook or the web for related topics. To facilitate recruitment via Google and Facebook (also Bing and Yahoo), we will work with young adults and www.PatientRecruitmentOnline.com to help develop our keywords, ads, and recruitment plans.  
PatientRecruitmentOnline.com is made up of experts in online marketing specifically for health-related study recruitment. They will assist us in refining our keywords, ads, recruitment campaign, homepage, and enrollment process.
untitled h1
4.2 inclusion and exclusion criteria

We will recruit adults aged 18-25 years. Men and women and individuals from all races/ethnicities will be eligible for participation. However, only participants at moderate to high risk of developing skin cancer will be enrolled. Participants will be screened with the Brief Skin Cancer Risk Assessment Tool 104. Risk items include sun sensitivity, sunburn history, number of moles/freckles, and latitude of childhood residence. Items are weighted, resulting in a 0-78 score. A cut-off of >27 denotes moderate to high skin cancer risk. Individuals with a history of skin cancer will be excluded. Based on our original UV4.me data, we estimate that approximately 50% of individuals who complete a screening form will be eligible by these criteria. Adult participants who are not eligible but are willing to provide informed consent will still be given access to UV4.me2 in order to promote goodwill with organizations and individuals and to gather preliminary data regarding additional dissemination and effects of UV4.me2 for older individuals and those at population-level risk for skin cancer.
untitled h1
4.3 procedures

Acceptability Testing. Once the enhanced intervention is fully developed, acceptability testing modeled after an NCI online design project will be conducted 105 as was done for the original UV4.me 106. We will focus on the new features added to UV4.me for UV4.me2 (e.g., the mobile version), a prototype study Facebook page, and the recruitment ads and strategies. Testing will involve consulting users about attractiveness, comprehension, population acceptability, and persuasion of materials. A series of questions and structured guides will be prepared in advance relating to the elements to be evaluated is in the original UV4.me. Acceptability testing will be conducted using focused interviews with approximately 20 participants eligible for and recruited using methods similar to the RCT. The first half of sessions will be conducted in person, and the second half will be conducted remotely via the web and telephone.  
Participants will use a device (computer, tablet, or smartphone) with access to the online intervention. The iterative process will involve discussion of an intervention element, followed by interaction with that part of the intervention, followed by discussion reacting the part, etc. The sessions will be video- or audiorecorded and transcribed, and the moderator will take notes.  
Usability Testing. In usability testing, users do typical tasks with a product, or explore it freely, while their behaviors are observed and recorded to identify design flaws that cause user errors/difficulties 107.  
As in the original UV4.me 106, we will follow the NCI usability guidelines 108 on planning, analyzing, developing, testing, and refining web interventions. We will conduct usability testing with 12-15 participants as recommended by Bastien 107. The evaluator notes the frequency/duration of behaviors that can indicate user problems/difficulties as well as measures such as time to finish a task, time recovering from errors, number of wrong choices, observations of frustrations, confusion, satisfaction, etc. 107. Morae software video-records the session and provides descriptive statistics on behaviors observed (frequencies, duration, etc.) and behavior patterns 107. We will conduct half of the usability sessions via the web using similar software without video.  
Randomized Controlled Trial. RCT participants will complete a baseline survey focusing on exposure and protection behaviors that takes a maximum of approximately 15 minutes on average. The study statistician will create a randomization scheme to randomize enrolled participants on a 2:2:1 basis to either the UV4.me2, UV4.me, or the e-pamphlet condition, respectively. Participants will not be informed of the differences in the intervention conditions beyond that they will all be online skin cancer risk reduction interventions. Participants will then be asked to complete follow-up assessments at 3, 6, and 12 months post-baseline.
untitled h1
4.3.1 overview of intervention conditions

E-Pamphlet Condition: A free non-interactive e-pamphlet ("Skin Cancer Prevention and Early Detection" from the American Cancer Society) will be accessible via our website. We chose to include a pamphlet condition for the following reasons: 1) such pamphlets are widely available to the public and used by dermatologists and primary care providers, 2) to compare the effects of our intervention to a minimal intervention, 3) to compare our effects to prior studies that used pamphlet or minimal interventions, 4) to include a standardized rather than a variable intervention, 5) we also considered using an intervention for another health issue (e.g., nutrition, physical activity, sleep) but anticipated that skin protection organizations would be less likely to participate in that case, and 6) a no-treatment condition could affect adoption and reach and could be considered unethical for those with moderate to high skin cancer risk.  
Original UV4.me: UV4.me is targeted to young adults, personally tailored, and includes interactive, multimedia, and goal-setting components. It includes 12 modules with content related to a specific topic important in terms of risk or protective behaviors: Why do people tan? To tan or not to tan? Indoor tanning, UV & looks, UV & health, Skin cancer, Skin damage, Sunscreen, Shade, Clothes, Skin exams, and Sunless tanning. Several more general sections (e.g., avatar, MyStuff - a printable summary of tailored goals/recommendations) are also included. Tailoring algorithms were created to direct participants to focus on certain modules based on their responses to a few initial questions (e.g., the indoor tanning module was recommended for indoor tanners). Throughout the program, participants are asked questions and provided with tailored feedback (e.g., "Do you know people who tan? If so, how likely are they to affect your choice to tan or not?"). A number of interactive elements (e.g., videos, games) were created to encourage implementation. For example, at the end of each module, participants could choose to set a goal for the next two weeks or not (e.g., "For the next two weeks, I will not use a tanning bed."). Enhanced UV4.me2: New features/strategies were chosen based on participant feedback from the original UV4.me grant (e.g., suggested strategies to make UV4.me more interactive), our data (e.g., number of mobile users who tried to access UV4.me), and reviews/models of effective e-Health interventions 65,66 and implementation strategies 69,70,72. Several enhancements are expected to improve reach, implementation, and behavioral outcomes simultaneously:
1. Mobile Site. A mobile version will likely improve reach directly because more people will have access and may improve effectiveness indirectly by facilitating ongoing implementation. The mobile platform will be developed for both Android and Apple devices. 2. Incentives. The enhanced UV4.me2 site will also offer incentives in the form of clickable coupons and links to free samples for sun protection products (e.g., sunscreen). Appropriately selected incentives generally reinforce behavioral implementation, retention, and health behavior change, including in online trials, especially among those initially least motivated and/or for behaviors that are not intrinsically enjoyable (e.g., applying sunscreen) 71,109-115. These incentives will be used to reinforce behavioral implementation, retention, and health behavior change. 3. Behavioral Tracking & Feedback. Users will be allowed to set goals and enter relevant behavioral events (e.g., sunbathing, indoor tanning, sunscreen use) over time and see them summarized graphically on the homepage and receive motivating feedback (e.g., "Great job, your tanning is going down over time!"). Behavior tracking and feedback are well-established empirically-supported behavior change techniques including for internet interventions 116,117. As with the original UV4.me site, participants will be able to set "alerts" to notify them (e.g., email, SMS texting) to remind them to work on their behavioral goals. 4. Peer Interaction. We will include an open-text component that will encourage participants to think, write, and communicate about relevant topics. Peer, normative, and social factors have a powerful influence on individual behavior, especially among adolescents and young adults, and social context was a factor identified in the review/model of effective eHealth interventions by Morrison and colleagues 66, has been shown to increase intervention implementation 118,119, and is consistent with other reviews of online interventions 65,69,70 and Bandura's Social Cognitive Theory 120. There will be several spaces within the UV4.me2 website in which participants will be able to enter open-text comments and upload images (e.g., "Tell us about someone you know who has had skin cancer."). These open-text areas within the website will also all be linked to and organized in one central "bulletin board" on the homepage so that users can find them easily without searching through the entire website. Participants will be able to choose to submit material privately (for themselves only) or publicly (for staff and all UV4.me2 users).  
We will be able to monitor and control this material such that what is submitted for online "publication" will initially be reviewed by study staff before public posting to ensure that material is appropriate (e.g., not profane, abusive, or irrelevant) and not un-helpful (e.g., does not recommend tanning). If most submissions are appropriate for a period of time, we will then move to an immediate posting procedure without staff pre-review with users being permitted to "flag" inappropriate content to alert staff to remove it.  
5. Ongoing News Updates. We plan to add new material (e.g., news/media stories, new research, events such as runs for melanoma charities) to UV4.me2 at least weekly with biweekly notifications to interested users (e.g., email, SMS texting).  
4.4 Measures Screener. Participants will be screened with Dr. Glanz's 9-item Brief Skin Cancer Risk Assessment Tool (see Inclusion Criteria above) 104. Internal and test-retest reliability compare favorably to those reported in the literature for similar items/scales 104. Participants will be required to understand English and have access to the internet via any modality (e.g., computer, smartphone) at least weekly.  
Demographics. Standard demographic items used by the census will assess: sex, age/birthdate, race/ethnicity, region of residence, education, income, and employment. We will include items on web use from the Health Information National Trends Survey 121, and will inquire about family history of skin cancer.  
Social Desirability. Social Desirability will be assessed using the 4-item Brief Social Desirability Scale that has demonstrated adequate internal reliability and validity among young adults 122. Reach. We will assess eligibility and enrollment overall and by recruitment source (i.e., skin protection organizations, Google Adwords, Facebook ads, consumer research panel, word of mouth) over 12-18 months in order to assess reach/potential generalizability. Sources will be tracked by placing a unique identifying pixel in each of our authorized web ads so that we know when an individual referred from a pre-specified source accesses the study website. Additionally, we will query participants as to how they found out about the study to identify informal word of mouth diffusion (e.g., unpaid study Facebook page, earned media). The number of individuals who click on the web ads, access the study homepage, complete a screening form, are eligible/ineligible, and use the website without enrolling will be assessed. Organizational sources will be monitored and queried as to strategies (i.e., posting the study URL on their website, additional promotion) used to encourage individual participation. As we have previously, we will use Google Analytics to observe user characteristics and behavior in aggregate in terms of internet source including diffusion via word of mouth, geographic location, technology used (i.e., mobile or not),
eligibility/enrollment rates, and so on. We will ask organizations to set up or allow us to set up a Google Analytics "goal" to assess such user characteristics and behavior in terms of individuals accessing the study ad from within the organization's website. We will also query organizations about their number of users/members and website views if they are monitoring these. Efficacy. Skin cancer-related behavioral outcomes will initially be assessed at baseline and 3 months later. They will also be assessed within UV4.me2 via behavioral tracking. We will assess sun protection (e.g., sunscreen use, clothing, shade) and UV exposure (e.g., sunburns, intentional/incidental sun exposure, indoor tanning) using items adapted from Glanz and colleagues and Ingledew and colleagues that our team has cognitively tested and assessed psychometrically with young adults 123-125. Sunburns, indoor tanning, and sunless tanning will be secondary outcomes since engagement in these activities is less frequent than sun exposure and protection in general. Negative effects will be defined as an increase in sunburns. Several studies have demonstrated the reliability and validity of self-report questionnaires of UV exposure and protection compared to observation and objective measures with no systematic bias identified among various populations 126-130. As indicators of quality of life, we will inquire as to the amount of perceived benefits of and barriers to UV exposure and protection 62,131-134.  
Maintenance. Skin cancer-related behavioral outcomes (described above) will be assessed at 6 and 12 months. We will also assess maintenance of reach, intervention implementation, and cost over time.  
Implementation. Using BeHealth's data management system, we will record whether, how frequently, and for how long participants logged into the interventions, how many sections are completed, use of the enhanced features, and whether these variables are associated with behavioral outcomes. Whether individuals click into the pages with discounts/free samples will be noted. Whether and when a user accesses the e-pamphlet will be noted as well. We will ask participants to give Likert ratings of their satisfaction with and perceived helpfulness of the study/intervention and selected components (e.g., the enhanced features) both from within the intervention and a one-month follow-up inquiry. The 4-item App Subjective Quality scale from the Mobile App Rating Scale will be used to assess satisfaction (a = .93)
135. We have experience using Google Analytics "goals" to track how frequently pages within modules are accessed. These implementation variables will also be linked to participant source (i.e., skin protection organizations, Google Adwords, Facebook ads, consumer research panel) when possible so that sources can be compared by participant intervention implementation. Cost. We will assess total and incremental costs and evaluate the cost-effectiveness of the interventions. We will first estimate costs from a payer perspective, capturing the explicit resources required to deliver the program after all start-up costs (e.g., development/programming costs) have been incurred. We will also estimate costs from a societal perspective, which includes all costs, regardless of who bears them.  
These will include all explicit costs as well as the implicit program costs related to participant time (i.e., opportunity costs 136. We will also denote whether costs are related to research or intervention delivery.  
We will collect cost data using a modified version of prior cost surveys developed by RTI 137,138 and adapted by Dr. Honeycutt for many interventions 139,140. The surveys capture all relevant labor- and non-labor-related inputs necessary to quantify costs. We will quantify participant time spent on the intervention using data from BeHealth's system and valuing participant time using age- and sex-specific wage rates, netting out research incentive payments to participants. We will quantify labor costs as program staff time (via questionnaires) valued using actual or estimated wages. Non-labor costs will be collected using program billing records and include materials/supplies used to support program activities and costs for facilities and contracted services. To evaluate the cost-effectiveness of UV4.me2 versus UV4.me and the e-pamphlet, we will combine cost estimates with effectiveness outcomes to estimate the cost per incident of UV exposure and sunburn averted.  
Acceptability/Usability Testing. Participants who complete testing will be eligible to receive $25 in giftcards or cash equivalent.  
RCT. In addition to discounts and free samples, enrolled participants in each intervention condition will be eligible to earn the following incentives in gift-cards or cash equivalent for completing study assessments: 1 month = $10, 3 months = $15, 6 months = $20, 12 months = $25, plus a $25 bonus for completing all time-points, and $5 for logging into the interventions for the first time, for a possible total of $100 throughout the year, plus a chance in a raffle for $100 at each time-point. We chose this level of incentives to balance two issues: 1) we wanted to encourage young adults to complete the longitudinal study assessments, but 2) when the research phase is complete, individuals will not be paid for using the program (other than the discounts and free products).
untitled h1
5.0 risks to participants

The main potential risks are breach of confidentiality and mild emotional distress. Some participants may become mildly distressed when considering their risk for skin cancer.
untitled h1
6.0 potential benefits to participants

We believe that the UV4.me and UV4.me2 internet interventions may reduce skin cancer risk behaviors among participants and thus ultimately reduce their skin cancer risk.
untitled h1
7.0 provisions to maintain the

The data that will be collected will not be sensitive in nature. Therefore, any potential breach of confidentiality would likely not be serious. Research staff will be carefully trained and supervised to maintain confidentiality of data (e.g., by password protecting electronic data). Data collected through the Internet will be obtained through secured means and stored on password protected servers as described above. All data on servers are password-protected and limited to authorized research personnel. This high-level architecture is the same system that has been IRB-approved for use in multiple randomized controlled trials of Internet interventions with which BeHealth has been involved.  
Self-report psychosocial data will be collected directly from participants on-line using encrypted and password protected data collection software designed for specifically for that purpose. Passwordprotected data files will be backed-up onto tape on a daily basis. Data will be maintained in such a way as to facilitate confidential data sharing. Data will be encrypted for transmission. Only members of the research team, all of whom will have been trained in research with human subjects and the importance of maintaining confidentiality, will have access to the data. The data will be labeled with participant numbers only. 8.0 Costs to Participants We anticipate no costs to participants during involvement in this research study.  
9.0 Consent Process For the acceptability and usability testing, participants will have an opportunity to read the informed consent and HIPAA documents and review them with a staff member (or others) and ask questions before signing.  
For the RCT, participants will be screened, provide consent, and if eligible, complete a survey (< 15 min.)
all online. Participants will have an opportunity to read the informed consent and HIPAA documents and review them with a staff member (or others) and ask questions.
untitled h1
10.0 off-study criteria

Based on our data, experience, the enhanced study procedures, and the consumer research panel's experiences with young adults, we anticipate attrition of approximately 20% at each time-point. For dropouts, the reason for attrition will be analyzed if available to search for evidence of informative missingness. 11.0 Drugs and Devices Not applicable.  
12.0 Multi-Site Research Study Not applicable.  
Reach. We will use a one-sample test of a binomial proportion (two-sided, α=0.05) to determine whether the fraction of eligible subjects who enroll and complete the baseline is higher in this project than the prior study. Assuming that 70% of eligible subjects enroll and complete the baseline, we will need access to 2,143 (i.e., 1500/0.70) eligible subjects. Given this sample size, we will be able to detect an increased enrollment and baseline completion proportion with at least 98% power if the true proportion is 70%.  
Demographic data from enrolled participants will be compared to census data using chi-square goodness of fit tests (two-sided, α=0.05). We will also use the same methods to compare enrollment rates and representativeness separately by recruitment source (i.e., skin protection organizations, Google Adwords, paid Facebook ads, consumer research panel, word of mouth [e.g., unpaid Facebook page, earned media]). Efficacy. We will compare demographics across randomization arms using chi-squares and ANOVAs for categorical and continuous data, respectively. The primary outcomes are baseline to 3-month changes in UV exposure and protection. We will use multivariable linear regression to compare the effectiveness of UV4.me2 with UV4.me, and UV4.me2 with the e-pamphlet. We will not compare UV4.me to the epamphlet because UV4.me compared to a control condition was established in the prior study. Covariates in these models will include study arm, recruitment month/season, US region, and demographic factors identified as significantly imbalanced across arms (confounders). Analyses will be performed separately for the two primary outcomes. Variance stabilizing transformations may be applied as needed. Hypothesis tests will be two-sided with a 1.25% Type I error to account for multiple testing (2 intervention comparisons x 2 outcomes). Based on our data, experience, the enhanced study procedures, and the consumer research panel's experiences with young adults, we anticipate attrition of approximately 20% at each time-point. With 1500 young adults at baseline, we anticipate collecting 3-month follow-up data from 1200 participants, with 240 from the e-pamphlet arm and 480 participants from each of the other arms. The means (SDs) of the baseline to 3-month changes in UV exposure and protection indexes from the prior study control arm were 0.34 (0.74) and -0.37 (0.90), respectively. Given these parameters, we will have 80% power to detect small standardized effect sizes of 0.22 between the UV4.me2 and UV4.me arms, and 0.26 between the UV4.me2 and the e-pamphlet. These correspond to small detectable differences between intervention arms of 0.16 and 0.19 in baseline to 3-month changes in UV exposure and protection indexes, respectively. We will be able to detect slightly larger differences when the UV4.me2 and e-pamphlet arms are compared (detectable differences - UV exposure: 0.20; protection: 0.24). Analyses will assume that missingness depends on observed data (i.e., missing at random; MAR)
141. For drop-outs, the reason for attrition will be analyzed if available to search for evidence of informative missingness. Baseline variables will be compared by whether participants drop out or not.  
Significant variables will be added to the model to strengthen our MAR assumption. As a sensitivity analysis, we will use a shared parameter model with the assumption of missing not at random 142. If large changes in parameter estimation are found, we will use the results from the shared parameter model. Maintenance. We will compare baseline to 6- and 12-month changes in UV exposure and protection across study arms. Tests will be two-sided with a 5% type I error for these secondary analyses. We anticipate having follow-up data from approximately 960 and 768 participants at 6- and 12-months, respectively. This will allow for 154 participants in the e-pamphlet arm and 307 in each of the other two arms at 12 months. Detectable differences with 80% power are presented in Table 3.  
Implementation. Using BeHealth's Wasabi data management system, we will create a summary index representing how frequently and for how long participants are logged into the interventions and how many sections are completed and determine whether this index is associated with behavioral outcomes. Whether and when a user accesses the e-pamphlet will be noted. We have experience using Google Analytics "goals" to track how frequently individual pages within modules are accessed. We will ask participants about their satisfaction with the study/intervention and selected components (e.g., the enhanced features) both from within the interventions and a one-month follow-up inquiry. These implementation variables will also be linked to participant source (i.e., skin protection organizations, Google Adwords, Facebook ads, consumer research panel) when possible so that sources can be compared in terms of participant intervention implementation. We will use the methods described for Aim 2 to compare these utilization and satisfaction measures across study arms, and Spearman's correlation to measure the level of association between implementation metrics and longitudinal changes in UV exposure and protection indexes within and across study arms. Tests will be two-sided with a 5%
type I error for these secondary analyses. We anticipate 1350 participants at 1-month follow-up (epamphlet = 270, others = 540). Whether individuals click into the pages with discounts and free samples from skincare companies will be noted. We will assess whether accessing more incentives within the interventions is associated with greater intervention utilization and better behavioral outcomes. Cost. We will estimate mean and median costs for each intervention. We will distinguish between development costs (one-time capital investment or "sunk" costs) and implementation costs, or the costs to maintain the interventions. We will also denote whether costs are related to research or intervention delivery. Analyses will distinguish between fixed costs (costs that do not vary with enrollment, e.g.,
server maintenance, data storage), and variable costs, which increase with additional participants. Because most costs are expected to be fixed, the mean cost per participant will be driven largely by the number enrolled. To explore costs of scale-up, we will conduct sensitivity analyses around additional dissemination efforts and higher take-up rates. We will also estimate the cost-effectiveness of UV4.me2 relative to the other conditions, calculating cost-effectiveness ratios as the difference in estimated costs divided by the difference in estimated effectiveness to determine the incremental cost per sunburn (or incident of UV exposure) averted for each intervention pair 141,142. We will also assess and estimate medical cost (e.g., OTC medication use, healthcare visits) offsets resulting from averted sunburns. To estimate longer-term cost-effectiveness, we will develop an Excel-based model to estimate qualityadjusted life year (QALY) gains associated with each sunburn (which increases melanoma risk) averted.  
For the cost-effectiveness modeling, we will use data from the literature on the probability of developing various forms of skin cancer as a result of sunburns and on QALY losses associated with various cancer outcomes to estimate the cost per QALY gained. Expressing cost-effectiveness as cost per QALY gained will facilitate comparison of the UV4.me2 intervention with other preventive health interventions. For all cost-effectiveness analyses, we will perform one-way (and n-way) sensitivity analyses to examine the impact of varying input values over a plausible range.
untitled h1
14.0 data safety monitoring plan

Computerized data collection allows for greater accuracy and completeness than other types of data collection. Computerized skip logic and data validation procedures will be utilized (e.g., "Please enter a number rather than a word."). Participants will be encouraged to complete a majority of the surveys. Raw data will be downloaded into password protected Excel and SPSS spreadsheets for cleaning, scoring, and initial descriptive analyses. Data cleaning will involve checks for random and inconsistent responding. Analyses will be conducted by the biostatistician (see Analyses section).
untitled h1
15.0 adverse events

Adverse events will be reported to the PI immediately, and the PI will report adverse events and planned interventions to the FCCC IRB within the required time-frame. Any subsequent AE will be reported along with prior ones for monitoring purposes. If 3 or more adverse events are reported, the study team will assess potential causes of the adverse events and, if events are clearly linked to study participation, consider discontinuing the study.
untitled h1
16.0 quality assurance procedures and

Information about study subjects will be kept confidential and managed according to the requirements of the Health Insurance Portability and Accountability Act of 1996 (HIPAA). Those regulations required a signed subject authorization informing the subject of the following: The protected health information (PHI) that will be collected from patient; who will have access to that information and why; who will use or disclose that information; the rights of a research subject to revoke their authorization or use their PHI.  
In the event that a participant revokes authorization to collect or use PHI, the investigator, by regulation, retains the ability to use all information prior to the revocation of subject authorization. Data collected through the Internet will be obtained through secured means and stored on password protected servers as described above. These data will be labeled with participant numbers only. All data on servers are password-protected and limited to authorized research personnel.  
We will employ several strategies to minimize repeat enrollment and enhance data quality, as recommended by the literature 98,143-151. These will include 1) verification of enrollment of participants with non-unique Personal Health Information (e.g., IP [Internet Protocol] address, email address, telephone number, name), and 2) separate analysis of data that indicates patterned (e.g., responding "always" to several items in a row), careless (e.g., too quick, illogical, gibberish, profane), or potentially fraudulent responding. We will create a summary index of such variables and conduct a latent class analysis to make a final decision about data inclusion/exclusion 146,152 as we did for the original UV4.me. The consumer research panel also employs such strategies to ensure data accuracy from its panel members such as providing researchers with panel entry data such as demographics, geolocation, socioeconomics, and health/behaviors to confirm eligibility and response consistency. By employing these procedures, we believe that we will be able to keep poor quality questionnaires relatively low at 5%
of initially eligible participants, which is consistent with typical inattention 144,147,149. Thus, we will plan to over-recruit by as much as 5% until we have the required number of quality baseline surveys from unique eligible participants.
untitled h1
17.0 participant informed consent

Participants in the usability and acceptability phase will indicate their consent either written or electronic form, depending on the circumstances of their participation. Participants in the RCT will give electronic consent before accessing the baseline survey.
untitled h1
18.0 references

1. USDHHS. The Surgeon General's Call to Action to Prevent Skin Cancer. Washington, DC2014.  
2. American Cancer Society. How many people get melanoma skin cancer? 2009; http://www.cancer.org/docroot/CRI/content/CRI_2_2_1X_How_many_people_get_melanoma_skin_canc er_50.asp. Accessed February, 12, 2010. 3. Donaldson MR, Coldiron BM. No end in sight: the skin cancer epidemic continues. Semin. Cutan. Med. Surg. 2011;30(1):3-5. 4. Linos E, Swetter SM, Cockburn MG, Colditz GA, Clarke CA. Increasing burden of melanoma in the United States. J. Invest. Dermatol. 2009;129(7):1666-1674. 5. Rogers HW, Weinstock MA, Harris AR, et al. Incidence estimate of nonmelanoma skin cancer in the United States, 2006. Arch. Dermatol. 2010;146(3):283-287.  
6. Sutton A, Crew A, Wysong A. Redefinition of Skin Cancer as a Chronic Disease. JAMA
Dermatol. 2015:1-2.  
7. American Cancer Society. Skin Cancer:Basal and Squamous Cell. Learn About Cancer 2012; http://www.cancer.org/Cancer/SkinCancer-BasalandSquamousCell/DetailedGuide/skin-cancer-basal-andsquamous-cell-what-is-basal-and-squamous-cell. Accessed August 1, 2012.  
8. Baxter AJ, Hughes MC, Kvaskoff M, et al. The Queensland Study of Melanoma: environmental and genetic associations (Q-MEGA); study design, baseline characteristics, and repeatability of phenotype and sun exposure measures. Twin Research and Human Genetics 2008;11(2):183-196. 9. Centers for Disease Control and Prevention. Skin Cancer: Risk Factors. 2011; http://www.cdc.gov/cancer/skin/basic_info/risk_factors.htm. Accessed July 30, 2012. 10. Chuang TY, Brashear R. Risk factors of non-melanoma skin cancer in United States veterans patients: a pilot study and review of literature. J. Eur. Acad. Dermatol. Venereol. 1999;12(2):126-132. 11. Goldberg MS, Doucette JT, Lim HW, Spencer J, Carucci JA, Rigel DS. Risk factors for presumptive melanoma in skin cancer screening: American Academy of Dermatology National Melanoma/Skin Cancer Screening Program experience 2001-2005. J. Am. Acad. Dermatol.  
2007;57(1):60-66. 12. Grulich AE, Bataille V, Swerdlow AJ, et al. Naevi and pigmentary characteristics as risk factors for melanoma in a high-risk population: a case-control study in New South Wales, Australia. Int. J. Cancer. 1996;67(4):485-491. 13. Markovic SN, Erickson LA, Rao RD, et al. Malignant melanoma in the 21st century, part 1: epidemiology, risk factors, screening, prevention, and diagnosis. Mayo Clin. Proc. 2007;82(3):364-380. 14. Oberyszyn TM. Non-melanoma skin cancer: importance of gender, immunosuppressive status and vitamin D. Cancer Lett. 2008;261(2):127-136.  
15. Perkins JL, Liu Y, Mitby PA, et al. Nonmelanoma skin cancer in survivors of childhood and adolescent cancer: a report from the childhood cancer survivor study. J. Clin. Oncol. 2005;23(16):37333741. 16. Psaty EL, Scope A, Halpern AC, Marghoob AA. Defining the patient at high risk for melanoma. Int. J. Dermatol. 2010;49(4):362-376.  
17. Qureshi AA, Zhang M, Han J. Heterogeneity in host risk factors for incident melanoma and nonmelanoma skin cancer in a cohort of US women. J. Epidemiol. 2011;21(3):197-203.  
18. Siskind V, Aitken J, Green A, Martin N. Sun exposure and interaction with family history in risk of melanoma, Queensland, Australia. Int. J. Cancer. 2002;97(1):90-95. 19. Etzkorn JR, Parikh RP, Marzban SS, et al. Identifying risk factors using a skin cancer screening program. Cancer control : journal of the Moffitt Cancer Center. 2013;20(4):248-254.  
20. Armstrong BK, Kricker A. How much melanoma is caused by sun exposure? Melanoma Res.  
1993;3(6):395-401.  
21. Dusza SW, Halpern AC, Satagopan JM, et al. Prospective study of sunburn and sun behavior patterns during adolescence. Pediatrics. 2012;129(2):309-317. 22. Lazovich D, Forster J. Indoor tanning by adolescents: prevalence, practices and policies. Eur. J. Cancer. 2005;41(1):20-27. 23. Skin Cancer Foundation. Preventing Skin Cancer. 2012; http://www.skincancer.org/prevention/sun-protection/prevention-guidelines/preventing-skin-cancer.  
Accessed July 31, 2012. 24. Swerdlow AJ, Weinstock MA. Do tanning lamps cause melanoma? An epidemiologic assessment. J. Am. Acad. Dermatol. 1998;38(1):89-98. 25. Vishvakarman D, Wong JC. Description of the use of a risk estimation model to assess the increased risk of non-melanoma skin cancer among outdoor workers in Central Queensland, Australia. Photodermatol. Photoimmunol. Photomed. 2003;19(2):81-88. 26. Gies PH, Roy CR, Toomey S, McLennan A. Protection against solar ultraviolet radiation.  
Mutation Research/Fundamental and Molecular Mechanisms of Mutagenesis. 1998;422(1):15-22.  
27. Moloney FJ, Collins S, Murphy GM. Sunscreens: safety, efficacy and appropriate use. American journal of clinical dermatology. 2002;3(3):185-191. 28. Lim HW, Cyr WH, DeFabo E, et al. Scientific and regulatory issues related to indoor tanning. J.  
Am. Acad. Dermatol. 2004;51(5):781-784. 29. Coups EJ, Manne SL, Heckman CJ. Multiple skin cancer risk behaviors in the U.S. population. Am. J. Prev. Med. 2008;34(2):87-93.  
30. Heckman CJ, Coups EJ, Manne SL. Prevalence and correlates of indoor tanning among US
adults. J. Am. Acad. Dermatol. 2008;58(5):769-780. 31. Freedman KS, Nelson NM, Feldman LL. Smoking initiation among young adults in the United States and Canada, 1998-2010: a systematic review. Prev Chronic Dis. 2012;9(E5). 32. Andrews JA, Tildesley E, Hops H, Li F. The influence of peers on young adult substance use. Health Psychol. 2002;21(4):349-357. 33. Galliher RV, Kerpelman JL. The intersection of identity development and peer relationship processes in adolescence and young adulthood: contributions of the special issue. J. Adolesc.  
2012;35(6):1409-1415.  
34. Brownson RC, Colditz G, Proctor E, eds. Dissemination and Implementation Research in Health. Oxford University Press; 2012.  
35. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am. J. Public Health. 1999;89(9):1322-1327. 36. RE-AIM. http://www.re-aim.hnfe.vt.edu/ Accessed Jan 5, 2015.  
37. Blackman KC, Zoellner J, Berrey LM, et al. Assessing the internal and external validity of mobile health physical activity promotion interventions: a systematic literature review using the RE-AIM framework. J Med Internet Res. 2013;15(10):e224. 38. Carnevale TD. Universal adolescent depression prevention programs: a review. J. Sch. Nurs. 2013;29(3):181-195. 39. Glasgow RE, Klesges LM, Dzewaltowski DA, Estabrooks PA, Vogt TM. Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues. Health Educ. Res. 2006;21(5):688-694.  
40. Kessler RS, Purcell EP, Glasgow RE, Klesges LM, Benkeser RM, Peek CJ. What does it mean to
"employ" the RE-AIM model? Eval. Health Prof. 2013;36(1):44-66. 41. Klesges LM, Dzewaltowski DA, Glasgow RE. Review of external validity reporting in childhood obesity prevention research. Am. J. Prev. Med. 2008;34(3):216-223.  
42. Shih ST, Carter R, Sinclair C, Mihalopoulos C, Vos T. Economic evaluation of skin cancer prevention in Australia. Prev. Med. 2009;49(5):449-453. 43. Guy GP, Jr., Thomas CC, Thompson T, et al. Vital signs: melanoma incidence and mortality trends and projections - United States, 1982-2030. MMWR. Morb. Mortal. Wkly. Rep. 2015;64(21):591596.  
44. Gordon LG, Rowell D. Health system costs of skin cancer and cost-effectiveness of skin cancer prevention and screening: a systematic review. Eur. J. Cancer Prev. 2014. 45. Do sunscreens have a role in preventing skin cancer? Drug Ther Bull. . 2011;49(6):69-72. 46. Gordon LG, Scuffham PA, van der Pols JC, McBride P, Williams GM, Green AC. Regular sunscreen use is a cost-effective approach to skin cancer prevention in subtropical settings. J. Invest. Dermatol. 2009;129(12):2766-2771.  
47. Stern R, Weinstein M, SG B. Risk reduction for nonmelanoma skin cancer with childhood sunscreen use. Arch. Dermatol. 1986;122(5):537-545. 48. Gibbons FX, Gerrard M, Lane DJ, Mahler HI, Kulik JA. Using UV photography to reduce use of tanning booths: a test of cognitive mediation. Health Psychol. 2005;24(4):358-363. 49. Greene K, Brinn LS. Messages influencing college women's tanning bed use: statistical versus narrative evidence format and a self-assessment to increase perceived susceptibility. Journal of Health Communication. 2003;8(5):443-461. 50. Jackson KM, Aiken LS. Evaluation of a multicomponent appearance-based sun-protective intervention for young women: uncovering the mechanisms of program efficacy. Health Psychol.  
2006;25(1):34-46.  
51. Roberts DC, Black D. Comparison of interventions to reduce sun exposure. Behav. Med.  
2009;35(2):67-76. 52. Mahler HI, Kulik JA, Gerrard M, Gibbons FX. Effects of photoaging information and UV photo on sun protection intentions and behaviours: a cross-regional comparison. Psychol Health.  
2013;28(9):1009-1031.  
53. Turrisi R, Mastroleo NR, Stapleton J, Mallett K. A comparison of 2 brief intervention approaches to reduce indoor tanning behavior in young women who indoor tan very frequently. Arch. Dermatol.  
2008;144(11):1521-1524.  
54. Aarestrup C, Bonnesen CT, Thygesen LC, et al. The effect of a school-based intervention on sunbed use in Danish pupils at continuation schools: a cluster-randomized controlled trial. J. Adolesc. Health. 2014;54(2):214-220.  
55. Lenhart A, Purcell K, Smith A, Zickuhr K. Social media and mobile internet use among teens and young adults. Washington, DC: Pew Research Center;2010. 56. Griffiths F, Lindenmeyer A, Powell J, Lowe P, Thorogood M. Why are health care interventions delivered over the internet? A systematic review of the published literature. Journal of medical Internet research. 2006;8(2):e10.  
57. Tate DF, Finkelstein EA, Khavjou O, Gustafson A. Cost effectiveness of internet interventions: review and recommendations. nn. Behav. Med. 2009. 58. Bichakjian CK, Schwartz JL, Wang TS, Hall JM, Johnson TM, Biermann JS. Melanoma information on the Internet: often incomplete--a public health opportunity? J. Clin. Oncol.  
2002;20(1):134-141.  
59. Heckman CJ, Darlow S, Ritterband L, Handorf E, Manne S. Efficacy of an Intervention to Alter Skin Cancer Risk Behaviors in Young Adults. Am. J. Prev. Med. 2016. 60. Ritterband LM, Thorndike FP, Cox DJ, Kovatchev BP, Gonder-Frederick LA. A behavior change model for internet interventions. nn. Behav. Med. 2009. 61. Fishbein M, Hennessy M, Yzer M, Douglas J. Can we explain why some people do and some people do not act on their intentions? Psychology, health & medicine. 2003;8(1):3-18.  
62. Cafri G, Thompson JK, Roehrig M, van den Berg P, Jacobsen PB, Stark S. An investigation of appearance motives for tanning: The development and evaluation of the Physical Appearance Reasons For Tanning Scale (PARTS) and its relation to sunbathing and indoor tanning intentions. Body Image.  
2006;3(3):199-209.  
63. Danoff-Burg S, Mosher CE. Predictors of tanning salon use: behavioral alternatives for enhancing appearance, relaxing and socializing. Journal of health psychology. 2006;11(3):511-518.  
64. Hillhouse J, Turrisi R, Stapleton J, Robinson J. A randomized controlled trial of an appearancefocused intervention to prevent skin cancer. Cancer. 2008;113(11):3257-3266.  
65. Litvin EB, Abrantes AM, Brown RA. Computer and mobile technology-based interventions for substance use disorders: an organizing framework. Addict. Behav. 2013;38(3):1747-1756.  
66. Morrison LG, Yardley L, Powell J, Michie S. What design features are used in effective e-health interventions? A review using techniques from Critical Interpretive Synthesis. Telemedicine journal and e-health : the official journal of the American Telemedicine Association. 2012;18(2):137-144. 67. Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res. 2010;12(1):e4.  
68. Williams AL, Grogan S, Clark-Carter D, Buckley E. Appearance-based interventions to reduce ultraviolet exposure and/or increase sun protection intentions and behaviours: a systematic review and meta-analyses. British journal of health psychology. 2013;18(1):182-217. 69. Brouwer W, Oenema A, Crutzen R, de Nooijer J, de Vries NK, Brug J. An exploration of factors related to dissemination of and exposure to internet-delivered behavior change interventions aimed at adults: a Delphi study approach. J Med Internet Res. 2008;10(2):e10. 70. Crutzen R, de Nooijer J, Brouwer W, Oenema A, Brug J, de Vries NK. Internet-delivered interventions aimed at adolescents: a Delphi study on dissemination and exposure. Health Educ. Res.  
2008;23(3):427-439. 71. Crutzen R, de Nooijer J, Brouwer W, Oenema A, Brug J, de Vries NK. Strategies to facilitate exposure to internet-delivered health behavior change interventions aimed at adolescents or young adults:
a systematic review. Health Educ. Behav. 2011;38(1):49-62. 72. Powell BJ, McMillen JC, Proctor EK, et al. A compilation of strategies for implementing clinical innovations in health and mental health. Med. Care Res. Rev. 2012;69(2):123-157. 73. Geller AC, Glanz K, Shigaki D, Isnec MR, Sun T, Maddock J. Impact of skin cancer prevention on outdoor aquatics staff: the Pool Cool program in Hawaii and Massachusetts. Prev. Med.  
2001;33(3):155-161.  
74. Glanz K, Geller AC, Shigaki D, Maddock JE, Isnec MR. A randomized trial of skin cancer prevention in aquatics settings: The pool cool program. Health Psychol. 2002;21(6):579-587. 75. Glanz K, Escoffery C, Elliott T, Nehl EJ. Randomized Trial of Two Dissemination Strategies for a Skin Cancer Prevention Program in Aquatic Settings. Am. J. Public Health. 2014:e1-e9. 76. Glanz K, Steffen A, Elliott T, O'Riordan D. Diffusion of an effective skin cancer prevention program: design, theoretical foundations, and first-year implementation. Health Psychol. 2005;24(5):477487. 77. Hall D, Kline M, Glanz K. Analysis of participatory photojournalism in a widely disseminated skin cancer prevention program. Health Promot Pract. 2011;12(5):666-672. 78. Hall DM, Escoffery C, Nehl E, Glanz K. Spontaneous diffusion of an effective skin cancer prevention program through Web-based access to program materials. Prev Chronic Dis. 2010;7(6):A125.  
79. Hall D, Elliott T, Nehl E, Glanz K. Effectiveness of a targeted, peer-driven skin cancer prevention program for lifeguards. International Journal of Aquatic Research and Education. 2008;2:287-297. 80. Hall DM, McCarty F, Elliott T, Glanz K. Lifeguards' Sun Protection Habits and Sunburns:
Association With Sun-Safe Environments and Skin Cancer Prevention Program Participation. Arch.  
Dermatol. 2009;145(2):139-144.  
81. Escoffery C, Glanz K, Elliott T. Process evaluation of the Pool Cool Diffusion Trial for skin cancer prevention across 2 years. Health Educ. Res. 2008;23(4):732-743. 82. Escoffery C, Glanz K, Hall D, Elliott T. A multi-method process evaluation for a skin cancer prevention diffusion trial. Eval. Health Prof. 2009;32(2):184-203. 83. Rabin BA, Nehl E, Elliott T, Deshpande AD, Brownson RC, Glanz K. Individual and setting level predictors of the implementation of a skin cancer prevention program: a multilevel analysis.  
Implement Sci. 2010;5:40.  
84. Elliott T, Glanz K, Nehl EJ. Characteristics of swimming pools with high rates of objectively measured sunscreen use. J. Am. Acad. Dermatol. 2009;60(4):684-688. 85. Glanz K, Isnec MR, Geller A, Spangler K. Process evaluation of implementation and dissemination of a sun safety program at swimming pools. In: Steckler A, Linnan L, eds. Process Evaluation in Public Health Interventions. San Francisco, CA: Jossey-Bass; 2002:58-82. 86. Hall D, Dubruiel N, Elliott T, Glanz K. Linking agents' activities and communication patterns in a study of the dissemination of an effective skin cancer prevention program. J. Public Health Manag. Pract. 2009;15(5):409-415. 87. Amon KL, Campbell AJ, Hawke C, Steinbeck K. Facebook as a recruitment tool for adolescent health research: a systematic review. Academic pediatrics. 2014;14(5):439-447 e434. 88. Batterham PJ. Recruitment of mental health survey participants using Internet advertising: content, characteristics and cost effectiveness. International journal of methods in psychiatric research. 2014;23(2):184-191. 89. Chu JL, Snider CE. Use of a social networking web site for recruiting Canadian youth for medical research. J. Adolesc. Health. 2013;52(6):792-794.  
90. Fenner Y, Garland SM, Moore EE, et al. Web-based recruiting for health research using a social networking site: an exploratory study. J Med Internet Res. 2012;14(1):e20. 91. Lohse B. Facebook is an effective strategy to recruit low-income women to online nutrition education. Journal of nutrition education and behavior. 2013;45(1):69-76. 92. Morgan AJ, Jorm AF, Mackinnon AJ. Internet-based recruitment to a depression prevention intervention: lessons from the Mood Memos study. J Med Internet Res. 2013;15(2):e31.  
93. Park BK, Calamaro C. A systematic review of social networking sites: innovative platforms for health research targeting adolescents and young adults. Journal of nursing scholarship : an official publication of Sigma Theta Tau International Honor Society of Nursing / Sigma Theta Tau.  
2013;45(3):256-264.  
94. Ramo DE, Prochaska JJ. Broad reach and targeted recruitment using facebook for an online survey of young adult substance use. J Med Internet Res. 2012;14(1):e28.  
95. Lane TS, Armin J, Gordon JS. Online Recruitment Methods for Web-Based and Mobile Health Studies: A Review of the Literature. J Med Internet Res. 2015;17(7):e183.  
96. Alexa_Internet_Inc. The top 500 sites on the web. 2012; http://www.alexa.com/topsites.  
Accessed April 21, 2015.  
97. Chang LL, Krosnick JA. National surveys via RDDTelephone Interviewing Versus the Internet:
Comparing Sample Representativeness and Response Quality. Public Opin. Q. 2009;73(4):641-678. 98. Craig BM, Reeve BB. Methods Report on the PROMIS Valuation Study: Year 1. 2013. 99. Liu H, Cella D, Gershon R, et al. Representativeness of the Patient-Reported Outcomes Measurement Information System Internet panel. J. Clin. Epidemiol. 2010;63(11):1169-1178. 100. National_Cancer_Institute. Food Attitudes and Behaviors (FAB) Survey. 2013; http://cancercontrol.cancer.gov/brp/fab/index.html. Accessed August 28, 2014. 101. National_Cancer_Institute. FLASHE: Family Life, Activity, Sun, Health, and Eating Study. 2014; http://cancercontrol.cancer.gov/brp/hbrb/flashe.html. Accessed August 28, 2014. 102. Stein K, Dyer M, Crabb T, et al. A pilot Internet "value of health" panel: recruitment, participation and compliance. Health and quality of life outcomes. 2006;4:90. 103. West R, Gilsenan A, Coste F, et al. The ATTEMPT cohort: a multi-national longitudinal study of predictors, patterns and consequences of smoking cessation; introduction and evaluation of internet recruitment and data collection methods. Addiction. 2006;101(9):1352-1361. 104. Glanz K, Schoenfeld E, Weinstock MA, Layi G, Kidd J, Shigaki DM. Development and reliability of a brief skin cancer risk assessment tool. Cancer Detect. Prev. 2003;27(4):311-315.  
105. Atkinson NL, Massett HA, Mylks C, Hanna B, Deering MJ, Hesse BW. User-centered research on breast cancer patient needs and preferences of an Internet-based clinical trial matching system. J Med Internet Res. 2007;9(2):e13. 106. Heckman C, Darlow S, Munshi T, et al. Development of an Internet Intervention to Address Behaviors Associated with Skin Cancer Risk among Young Adults. Internet Interv. 2015;2(3):340-350.  
107. Bastien JM. Usability testing: a review of some methodological and technical aspects of the method. Int J Med Inform. 2009. 108. Services USDoHaH. The Research-Based Web Design & Usability Guidelines, Enlarged/Expanded edition. . Washington, DC: Government Printing Office; 2006. 109. Giles EL, Robalino S, McColl E, Sniehotta FF, Adams J. The effectiveness of financial incentives for health behaviour change: systematic review and meta-analysis. PloS one. 2014;9(3):e90347.  
110. Mantzari E, Vogt F, Marteau TM. Financial incentives for increasing uptake of HPV
vaccinations: a randomized controlled trial. Health Psychol. 2015;34(2):160-171. 111. Mantzari E, Vogt F, Shemilt I, Wei Y, Higgins JP, Marteau TM. Personal financial incentives for changing habitual health-related behaviors: A systematic review and meta-analysis. Prev. Med.  
2015;75:75-85.  
112. Alexander GL, Divine GW, Couper MP, et al. Effect of incentives and mailing features on online health program enrollment. Am. J. Prev. Med. 2008;34(5):382-388. 113. Khadjesari Z, Murray E, Kalaitzaki E, et al. Impact and costs of incentives to reduce attrition in online trials: two randomized controlled trials. J Med Internet Res. 2011;13(1):e26.  
114. Schierenberg A, van Amsterdam J, van den Brink W, Goudriaan AE. Efficacy of contingency management for cocaine dependence treatment: a review of the evidence. Curr Drug Abuse Rev.  
2012;5(4):320-331. 115. Young JM, O'Halloran A, McAuley C, Pirotta M, Forsdike K, Currow D. Unconditional and conditional incentives differentially improved general practitioners' participation in an online survey:
randomized controlled trial. J. Clin. Epidemiol. 2014.  
116. Manzoni GM, Pagnini F, Corti S, Molinari E, Castelnuovo G. Internet-based behavioral interventions for obesity: an updated systematic review. Clin Pract Epidemiol Ment Health. 2011;7:19-28.  
117. Prestwich A, Kellar I, Parker R, et al. How can self-efficacy be increased? Meta-analysis of dietary interventions. Health psychology review. 2014;8(3):270-285. 118. Brouwer W, Kroeze W, Crutzen R, et al. Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review. J Med Internet Res. 2011;13(1):e2. 119. Franklin VL, Greene A, Waller A, Greene SA, Pagliari C. Patients' engagement with "Sweet Talk" - a text messaging support system for young people with diabetes. J Med Internet Res. 2008;10(2):e20. 120. Bandura A. Social cognitive theory of mass communication. In: Bryant J, Oliver MB, eds. Media Effects: Advances in Theory and Research. New York, NY: Routledge; 2002:94-124. 121. National_Cancer_Institute. Health Information National Trends Survey. http://hints.cancer.gov/. Accessed August 28, 2014. 122. Haghighat R. The Development of the Brief Social Desirability Scale (BSDS). Europe's Journal of Psychology. 2007;3(4). 123. Glanz K, Yaroch AL, Dancel M, et al. Measures of sun exposure and sun protection practices for behavioral and epidemiologic research. Arch. Dermatol. 2008;144(2):217-222.  
124. Lazovich D, Stryker JE, Mayer JA, et al. Measuring nonsolar tanning behavior: indoor and sunless tanning. Arch. Dermatol. 2008;144(2):225-230. 125. Ingledew DK, Ferguson E, Markland D. Motives and sun-related behaviour. Journal of health psychology. 2010;15(1):8-20.  
126. Glanz K, Mayer JA. Reducing ultraviolet radiation exposure to prevent skin cancer methodology and measurement. Am. J. Prev. Med. 2005;29(2):131-142.  
127. Glanz K, McCarty F, Nehl EJ, et al. Validity of self-reported sunscreen use by parents, children, and lifeguards. Am. J. Prev. Med. 2009;36(1):63-69. 128. Glanz K, Gies P, O'Riordan DL, et al. Validity of self-reported solar UVR exposure compared with objectively measured UVR exposure. Cancer Epidemiol. Biomarkers Prev. 2010;19(12):3005-3012. 129. O'Riordan DL, Glanz K, Gies P, Elliott T. A pilot study of the validity of self-reported ultraviolet radiation exposure and sun protection practices among lifeguards, parents and children. Photochem.  
Photobiol. 2008;84(3):774-778.  
130. O'Riordan DL, Nehl E, Gies P, et al. Validity of covering-up sun-protection habits: Association of observations and self-report. J. Am. Acad. Dermatol. 2009;60(5):739-744.  
131. Buller DB, Reynolds KD, Yaroch A, et al. Effects of the Sunny Days, Healthy Ways curriculum on students in grades 6 to 8. Am. J. Prev. Med. 2006;30(1):13-22.  
132. Buller DB, Taylor AM, Buller MK, Powers PJ, Maloy JA, Beach BH. Evaluation of the Sunny Days, Healthy Ways Sun Safety Curriculum for Children in Kindergarten through Fifth Grade. Pediatr. Dermatol. 2006;23(4):321-329. 133. Cafri G, Thompson JK, Jacobsen PB. Appearance reasons for tanning mediate the relationship between media influence and UV exposure and sun protection. Arch. Dermatol. 2006;142(8):1067-1069. 134. Robinson NG, White KM, Young RM, et al. Young people and sun safety: the role of attitudes, norms and control factors. Health Promot J Austr. 2008;19(1):45-51. 135. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR mHealth and uHealth.  
2015;3(1):e27.  
136. Foster EM, Dodge KA, Jones D. Issues in the Economic Evaluation of Prevention Programs.  
Appl Dev Sci. 2003;7(2):76-86. 137. Honeycutt AA, Khavjou OA, Jones DJ, Cuellar J, Forehand RL. Helping the Noncompliant Child: An Assessment of Program Costs and Cost-Effectiveness. J Child Fam Stud. 2015;24(2):499-504.  
138. Khavjou OA, Honeycutt AA, Hoerger TJ, Trogdon JG, Cash AJ. Collecting costs of community prevention programs: communities putting prevention to work initiative. Am. J. Prev. Med. 2014;47(2):160-165. 139. French MT, Dunlap LJ, Zarkin GA, McGeary KA, McLellan AT. A structured instrument for estimating the economic cost of drug abuse treatment. The Drug Abuse Treatment Cost Analysis Program
(DATCAP). J. Subst. Abuse Treat. 1997;14(5):445-455.  
140. Zarkin G, Dunlap L, Homsi G. The substance abuse services cost analysis program (SASCAP): A
new method for estimating drug treatment services costs. Eval. Program Plann. 2004;27(1):35-43. 141. Little RJA, Rubin DB. Statistical Analysis with Missing Data. 2nd edition ed. New York: John Wiley; 2002. 142. Ten Have TR, Pulkstenis E, Kunselman A, Landis JR. Mixed effects logistic regression models for longitudinal binary response data with informative dropout. Biometrics. 1998;54:367-383. 143. Bauermeister J, Pingel E, Zimmerman M, Couper M, Carballo-Dieguez A, Strecher VJ. Data Quality in web-based HIV/AIDS research: Handling Invalid and Suspicious Data. Field methods.  
2012;24(3):272-291.  
144. Bowen AM, Daniel CM, Williams ML, Baird GL. Identifying multiple submissions in Internet research: preserving data integrity. AIDS Behav. 2008;12(6):964-973.  
145. Bull SS, Vallejos D, Levine D, Ortiz C. Improving recruitment and retention for an online randomized controlled trial: experience from the Youthnet study. AIDS Care. 2008;20(8):887-893.  
146. Huang JL, Curran PG, Keeney J, Poposki EM, DeShon RP. Detecting and Deterring Insufficient Effort Responding to Surveys. Journal of Business Psychology. 2012;27:99-114.  
147. Konstan JA, Rosser BRS, Ross MW, Stanton J, Edwards WM. The Story of Subject Naught: A
Cautionary but Optimistic Tale of Internet Survey Research. Journal of Computer Mediated Communication. 2005;10(2). 148. Koo M, Skinner H. Challenges of internet recruitment: a case study with disappointing results. J
Med Internet Res. 2005;7(1):e6.  
149. Meade AW, Craig SB. Identifying careless responses in survey data. Psychol Methods.  
2012;17(3):437-455.  
150. Reips UD. Standards for Internet-based experimenting. Exp Psychol. 2002;49(4):243-256.  
151. Quach S, Pereira JA, Russell ML, et al. The good, bad, and ugly of online recruitment of parents for health-related focus groups: lessons learned. J Med Internet Res. 2013;15(11):e250.  
152. Meade AW, Craig SB. Identifying careless responses in survey data. Psychological Methods.  
2012;17(437-55).