Frequency of Gobbledygook. Content was evaluated using a scoring matrix based on materials available on the Center for Disease Control website. The Patient Education Material Assessment Tool for Print (PEMAT-P) was used to assess usability and actionability.
A total of 141 (97%) PEMs met inclusion criteria and were analyzed for readability, content and quality.
The mean readability was above the recommended 6 th grade reading level (P<0.001). Content was variable across PEMs. The PEMAT-P scores reflected good understandability with a median score of 83% (IQR 75-87%), while actionability was poor with a median score of 41% (IQR 40-60%).
Despite availability of web-based PEMs for COVID-19, the readability was significantly higher than the NIH and USDHHS recommended 6 th grade reading level and actionability of PEMs was low. It is critical to provide readable patient education materials on COVID-19 to effectively disseminate accurate information and facilitate patients' understanding of the virus, how it spreads, and how to protect themselves.
Keywords: COVID-19, SARS-CoV-2, patient education materials, readability
 COVID-19 PEMs are readily available on U.S. academic medical websites.  Content of COVID-19 PEMs is highly variable.  PEMs on COVID-19 are written above the recommended 6 th grade reading level.  PEMs on COVID-19 are critical to disseminate accurate information.
The novel Coronavirus, SARS-CoV-2, and subsequent COVID-19 pandemic has led to a public health crisis. [1, 2] In the United States, the COVID-19 outbreak has had profound economic and social impact as a result of statewide quarantines, social distancing and public face mask mandates. One significant consequence has been the effect on the healthcare system, which as of September 2020 has seen over 6 million cases and 185,000 fatalities nationwide.
[3]
Patients infected with SARS-CoV-2 can present with a myriad of symptoms, which can range from mild to severe. Typical symptoms include fever, cough, shortness of breath, and muscle pain. [4] Severe complications have been reported in 33% of patients, including acute respiratory distress syndrome, septic shock and severe pneumonia. [5, 6] Currently, there is no treatment or approved vaccine against SARS-CoV-2 and cases worldwide continue to rise.
Our understanding of this infection has evolved and expanded since the start of the pandemic.
However, there continues to be a number of uncertainties regarding the wide-ranging manifestations of the disease, it's appropriate treatment and management, and optimal ways of preventing transmission. [7- 10] Additionally, there are many misconceptions among the general public and a need for reliable information for the public. The internet is a frequently used resource for providing patient education materials (PEM). Studies show that of the 78% of adults in the US who have internet access, 83% of them utilize the internet to search for health information.
[11] However, much of the health information available online is written above the sixth grade reading level recommended for PEM by the National Institute of Health (NIH) and U.S. Department of Health and Human Services (USDHHS). [12] It is critical to maximize the readability of patient education materials on COVID-19 to effectively disseminate accurate information. Increasing public knowledge about this infection can potentially improve health outcomes by teaching early identification of symptoms and effective methods for preventing the spread of disease. The primary aim of this study was to evaluate the readability, content, and quality of web-based patient education materials on COVID-19 from U.S. academic medical centers.
This article was prepared in accordance with the guidelines for "Strengthening the Reporting of Observational Studies in Epidemiology" (STROBE). [13] The names of U.S. medical schools were obtained from the Association of American Medical Colleges (AAMC) website (n=145). Institutional websites, hospital websites and heath encyclopedia websites associated with each schools' medical center were identified using the Google search engine. All websites were searched for PEMs using the following key words: coronavirus, COVID, COVID-19, and SARS-CoV-2. Search for PEMs occurred from June 17, 2020 through June 26, 2020. PEMs were excluded if they were written in languages other than English or information was only presented in graphic, pictoral, video, or tabular format. PEMs that were linked from non-academic websites were also excluded. All PEMs were evaluated for readability, content, and quality.
Text from PEMs was copied into a plain text document using Microsoft Word and edited. Text unrelated to educational material, including advertisements, website URLs, copyright information, navigation tools, addresses, and citations, was removed as it may influence readability scores. Frequency of Gobbledygook (Gunning FOG). [15, 16] Each uses a unique formula to evaluate readability based on a combination of mean number of syllables per word, mean number of words per sentence, numbers of sentences and numbers of polysyllabic words.[14, 17, 18] Each readability index reflects an estimated grade level that is required to read and comprehend a text without difficulty, therefore the lower the index, the easier the text is to read. A score of 6 in any of the indices reflects a 6 th grade reading level.
The content of each PEM was evaluated using a scoring matrix developed by the authors (JK, EL, PT, WG, CE) based on COVID-19 educational materials found on the CDC website on June 10, 2020. [19] The matrix consisted of 67 items organized into 10 domains: (1) definition of coronavirus, (2) signs and symptoms of the illness, (3) testing, (4) disease spread, (5) primary prevention, (6) what to do if ill, (7) timing of exposure to others after the disease, (8) at-risk populations, (9) what to do if you have had an exposure, and (10) additional resources.
The quality of PEMs was evaluated using the Patient Education Material Assessment Tool for Print (PEMAT-P). [20] PEMAT-P is a validated tool used to assess the quality of PEM by evaluating the understandability and actionability of a document. [20] Understandability is a metric that appraises the content, word choice, style, organization, use of numbers, lay out, design, and use of visual aids.
Actionability is whether an action is identifiable in the text and how easily that action can be acted on.
PEMAT-P scores range from 0-100% in either domain. A score of less than 70% reflects poor performance. [20] Descriptive statistics were used to summarize the data. The mean readability scores of the PEMs were compared, using a one-sample t test, to the scores that indicted a 6 th grade reading level. Content proportions were analyzed using binomial distribution, with 95% confidence intervals (CIs). All data were analyzed with STATA SE (Version 12, College Station, TX). This study was exempt from Institutional Review Board (IRB) approval per Northwestern University's IRB guidelines.
Of the 145 US medical schools identified, 100% of their associated medical centers had PEMs on COVID-19. Four of the Spanish-language PEMs were excluded, leaving 141 (97%) PEMs that were analyzed for readability, content, and quality.
The mean readability of PEMs was above the recommended 6 th grade reading level by all indices (Table 1) . When each individual PEM was evaluated, only one PEM (0.7%) had a FKGL readability at or below the 6 th grade reading level. Two PEMs (1.4%) had a SMOG readability at or below the 6 th grade reading level, while three (2%) met this threshold using the Gunning FOG index.
The PEMs were evaluated for the inclusion of 67 items across 10 domains (Table 2) . Fever was the most frequently documented symptom in the PEMS (98% of PEMs). Ninety-seven percent of PEMs instructed patients to contact their health care provider if they were ill. Sixty two percent of PEMs discussed seeking immediate medical care, yet very few referred to the specific signs that should prompt such care: trouble breathing 43%, persistent chest pain 30%, inability to stay awake 28%, and blue lips or face 24%. Several websites discussed at-risk populations, with 63% discussing the elderly (age >65 years) for being at risk for COVID-19, and only 15% of PEMs referencing obesity as a risk factor. While many PEMs addressed testing, only 26% addressed the implications of a positive test and only 21% addressed the implications of a negative test. Even fewer (16%) discussed the asymptomatic spread of disease. While the use of face coverings was suggested in 75% of PEMs, just 38% described the proper use of face masks. Criteria for return after illness was infrequently discussed with fewer than 15% of PEMs discussing possible criteria for return to work based on symptoms or days since a positive test. The CDC was the most commonly cited additional resource (94% of PEMs).
The PEMAT-P scores for PEMs reflected good understandability with a median score of 83% (IQR 75-87%). Sixteen (11.6%) PEMs had below 70% score suggesting poor understandability.
Actionability of the PEMs was poor with a median score of 41% (IQR 40-60%). Only 16 (11.6%) PEMs had an actionability above 70% suggesting good actionability.
In this study, we have demonstrated that although all medical centers associated with U.S. medical schools had web-based patient education materials about COVID-19, the readability was significantly higher than the NIH and USDHHS recommended 6 th grade reading level and actionability of these PEMs was low.
As literacy of the general public is limited, comprehension of the general public should also be considered. [21] The importance of comprehension of PEMs on COVID-19 should ideally allow the public not only to understand material but also to act upon it. While the majority of the PEMs had good understandability, the actionability was considered poor in most cases. Despite high reading levels of PEMs, the layout and presentation of the material, which is not assessed by the readability indices, allows for improved understanding of the PEMs. Actionability is particularly important for COVID-19 PEMs as a community faced with a global pandemic should be able to take steps to protect itself and how to proceed if illness occurs. Improvement of actionability could use decision trees emphasizing test results and symptoms to determine quarantine length, testing, and treatment as well as when to seek emergency medical care.
Our content rubric was based on the CDC's coronavirus resources. [19] As the nation's health protection agency, the CDC is the leading federal agency for America's public health initiatives, and the vast majority of PEMs listed the CDC as an additional resource. Our data revealed that almost all PEMs addressed the three most common COVID-19 symptoms: fever, cough, and shortness of breath. [22] However, PEMs frequently did not detail other symptoms associated with COVID-19 such as the associated gastrointestinal symptoms, which account for 18% of presenting symptoms.
[23] Given the contagious nature of this disease, [24] it is imperative that the infected public not only recognize symptoms of the disease, but also employ safety measures to decrease risk of exposure to others.
[25] While the majority of PEMs addressed social distancing, hand hygiene, and use of face coverings, few described the proper use of masks, which has been associated with flattening of the curve in many areas in the United States. [26] Almost all PEMs emphasized the importance of contacting a health care provider if a patient is ill with COVID-19 like symptoms, however few addressed emergency warning signs to seek immediate care. Further, only half of PEMs emphasized that ill individuals should isolate from other people. Testing is key to identifying and limiting spread of the novel coronavirus. [27] Description of testing types may prove useful in public decision making as polymerase chain reaction (PCR) and immunoglobulin M (IgM) testing is reflective of current infection, while immunoglobulin G (IgG) tests for past infection. While many PEMs discussed testing for the virus, less than a third discussed what the results of a positive or a negative test mean. Testing for the virus has evolved over time, with some tests having a false negative rate as high as 29%. [28] The possibility and implications of a false negative is important to include in PEMs as a negative result may lead to a false sense of security. [29] As the pandemic has progressed, criteria for safe return after illness has continued to change.
[30] Yet, it was rare for PEMs to address when patients could return after symptoms abate or after an asymptomatic positive test result. [31] Interestingly, it was rare for PEMs to describe the high-risk populations for COVID-19.
Early data released by the World Health Organization showed how those with co-existing disease are at greater risk, particularly those with obesity, hypertension, chronic lung disease, diabetes mellitus and cardiovascular disease. [32] Only a quarter of those at-risk populations were addressed within the PEMs; furthermore, risk factors of obesity and residence in long-term care facilities were mentioned least. This is of concern as 68% of Americans are considered obese. [33, 34] Furthermore, as seen in other viral diseases such as influenza, the elderly are disproportionately affected. [34] Future PEMs need to ensure they address those that are most at risk.
Our study must be evaluated in the context of its limitations. PEMs were gathered at one point in time in an evolving globing pandemic and information provided may be outdated by the time patients are searching for it and using it. As guidelines from the CDC continue to change, our rubric was created prior to identifying PEMs in an attempt to minimize discrepancies in the rubric content and PEMs. The PEMs could only be collected from websites functioning properly; when a website was malfunctioning, the data collected were based on what could be readily obtained by the general public. While we chose to evaluate academic medical center online PEMs, patients may prefer to use non-academic websites when searching online health information. We chose to focus on academic medical centers, as their content may be more accurate than the information found elsewhere on the Internet. Additionally, there are multiple scoring instruments that can be used to assess readability scores; three validated scores were used for this paper to provide a varied evaluation, but these choices were not exhaustive. [15, 16] 
The public uses the Internet for health care concerns, and there is a plethora of COVID-19 materials available. Our study found that the readability, content, and quality of Web-based patient education materials addressing COVID-19 is lacking. The coronavirus global outbreak has been called a once-in-a-century pandemic, [35] necessitating adequate and effective communication from the medical community to the general public. The best way to do so is to create materials that are written at the recommended reading level, and are understandable, and are actionable. 