News media influence our understanding of, our beliefs about, and our attitudes toward what is happening around us [12] . On the one hand, by selecting what to report-which is not always aligned with what people want to know [27] -news media can create biased public awareness ("agenda-setting") [30] . For instance, the proportion of Gallup poll respondents who called the Gulf crisis the most important national problem could be simply explained by the amount of its media coverage [22] . On the other hand, by determining how to report ("framing"), news media can potentially alter public attitudes on particular issues [2, 7, 28, 34, 42] . Framing is implemented by emphasizing certain aspects of an issue more than other aspects and making them more salient [7, 13] . For example, when reporting on the issue of poverty, some news media may focus on unemployed individuals, while others may focus on national policies. Such differences in focus have been shown to be pivotal; according to one study [21] , those who are exposed to the former framing of poverty were more likely to blame it on individual failings, while those exposed to the latter were more likely to blame it on the government or other forces beyond their control. Therefore, to understand their influence on the public, it is crucial to study framing-in addition to the agenda setting-in news media.
In contrast to the agenda-setting and selection bias, which have been extensively studied [22, 26, 39] , media framing has posed challenges to researchers because of the inherent complexity and vagueness of the concept. Media frames are difficult to concretely operationalize as they are often abstract and subtle [5, 29] . Most of the studies on media frames are thus conducted based on manual labels obtained for a handful of specific issues and issue-specific frames, such as "global warming" vs. "climate change" [41] , "gay civil unions" vs. "homosexual marriage" [35] , "punishment" vs. "innocence" (in relation to the death penalty) [2] , and more [40] . Finding a systematic approach to media framing that transcends these issues remains a critical challenge.
A prominent trend in the research that aims to address this limitation is the annotation of a large corpus of news articles with a standardized set of media frames that are universal across multiple issues. Probably the most notable such effort is the "Media Frames Corpus" (MFC) [6] , which labels 20,037 news articles from 13 U.S. national newspapers on the topics of immigration, smoking, and same-sex marriage with one of 15 topic-agnostic general media frames (shown in Table 1 ) [5] .
Here, leveraging the MFC, we develop a general media frame classifier and apply it to the near-complete set of news articles published in the New York Times between January 2000 and December 2017, aiming to understand the temporal and system-wide patterns. The New York Times archive has been a subject of many studies [8, 11, 36] because it does not only exhibit one of the largest digitized news article databases yet available but also is one of the most prominent agenda-setting media outlets, widely read by the public and policy makers [17, 25, 33] .
Our media frame classifier enables us to investigate media frames of the large-scale news corpus that cannot be manually analyzed due to its scale. In contrast to previous studies focusing on news about particular issues over shorter periods, our study infers general media frames from 1.5M news articles over 18 
To estimate the prevalence of each media frame, we first build an article-level media frame classifier. It is trained to predict the primary frame of an individual article using the labeled Media Frames Corpus (MFC) 3 . We used the state-of-the-art language representation model, BERT (Bidirectional Encoder Representations from Transformers) [9] . We fine-tuned the pre-trained BERT-base model 4 with our training data using a small learning rate, 0.0002, a maximum sequence number of 128, a batch size of 32, and a number of training epochs of 3. Since the MFC dataset is imbalanced, we added class weights in the loss function. The performance of the model, trained on 11k articles, is F 1 =71.34, based on a test set of 1,138 articles. The best accuracy yet reported among similar approaches using the same corpus, was 58.4% on the "Immigration" subset, in the previous study by Ji and Smith (2017) , which is excelled by our presented method. In addition to the quantitative evaluation, we present qualitative validation of the identified frames using the keywords of the articles. New York Times articles include META keywords in the form of Table 2 shows the top 10 most frequent META keywords and their top two most frequent frames. We also present the prevalence of the most frequent frame. We can see intuitive agreements between the top META keywords and the detected frames. For example, the Cultural identity and Quality of life frames are frequently found in articles with culture-, art-, and entertainment-related keywords, such as Books and literature, Music, Baseball, and Reviews. The Security and defense frame is found in articles about Terrorism, and the Political frame in articles with the keywords Politics and government and United States politics and government.
Similarly to the keyword-level analysis, we now compare frequent frames between news sections. New York Times articles embed sections in a full URL. For example, the URL https://www. nytimes.com/2020/02/21/health/coronavirus-cases-usa.html shows that the article appears in the 'Health' section. Table 3 shows the two most frequent frames in the top five news sections, including the prevalence of the most frequent frame. We note that the 'Sports' section (1st) has 146,732 articles, and the 'Arts' section (5th) has 
To characterize articles of each frame, we investigate unigrams that are over-represented in each frame. To reliably compute the overrepresentation, we use log-odds ratios with informative Dirichlet priors [31] . This method estimates the log-odds ratio of each word w between two corpora i and j given the prior frequencies obtained from a background corpus α. The log-odds ratio for word w, δ i−j w is estimated as:
where n i (resp. n j ) is the size of corpus i (resp. j), y i w (resp. y j w ) is the count of word w in corpus i (resp. j), α 0 is size of the background corpus, and α w is the frequency of word w in the background corpus. Furthermore, the method provides an estimate for the variance of the log-odds ratio and z-score, as follows:
We identify words associated with each frame with an exploratory analysis using the log-odds ratio. To remove noisy estimates and rare words, we select words that appear at least 100 times in the entire corpus. We then extract all unigrams from the news articles and compute their log-odds ratio using Eq. (1). For the prior, background word frequency is computed from the entire New York Times dataset, rather than the sample. The words are then ranked by their estimated z-scores, computed using Eq. (3). The 20 most over-represented unigrams for each media frame are presented in Table 1 . The top five words for the Crime and punishment frame are police, prosecutors, charges, officers, and arrested. For the Cultural identity frame, culture-related words like theater, museum, game, and music are shown in the table, while for Quality of life, familyrelated words like mother, fathers, family, beloved, and life are appeared. For the Fairness and equality frame, the top five words are editor, article, writer, editorial, and op, indicating that this frame tends to appear in Opinion articles in the New York Times. Overall, the over-represented words show a strong association with their corresponding frame. This improves our understanding of how different media frames are used in news reporting.
In this section, we answer our first research question, RQ1: What are the long-term trends of framing? We examine the temporal abundance of each frame across all news articles published in the New York Times in the period under investigation. As each article is assigned a single media frame, we simply calculate the monthly average fractions of each frame and tack them (the "Other" frame is excluded). Figure 1 shows the evolution of each frame over time. The first long-term trend that we can observe is the consistent prominence of the Cultural identity (orange), Quality of life (gray), and Economic (light orange) frames, as well as the gradual shift from economicto culture-and quality of life-related frames. This long-term trend may be a reflection of the general societal trend toward more sophisticated cultural needs [18] and market forces [19] The Political frame (purple) fluctuates with the election cycle. Its two strongest surges are in 2000 and 2016, coinciding with the two presidential elections that elected Republican presidents and created a lot of controversy. It is also interesting to see that the proportion for the Political frame stays high after the 2016 presidential election, indeed higher than before; the monthly average of in the 12 months from January 2017 (0.100) is higher than for the 12 months to September 2016 (0.089), and the difference is statistically significant by the Mann-Whitney U Test (U=39.0, p = 0.030). The other prominent peak corresponds to the Iraq War. We observe a surge in the Security and defense frame in 2001 after the September 11 attacks, but overall it has become less prominent over time.
The External regulation frame surged during two time periods: from 2001 till mid-2003, probably due to the September 11 attacks, and after 2017, due to the Muslim travel ban and the strengthening of immigration regulation in the U.S. The Policy prescription frame formerly comprised 5% of all articles on average, but this suddenly dropped to 2.5% in late 2004 and has shown little change since then. The remaining frames show a low but consistent level over time; on average, the Morality, Legality, Health, Crime, and Public opinion frames are included in 4.2%, 4.4%, 5.1%, 5.2%, and 2.6% of all articles, respectively.
In summary, the four most prominent frames -Cultural identity, Quality of life, Economic, and Political -compete with each other, and their historical prevalence reflected major social events. Furthermore, the Cultural identity frame steadily increases in prevalence.
While the total prevalence of each frame is fairly stable over time, this does not necessarily mean that the issues or the topics are also stable. To examine the temporal evolution of frames in relation to different topics, we extract the most representative words for each frame over time by using the log-odds ratio. We first extract articles representing each frame published in each year. We then extract all unigrams from each one-year sample and compute their log-odds ratios against the all-other-years sample using Eq. (1). For the prior, background term frequency is computed for all articles relating to the corresponding frame. The terms are then ranked by their estimated z-scores, computed using Eq. (3). We present the top 15 most representative words in the Economic and Security & defense frames for a selected time period in Table 4 . The words in each year clearly show the issues presented with the corresponding frame. For example, 'giuliani', mayor of New York City during 9/11, and 'dynegy', a company that filed for bankruptcy protection, appear in 2001, 'katrina', 'orleans', and 'hurricane' in 2005, 'goldman', 'greece', 'banks', and 'ipad' in 2010, 'eurozone', 'puerto rico', and 'volkswagen' in 2015, and 'bitcoin', 'obamacare', 'equifax', and 'amazon' in 2017. The hurricane-related words in 2005 support previous findings that cost is one of the most important dimensions in reporting on natural disasters [20] .
The Security frame also reveals relevant keywords, for example, 'taliban', 'hijacked', and 'osama bin laden' in 2001, 'orleans' and 'hurricane' in 2005, 'wikileaks', 'bp', and 'spill' in 2010, 'islamic', 'paris', 'isis', and 'hebdo' in 2015, and 'marawi', 'rohingya', 'hacking', and 'korea' in 2017. Conflicts in the Middle East, North Korea, and the Philippines, and recent privacy issues around Facebook are framed as threats to the United States (e.g., the U.S. supports the Philippine Government around Marawi conflicts) and thus relate to the Security frame.
Some issues can be associated with multiple frames because they are inherently multidimensional [40] . For example, the 9/11 attacks in 2001 and Hurricane Katrina in 2005 had a massive impact, and thus can be expected to be reported with various frames. Our results show an interesting contrast between the two frames for a news event. After the 9/11 attacks, the Economic articles tend to provide more reporting on companies (e.g., 'swissair' or 'dynegy'). In contrast, the Security and defense articles focus more on the hijackers and their affiliates. Likewise, we see that Hurricane Katrina not only had massive economic impact (e.g., the value of houses with three 'bedrooms'), but also naturally brought up the security of the nation against natural disasters (e.g., 'homeowners' and 'privatization').
We now move on to our second research question: RQ2: What are the framing patterns at the issue level? While the New York Times shows a fairly stable temporal pattern of frame abundance with some exceptions, specific issues may not be stable at all. Our question, for instance, in relation to the topic of abortion, whether there has been a consistent framing.
To quantitatively estimate the temporal change in issue-level framing, we use keyword matching to collect news articles around a particular issue and examine the temporal evolution of prominent frames. Two different types of issues are considered: 1) policies (e.g., 'Abortion', 'Smoking', and 'Immigration') and 2) events and crises (e.g., 'Football', 'Hurricane', and 'Shooting'). These issues are chosen as illustrative examples to show different patterns of frame dynamics; similar dynamics are shared across other issues. Figure 2 shows a stream graph of the total counts of news articles around various issues over time (at the month level), broken down into the different media frames. For 'Abortion' (Fig.2(b) ), we find that the Political and Legality frames dominate, while the presence of Morality is more noticeable compared with other issues. For 'Smoking' (Fig.2(c) ), we see that the Cultural, Quality of life, and Health frames are consistently more prevalent than other frames. We also see that the Policy and External regulation frames show a sudden increase in 2003, which may be due to New York City's smoke-free law that banned smoking in almost all restaurants and bars in 2003. For 'Immigration' (Fig.2(d) ), we see the Security frame increasing in prevalence in 2001 after the September 11 attacks, and the prevalence of the Political frame increasing around the 2006 midterm elections. This suggests that the discussion on immigration tended to be increasingly associated with security after September 11, and became one of the key issues in subsequent elections.
For 'Football' (Fig.2(e) ), we see a cyclic pattern with a recent decreasing trend that synchronizes with the playing seasons. The two frames relating to entertainment, the Cultural and Quality of (a) Legend life frames, have the strongest prevalence, as expected. For 'Hurricane' (Fig.2(f) ), we see sporadic surges in the volume of news articles as hurricanes hit the U.S. Generally, the Economic frame seems to be the most prominent in the early stage of hurricanes, prevalence gradually decreasing, while the Cultural frame appears consistent over time. The importance shown by the Economic frame corroborates the results of previous studies [20] . Lastly, 'Shooting' has consistently received great attention with a few surges around mass shooting incidents ( Fig.2(g) ). The most prominent frames are the Crime, Cultural, and Quality of life frames, but in recent years, the Political and Legality frames increased in prevalence, probably due to the more active public discussion of gun control laws. We investigate this phenomenon more closely in Section 7.
The results indicate that the eminence profile of general frames can be used as a signature for characterizing events or issues. Moreover, tracking the most prominent frames on a particular issue can reveal how the focus of the media changes over time.
We now focus on the shorter time span and track how issues develop from the perspective of media frames. Are there any framing patterns moving between the early, mid, and late stages of issue development? To answer this question, we focus on news articles about two mass shootings (in Orlando in 2016 and Las Vegas in 2017) and two hurricanes (Katrina and Sandy), which received much attention from news media. We extract the news articles by keyword matching and limit results to those published within four weeks from the event (for the hurricane event dates, we used 2005/08/23 for Katrina and 2012/10/22 for Sandy, i.e., the dates the hurricanes were formed). The four-week window was chosen based on previous studies of media coverage of mass shootings [8, 33, 33] . Compared to mass shootings, natural disasters produce a much more diverse news lifespan, even lasting multiple years [20] . We consider a month as a long-enough time window to capture the most fundamental news cycle and levels of public attention. We then divide the news articles into three groups, which relate to the early, mid, and late stages of issue development, respectively, based on publication dates.
We find diverse framing patterns along with the issue development. Hurricane Katrina, the costliest hurricane in U.S. history, was described most commonly with the Economic frame, while Hurricane Sandy was reported most commonly with the Cultural identity frame in the early and late stages, with the Quality of life frame prevailing in the mid stage. The Orlando nightclub shooting, the deadliest incident of violence against LGBT people in U.S. history, was reported most commonly with the Morality frame in the early stage and with the Political frame in the late stage. The Las Vegas shooting was reported with the Crime and punishment frame in the early stage and the Cultural identity and Quality of life frames in the later stages.
We, however, consistently find framing convergence across the different issues, meaning that the dominance of the most dominant frame at the late stage is stronger than that of its counterpart at the early stage. For example, while the prevalence of the Morality frame in the early stage of the Orlando shooting coverage was 18.6%, that of the Political frame in the late stage reached 40.7%. Similarly, the prevalence of the Economic frame increased from 21.4% at the early stage to 27.5% at the late stage in the coverage of Hurricane Katrina. This tendency is related to frame-changing [8, 32] . To keep an issue fresh to readers, journalists shift frames over time within an issueattention cycle. For example, the emphasis of the news coverage of the Columbine school shooting moved from a focus on personal details to one on societal problems over the course of a month [8] , which is somewhat aligned with the framing convergence from the Mortality to the Political frame in relation to the Orlando shooting.
The framing convergence can be driven by a combination of multiple potential mechanisms. At the early stage, there are lots of unknowns. News media need to report what is happening at the present moment from multiple perspectives. Moreover, the number of articles is high. Media are thus simply obligated to use diverse frames to report an issue at the early stage. Actual convergence on the importance of certain aspects may then take place in social discourses. While various perspectives are reported at the early stage, as time goes by, the public and the news media may form stronger consensus on the most important perspectives [44] .
Here, we investigate our next research question: RQ3: How is each frame delivered? Is there a specific linguistic style (e.g., sentiment)? Intuitively, some media frames might be likely to seek to evoke a certain sentiment. For example, news articles with the Crime and punishment frame may be typically written with a negative sentiment. As the sentiment of the news articles can influence user perception, news reading behavior, and propagation dynamics [38] , understanding the association between media frames and their sentiment helps to assess the impact of these frames. We measure sentiment of each news article by using the widely adopted VADER Sentiment Analyzer [4] . While the sentiment of a word can be changed by the context [1] , a lexicon-based method has been validated with a variety of massive text data, such as books [37] , chats [24] , song lyrics [10] , news articles [38] , and tweets [15] . It should be noted that the sentiment of a headline can be taken as a proxy for the key sentiment of the corresponding article, because headlines are expected to express a concise summary of the article [3] . Figure 3 shows the number of articles with each frame (x-axis) and their average sentiments (y-axis), which are computed monthly. The clustering of the dots from the same frame is visible. In other words, each media frame is delivered with a particular sentiment, and such associations are quite stable across different events occurring in the last 18 years. Figure 4 shows the monthly average sentiments of the news articles for each frame over time. As the closely clustered dots for each frame in Figure 3 imply, in general, the sentiments associated with each frame are stable (average variation<0.1) over time. The solid black line represents the monthly average sentiment. Interestingly, the line is close to zero with a slightly negative bias, meaning that More interestingly, the aggregated neutral sentiment is also observed in a set of articles on specific issues that could be expected to have negative sentiment. Figures 5 and 6 show the sentiment of the frames, as calculated from news articles relating to Hurricanes Katrina and Sandy, respectively. We extract news articles by using the keywords 'Hurricane Katrina' and 'Hurricane Sandy', and limit the articles to those published in the month after the hurricanes formed.
In contrast to our expectation that disaster-related news be written with a negative sentiment, the aggregated sentiment is, to some extent, neutralized by positive news. For example, news about aid or relief can be delivered through various frames, including the Economic or Capacity and resources frames. Or, dramatic and touching human-interest stories are delivered through the Cultural identity frame. Thus, even disaster-related news does not solely convey a negative mood but tends to be balanced. While it is debatable whether this 'balanced' sentiment is an intentional consequence of editorial decisions [16] , we would note that, in spite of some media frames that are typically written in a negative tone, the aggregated sentiment of temporally or issue-specific news articles can be neutral.
In this section, we focus on mass shootings in the United States as a case study to demonstrate how issue-agnostic frames can help us understand the differences in framing among similar events. In contrast to the previous media studies on disasters [20] or shootings [33] , which relied on manual coding, we demonstrate the potential of large-scale media frame research. In Section 5.1, we have shown that the media coverage of the two mass shootings went through different development patterns in terms of the issue framing. We now extend this analysis on a more systematic footing. We first compile a list of 11 mass shootings that occurred in the U.S. from 2000 to 2017 [43] , shown in Table 5 . We then extract articles appearing during the month after the incident using 'shooting' and its location as keywords. One month is the most commonly used time window for studying the media coverage of shootings [8, 33, 33] . For each of the 11 shootings, we compute the proportion of each media frame and construct a 14-dimensional vector whose dimension maps onto the corresponding frame. Figure 7 shows the clusters of mass shootings by hierarchical clustering of mass shooting vectors. We use Euclidean distance as the distance metric and the Ward variance minimization algorithm as the linkage method. We refer to the green cluster (which covers the Orlando nightclub shooting and the Sandy Hook Elementary School shooting) as C д , the red cluster (which covers the Fort Hood shooting, the Washington Navy Yard shooting, the San Bernardino attack, and the Virginia Tech shooting) as C r , and the sky-blue cluster (which covers the rest of the shootings) as C s . The average prevalences of each media frame for C д , C r , and C s are presented in Table 6 . We also mark the discriminating frames if the average proportion of a particular frame in one cluster is one standard deviation away (either above (up arrow ↑) or below (down arrow ↓)) from that in the other two clusters.
The discriminating frames show which perspectives were more emphasized in the reporting of each cluster of mass shootings. We find that the two shootings (C д ), whose victims were minority groups, tend to produce more articles on public safety, morality, Table 6 : Average prevalences of each media frame for mass shooting clusters.
and political impact. Considering that the Sandy Hook Elementary School shooting provoked a national debate on gun control and the Orlando gay nightclub shooting occurred during the final stages of the 2016 United States presidential election campaign, it is understandable that the Political frame is particularly prominent. The four shootings (C r ), respectively at the military base (Fort Hood), the Naval Sea Systems Command (Washington Navy), the government-funded public benefit corporation (San Bernardino), and the public university (Virginia), are reported with a stress on crime and threats to welfare. The remaining five mass shootings in C s show a higher prevalence of the Cultural identity and Quality of life frames than the other two clusters. The Cultural identity frame is used when explaining the motives of suspects, and the Quality of life frame is used for a human-interest story.
The discriminating frames with a down arrow in Table 6 can be understood to represent the news stories that were not published. Since the number of articles published per day is still limited, even in the online era, newsrooms may first develop more critical stories and delay others (and sometimes cancel them if the story loses news value due to delays).
This work has built a media frame classifier and used it to conduct a systematic media frame analysis of 1.5 million news articles from the New York Times published from 2000 to 2017. We fine-tuned the pre-trained BERT-base model using the Media Frames Corpus and achieved a higher performance than the best-existing method, which uses discourse structure and a recursive neural network. We addressed research questions about the long-term trends, issue-level framing dynamics, and delivery of frames in news reporting. We also showed that issue-agnostic media framing can be a helpful tool to differentiate articles about similar events (e.g., mass shootings).
We found that, although the most prominent media frames are fairly stable, there exist long-term trends of highly predictive fluctuation in relation to major social events. The gradual shift from economic-to culture-and quality of life-related frames could be explained by the societal trend toward more sophisticated cultural needs [18] and the influences of market forces [19] over time. We also found that the eminence profile of topic-agnostic frames can be used as a signature for characterizing events or issues. By digging into the framing patterns of each issue, we discovered frame convergence, whereby the dominance of the most dominant frame at the late stage is stronger than that of its counterpart at the early stage. This might reflect the many unknowns in the early stage and the consensus formed in the late stage. Moreover, we studied the association between media frames and average sentiment. We showed that even though some frames (e.g., Crime) are strongly connected to a specific one (e.g., negative), the aggregate tone across the time and topics is almost neutral. By delving into the news coverage of mass shootings in the United States from 2000 to 2017, we revealed three clusters of framing patterns organized primarily: around who the victims were, where the shooting occurred, and the motives of the suspect(s).
For future work, we consider several research directions. First, we aim to leverage the multilingual word embeddings to port our classifier to non-English news articles. In these contexts, media frame classification is often conducted by extracting representative English words from the MFC, translating the word into the target language, and extending the lexicons [14] . We will explore the potential of the multilingual word embeddings to simplify the process of detecting media frames in non-English news articles. Second, we will extend our analysis to the comparison of multiple news media outlets. Although the New York Times is an authoritative source for agenda-setting and plays a critical role in shaping public opinion, comparing various news media outlets from the perspective of their prominent frames on different issues will allow a more comprehensive picture. For example, it would be interesting to examine the political leanings of the news media in terms of the dominant frames on certain issues. Last but not least, with the advancement of neural language models, framing detection will become more accurate and be applied to more and more diverse text data. As the general media frames are universal across different topics, they may serve as a useful means of categorizing social media posts (e.g., tweets) across issues, providing a new tool to study the interaction between media and the public.