O N AUGMENTING THE REFERENCES SECTION WITH A CITATION NETWORK VISUALIZATION
A BSTRACT
Researchers often ﬁnd relevant articles by looking at the references section.
We conducted user interviews with researchers about their workﬂow and their needs when carrying out literature research.
Based on this study, we identify a set of problems encountered by researchers.
We then propose to embed classiﬁed citation networks into articles as a solution, and to complement this graph with optional comments about references by authors.
We demonstrate this idea by implementing it for this article.
We argue that our solution helps increase inclusivity and improves the efﬁciency of reading scientiﬁc articles.
I NTRODUCTION
In the ﬁeld of Machine Learning (ML), the volume of conference submissions and the acceleration of scientiﬁc article publication as a means of dissemination of discovery are at an all time high.
In order to uncover contemporary problems faced by researchers when reading scientiﬁc articles, we conducted user experience (UX) interviews with 20 researchers at various career stages (Section 2).
From this study, we distilled a few key problems that were faced by our interviewees (Section 3).
The proposed solution (Section 4) extends core aspects of research on citations over the past few decades, which we summarize below.
1. R ELATED WORK ON CITATION TYPES AND NETWORK
Citation types Various guidelines for reading articles (Purugganan & Hewitt, 2004; Keshav, 2007;
Eisner, 2009) describe the importance of citations and discovering related antecedent articles.
Note that in this article we use the term citation types to capture different sets of semantics/classiﬁcation labels for citations (i.e., a directed edge from a article to an antecedent article).
Moravcsik & Murugesan (1975)
ﬁrst proposed four citation types according to their motivation (Table 1).
Subsequently, Spiegel-Rosing (1977) expanded this set to thirteen different citation types.
Teufel et al.
(2006) and Han et al.
(2016) used a subset of these citation types (see Table 2 for the former) and as labels for a citation classiﬁcation task.
Similarly, inspired by the “organic or perfunctory” citation type in Moravcsik & Murugesan (1975), Dong & Sch¨afer (2011) proposed four citation types which are deemed to be the most general and mutually exclusive and used them as labels (Table 3).
There has also been a rich body of research on automated citation types classiﬁcation (Xu et al., 2013;
Bakhti et al., 2018), which helps assigning citation types at scale and standardizing the citation types to be assigned to citations.
Citation network Citation types are commonly thought of as edge labels in a citation network.
Various visualization tools have been built to aid with navigation in these networks.
Some examples are CiteWiz (Elmqvist & Tsigas, 2007), Apolo (Chau et al., 2011), CircleView (Bergstr¨om &
Whitehead Jr, 2006), and the tools proposed in Sch¨afer & Kasterka (2010) and Weitz & Sch¨afer
In the user interviews we conducted, none of the researchers interviewed mentioned any of these tools to be part of their workﬂow.
Chau et al.
(2011) found a strong preference in using a network visualization component over a list format (i.e., a table) to display citation network data.
Our proposed solution builds upon the concept of using spatial arrangement to manage the exploration of context when navigating citations.
[Table]
Conceptual
[Table]
operational
[Table]
of theory vs. use of method.
Evolutionary or juxtapositional: work in the prime article is based on the cited work vs. an alternative to it.
Organic or perfunctory: work is critical for understanding and reproducing the citing article vs. a general acknowledgement out of politeness, policy, or piety as coined by Ziman (1968).
Conﬁrmative or negational: the citing article conﬁrms the cited article vs. disputes the cited article.
[Table]
support
[Table]
Neutral description of cited work, or not enough textual evidence for above categories or unlisted citation function.
1. R EFLEXIVITY STATEMENT
We are a group of industry professionals from different sectors (ﬁnance, edTech, and e-commerce) and academics.
Our motivation for conducting this study comes from the realization that navigating
[Table]
and accessing the most relevant academic research output when working in the industry is challenging.
To this end, we started interviewing researchers in academia and industry in order to understand their workﬂow in the hopes that we are able to learn how they are navigating and accessing research content.
The citation network visualization is motivated by problems we discovered in our UX interviews with researchers.
U SER EXPERIENCE INTERVIEWS
Over the course of three months, we conducted extensive contextual interviews with 20 researchers at various careers stages.
We focused on understanding the researchers’ problems when they are conducting research.
2. R ESEARCHER PROFILE SUMMARY
All 20 researchers were from ﬁelds with a computational component.
Furthermore, at least one of the following requirements needed to be satisﬁed:
• The research area is interdisciplinary (e
g., Computational Neuroscience) or has applications within other disciplines (e
g., ML).
• The research area has industrial applications.
• The researcher has had past experiences in disseminating research knowledge to a nonexpert audience.
The researchers interviewed came from diverse backgrounds:
• 10 interviewees had industry research experience.
• 6 interviewees were professors with experience in leading research groups and/or supervising graduate students.
• 10 interviewees were specialized in Computer Science/ML.
The others came from various ﬁelds with a computational component and an interface with data science: Applied Mathematics, Computational Neuroscience, Computational Biology, Computational Physics, Biomedical Sciences, and Quantum Computing.
We tagged researchers according to the following roles (where a researcher can have multiple roles):
• Academics going into a new ﬁeld (new researcher, NR ).
• Industry researchers following academic research (industry researcher, IR ).
• Research producers in academia (research producer, RP ).
2. I NTERVIEW FORMAT
Most interviews took an hour, with some taking less time due to time constraints of the interviewees, and some taking longer in the form of multiple calls with the same interviewee.
The interviews took place over a video call and were conducted by two interviewers: the interviewer with a better understanding of the interviewee’s area would lead the interview, whereas the second interviewer would take notes used for post-call summary and analysis.
The interviewees were informed beforehand that the topic of the interview would be based on the broad question: How can we make research more accessible with respect to navigating and understanding a research area?
The broad question served as a starting point for interviewees to express their thoughts and past experiences.
We followed standard UX research guidelines of keeping questions open-ended and letting interviewees guide the discussion towards the problems they felt were most relevant 2 .
Whenever a problem was brought up by researchers we interviewed, we tried to get to the root causes that triggered it.
For example, “ I would like to read several articles on the same result. ” would be followed up by prompting what exactly the researcher is looking to establish by going through several versions of the result.
We also directly asked the researchers to qualify the degree of inconvenience associated with a speciﬁed problem.
This allowed us to identify the most relevant problems to address.
We also explored the current solutions / coping mechanisms the researchers might have.
This was helpful in (1) identifying the incumbent best solutions, and (2) understanding whether the problem they expressed warrants a new solution.
In fact, some of the problems were addressable by current tools.
I NTERVIEW RESULTS AND KEY PROBLEMS
Based on the interview notes, we performed a card sorting exercise.
We noted the most common questions and issues stated by interviewees, and then categorized them into a high-level problem statement.
Throughout the interviews, these questions and issues (re-stated as questions) were often asked:
1. (Q1)
What are the key articles to read for a review of the ﬁeld?
(Q2)
What articles are the most accessible and self-contained, in terms of giving an intuitive understanding without an overload of jargon and cross-references?
(Q3)
When reading an article, what are the key reference articles to better understand the latter?
(Q4)
What are the most relevant articles citing an article of interest?
(Q5)
What are competing methodologies solving the same research problem addressed by a given research article?
We then categorized the questions and issues to the following high-level problem statements 4 :
• Not knowing what to read when exploring a new area of research (NR, IR).
• Inefﬁcient consumption methods (reading texts/watching videos) of research works (NR, IR).
• Cumbersome ways to curate/track/archive explored literature (NR, IR, RP).
Researchers used various solutions to attempt to answer Q1-Q5 and therefore address the above problems but none of the researchers interviewed mentioned being completely satisﬁed with them 5 , and most found the tools to be of limited usability.
We observed (1) “Tool Fatigue”, where current solutions are deemed to be cumbersome to use as they are often third-party tools and are away from the reading experience, and (2) that they do not adequately answer the question of why an article is being cited.
The latter is often mentioned when Q1-Q5 were discussed during the interviews.
P ROPOSAL TO HELP ANSWER Q1-Q 4. E MBEDDING A CITATION NETWORK VISUALIZATION
In this position article, we hope to surface contemporary research in citation types and network in order to help answer Q1-Q5.
Citation network visualization, along with citation types as edge labels, can be used to quickly and more accurately ﬁnd out why an article is being cited.
We would also like to propose the idea that the article itself should either store or point to the network visualization, which helps with the “Tool Fatigue” problem we observed.
As an example, Figure 1 displays the citation network of Article 1 (black vertex), which contains four cited articles with three citation types.
Let T0, T1, and T2 be shorthands for Table 1, Table 2, and Table 3, respectively.
In the following, we propose a few prototypical user ﬂows for how a researcher might answer Q1-Q5 by retrieving articles using the following citation types:
1. (Q1) Evolutionary (T1), CoCoGM (T2), or Fundamental idea (T3), since the current article is based on or compared against the cited article.
2. (Q2) Background/technical basis (T3), which implies that the cited article contains background high-level/technical information.
(Q3)
Organic (T1) or PModi (T2), which implies that the current article builds upon the ideas in the cited article.
4. (Q4) CoCoXY/PSim (T2) or Fundamental idea (T3), which implies that the citing article extends the
5. (Q5) PSim (T2) or Comparison (T3), which implies similarities between the current and the cited article.
While in the above user ﬂows we decided which citation types to use heuristically, we argue that the above lookup process helps with answering Q1-Q5 in a timely manner.
In order to demonstrate this idea, we build a rudimentary solution that uses this article as the center of the network.
First, we construct this article’s citation network neighborhood and derived the citation types based on
T2 (Figure 2).
Then we build and host a D3 visualization 6 .
Lastly, in order to make the visualization immediately accessible from this article, we provide a shortened URL and a QR code (See
Section 7).
4. A UGMENTING CITATION TYPES WITH AUTHOR COMMENTS
While we presented citation types as an approach to answering Q1-Q5, we believe that it lacks general expressiveness.
In Figure 2, we observe relatively large numbers of citations of the same types ( PBas and PMot ).
This makes it difﬁcult for readers to differentiate between references of the same citation type.
To supplement the citation visualization, we propose a workﬂow where authors are given the option to input a free-form comment for every reference.
This enables authors to add information that is not conveyable by the generic citation types.
We anticipate that the literature review sections of papers will especially beneﬁt from such comments since there could be many citations of the same type.
Therefore, this extra expressiveness can help address the deluge of citations problem mentioned in
Section 3.
In the dynamic network visualization that we propose (see Section 7), comments are displayed when hovering over the edges.
We have provided some example comments for the following references:
Chau et al.
(2011) and Ziman (1968).
Q UESTIONS AND FUTURE EXPLORATIONS
A few natural questions arise stated here to motivate future explorations:
Citation types
Which citation type schema should be used?
Using a standardized schema allows global statistics of the network to be computed (Sch¨afer & Kasterka, 2010), such as “Return past articles that are similar to Article 0”.
This can be obtained, for example, by using PSim (T2).
Moreover, a standardized schema will minimize the effort needed by readers when navigating the citation networks.
We conjecture that the most expressive schema would work best (e.g., T2), and envision that the schema will rarely evolve as long as the typical construct of a scientiﬁc article remains the same.
Who should be performing the citation type classiﬁcation?
While article authors know their article the best, different authors may interpret the citation types differently, even though they are citing a article similarly.
Third-party annotators (Sch ¨ afer & Kasterka (2010) proposed crowd-based classiﬁcation), who are more likely less familiar than the authors, might be able to more consistently classify the citations, as they might perform the task more often than individual authors.
Lastly, automated techniques (Xu et al., 2013; Bakhti et al., 2018) can perform classiﬁcation in a scalable manner.
We think that moving forward, the cheapest way to get started and evaluate these visualizations is if authors themselves could include these visualizations along with their articles.
In our proposal, the visualizations are hosted separately, so they could evolve and are version-controlled.
For past articles, automated techniques could be a great approach.
What are good properties of citation types?
When deﬁning citation types with the purpose of searches in the citation graph, properties such as transitivity would be helpful.
For example, if PSim is transitive, then if A cites B with type PSim , and B cites C with type PSim , then if C were to be cited in A, it would also be cited with type PSim .
Exploration of such properties for speciﬁc citation types could lead to useful global properties of the citation network.
Citation network and visualization
What is the best way of providing the visualization?
Is a shortened URL presented as a footnote of the references section the best article element for accessing the visualization?
Other options include embedding a QR code or the visualization itself in the article.
However, embedding the visualization directly will remove the dynamism of the visualization.
What is the easiest way for authors to include the visualization?
We envision a workﬂow that includes a plugin for the (raw) bibliography section of the article.
Here, authors could assign a citation types to their bibliographic entries, and work with a third party service that automatically creates and hosts a visualization.
Should higher-order neighborhoods be included?
Our interviewees often mentioned that they needed to quickly ﬁnd out which article is cited and why it is cited.
This dovetails with viewing the ﬁrst-order neighborhood of the network.
However, Q1, Q2, and Q5 could beneﬁt from higher-order neighborhoods, as those questions involve more global statistics of the network.
Understanding of use cases for such higher order neighborhoods are related to understanding the properties of citation types (Point 3 of the ’Citation types’ section above).
In the current form, we think that while there is a beneﬁt from having second and third-order neighborhoods, the reader will encounter diminishing returns since the visualization would be less effective as it would contain many edges.
Further exploration on visualization types will need to be conducted to enable higher-order visualizations.
How can we evaluate the effectiveness of the visualizations?
If the visualizations are served via the reader’s web browser, then we can apply web measurement techniques (Kohavi et al., 2020) to understand aggregated user behavior.
Furthermore, we can compare different visualization aspects, such as citation schema, graph layout, colour palette, existence of legend (which may clutter the responsive mobile view), expressing richer semantics on the edges (e.g., bolder lines to indicate stronger dependence), etc.
Augmenting citation types with comments
How can one measure the efﬁcacy of providing comments?
While authors’ comments can increase the expressiveness of information provided per reference, they also increase the amount of information being stored in the network visualization.
In order to understand the impact of comments, we think that metrics like (1) authors’ likelihood to input a comment, and (2) users’ likelihood to engage with a reference in the network visualization given that it has a comment, can be used to measure efﬁcacy.
How can we increase authors’ likelihood to input a comment?
If we can prove that authors’ comments are helpful for readership (see the previous question), we believe that there are two design goals to achieve: (1) make the workﬂow to add comments as simple as possible, and (2) develop incentives for authors to add comments.
For (2), a possible incentive is a way for authors to refer back to their own comments on references while they are developing the literature review section.
A CCESSIBILITY STATEMENT
We commit to providing solutions that are accessible to the widest possible audience.
We recognise the importance of making research accessible to people with a diverse range of hearing, movement, sight, and cognitive abilities.
The solution we propose is inherently aligned with improving the dissemination of information by diversifying the format available for reference sections that are typically exclusively presented in a ﬂat textual format.
Article readers are given an alternative way to interact with and visualize reference sections, raising the standard of the article reading user experience (UX) for researchers.
The accompanying visualization tool to this solution is web-based and enables users to:
• change colours and contrast levels; • zoom in and out to the desired size; • access the graphic through a variety of devices including mobile phones and tablets.
In addition, the colour palette used is Web Accessibility Content Guidelines (WCAG) 2.1 Level
AA compliant.
It was a deliberate choice to adhere to Level AA rather than Level AAA as the primary objective of colouring the edges of the citation network is to distinguish between colours.
Level AAA would have not provided the necessary contrast between colours.
Font, headings, and structure adhere to guidelines provided by the British Dyslexia Association based on WCAG 2.1.
How accessible is the website?
The website solution is currently a prototype and by deﬁnition has minimal features.
Future improvements could be made to accessibility depending on uptake of the solution.
These may include the ability to:
• navigate most of the website using just a keyboard; • navigate most of the website using speech recognition software; • listen to most of the website using a screen reader (including the most recent versions of JAWS, NVDA, and VoiceOver); • generate article copies of the visual.
These are currently limitations to the solution.
New accessibility features will be considered and prioritized based on user feedback.
Feedback and contact information To give feedback on accessibility and to get more information, please contact p.manggala@uva.nl.
A CKNOWLEDGMENTS
We thank all the researchers we interviewed.
We are grateful to two anonymous referees for many helpful comments
R EFERENCES A DD - ON
[Figure]