Optimized Web-Crawling of Conversational Data from Social Media and Context-Based Filtering (2020)

Abstract: Building Chabot’s requires a large amount of conversational data. In this paper, a web crawler is designed to fetch multi-turn dialogues from websites such as Twitter, YouTube and Reddit in the form of a JavaScript Object Notation (JSON) file. Tools like Twitter Application Programming Interface (API), LXML Library, and JSON library are used to crawl Twitter, YouTube and Reddit to collect conversational chat data. The data obtained in a raw form cannot be used directly as it will have only text metadata such as author or name, time to provide more information on the chat data being scraped. The data collected has to be formatted for proper use case and the JSON library of python allows us to format the data easily. The scraped dialogues are further filtered based on the context of a search keyword without introducing bias and with flexible strictness of classification.

## introduction real-world data remains a necessary part of training system models. the digital streams that individuals produce are quite useful in the data analysis domain, like natural language processing and machine learning. social networking applications like twitter, youtube and reddit contain a large volume of data that are quite useful for various algorithms. naturally, the need to make information easily accessible to all leads to deploying a conversational agent. in order to build a chat model, a huge volume of conversational text data is required. twitter is a microblogging service that allows individuals to post short messages called tweets that appear on timelines. these tweets were limited to 140 characters, which has been later expanded to 280 characters and prone to change again in the future. tweets consist of two kinds of metadata that are entities and places. tweet entities are hashtags, user-mentions, images, and places in the real world's geographical locations. metadata and short prose add to fewer than 280 characters can link to webpages, twitter users. twitter timelines are categorized into the home timeline and user timeline. timelines are collections of tweets in chronological order. twitter api uses representational state transfer (rest) api to crawl and collect a random set of sample public tweets. the api allows users to explore and search for trending topics, tweets, hashtags, and geographical locations. youtube, a video-sharing website, allows users to view, upload, rate, report on videos. it contains a wide variety of videos such as tv show clips, music videos, documentaries. it also provides a platform for users to communicate and describe their thoughts about what they watch through comments. reddit is a social news platform where registered users may submit links, images, text, posts and also upvote or downvote the posts posted by other users. posts are organized based on boards created by the user called subreddit. it is also a platform for web content rating and discussions. reddit stores all of its content in json format which can be viewed on the browser by extending the reddit link with the extension '.json'. real-time datasets are needed to build a model to generate accurate output. as the available datasets are insufficient and do not contain realistic examples, there is a need to build a crawler which would scrape conversational data. building crawlers for each website would allow collection of conversational data. this would help in the creation of datasets of conversational data. ## literature survey a focused crawler is designed to crawl and retrieve specific topics of relevance. the idea of the focused crawler is to selectively look for pages that are relevant while traversing and crawling the least number of irrelevant pages on the web. context focused crawlers (cfc) use a limited number of links from a single queried document to obtain all relevant pages to the document, and the said obtained documents are relevant con-cerning the context. this data obtained is then used to train a classifier that would detect the context of documents and allow classification of them into categories based on the link distance from a query to target. features of a crawler are- each website comes with the inclusion of a file known as robot.txt. it is a standardized practice where robots or bots are communicated with the website through this protocol. this standard provides the necessary instructions to the crawler about the status of the website and whether it is allowed to scrape the data off of the website. this is used to inform crawlers whether the website can be crawled either partially or fully, if the website cannot be crawled as per the robot.txt then the server blocks any such requests and can even lead to blocking of ip's. websites have robot.txt, which prevent the use of crawlers that attempt to scrape large data from their website. any request for a large amount of data is blocked almost immediately. to prevent such a case where the crawler should not be blocked, a list of publicly available proxy servers as described in @xcite is used to scale and crawl the website. twitter is a popular social media platform used for communication. crawling such a website can be useful to gain conversational data, and such information can be targeted based on the topic; one such example is a perception on the internet of things as shown in bian j (2017) or assessing the adequacy of gender identification terms on intake forms as described in hicks a (2015). ## conclusion the proposed crawler can fetch multi-turn dialogues from twitter, youtube. the crawler can scrape conversational data from twitter, youtube. unstructured data retrieved from the crawler is converted to well-formatted data. streaming api has been used to crawl twitter and retrieve random sets of public tweets. quoted retweets or retweets with comments are filtered from the collected set of public tweets. the quoted retweet, with its contained parent tweet, is outputted to the json file. youtube crawling is made easy without any limitations as like which the youtube data v3 api has, and getting comments has never been this easy and fast before. reddit crawler parses the json object from the response downloaded from the spider's request and can get the comments of the posts.