Scrap LinkedIn Company Data using Voyager Api
Tags: python, programming, scraping, opensource
Published: 2023-08-29 17:35:47

<h3 data-pm-slice="1 3 []">What is LinkedIn Scrapping?</h3>
<a class="attrlink" href="https://www.bytescrum.com/" target="_blank" rel="noopener noreferrer nofollow">LinkedIn scraping</a> is the process of extracting data from <a class="attrlink" href="https://www.bing.com/ck/a?!&amp;&amp;p=c05d028a9f4dffd0JmltdHM9MTY5MDkzNDQwMCZpZ3VpZD0wM2NkZjZjYy1jYmI1LTZiMGEtMTZlZi1lNWU3Y2ExODZhYTEmaW5zaWQ9NTIyNQ&amp;ptn=3&amp;hsh=3&amp;fclid=03cdf6cc-cbb5-6b0a-16ef-e5e7ca186aa1&amp;psq=linkedin&amp;u=a1aHR0cHM6Ly9pbi5saW5rZWRpbi5jb20v&amp;ntb=1" target="_blank" rel="noopener noreferrer nofollow">LinkedIn</a>'s website using automated scripts or tools. It involves collecting information from profiles, company pages, job postings, and other publicly available data.

On the other hand, web scraping involves automatically gathering data from websites by sending HTTP requests and extracting specific information from HTML content.

<strong>Note:</strong> However, violating <a class="attrlink" href="https://pypi.org/project/linkedin-scraper/" target="_blank" rel="noopener noreferrer nofollow">LinkedIn</a>'s terms of service can lead to account suspension, IP blocking, or legal action.

Reviewing the website's terms of service and API usage guidelines is essential to avoid legal or ethical issues.
<h3>What is Voyager's REST API?</h3>
Voyager's REST API enables developers to integrate enterprise applications with the geospatial search platform. It provides a web-based interface for exploring and interacting with Voyager's data and services.

Voyager offers two types of <strong>open</strong> <strong>interfaces: XML over HTTP and RESTful APIs.</strong> <a class="attrlink" href="https://towardsdatascience.com/using-browser-cookies-and-voyager-api-to-scrape-linkedin-via-python-25e4ae98d2a8" target="_blank" rel="noopener noreferrer nofollow"><strong>LinkedIn developed Voyager-AP</strong>I</a>, a new API service, to provide a more resilient platform for web and mobile applications based on the Play framework and GraphQL query language.

<strong>How to Obtain JSESSIONID and CSRF Token:</strong>

<strong>JSESSIONID:</strong> When a user accesses a website, the web server generates a unique identifier called the <strong>JSESSIONID.</strong>
<ul>
 	<li>It is often kept in the web browser used by the user as a cookie and sent along with future requests to identify the user's session.</li>
 	<li>The <strong>JSESSIONID</strong> may be obtained by viewing the cookies in your browser's developer tools when visiting the page.</li>
 	<li>Look for a cookie with the string "JSESSIONID" or anything similar. Please remember that certain websites may use distinct cookie names for session tracking.</li>
</ul>
<strong>CSRF Token</strong>: The <strong>CSRF</strong> token <strong>(Cross-Site Request Forgery token)</strong> is a security measure to prevent hostile websites from sending unauthorized requests.
<ul>
 	<li>It is frequently necessary when performing POST, PUT, or DELETE requests to web services.</li>
</ul>
<ul>
 	<li>The method for obtaining the <strong>CSRF</strong> token differs according to the website or online service. In other circumstances, the token is concealed in a web page's form and may be extracted using web scraping techniques.</li>
</ul>
<ul>
 	<li>When performing an initial GET call to a web service, the <strong>CSRF</strong> token may be sent in the response headers. In such circumstances, the token may be extracted from the response headers and used in subsequent requests as needed.</li>
</ul>
<strong>Step by Step guidelines using the </strong><a class="attrlink" href="https://developer.mozilla.org/en-US/docs/Glossary/Python" target="_blank" rel="noopener noreferrer nofollow"><strong>Python</strong></a><strong> module for performing HTTP requests.</strong>
<ol>
 	<li>It imports the requests library, a well-known <a class="attrlink" href="https://www.geeksforgeeks.org/scrape-linkedin-using-selenium-and-beautiful-soup-in-python/" target="_blank" rel="noopener noreferrer nofollow">Python</a> module for performing HTTP requests.</li>
</ol>
<ol start="2">
 	<li>It assigns a user-agent string to the headers variable. The user agent identifies the type of client making the server request.</li>
</ol>
<ol start="3">
 	<li>It defines the variable company_link, which holds the business information API endpoint URL. The URL appears to be of the form <a href="http://linkedin.com/voyager/api/entities/companies" target="_blank" rel="noopener noreferrer nofollow">linkedin.com/voyager/api/entities/companies</a><a class="attrlink" href="https://www.linkedin.com/voyager/api/entities/companies/company_id" target="_blank" rel="noopener noreferrer nofollow">..</a>, where company_id is changed with the company's particular entity ID.</li>
</ol>
<ol start="4">
 	<li>The script then builds requests.session() object, which enables a persistent session and cookie setting.</li>
</ol>
<ol start="5">
 	<li>It uses the s.cookies property to set the session's needed cookies (li_at and JSESSIONID). These cookies may be required for authentication or permission while using the <a class="attrlink" href="https://medium.com/nerd-for-tech/linked-in-web-scraper-using-selenium-15189959b3ba" target="_blank" rel="noopener noreferrer nofollow">LinkedIn</a> API.</li>
</ol>
<ol start="6">
 	<li>The script sets the session's user-agent and CSRF token headers to simulate a standard web browser request.</li>
</ol>
<ol start="7">
 	<li>It uses the session object s.get(company_link) to send an HTTP GET call to the company_link URL.</li>
</ol>
<ol start="8">
 	<li>The API answer has been received and looks to be in JSON format.</li>
</ol>
<ol start="9">
 	<li>The JSON response is then processed into a <a class="attrlink" href="https://www.linkedin.com/learning/web-scraping-with-python" target="_blank" rel="noopener noreferrer nofollow">Python</a> dictionary, and the script publishes the dictionary's contents, which contain corporate information like staff count, website URL, company type, industries, description, and more.</li>
</ol>
```python
def getdatafromvoyagerlinkedin(company_id):
    headers = {"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36",
               }

    company_link = f'https://www.linkedin.com/voyager/api/entities/companies/{company_id}'

    with requests.session() as s:
        s.cookies['li_at'] = "AQEDATf5D_XXXXXXXXXXXXXXu"
        s.cookies["JSESSIONID"] = "ajax:1XXXXXXXXXXXXXX0"
        s.headers = headers
        s.headers["csrf-token"] = s.cookies["JSESSIONID"].strip('"')
        response = s.get(company_link)
        response_dict = response.json()
        return response_dict


print(getdatafromvoyagerlinkedin(16198870))
# Output
#{'employeeCountRange': '11-50',
 #'specialties': ['Web Development', 
#'Mobile App Development', 'Web Design', 
#'Node', 'Flutter', 'Ionic', 'AWS', 'Digital Ocean', 'Laravel', 'WordPress', 
#'React Native', 'React JS experts', 'PHP development', 'PrestaShop', 'OpenCart',
# 'SEO and SEM', 'Joomla'],
 #'entityUrn': 'urn:li:fs_company:16198870', 
#'websiteUrl': 'https://www.bytescrum.com', 
#'companyType': 'Privately Held', 'foundedDate': {'year': 2017}, 
#'entityInfo': {'objectUrn': 'urn:li:company:16198870',
#'trackingId': 'ymxYkL2eSUSOWQIt+Mn3xQ=='}, 
#'industries': ['Information Technology and Services'],
 #'description': 'ByteScrum taps into its strong business acumen to find solutions to the unique set of challenges and constraints imposed by each new project and delivers solutions that fill performance gaps. 
#Our founders understood for the first time how good software development services can transform the needs of entire business communities, especially emerging technologies.
# We have a proven track record of successfully meeting deadlines and executing the most complex projects within budget while consistently maintaining the highest quality.\n\nOur specialities:\n\n● Mobile app development
 #(Android and iOS)\n● Web app development (MERN, MEAN, Vue JS, PHP, Laravel, WordPress)\n● Custom Software development\n● Web designing (PSD to HTML/WordPress)\n● Api integration \n● 
#CMS development\n● Web and app service integration\n● SEO and SEM services\n\nContact Us:\nhttps://www.bytescrum.com/contact-us/', 
#'basicCompanyInfo': {'headquarters': 'Lucknow',
 #'followingInfo': {'entityUrn': 'urn:li:fs_followingInfo:urn:li:company:16198870', 
#'dashFollowingStateUrn': 'urn:li:fsd_followingState:urn:li:fsd_company:16198870', 'following': False, 'trackingUrn': 'urn:li:company:16198870', 'followingType': 'DEFAULT'},
# 'miniCompany': {'objectUrn': 'urn:li:company:16198870', 'entityUrn': 'urn:li:fs_miniCompany:16198870', 'name': 'ByteScrum Technologies Private Limited',
# 'showcase': False, 
#'active': True,
# 'logo': {'com.linkedin.common.VectorImage': {'artifacts': [{'width': 200, 'fileIdentifyingUrlPathSegment': '200_200/0/1653201669588?e=1698883200&amp;v=beta&amp;t=GE_5HHCt3u_xxKWDV1d3KmNBx0-AJXvIyjkIxSaXp-E',
# 'expiresAt': 1698883200000, 'height': 200}, {'width': 100, 'fileIdentifyingUrlPathSegment': '100_100/0/1653201669588?e=1698883200&amp;v=beta&amp;t=rbIH_vzfS4YkrOV-inNhuY9XXdbj28K9l4ZY_4-I41o', 
#'expiresAt': 1698883200000, 'height': 100}, {'width': 400, 'fileIdentifyingUrlPathSegment':
# '400_400/0/1653201669588?e=1698883200&amp;v=beta&amp;t=rARzTyswXT1D9vObNkCAh9ljFivi4r6T0QxC_WwLVvQ', 'expiresAt': 1698883200000, 'height': 400}], 
#'rootUrl': 'https://media.licdn.com/dms/image/C4D0BAQHzTgUzh6WpUw/company-logo_'}}, 'universalName': 'bytescrum', 'dashCompanyUrn': 'urn:li:fsd_company:16198870', 
#'trackingId': 'XXXXXXXXXXXXXXXXX'}}}
```
LinkedIn now provides a unique ID to each company page, making it easy to discover. Once you have that, replace that number in the code with one.

<a href="http://linkedin.com/voyager/api/entities/companies{company" target="_blank" rel="noopener noreferrer nofollow"><em>linkedin.com/voyager/api/entities/companies{company</em></a><em> id}</em>
<h3>Conclusion</h3>
<a class="attrlink" href="https://github.com/topics/linkedin-scraper?l=python" target="_blank" rel="noopener noreferrer nofollow">LinkedIn scraping</a> and web <a class="attrlink" href="https://scrapeops.io/python-scrapy-playbook/python-scrapy-linkedin-people-scraper/" target="_blank" rel="noopener noreferrer nofollow">scraping </a>are two ways to obtain information from <a class="attrlink" href="https://blog.devgenius.io/how-to-build-a-scraping-tool-for-linkedin-in-7-minutes-tool-data-science-csv-selenium-beautifulsoup-python-a673f12ac579" target="_blank" rel="noopener noreferrer nofollow">LinkedIn</a>'s website. While <a class="attrlink" href="https://www.linkedin.com/pulse/how-easy-scraping-data-from-linkedin-profiles-david-craven/" target="_blank" rel="noopener noreferrer nofollow">LinkedIn scraping</a> is concerned with gathering publically accessible data, web scraping is concerned with mechanically pulling particular information from websites using HTTP queries and HTML parsing.

Following LinkedIn's terms of service is critical to prevent potential penalties such as account suspension or legal action.<a class="attrlink" href="https://www.scraperapi.com/blog/linkedin-scraper-python/" target="_blank" rel="noopener noreferrer nofollow"> Voyager's REST API </a>enables corporate applications to integrate with their geographic search engine seamlessly.

The JSESSIONID is a unique identification provided by a website server, whereas the CSRF token protects against unwanted requests. Both are accessible via web browser cookies, web scraping tools, or response headers.

Knowing data extraction regulations and ethical issues is critical while working with web scraping and APIs like Voyager.

If you find this post exciting, find more exciting posts on the <a class="attrlink" href="http://blog.learnhub.africa/" target="_blank" rel="noopener noreferrer nofollow">Learnhub Blog</a>; we write everything tech from <a class="attrlink" href="https://blog.learnhub.africa/category/cloud-computing/" target="_blank" rel="noopener noreferrer nofollow">Cloud computing</a> to <a class="attrlink" href="https://blog.learnhub.africa/category/frontend/" target="_blank" rel="noopener noreferrer nofollow">Frontend Dev</a>, <a class="attrlink" href="https://blog.learnhub.africa/category/security/" target="_blank" rel="noopener noreferrer nofollow">Cybersecurity</a>, <a class="attrlink" href="https://blog.learnhub.africa/category/data-science/" target="_blank" rel="noopener noreferrer nofollow">AI</a>, and <a class="attrlink" href="https://blog.learnhub.africa/category/blockchain/" target="_blank" rel="noopener noreferrer nofollow">Blockchain</a>.

Culled from <a href="https://hashnode.com/@bytescrum" target="_blank" rel="noopener noreferrer nofollow">ByteScrum</a>
<h2>Resources</h2>
<ul>
 	<li><a class="attrlink" href="https://blog.learnhub.africa/2023/05/26/20-essential-python-extensions-for-visual-studio-code/" target="_blank" rel="noopener noreferrer nofollow">20 Essential Python Extensions for Visual Studio Code</a></li>
 	<li><a class="attrlink" href="https://blog.learnhub.africa/2023/04/27/using-python-for-web-scraping-and-data-extraction/" target="_blank" rel="noopener noreferrer nofollow">Using Python for Web Scraping and Data Extraction</a></li>
 	<li><a class="attrlink" href="https://blog.learnhub.africa/2023/06/06/13-powerful-python-snippets-to-automate-tasks/" target="_blank" rel="noopener noreferrer nofollow">13 Powerful Python Snippets To Automate Tasks</a></li>
 	<li><a class="attrlink" href="https://www.python.org/about/gettingstarted/" target="_blank" rel="noopener noreferrer nofollow">Getting Started with Python</a></li>
 	<li><a class="attrlink" href="https://blog.learnhub.africa/2023/07/21/scraping-using-python-in-step-by-step-approach/" target="_blank" rel="noopener noreferrer nofollow">Web scraping with Python</a></li>
</ul>