Classify Website Content using Langchain, OpenAI and Browserless
Tags: nocode, lowcode, backend, javascript
Published: 2023-05-23 15:05:52

<p>In the fast-paced digital era, the internet is teeming with an overwhelming amount of websites and online content. Finding the right information amidst this vast sea of data can be a daunting and time-consuming task.</p>
<p>Fortunately, with the advent of advanced language models and frameworks such as <a href="https://js.langchain.com/docs/">Langchain</a>, we now have access to powerful AI tools that can streamline various tasks. </p>
<p>In this blog, you’ll explore the capabilities of the <a href="https://demo.rowy.io/table/langchainURLClassifier#sideDrawer=%22table-information%22">Langchain URL Classifier app</a> (powered by Rowy) and discover how it harnesses Langchain to effectively classify websites.</p>
<h2 id="langchainurlclassifieranoverview">Langchain URL Classifier - An Overview 🔎</h2>
<p>The Langchain URL Classifier is a low-code demo app of how to <strong>leverage Langchain framework</strong> along with <strong>OpenAI</strong> to classify URLs and website content coupled with <strong>Browserless</strong> tool to make it more accurate. This app will take a website URL and give out a category of the website along with what is the purpose of each website. </p>
<p>By following this tutorial, you'll acquire a complete and functional app backend for URL classification. It serves as an excellent foundation that can be customized and adapted to suit your specific use cases as needed.</p>
<p><img src="https://firebasestorage.googleapis.com/v0/b/website-a1s39m.appspot.com/o/blog%2F6HEbFhFrwOjGlZ60A9hj%2FcontentImages%2FSMY22OMaFnWFC9u2ttcK-Langchain1.png?alt=media&token=354b5ff8-54c7-411e-8233-4a0a8d73cefd" alt="langchain1.png" /></p>
<h3 id="whatislangchain">What is Langchain? 🦜🔗</h3>
<p><a href="https://js.langchain.com/docs/">Langchain</a> is an <strong>AI framework</strong> designed for developing applications powered by language models. It distinguishes itself by going <strong>beyond mere language model integration</strong>, incorporating key principles that make applications more data-aware and dynamic.</p>
<p>By connecting language models to additional data sources, Langchain unlocks new possibilities for developing intelligent applications.</p>
<p>The Langchain URL Classifier primarily uses <strong>LLM Chains</strong> that enable us to connect several components to form a single, coherent application. We can, for example, build a chain that receives user input, formats it with a <code>PromptTemplate</code>, and then delivers the prepared result to an LLM. We can create more sophisticated chains by merging multiple chains or chains with other components.</p>
<h2 id="prerequisites">Pre-requisites ✅</h2>
<ul>
<li>A Rowy Account with <strong>Rowy Run</strong> set up. If you don’t have one, create one on <a href="www.rowy.app">www.rowy.app</a> - It is fully <a href="https://www.rowy.io/pricing">free</a> for one project with unlimited database tables and functions (For more information on setting up Rowy, check out our documentation at https://docs.rowy.io/setup/install)</li>
</ul>
<h2 id="settingupthelangchainurlclassifier">Setting up the Langchain URL Classifier ✨</h2>
<p>Get started with the following step-by-step instructions to create a Langchain URL Classifier backend app 🚀</p>
<h3 id="1createatableonrowy">1. Create a table on Rowy</h3>
<p>Visit <a href="www.rowy.io">www.rowy.io</a> and click "<strong>Sign In</strong>". Once you've created your Workspace and set up your Project, select the <strong>Langchain URL Classifier</strong> template from the "<strong>Create a Table</strong>" section at the top of the page.</p>
<p><img src="https://firebasestorage.googleapis.com/v0/b/website-a1s39m.appspot.com/o/blog%2F6HEbFhFrwOjGlZ60A9hj%2FcontentImages%2FAxmu7KZM8rfcocuHFWiC-Langchain2.png?alt=media&token=fbea2d9e-fb9c-4e31-886f-5675fda89988" alt="langchain2.png" /></p>
<p>The template setup wizard will guide you through the steps to set up your table, add your secret keys, and deploy your cloud functions – all in one go!</p>
<h3 id="2settingupyoursecrets">2. Setting up your Secrets 🤫</h3>
<p>As part of the template setup process, you'll be prompted to add two secrets: your <strong>OpenAI API Key</strong> and <strong>Browserless API Key.</strong> Here's how to retrieve these secret keys for your project:</p>
<h4 id="21retrievingtheopenaiapikey">2.1 Retrieving the OpenAI API Key ✅</h4>
<ul>
<li>Go to the OpenAI Account Dashboard and create an account if you haven’t already.</li>
<li>Click on Create a new secret key.
<img src="https://firebasestorage.googleapis.com/v0/b/website-a1s39m.appspot.com/o/blog%2F6HEbFhFrwOjGlZ60A9hj%2FcontentImages%2F0kfAaVff1wokD2pMVeA1-Langchain3.png?alt=media&token=c0b57eff-8b7a-4693-8b11-320a54829ea8" alt="langchain3.png" /></li>
<li>Add a name for your secret key, let’s say <strong>Rowy Langchain</strong>. Click <strong>Create</strong>.</li>
<li>Now, your OpenAI API Key is generated! 🥳 Copy and store it somewhere temporarily.
<img src="https://firebasestorage.googleapis.com/v0/b/website-a1s39m.appspot.com/o/blog%2F6HEbFhFrwOjGlZ60A9hj%2FcontentImages%2FSio9WSSQfA8AuQWapMFz-Langchain4.png?alt=media&token=d0883cd0-e45c-4b39-a7e8-5d974890e9b6" alt="langchain4.png" />
In the Template Setup Wizard, when prompted to add your OpenAI API Key, click on the <strong>Add a key in Secret Manager</strong> button. This will redirect you to the <a href="https://console.cloud.google.com/security/secret-manager">GCP Secret Manager</a>. Add your <strong>OpenAI API Key</strong> as shown below and click <strong>Create a Secret</strong>.
<img src="https://firebasestorage.googleapis.com/v0/b/website-a1s39m.appspot.com/o/blog%2F6HEbFhFrwOjGlZ60A9hj%2FcontentImages%2F9Owk3LYBLJ1wXDYj7mN8-Langchain5.png?alt=media&token=64046148-3aa6-4f0c-b82b-1d1bcc34d0a2" alt="langchain5.png" />
Now, you can go back to the setup process. In the Secret dropdown itself, click <strong>Refresh</strong> to see your <strong>OpenAI API Key</strong> and select it.</li>
</ul>
<h4 id="22retrievingthebrowserlessapikey">2.2 Retrieving the Browserless API Key ✅</h4>
<ul>
<li>Navigate to the <a href="https://cloud.browserless.io/account/">Browserless Dashboard</a> and create an account, if you haven’t already.
<img src="https://firebasestorage.googleapis.com/v0/b/website-a1s39m.appspot.com/o/blog%2F6HEbFhFrwOjGlZ60A9hj%2FcontentImages%2FTOA3dpO8Nh7BDN6hrVRb-Langchain7.png?alt=media&token=962a0c42-50bd-42f1-9b8b-fdf53db425b2" alt="langchain7.png" /></li>
<li>Select a <a href="https://www.browserless.io/sign-up-cloud-unit/">plan</a> as per your requirement. (You can also opt for the free tier)
<img src="https://firebasestorage.googleapis.com/v0/b/website-a1s39m.appspot.com/o/blog%2F6HEbFhFrwOjGlZ60A9hj%2FcontentImages%2FR9IgOAlVL4udKe73ebC9-Langchain8.png?alt=media&token=0fa8b4fb-a047-4c14-b956-ee9283525cab" alt="langchain8.png" /></li>
<li>Verify your email address according to the prompted steps on the portal.</li>
<li>Your <strong>Browserless API Key</strong> is generated! 🥳 Copy the generated API key from the dashboard.
<img src="https://firebasestorage.googleapis.com/v0/b/website-a1s39m.appspot.com/o/blog%2F6HEbFhFrwOjGlZ60A9hj%2FcontentImages%2FXc81BmUF6vmyd6COiy8e-Langchain9.png?alt=media&token=a767a6f0-ce45-4389-a69b-98ba2c9e84b7" alt="langchain9.png" /></li>
</ul>
<p>In the Template Setup Wizard, when prompted to add your <strong>Browserless API Key</strong>, click on the <strong>Add a key in Secret Manager</strong> button. This will redirect you to the <a href="https://console.cloud.google.com/security/secret-manager">GCP Secret Manager</a>. Add your API Key as shown below and click <strong>Create a Secret</strong>.</p>
<p><img src="https://firebasestorage.googleapis.com/v0/b/website-a1s39m.appspot.com/o/blog%2F6HEbFhFrwOjGlZ60A9hj%2FcontentImages%2FE5uN4Mvn6a5z5DLASW56-Langchain10.png?alt=media&token=91d0ce17-4ffa-4b6a-b5e7-839e9aef3b7c" alt="langchain10.png" /></p>
<p>Now, you can go back to the setup process. In the Secret dropdown itself, click <strong>Refresh</strong> to view your <strong>Browserless API Key</strong> and select it.</p>
<h3 id="3deployingcloudfunctions">3. Deploying Cloud Functions 🚀</h3>
<p>The template set up process will then prompt you to deploy the automations. Click the “<strong>Deploy Functions</strong>” button to set your derivatives into gear. Once the deployment completes, click “<strong>Proceed</strong>” and you will be redirected to your Langchain URL Classifier application.
<img src="https://firebasestorage.googleapis.com/v0/b/website-a1s39m.appspot.com/o/blog%2F6HEbFhFrwOjGlZ60A9hj%2FcontentImages%2F6Hk6DtZrWM9BeKYLhbkF-Langchain11.png?alt=media&token=adfd6373-ec74-4493-91f8-056e3f239454" alt="langchain11.png" /></p>
<p>Now, you’re all set to get hands on with the application! 🎉 Connect it to the frontend of your choice, experiment with the logic, or feel free to use the default Rowy template as a starting point.</p>
<h2 id="simplifyingweburlclassification">Simplifying Web URL Classification 💫</h2>
<p>The Langchain URL Classifier primarily consists of two automated data fields, i.e. <strong>Website Category</strong> and <strong>Website Summary</strong>. Every time a new row is added, the user can <strong>specify the URL of the Website</strong> in the URL field. The derivative functions listen to this field and dynamically generate results.</p>
<p><img src="https://firebasestorage.googleapis.com/v0/b/website-a1s39m.appspot.com/o/blog%2F6HEbFhFrwOjGlZ60A9hj%2FcontentImages%2FnifejvSWBgNbh18yeVjx-Langchain12.png?alt=media&token=3a4f5f87-495b-4ef7-a6fb-cb1507ee55ab" alt="langchain12.png" /></p>
<h3 id="websitecategory">Website Category</h3>
<p>The Website Category field uses an instance of the <code>LLMChain</code> class provided by <strong>Langchain</strong>. The <strong>Web Scraping service of the Browserless API</strong> retrieves the HTML content of the website. Based on the processing and extraction of this HTML content, the language model generates a response based on the provided prompt, which includes classifying the website into a category.</p>
<h3 id="websitesummary">Website Summary</h3>
<p>The Website Summary field uses a similar logic to generate the summary of the HTML content provided by the web scraping service.</p>
<h2 id="conclusion">Conclusion</h2>
<p>The Langchain URL Classifier is an innovative application that utilizes the power of language models to simplify web URL classification. By leveraging the capabilities of the Langchain framework, this tool categorizes web URLs based on the content and purpose of the respective websites.</p>
<p>This blog highlights the use case of Rowy as a platform for leveraging large language models. It focuses on the Langchain URL Classifier, an application that dynamically generates website categories and summaries based on the provided URL. By simplifying web content navigation and enhancing information retrieval, the Langchain URL Classifier offers an efficient solution for finding relevant information.</p>
<p>If you enjoyed our content, please give us a <a href="https://twitter.com/rowyio">follow on Twitter</a> or <a href="https://rowy.io/discord">join our Discord</a> in case of any questions. We look forward to hearing from you! 💜</p>