!function(){function t(t){document.documentElement.setAttribute("data-theme",t)}var e=function(){var t=null;try{t=new URLSearchParams(window.location.search).get("docusaurus-theme")}catch(t){}return t}()||function(){var t=null;try{t=localStorage.getItem("theme")}catch(t){}return t}();t(null!==e?e:"light")}(),document.documentElement.setAttribute("data-announcement-bar-initially-dismissed",function(){try{return"true"===localStorage.getItem("docusaurus.announcement.dismiss")}catch(t){}return!1}())

[Skip to main content](#__docusaurus_skipToContent_fallback)

LangChain v0.2 is coming soon! Preview the new docs [here](/v0.2/docs/introduction/).

[

![🦜️🔗 Langchain](/v0.1/img/brand/wordmark.png)![🦜️🔗 Langchain](/v0.1/img/brand/wordmark-dark.png)

](/v0.1/)[Docs](/v0.1/docs/get_started/introduction/)[Use cases](/v0.1/docs/use_cases/)[Integrations](/v0.1/docs/integrations/platforms/)[API Reference](https://api.js.langchain.com)

[More](#)

*   [People](/v0.1/docs/people/)
*   [Community](/v0.1/docs/community/)
*   [Tutorials](/v0.1/docs/additional_resources/tutorials/)
*   [Contributing](/v0.1/docs/contributing/)

[v0.1](#)

*   [v0.2](https://js.langchain.com/v0.2/docs/introduction)
*   [v0.1](/v0.1/docs/get_started/introduction/)

[🦜🔗](#)

*   [LangSmith](https://smith.langchain.com)
*   [LangSmith Docs](https://docs.smith.langchain.com)
*   [LangChain Hub](https://smith.langchain.com/hub)
*   [LangServe](https://github.com/langchain-ai/langserve)
*   [Python Docs](https://python.langchain.com/)

[Chat](https://chatjs.langchain.com)[](https://github.com/langchain-ai/langchainjs)

Search

*   [Providers](/v0.1/docs/integrations/platforms/)
    
    *   [Providers](/v0.1/docs/integrations/platforms/)
    *   [Anthropic](/v0.1/docs/integrations/platforms/anthropic/)
    *   [AWS](/v0.1/docs/integrations/platforms/aws/)
    *   [Google](/v0.1/docs/integrations/platforms/google/)
    *   [Microsoft](/v0.1/docs/integrations/platforms/microsoft/)
    *   [OpenAI](/v0.1/docs/integrations/platforms/openai/)
*   [Components](/v0.1/docs/integrations/components/)
    
    *   [LLMs](/v0.1/docs/integrations/llms/)
        
    *   [Chat models](/v0.1/docs/integrations/chat/)
        
    *   [Document loaders](/v0.1/docs/integrations/document_loaders/)
        
    *   [Document transformers](/v0.1/docs/integrations/document_transformers/)
        
        *   [html-to-text](/v0.1/docs/integrations/document_transformers/html-to-text/)
        *   [@mozilla/readability](/v0.1/docs/integrations/document_transformers/mozilla_readability/)
        *   [OpenAI functions metadata tagger](/v0.1/docs/integrations/document_transformers/openai_metadata_tagger/)
    *   [Document compressors](/v0.1/docs/integrations/document_compressors/)
        
    *   [Text embedding models](/v0.1/docs/integrations/text_embedding/)
        
    *   [Vector stores](/v0.1/docs/integrations/vectorstores/)
        
    *   [Retrievers](/v0.1/docs/integrations/retrievers/)
        
    *   [Tools](/v0.1/docs/integrations/tools/)
        
    *   [Agents and toolkits](/v0.1/docs/integrations/toolkits/)
        
    *   [Chat Memory](/v0.1/docs/integrations/chat_memory/)
        
    *   [Stores](/v0.1/docs/integrations/stores/)
        

*   [](/v0.1/)
*   [Components](/v0.1/docs/integrations/components/)
*   [Document transformers](/v0.1/docs/integrations/document_transformers/)
*   @mozilla/readability

On this page

@mozilla/readability
====================

When ingesting HTML documents for later retrieval, we are often interested only in the actual content of the webpage rather than semantics. Stripping HTML tags from documents with the MozillaReadabilityTransformer can result in more content-rich chunks, making retrieval more effective.

Setup[​](#setup "Direct link to Setup")
---------------------------------------

You'll need to install the [`@mozilla/readability`](https://www.npmjs.com/package/@mozilla/readability) and the [`jsdom`](https://www.npmjs.com/package/jsdom) npm package:

*   npm
*   Yarn
*   pnpm

    npm install @mozilla/readability jsdom

    yarn add @mozilla/readability jsdom

    pnpm add @mozilla/readability jsdom

Though not required for the transformer by itself, the below usage examples require [`cheerio`](https://www.npmjs.com/package/cheerio) for scraping:

*   npm
*   Yarn
*   pnpm

    npm install cheerio

    yarn add cheerio

    pnpm add cheerio

tip

See [this section for general instructions on installing integration packages](/v0.1/docs/get_started/installation/#installing-integration-packages).

*   npm
*   Yarn
*   pnpm

    npm install @langchain/community

    yarn add @langchain/community

    pnpm add @langchain/community

Usage[​](#usage "Direct link to Usage")
---------------------------------------

The below example scrapes a Hacker News thread, splits it based on HTML tags to group chunks based on the semantic information from the tags, then extracts content from the individual chunks:

    import { CheerioWebBaseLoader } from "langchain/document_loaders/web/cheerio";import { MozillaReadabilityTransformer } from "@langchain/community/document_transformers/mozilla_readability";import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";const loader = new CheerioWebBaseLoader(  "https://news.ycombinator.com/item?id=34817881");const docs = await loader.load();const splitter = RecursiveCharacterTextSplitter.fromLanguage("html");const transformer = new MozillaReadabilityTransformer();const sequence = splitter.pipe(transformer);const newDocuments = await sequence.invoke(docs);console.log(newDocuments);/*  [    Document {      pageContent: 'Hacker News new | past | comments | ask | show | jobs | submit login What Lights\n' +        'the Universe’s Standard Candles? (quantamagazine.org) 75 points by Amorymeltzer\n' +        '5 months ago | hide | past | favorite | 6 comments delta_p_delta_x 5 months ago\n' +        '| next [–] Astrophysical and cosmological simulations are often insightful.\n' +        "They're also very cross-disciplinary; besides the obvious astrophysics, there's\n" +        'networking and sysadmin, parallel computing and algorithm theory (so that the\n' +        'simulation programs are actually fast but still accurate), systems design, and\n' +        'even a bit of graphic design for the visualisations.Some of my favourite\n' +        'simulation projects:- IllustrisTNG:',      metadata: {        source: 'https://news.ycombinator.com/item?id=34817881',        loc: [Object]      }    },    Document {      pageContent: 'that the simulation programs are actually fast but still accurate), systems\n' +        'design, and even a bit of graphic design for the visualisations.Some of my\n' +        'favourite simulation projects:- IllustrisTNG: https://www.tng-project.org/-\n' +        'SWIFT: https://swift.dur.ac.uk/- CO5BOLD:\n' +        'https://www.astro.uu.se/~bf/co5bold_main.html (which produced these animations\n' +        'of a red-giant star: https://www.astro.uu.se/~bf/movie/AGBmovie.html)-\n' +        'AbacusSummit: https://abacussummit.readthedocs.io/en/latest/And I can add the\n' +        'simulations in the article, too. froeb 5 months ago | parent | next [–]\n' +        'Supernova simulations are especially interesting too. I have heard them\n' +        'described as the only time in physics when all 4 of the fundamental forces are\n' +        'important. The explosion can be quite finicky too. If I remember right, you\n' +        "can't get supernova to explode",      metadata: {        source: 'https://news.ycombinator.com/item?id=34817881',        loc: [Object]      }    },    Document {      pageContent: 'heard them described as the only time in physics when all 4 of the fundamental\n' +        'forces are important. The explosion can be quite finicky too. If I remember\n' +        "right, you can't get supernova to explode properly in 1D simulations, only in\n" +        'higher dimensions. This was a mystery until the realization that turbulence is\n' +        'necessary for supernova to trigger--there is no turbulent flow in 1D. andrewflnr\n' +        "5 months ago | prev | next [–] Whoa. I didn't know the accretion theory of Ia\n" +        'supernovae was dead, much less that it had been since 2011. andreareina 5 months\n' +        'ago | prev | next [–] This seems to be the paper',      metadata: {        source: 'https://news.ycombinator.com/item?id=34817881',        loc: [Object]      }    },    Document {      pageContent: 'andreareina 5 months ago | prev | next [–] This seems to be the paper\n' +        'https://academic.oup.com/mnras/article/517/4/5260/6779709 andreareina 5 months\n' +        "ago | prev [–] Wouldn't double detonation show up as variance in the brightness?\n" +        'yencabulator 5 months ago | parent [–] Or widening of the peak. If one type Ia\n' +        'supernova goes 1,2,3,2,1, the sum of two could go 1+0=1 2+1=3 3+2=5 2+3=5 1+2=3\n' +        '0+1=1 Guidelines | FAQ | Lists |',      metadata: {        source: 'https://news.ycombinator.com/item?id=34817881',        loc: [Object]      }    },    Document {      pageContent: 'the sum of two could go 1+0=1 2+1=3 3+2=5 2+3=5 1+2=3 0+1=1 Guidelines | FAQ |\n' +        'Lists | API | Security | Legal | Apply to YC | Contact Search:',      metadata: {        source: 'https://news.ycombinator.com/item?id=34817881',        loc: [Object]      }    }  ]*/

#### API Reference:

*   [CheerioWebBaseLoader](https://api.js.langchain.com/classes/langchain_document_loaders_web_cheerio.CheerioWebBaseLoader.html) from `langchain/document_loaders/web/cheerio`
*   [MozillaReadabilityTransformer](https://api.js.langchain.com/classes/langchain_community_document_transformers_mozilla_readability.MozillaReadabilityTransformer.html) from `@langchain/community/document_transformers/mozilla_readability`
*   [RecursiveCharacterTextSplitter](https://api.js.langchain.com/classes/langchain_textsplitters.RecursiveCharacterTextSplitter.html) from `langchain/text_splitter`

Customization[​](#customization "Direct link to Customization")
---------------------------------------------------------------

You can pass the transformer any [arguments accepted by the `@mozilla/readability` package](https://www.npmjs.com/package/@mozilla/readability) to customize how it works.

* * *

#### Help us out by providing feedback on this documentation page:

[

Previous

html-to-text

](/v0.1/docs/integrations/document_transformers/html-to-text/)[

Next

OpenAI functions metadata tagger

](/v0.1/docs/integrations/document_transformers/openai_metadata_tagger/)

*   [Setup](#setup)
*   [Usage](#usage)
*   [Customization](#customization)

Community

*   [Discord](https://discord.gg/cU2adEyC7w)
*   [Twitter](https://twitter.com/LangChainAI)

GitHub

*   [Python](https://github.com/langchain-ai/langchain)
*   [JS/TS](https://github.com/langchain-ai/langchainjs)

More

*   [Homepage](https://langchain.com)
*   [Blog](https://blog.langchain.dev)

Copyright © 2024 LangChain, Inc.