RESTful Web scraping in Scala, using Play Framework and Jsoup
Tags: scala, webdev, restapi, backend
Published: 2020-09-19 12:43:39

<!-- wp:paragraph -->
<p>Recently I have encountered a very cool site with cooking recipes, which had extremely poor UI, especially when using a mobile. There was no official API, and so I have decided to build a web service that would web scrape the content out of it, and publish it using RESTful API. In this post I will show you how to use <a href="https://www.scala-lang.org/">Scala</a>, <a href="https://www.playframework.com/" target="_blank" rel="noreferrer noopener">Play Framework</a> and <a href="https://jsoup.org/" target="_blank" rel="noreferrer noopener">Jsoup</a> to build such service. Enjoy!</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>You will need</h2>
<!-- /wp:heading -->

<!-- wp:list -->
<ul><li>Scala - I use the version 2.13.1</li><li>Sbt - install from official <a href="https://www.scala-sbt.org/" target="_blank" rel="noreferrer noopener">website</a></li><li>IDE of choice - IntelliJ or VS Code, I use latter</li><li>Around 15 minutes of your time</li></ul>
<!-- /wp:list -->

<!-- wp:heading -->
<h2>Creating empty Play project</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>We will start off by creating a Play project. You can of course start from the scratch and add the code iteratively, but for the scope of this tutorial, I will suggest using on of the Play's example projects as a starter. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>The <a href="https://github.com/playframework/play-samples/tree/2.8.x/play-scala-rest-api-example">Scala Rest API project</a> have almost everything we need. This project serves as a nice backbone to our use case, which we will be able to extend with ease.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Let's start by cloning the Play Framework samples repository, and opening the Scala Rest API example like this:</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock {"language":"md"} -->
<pre class="EnlighterJSRAW" data-enlighter-language="md" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">git clone https://github.com/playframework/play-samples.git
cd play-scala-rest-api-example</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>After this, try running this command - the Play application should shoot off, and you should be able to access it at <code>http://localhost:9000</code></p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">sbt run</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>If the application builds and runs correctly, we are done with initializing the project. Now, you can open this folder with your IDE of choice - I will use VS Code</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Adding Jsoup dependency</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Next, we will add the Jsoup - an open source HTML parser for Java and JVM. To do this, open a <code>build.sbt</code> file and add the following in the <code>libraryDependencies</code> block:</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">"org.jsoup" % "jsoup" % "1.13.1"</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>Of course, make sure to use the latest version possible - at the time of authoring this article, <strong>1.13.1</strong> was the latest one.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Now, wait for <code>sbt</code> to refresh the dependencies, and we are ready to write some code!</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Defining an endpoint</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>The first coding step is to define an endpoint to Play framework. This is done, by adding route definitions to <code>conf/routes</code> file. In our case, I want to have a <code>v1/recipes</code> endpoint that is solely handled by <code>RecipeRouter</code> class. We can do this by adding the following:</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">->         /v1/recipes             v1.recipe.RecipeRouter</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>Now, any request that hits that URL, will be redirected to our router.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Next of, I am creating a package for my recipe related classes <code>app/v1/recipe</code>. In here, let's create an entry point that will handle the requests: <code>RecipeRouter</code></p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="java" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">package v1.recipe

import javax.inject.Inject

import play.api.routing.Router.Routes
import play.api.routing.SimpleRouter
import play.api.routing.sird._

class RecipeRouter @Inject()(controller: RecipeController) extends SimpleRouter {
  val prefix = "/v1/recipes"

  def link(id: String): String = {
    import io.lemonlabs.uri.dsl._
    val url = prefix / id.toString
    url.toString()
  }

  override def routes: Routes = {
    case GET(p"/") =>
      controller.index

    case GET(p"/all/$id") =>
      controller.showAll(id)
  }

}</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:heading -->
<h2>Connecting to website</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Going further, it's time to finally use <code>Jsoup</code> to get some content out of the website. In the previous section, we have defined <code>showAll</code> function, and that's the one I will be implementing here. We will be making use of the <code>id</code> parameter, which come in handy in many situations.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>So, let's move to <code>RecipeController</code> class, and define the first element: <code>sourceUrl</code>. I like to have it defined as a class field, so that it is accessible by all methods:</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">val sourceUrl: String = "https://kwestiasmaku.com"</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>Then, inside our method, let's get the content of this website. It can be done like this:</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">def showAll(pageId: String): Action[AnyContent] = Action { implicit request =>
    val htmlDocument = Jsoup.connect(s"${sourceUrl}/home-przepisy?page=${pageId}").get()

    val r: Result = Ok(htmlDocument)
    r
}</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>In the snippet above, we are connecting to our website, b using <code>Jsoup.connect</code> method, while passing the correct URL as a string. What we get, is an HTML document, that we just return for now. We will do some finer extraction in the next section.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>To test the endpoint, call the following URL in your browser, or any http client, like <code>curl</code>:</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">http://localhost:9000/v1/recipes/all/1</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>You should see a response, which is raw HTML document. Now, let's parse it and extract some meaningful data.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Extracting the content</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Now, let's extract some specific elements from the raw HTML. For this, we can use the <code>select</code> method of Jsoup. This method requires a CSS like selector syntax - if you have used CSS before, then it is as easy specifying the correct string - in our case, to extract single recipe HTML object, use the following:</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">import scala.jdk.CollectionConverters._

val recipesDomElements = htmlDocument.select("section#block-system-main .col").asScala</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>This query returns, collection-like structure, on which we can easily iterate. To extract specific elements from a single recipe, use the following:</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">final case class Recipe(title: String, href: String, img: String)

val recipeData = for(recipeElement &lt;- recipesDomElements)
      yield Recipe(
        recipeElement.select(".views-field-title a").html(),
        sourceUrl + recipeElement.select(".views-field-title a").attr("href"),
        recipeElement.select("img").attr("src")
      )</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>In here, we use a simple <code>for</code> loop, which yields the concrete data, by wrapping it into <code>Recipe</code> case class. Now, we have nicely cleaned data, and we can finally return it as JSON.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Returning a JSON</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Last step is to return our collection of <code>Recipe</code> case classes as a JSON. We can do this using Play's built in <code>Json</code> class, and its methods:</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">implicit val recipeWrites: OWrites[Recipe] = Json.writes[Recipe]

val r: Result = Ok(Json.toJson(recipeData))
r</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>An important thing to have in mind, is that before converting a case class into its JSON representation, we must declare implicit writer, so that the <code>Json</code> class knows how to write this specific data.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>And that's it! The complete <code>RecipeController</code> class implementation looks like the following:</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">package v1.recipe

import javax.inject.Inject
import org.jsoup.Jsoup
import play.api.mvc._
import play.api.libs.json.{Json, OWrites}

import scala.jdk.CollectionConverters._

class RecipeController @Inject()(val controllerComponents: ControllerComponents) extends BaseController {
  implicit val recipeWrites: OWrites[Recipe] = Json.writes[Recipe]
  val sourceUrl: String = "https://kwestiasmaku.com"

  def index: Action[AnyContent] = Action { implicit request =>
    val r: Result = Ok("hello world")
    r
  }

  def showAll(pageId: String): Action[AnyContent] = Action { implicit request =>
    val htmlDocument = Jsoup.connect(s"${sourceUrl}/home-przepisy?page=${pageId}").get()
    val recipesDomElements = htmlDocument.select("section#block-system-main .col").asScala

    val recipeData = for(recipeElement &lt;- recipesDomElements)
      yield Recipe(
        recipeElement.select(".views-field-title a").html(),
        sourceUrl + recipeElement.select(".views-field-title a").attr("href"),
        recipeElement.select("img").attr("src")
      )

    val r: Result = Ok(Json.toJson(recipeData))
    r
  }
}</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:heading -->
<h2>Summary</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>I hope you have found this post useful. If so, don’t hesitate to like or share this post. Additionally, you can follow me on my social media if you fancy so 🙂</p>
<!-- /wp:paragraph -->