Using Computer Vision to extract sprite pixel art
Tags: kotlin, java, opencv, pixelart
Published: 2020-05-27 12:53:09

---
title: Using Computer Vision to extract sprite pixel art
published: true
description: How to automate sprite sheet splitting using openCV
tags: kotlin, java, opencv, pixelart 
cover_image: https://raw.githubusercontent.com/viniciusccarvalho/static_blog_content/master/images/mario/nes_remix_2.jpg
---
# Introduction
Recently my 7yr old has been asking me to find some Mario Pixel Art for him to use as template to build minecraft structures.

<figure>
<img src="https://dev-to-uploads.s3.amazonaws.com/i/4giloazsy4qtkqqlwk48.png">
<figcaption>If you Google it, this is probably the top result<figcaption>
</figure>

There are quite a few like this one, high resolution and with guiding lines. Then one day your kid asks your for the [Hammer Mario](https://www.mariowiki.com/Hammer_Mario), that one was not easy to find.

My goal was to avoid having to use tools such as Gimp or [TexturePacker](https://www.codeandweb.com/texturepacker) to manipulate pixel art.

Another Google search and I found [Mario Universe](http://www.mariouniverse.com/) a website full of Mario [Spritesheets](https://en.wikipedia.org/wiki/Texture_atlas).

<figure>
<img src="https://dev-to-uploads.s3.amazonaws.com/i/rxirto8epr5voplq7h0w.png">
<figcaption>A perfect Super Mario Bros 3 Spritesheet/Texture Atlas
</figure>

But again, I don't want to cut each one of those sprites (187 in that picture alone). I wanted to automate all that.

<figure>
<img src="https://dev-to-uploads.s3.amazonaws.com/i/6gvk47u01fvp1oe5ntuw.png">
<figcaption>This is the end goal result</figcaption>
</figure>

# Enter openCV
My first thought was to use openCV. I'm not familiar at all with it, I won't pretend I am. But I know it's a powerful toolkit that is used for instance to detect objects in frames. There must be an easy way to do that for simple sprites.

Another Google search and I bumped into this [Stackoverflow](https://stackoverflow.com/questions/13584586/sprite-sheet-detect-individual-sprite-bounds-automatically) thread: BINGO!

That is exactly what I was looking for. I had to tweak the parameters a bit, for example my structuring element size was reduced to be a 1px wide.

I needed pixel perfect matching for the images, using anything but would merge some sprites with their neighbors as a single rectangle.

Also, I wanted this to be in Kotlin, the Python code is easy enough to use. But I intend to build a small page for my kid to choose each sprite and print it, writing in Kotlin would be easier to me.

So here's the snippet I ended up writing to save the sprites from an input image:

```kotlin
fun extractSprites(file: File) {
		val parentFolder = file.parentFile
		val output = File(parentFolder, file.nameWithoutExtension)
		output.mkdirs()

		val bufferedImage = ImageIO.read(file)
		val image = Imgcodecs.imread(file.absolutePath)
		val gray = Mat()
		val thresh = Mat()
		val close = Mat()
		val dilate = Mat()
		val hierarchy = Mat()
		val contours = mutableListOf<MatOfPoint>()

		Imgproc.cvtColor(image, gray, Imgproc.COLOR_BGR2GRAY)
		Imgproc.adaptiveThreshold(gray, thresh, 255.0,
				Imgproc.ADAPTIVE_THRESH_GAUSSIAN_C,
				Imgproc.THRESH_BINARY_INV, 5, 1.0)
		val kernel = Imgproc.getStructuringElement(Imgproc.MORPH_RECT,
				Size(1.0,1.0))
		Imgproc.morphologyEx(thresh, close,  Imgproc.MORPH_CLOSE,
				kernel, Point(-1.0, -1.0), 2)
		Imgproc.dilate(close, dilate, kernel,
				Point(-1.0, -1.0), 1)
		Imgproc.findContours(dilate, contours, hierarchy,
				Imgproc.RETR_EXTERNAL, Imgproc.CHAIN_APPROX_SIMPLE)


		for ((index, it) in contours.withIndex()) {
			val rect = Imgproc.boundingRect(it)
			val sprite = bufferedImage.getSubimage(rect.x,
					rect.y,
					rect.width,
					rect.height)
			val fileName = "${file.nameWithoutExtension}_$index"
			ImageIO.write(sprite, "png",
					File(output, "${fileName}.${file.extension}"))
			
		}

	}
```

The result creates indexed images such as `picture_<index>.<extension>` on a folder with the same name as the source image.

# Making those tiny images scalable

Now that I have an almost infinite supply of tiny 32x32 images, I had to find a way to scale them. Again the goal is to avoid using a tool, I wanted a website where my kid could go click on a pixel and get a full page print of it in high quality.

Well, turns out [Scalable Vector Graphics](https://en.wikipedia.org/wiki/Scalable_Vector_Graphics) or SVG for shorts exists for that. And they are nothing but a textual markup language. Converting an image into an SVG is very easy, no need for a tool or a framework for such simple task. All you need to do is read each pixel, and if the color of that pixel is not transparent write it as a `<rect>` element of an SVG document.

```kotlin
fun convertToSVG(image: BufferedImage, svgFolder: File, fileName: String) {
		val width = image.width
		val height = image.height
		val header = "<?xml version=\"1.0\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\" \n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns=\"http://www.w3.org/2000/svg\"\n     version=\"1.1\" width=\"$width\" height=\"$height\">\n";
		val svg = StringBuilder()
		svg.append(header)
		for(y in 0 until height){
			for(x in 0 until  width){
				val color = Color(image.getRGB(x, y), true)
				if(color.alpha != 0 ) {
					val rgbColor = Integer.toHexString(color.rgb).substring(2)
					svg.append("    <rect x=\"${x}px\" y=\"${y}px\" width=\"1px\" height=\"1px\" fill=\"#$rgbColor\"/>\n")
				}
			}
		}
		svg.append("</svg>")
		val output = File(svgFolder, "$fileName.svg")
		output.writeText(svg.toString())
	}
```


<figure>
<img src="https://dev-to-uploads.s3.amazonaws.com/i/jsxc18abctr9qkr9m5w3.png" width="320" height="560" style="float: left">
<figcaption> Original image scaled 20x<figcaption>
</figure>

<figure>

<img src="https://raw.githubusercontent.com/viniciusccarvalho/static_blog_content/master/images/mario/mario_3.svg">
<figcaption>SVG version</figcaption>
</figure>

Easy to see the difference right?

My next steps are to make all this consumable for my kid, a simple page where he can click on the sprite sheet, explode it and send to the printer in a large svg format with some guiding grids (pretty sure if there ain't any tool out there, adding those lines will be easy)

Hope you had as much fun reading this as I had writing it :)

Happy coding