Automated Lip Syncing in Mini Micro
Tags: miniscript, minimicro, programming, animation
Published: 2023-12-26 23:14:02

[Mini Micro](https://miniscript.org/MiniMicro) version 1.2, released this summer, contains a number of [new features](https://dev.to/joestrout/announcing-mini-micro-12-28id).  These include a new `Sound.amp` property, which tells you the amplitude of a playing sound.  This can be used for a lot of things — but in this post, I'll show how it can be used to animate the mouth of a character on screen as they talk.

The video above was created with the following program using this technique.

```
clear
mainImage = file.loadImage("mcp.png")
gfx.drawImage mainImage, 480-mainImage.width/2, 320-mainImage.height/2

jawSprite = new Sprite
jawSprite.image = file.loadImage("mcp-jaw.png")
jawSprite.x = 541; jawSprite.y = 151
jawSprite2 = new Sprite
jawSprite2.image = jawSprite.image
jawSprite2.x = 411; jawSprite2.y = 151
display(4).sprites = [jawSprite, jawSprite2]

list.swap = function(i,j)
	temp = self[i]
	self[i] = self[j]
	self[j] = temp
end function
c = jawSprite2.corners
c.swap(0,1); c.swap(2,3)  // (flips the image horizontally)
jawSprite2.setCorners c

openMouth = function(amount)
	c = jawSprite.corners
	dy = 32 * amount
	c[0][1] = c[1][1] - dy  // bottom inner corner
	c[3][1] = c[2][1] - dy  // top inner corner
	jawSprite.setCorners c
	c = jawSprite2.corners
	c[0][1] = c[1][1] - dy  // bottom inner corner
	c[3][1] = c[2][1] - dy  // top inner corner
	jawSprite2.setCorners c
end function

speech = file.loadSound("happyNewYear.ogg")

speech.play
amount = 0
while speech.isPlaying
	a = speech.amp
	target = a * 5  // (found experimentally)
	amount = (amount + target*4) / 5
	openMouth amount
	yield
end while
openMouth 0
```

The first half of this code is just setup: loading and drawing the background image (mcp.png), and then loading two sprites for the jaw.  It's two sprites because I was lazy and only prepared an image for the _right_ half of the jaw, which gets loaded as `jawSprite`; for the _left_ half of the jaw, I just make another sprite with the same image, but swap the left and right corners, inverting it (this is `jawSprite2`).

To animate the mouth, I first created the `openMouth` function, which again monkeys with the corners of the sprites.  This time it just moves the "inner" corners of the sprites (representing the center of the chin) down a few pixels, based on the given `amount` parameter.

Finally, the main loop at the bottom of the code does all the real work.  It loads and plays an audio file (happyNewYear.ogg), and then goes into a loop for as long as that speech sound is playing.  On each iteration of the loop, it reads the current amplitude of the sound using `speech.amp`.  Because this value changes very rapidly, I do a bit of smoothing (`amount = (amount + target*4)/5`), and then pass the result to the `openMouth` function we defined above.

And that's it!  With this simple technique, you can make a compelling illusion of a character talking.

All the code and resources are available over at [GitHub](https://github.com/JoeStrout/minimicro-lip-sync-demo).  Why not download it and give it a try yourself?  In either case, please let me know what you think in the comments below!