Puppeteer doesn’t remove files from temp directory
Tags: puppeteer, headless, temp, crontab
Published: 2023-03-19 07:11:21

<!-- wp:paragraph -->
<p>Frequently I am building bots. They are running on the server and gathering some data. For some of them, I use Puppeteer and headless Chromium instances. That combination works great. However, there is an issue. Puppeteer doesn't remove files from a temp directory. After a while, it could be a problem, because directory size is growing consistently. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>Did you meet the same issue? Let me help you fix it. </p>
<!-- /wp:paragraph -->

<!-- wp:more -->
<!--more-->
<!-- /wp:more -->

<!-- wp:heading -->
<h2>Good to know</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>This article is suitable only for people who run <a href="https://pptr.dev/" target="_blank" rel="noreferrer noopener">Puppeteer</a> on Linux (maybe on Mac?). If you are using Windows, unfortunately, I can't help. However, perhaps you could use a general idea and make the same for your case.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Long story short</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>I solved that issue by adding a crontab task, which deletes all files with a modification time older than 10 days. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>If you want to know more details read the following text.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Search Puppeteer temp files</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>Let's go into <code data-enlighter-language="generic" class="EnlighterJSRAW">/tmp</code> directory and check where are Puppeteer files.</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group=""># show files from /tmp filter by "puppeteer"
ls /tmp | grep puppeteer</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:image {"align":"center","id":2933,"width":317,"height":484,"sizeSlug":"full","linkDestination":"none"} -->
<figure class="wp-block-image aligncenter size-full is-resized"><img src="https://jakeroid.com/wp-content/uploads/2023/03/image.png" alt="" class="wp-image-2933" width="317" height="484"/></figure>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Here we go. Puppeteer creates many directories. Let's calculate how many directories we have and the total size of them. </p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group=""># count of directories
ls /tmp | grep puppeteer | wc -l

# total size of them 
du -hc /tmp/puppeteer* | grep total</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>I got 25782 directories and 27 gigabytes of size. It's huge... Let's move on and automate removing them.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Help Puppeteer remove temp files</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>If the mountain doesn't go to me, I will go to the mountain 😄. Let's write a crontab task to remove Puppeteer files that weren't modified for a long time. </p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p>First, let's write a command to find files older than 10 days (for example).</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group=""># find dirs and files that weren't modified 10+ days
find /tmp/puppeteer* -mtime +10</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>The command will find files that weren't modified 10 or more days ago. To delete them, we could extend the command like the following one.</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group=""># find and delete dirs and files that weren't modified 10+ days
find /tmp/puppeteer* -mtime +10 -exec rm -rf {} \;</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>Time to add it as a crontab task. Write the following command to open the crontab config.</p>
<!-- /wp:paragraph -->

<!-- wp:enlighter/codeblock -->
<pre class="EnlighterJSRAW" data-enlighter-language="generic" data-enlighter-theme="" data-enlighter-highlight="" data-enlighter-linenumbers="" data-enlighter-lineoffset="" data-enlighter-title="" data-enlighter-group="">crontab -e</pre>
<!-- /wp:enlighter/codeblock -->

<!-- wp:paragraph -->
<p>Here is my example. The crontab will run that command each day at 4:05 and delete all files related to Puppeteer. I used the full path to find the command to be sure crontab would use the correct one.</p>
<!-- /wp:paragraph -->

<!-- wp:image {"id":2939,"sizeSlug":"large","linkDestination":"none"} -->
<figure class="wp-block-image size-large"><img src="https://jakeroid.com/wp-content/uploads/2023/03/image-1-1024x78.png" alt="" class="wp-image-2939"/></figure>
<!-- /wp:image -->

<!-- wp:heading -->
<h2>Conclusion</h2>
<!-- /wp:heading -->

<!-- wp:paragraph -->
<p>That simple way will help Puppeteer to remove files from a temp directory. If you have any questions, ask them below in the comments section.</p>
<!-- /wp:paragraph -->

<!-- wp:paragraph -->
<p><em>If you are interested, I have one more article related to headless browsers. Check it out: <a href="https://jakeroid.com/blog/how-to-protect-content-against-scrapers-with-headless-browsers/" target="_blank" rel="noreferrer noopener">How to protect content against scrapers with headless browsers?</a></em></p>
<!-- /wp:paragraph -->