teleren/devto huggingface dataset:
Azure Functions: unzip large files
Tags: azure, azurefunctions, node, javascript
Published: 2023-02-09 14:06:01

I want to share my experience unzipping large files stored in [Azure Blob Storage](https://azure.microsoft.com/en-us/products/storage/blobs) using Azure Functions with Node.js.  

I've chosen to use Azure Functions on the [Consumption tier](https://azure.microsoft.com/en-us/pricing/details/functions/#pricing) (pay-as-you-go plan) because our unzip process had to run only once a day. Advantage of it - quite a low price.

Unfortunately, you cannot use a default [Azure Blob Storage trigger](https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-storage-blob-trigger) because from my experience it is unreliable when it comes to triggering functions for many uploaded files in parallel. There are better ways to trigger Azure Functions and my favourite is using [Azure Event Grid](https://learn.microsoft.com/en-us/azure/event-grid/overview) trigger.   

The full function code that can read, unzip and write back large files using Node.js Streams to ensure low memory footprint can be found bellow. 

> I used this function to unzip ~150MB size files. 

```javascript
const { BlobServiceClient } = require("@azure/storage-blob");
const unzipper = require("unzipper");

const ONE_MEGABYTE = 1024 * 1024;
const uploadOptions = { bufferSize: 4 * ONE_MEGABYTE, maxBuffers: 20 };

const AZURE_STORAGE_CONNECTION_STRING = process.env.IMPORT_AZURE_STORAGE_CONNECTION;
// Create the BlobServiceClient object which will be used to create a container client
const blobService = BlobServiceClient.fromConnectionString(AZURE_STORAGE_CONNECTION_STRING);

module.exports = async function (context, zipName) {
    context.log("Unzip: " + zipName);

    const importContainer = blobService.getContainerClient("zip-import");
    const processContainer = blobService.getContainerClient("zip-process");

    // Stream zip 
    const blobClient = importContainer.getBlobClient(zipName);
    const downloadBlockBlobResponse = await blobClient.download();
    const zipStream = downloadBlockBlobResponse.readableStreamBody.pipe(unzipper.Parse({ forceStream: true }));

    for await (const entry of zipStream) {
        const blockBlobClient = processContainer.getBlockBlobClient(entry.path);
        try {
            await blockBlobClient.uploadStream(entry, uploadOptions.bufferSize, uploadOptions.maxBuffers);
            context.log(`Uploaded ${entry.path} from unzipped ${zipName}`);
        } catch (error) {
            throw new Error(`Error while uploading unzipped file ${entry.path}: ${error}`);
        }
    }
};

```

As you can see from the code above, the function reads from the "zip-import" Blob Storage, unzips using [unzipper](https://www.npmjs.com/package/unzipper), a streaming cross-platform unzip tool, and writes back unzipped files into the "zip-process" Blob Storage.

Happy unzipping!
Script to Check the Timeout Value and Recent Execution Time of a Glue Job
Tags: awsglue
Published: 2023-02-09 14:01:30

I wrote a script that calculates the minimum, average, and maximum time taken from the recent execution history of the Glue Job and displays it along with the Timeout value set for the Glue Job.

It is displayed in this way.

```plain
name                                                                   timeout     min    avg    max
gluejob-test2                                                           172800      36     41     44
gluejob-test2-01                                                        172800      52     54     57
longgluejob-sample1                                                     172800    2621   3071   3725
longgluejob-sample2                                                     172800   20837  22760  27465
```

The unit of time is in seconds.

The source code is like this. It is written in Python.

```python
import sys
import boto3

def main():
    profile = sys.argv[1]
    list_jobs(profile)

def list_jobs(profile):
    session = boto3.session.Session(profile_name=profile)
    glue_client = session.client("glue")
    jobs = fetch_glue_job_list(glue_client)
    print("name                                                                   timeout     min    avg    max")
    for job in jobs:
        job_name = job["Name"]
        timeout = job["Timeout"] * 60 # the values returned by the API are in minutes.
        history = fetch_glue_job_history(glue_client, job_name)
        
        job_name_str = job_name.ljust(70)
        timeout_str = str(timeout).rjust(6)
        history_str = " ".join([str(h).rjust(6) for h in history])
        print(f"{job_name_str} {timeout_str} {history_str}")

def fetch_glue_job_list(glue_client):
    jobs = []
    res = glue_client.get_jobs()
    while True:
        for elem in res["Jobs"]:
            jobs.append(elem)
        if "NextToken" not in res:
            break
        res = glue_client.get_jobs(NextToken=res["NextToken"])
    return jobs

def fetch_glue_job_history(glue_client, job_name):
    MAX_COUNT = 100 # retrieve 100 as the recent execution history
    history = []
    res = glue_client.get_job_runs(JobName=job_name)
    while True:
        for run in res["JobRuns"]:
            if run["JobRunState"] == "SUCCEEDED":
                history.append(run["ExecutionTime"])
                if len(history) >= MAX_COUNT:
                    break
        if len(history) >= MAX_COUNT:
            break
        if "NextToken" not in res:
            break
        res = glue_client.get_job_runs(JobName=job_name, NextToken=res["NextToken"])

    if history:
        avg_value = int(sum(history) / len(history))
        max_value = max(history)
        min_value = min(history)
    else:
        avg_value = 0
        max_value = 0
        min_value = 0
    return [min_value, avg_value, max_value]

main()
```
Building MVC Applications in PHP Laravel
Tags: php, programming, laravel
Published: 2023-02-09 13:57:56

I've been writing a new series on @codemagazine for Building MVC applications in PHP Laravel.

You can find here links to all the issues under this series:

1. [Building MVC Applications in PHP Laravel: Part 1](https://www.codemag.com/Article/2205071/Building-MVC-Applications-in-PHP-Laravel-Part-1)

2. [Building MVC Applications in PHP Laravel: Part 2](https://www.codemag.com/Article/2207041/Building-MVC-Applications-in-PHP-Laravel-Part-2)

3. [Dependency Injection and Service Container in Laravel](https://www.codemag.com/Article/2212041/Dependency-Injection-and-Service-Container-in-Laravel)

4. [Mastering Routing and Middleware in PHP Laravel](https://www.codemag.com/Article/2301041/Mastering-Routing-and-Middleware-in-PHP-Laravel)

5. [Authentication in Laravel - Part 1](https://www.codemag.com/Article/2305081/Authentication-in-Laravel-Part-1)

6. [Authentication in Laravel, Part 2: Token Authentication](https://www.codemag.com/Article/2309081/Authentication-in-Laravel-Part-2-Token-Authentication)

More articles coming soon to this series. Stay tuned!

Happy Laraveling!
一个很好的爬虫库
Tags: python, webdev, spider
Published: 2023-02-09 13:48:53

DrissionPage 是一个基于 python 的网页自动化工具。

它既能控制浏览器，也能收发数据包，甚至能把两者合而为一，

可兼顾浏览器自动化的便利性和 requests 的高效率。

它功能强大，内置无数人性化设计和便捷功能。

它的语法简洁而优雅，代码量少，对新手友好。

http://g1879.gitee.io/drissionpage/#/README
Guide for Deploying PHP Laravel Apps on the Google Cloud Platform
Tags: laravel, googlecloud, php
Published: 2023-02-09 13:48:39

I've written a series for a beginner's guide for deploying PHP Laravel Apps on Google Cloud Platform.

Here are the links for all three articles in this series:

1. [Beginner’s Guide to Deploying PHP Laravel on the Google Cloud Platform: Part 1](https://www.codemag.com/Article/2111071/Beginner%E2%80%99s-Guide-to-Deploying-PHP-Laravel-on-the-Google-Cloud-Platform)

2. [Beginner’s Guide to Deploying PHP Laravel on the Google Cloud Platform: Part 2](https://www.codemag.com/Article/2201051/Beginner%E2%80%99s-Guide-to-Deploying-PHP-Laravel-on-the-Google-Cloud-Platform-Part-2)

3. [Beginner’s Guide to Deploying PHP Laravel on the Google Cloud Platform: Part 3](https://www.codemag.com/Article/2203051/Beginner%E2%80%99s-Guide-to-Deploying-PHP-Laravel-on-the-Google-Cloud-Platform-Part-3)


Happy Laraveling!
Remove specific files from your git repository
Tags: git, github
Published: 2023-02-09 13:38:45

Hi 👋. Welcome to this guide on removing specific files from your Git repository. Whether you're a seasoned Git user or just starting out, there will come a time when you need to clean up your repository and remove unwanted files. Perhaps you've accidentally committed sensitive information, or maybe you just need to clean up old, unnecessary files. Whatever the reason, removing files from your Git repository can seem like a daunting task, but it's actually quite simple once you know how. In this post, we'll take a step-by-step approach to explain how to safely remove specific files from your Git repository, and what to do if you've already pushed the files to a remote repository. Let's get started!

Use this command to remove file **only from the git** repository
```git
git rm --cached pathtothefilewhichyouwanttoremove
```
After running this command you should get output like this
```git
rm 'pathtofilewhichyouwanttoremove'
```
If you run `git status` at this point you will get output like this
```git
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        deleted:    js/one.js

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        js/one.js
        
```
Since I have deleted one.js file which is present in js folder, I got this as my output. Depending upon the path and filename of the file which you have deleted you will get similar output. 

If you wish to keep the file at the same location as before you may add it in the `gitignore` file. In this case I am adding `js/one.js` to my git ignore file. You should add relative path to the file which you don't want to appear in your git repository.
```git
js/one.js
```

If you wish to **delete that specific file from your machine** as well you this command 👇 instead
```git
git rm  pathtothefilewhichyouwanttoremove
```
Now you may commit these changes and push to your remote repository. After finishing the above mentioned steps if you head over to your remote repository the file which you have removed will not be present. If you have found this blog post useful leave a like 👍. 

Struck somewhere 🥶, comment down below so that I can help you out.


**Happy Hacking**
E-Learning platform - looking for DEVS
Tags: hiring, nextjs, elearning, education
Published: 2023-02-09 13:21:16

I am sharing information about a new project I will be working on (Online Qualification), it's an E-Learning platform focusing the global educational industry. 

Our aim is to offer education to developing countries, which are still exploring the benefits of the internet and online education. 

Traditionally, studying in the UK is a challenge for many when it comes to tuition fees, our platform will partner with UK qualification providers and deliver undergraduate and postgraduate course with an affordable fees.

We are still #hiring the dream team for this project, it's an incredible opportunity for those who are self-motivated and eager build a career within the company.

Available roles:
-> Senior Software Engineer / Architect
-> Full stack developer
-> Frontend developer
-> Backend developer
-> UI/UX Designer 

- Full time
- On site
- Location: London, Canary Wharf
LGPD e seu impacto no mercado
Tags: veracode, beginners, lgpd, security
Published: 2023-02-09 13:13:33

A Lei Geral de Proteção de Dados Pessoais (LGPD) é uma lei brasileira que regulamenta o tratamento de dados pessoais no Brasil e tem como objetivo proteger os direitos fundamentais de liberdade e privacidade dos titulares de dados. A lei entrou em vigor em agosto de 2020 e estabelece regras claras para o tratamento de dados pessoais por parte de empresas, organizações governamentais e outras entidades.

A LGPD tem um impacto significativo no mercado, pois exige que as empresas adotem práticas e processos mais rigorosos de proteção de dados pessoais, incluindo a gestão de riscos, a implementação de medidas técnicas e administrativas de segurança, a transparência nas informações fornecidas aos titulares de dados e a responsabilização por eventuais falhas na proteção de dados.

A lei também permite que os titulares de dados exijam a correção, apagamento ou portabilidade de seus dados pessoais, o que pode levar a uma maior conscientização dos direitos dos titulares de dados e ao aumento da demanda por soluções de privacidade e proteção de dados.

Além disso, a LGPD impõe penalidades financeiras severas às empresas que não cumprirem as suas disposições, o que pode ter um impacto significativo na saúde financeira das empresas e no seu valor de mercado. Portanto, as empresas precisam estar cientes da LGPD e implementar medidas para garantir o cumprimento da lei, o que pode incluir investir em tecnologia, treinamento de funcionários e revisão de políticas e processos.

Para saber como a Veracode e a M3Corp podem lhe ajudar em seu programa de segurança, entre em [contato](https://www.m3corp.com.br/contato/).
Static Typing in Programming Languages
Tags: webdev, programming, computerscience
Published: 2023-02-09 12:03:49

In programming, static typing is a method of verifying the type of a program at compile time, rather than at runtime. This means that the program is analyzed and checked for type compliance before it is executed. If any part of the program does not conform to the rules of the type system, the program is rejected and will not run.

For example, in most static languages, the expression "a" - 1 is not allowed, as the compiler knows that "a" is a string and 1 is an integer, and the subtraction operator can only be used with values of the same type. This prevents the program from being executed and helps catch errors early in the development process.



Many static typed languages, such as Java, require explicit type declarations, meaning that each variable must be declared with its type. For example, the Java function "public int subtract(int x, int y)" takes two integer arguments and returns an integer result.

However, other statically typed languages, such as Haskell, can infer types automatically. In this case, the function "subtract x y = x - y" is written without type declarations, but the language's powerful and strict type system can determine the type of each argument based on the type of the result.


In conclusion, static typing offers a more rigorous and predictable approach to programming, as the type of each expression in the program can be determined without executing the code. This helps catch errors early on, improves code quality, and leads to more stable and secure applications.
O que é o eLearning da Veracode?
Tags: veracode, beginners, tutorial, security
Published: 2023-02-09 13:13:24

O eLearning da Veracode é uma plataforma de treinamento em segurança de software que fornece aos desenvolvedores, gerentes de segurança e outros profissionais da área uma ampla variedade de cursos e recursos de treinamento para ajudá-los a melhorar suas habilidades e conhecimentos em segurança de software.

Os cursos do eLearning são projetados para cobrir uma ampla gama de tópicos relacionados à segurança de software, incluindo introdução à segurança de software, técnicas de teste de segurança, correção de vulnerabilidades de segurança, gerenciamento de ameaças e muito mais. Os cursos são oferecidos em vários formatos, incluindo vídeos, apresentações, aulas interativas e outros recursos, para ajudar os alunos a aprender de forma eficiente e eficaz.

Além dos cursos, o eLearning da Veracode também fornece uma ampla gama de recursos adicionais para ajudar os alunos a melhorar suas habilidades em segurança de software, incluindo fóruns de discussão, biblioteca de recursos, webinars ao vivo e muito mais.

O objetivo do eLearning da Veracode é ajudar os profissionais da área a compreender e aplicar as melhores práticas em segurança de software, para garantir que as aplicações estejam protegidas contra ameaças cibernéticas e mantenham a privacidade e a integridade de dados sensíveis.

Para saber mais sobre a Veracode e solicitar um ambiente de testes, entre em [contato](https://www.m3corp.com.br/contato/).
10 a Penny, one consideration before you specialise in a JavaScript framework
Tags: discuss, javascript, react
Published: 2023-02-09 13:11:43

Hi!

_This is the perspective of a hopeful entrepreneur_

When React was very new, it used to wind me up that people didn't call themselves JavaScript developers after all that is what they are writing, and only with Svelte and similar languages I can honestly thats not true anymore, its a language dressed as a framework.

I was told in a surreal encounter with an old school C programmer when I used to work in a corner shop/store, "Go where the money is" I always try to.

So i'm an experienced consultant with a business idea, I can write in any framework or language I want too, i'm at that stage I don't really mind what i'm using, but I see speed as important in my particular use case, fast is good - so I would choose the fastest, but what about hiring?

## If I got big
If the business really took off, hiring Svelte developers or python (on Graal) developers might be hard - I could choose React after all there are so many react developers that their average wage is actually lower than any other framework, because 10 a penny and for Svelte its almost twice as much in wage but half as much work out there, so do I pick the best by using the best performing stack? I have no idea whats right...
React for Beginners: Build a To-Do app with React and Tailwind CSS
Tags: webdev, javascript, react
Published: 2023-02-09 12:38:36

In this video tutorial, I show how to build a to-do app with ReactJS and Tailwind CSS. While building, you'll learn about different hooks, how react works, and how Tailwind can simplify development process.

Here's the link => https://www.youtube.com/watch?v=e3lloTY9V0s&t=2232s
Top 50 Linux Commands You Must Know as a Regular User.
Tags: cybersecurity
Published: 2023-02-09 12:38:17


**ls **- The most frequently used command in Linux to list directories
**pwd **- Print working directory command in Linux
**cd **- Linux command to navigate through directories
**mkdir **- Command used to create directories in Linux
**mv **- Move or rename files in Linux
**cp **- Similar usage as mv but for copying files in Linux
**rm **- Delete files or directories
**touch** - Create blank/empty files
**ln **- Create symbolic links (shortcuts) to other files
**cat **- Display file contents on the terminal
**clear **- Clear the terminal display
**echo **- Print any text that follows the command
**less **- Linux command to display paged outputs in the terminal
**man **- Access manual pages for all Linux commands
**uname **- Linux command to get basic information about the OS
**whoami **- Get the active username
**tar **- Command to extract and compress files in Linux
**grep** - Search for a string within an output
**head **- Return the specified number of lines from the top
**tail **- Return the specified number of lines from the bottom
**diff **- Find the difference between two files
**cmp **- Allows you to check if two files are identical
**comm **- Combines the functionality of diff and cmp
**sort **- Linux command to sort the content of a file while outputting
**export **- Export environment variables in Linux
**zip **- Zip files in Linux
**unzip **- Unzip files in Linux
**ssh **- Secure Shell command in Linux
**service **- Linux command to start and stop services
**ps **- Display active processes
**kill **and **killall **- Kill active processes by process ID or name
**df **- Display disk filesystem information
**mount **- Mount file systems in Linux
**chmod **- Command to change file permissions
**chown **- Command for granting ownership of files or folders
**ifconfig **- Display network interfaces and IP addresses
**traceroute **- Trace all the network hops to reach the destination
**wget **- Direct download files from the internet
**ufw **- Firewall command
**iptables** - Base firewall for all other firewall utilities to interface with
**apt, pacman, yum, rpm** - Package managers depending on the distro
**sudo **- Command to escalate privileges in Linux
**cal **- View a command-line calendar
**alias **- Create custom shortcuts for your regularly used commands
**dd **- Majorly used for creating bootable USB sticks
**whereis **- Locate the binary, source, and manual pages for a command
**whatis** - Find what a command is used for
**top **- View active processes live with their system usage
**useradd **and usermod - Add new user or change existing users data
**passwd** - Create or update passwords for existing users
Web automation & scraping with Playwright JS
Tags: webdev, javascript, playwright
Published: 2023-02-09 12:37:10

Playwright is a very powerful tool. You can use it to do end-to-end testing, web automation, creating bots and so on. 

In this tutorial, I show how we can strap up an application that goes and do stuff for us on Ebay.

Here's the link =>  https://www.youtube.com/watch?v=Z7zGUFZgkgA&t=930s
Self Host your Own Restream Application using Nginx and Docker Today
Tags: nginx, restream, docker
Published: 2023-02-09 11:53:02

Streaming content to multiple channels is something every streamer can get excited about, but their is a catch, highly-priced licenses. Well, that's where Nginx comes in as your saviour to self host your own solution.

Last year, i was tasked to stream to 8 platforms all at once, **3 Facebook pages**, **2 Twitter Pages**, and **3 Youtube accounts**. 

Restream had limits to its own licensing, the same applied to Castr.io. So, I chose the manual way of getting things done, create the solution myself. So, i cloned and improved a solution by `Rafalfaro18` on Github.

If you are in the same situation as i was, worry not, simply follow the steps below.

* Deploy a cloud server probably running a friendly distro such as Ubuntu, or CentOS.

> I use [Vultr](https://vultr.com), but if you have a Digital Ocean or Linode account, you can use that, there are no limitations.

* Login using a regular account that's a member of the Sudoers group.
* [Install Docker and Docker Compose](https://techpointmag.com/how-to-install-docker-and-docker-compose/) on the server.
* [Clone my Github repository](https://github.com/mjhumphrey4/self-hosted-restream-application).
* View contents of the README file and setup your streaming platforms.
* Deploy the application and make sure the container is running, and allow port `1935` through your firewall.
* Start sending your stream to the server, Nginx will broadcast it to all streaming platforms you set in the `.env` file.
* Go to your Youtube channel or other channels you set and verify that your stream is going through.

Just like that, you can break the chains all thanks to open-source. More over, once you are done streaming your content, you can safely delete the server to avoid any extra costs billed to your account. 

Now, compare that to a 1 month Restream license that will remain unused when you are not streaming.

I hope you find this post useful, if you do, a thumbs up would not be bad either.