teleren/devto huggingface dataset:
Scoopi Web Scraper
Published: 2022-12-02 12:50:28

In Java, it is easy to scrape with JSoup and HtmlUnit in Java, but the things get complicated when data is from large number of pages. Scraping libraries do well in scraping data from limited set of pages but they are not meant to handle thousands of pages.

[Scoopi Web Scraper](https://www.codetab.org/tutorial/scoopi-web-scraper/introduction/) is developed taking these aspects into consideration. It is built upon JSoup and HtmlUnit. Some of the features of Scoopi are 

- Scoopi is completely definition driven and no knowledge of coding is required. Data structure, task workflow and pages to scrape are defined with a set of YML definition files. It can be configured to use either JSoup or HtmlUnit as scraper.
- Query can be written either using Selectors with JSoup or XPath with HtmlUnit.
- Scoopi persists pages and parsed data to file system and recovers from the failed state without repeating the tasks that are completed.
- Scoopi is a multi-thread application which process pages in parallel for maximum throughput.
- Allows to transform, filter and sort the data
- In cluster mode, it can scale horizontally by distributing tasks across multiple nodes.
- Designed to run in various environments; in bare JVM or in Docker containers or even on high end container orchestration platforms such as Kubernetes.

Know more about Scoopi at [Scoopi Web Scraper](https://www.codetab.org/tutorial/scoopi-web-scraper/introduction/)
#FuzzingWeekly CW 48
Tags: opensource, security, testing, cybersecurity
Published: 2022-12-02 12:43:07

Looking for Remote Code Execution bugs in the Linux kernel > https://xairy.io/articles/syzkaller-external-network

Introducing Afl-Ruby: fuzz your Ruby programs using afl > https://robertheaton.com/2019/02/16/introducing-afl-ruby/

Burp Suite - Fuzzing for Credentials >
https://www.youtube.com/watch?v=MrWPpgUVf00
🎄JS Advent #2 - Mutable and Immutable methods 🎄
Tags: javascript, webdev, programming, beginners
Published: 2022-12-02 12:40:56

##### For the month of December I have challenged myself to write 24 articles about JS up until Xmas.
##### The second installment in this series is around Mutable and Immutable array methods.

#### What is a mutable method?
- A mutable method is a method which changes the state of the array it is called on.
eg. `splice`, `pop`, `push`, `unshift`, `shift`, `reverse`

#### What is an immutable method? 
- An immutable method is a method which does not change the state of the array it is called on and sometimes returns a new array instead.
eg. `concat`, `map`, `slice`, `filter`

#### Examples of array mutation methods
pop:
```javascript
let arr = [1,2,3,4];
console.log(arr); // [1,2,3,4]
arr.pop();
console.log(arr); // [1,2,3]
```

push:
```javascript
// reset the array
let arr = [1,2,3,4];
arr.push(5);
console.log(arr); // [1,2,3,4,5]
```

reverse:
```javascript
 // reset the array
let arr = [1,2,3,4];
arr.reverse();
console.log(arr); // [4,3,2,1]
```

unshift:
```javascript
// reset the array
let arr = [1,2,3,4]; 
arr.unshift(1);
console.log(arr); // [1,1,2,3,4]
```

- __The value of the original array has changed in every instance__.

## Examples of array immutable methods
filter:
```javascript
let arr = [1,2,3,4];
let newArray = arr.filter(el => el > 2);
console.log(arr); // [1,2,3,4]
console.log(newArray); // [3,4]
```

map:
```javascript
let arr = arr.map(el => el * 2);
console.log(arr); // [1,2,3,4]
console.log(newArray); // [2,4,6,8]
```

concat:
```javascript
let newArray = arr.concat([5,6,7]);
console.log(arr); // [1,2,3,4]
console.log(newArray); // [1,2,3,4,5,6,7]
```

slice:
```javascript
let newArray = arr.slice(2)
console.log(arr); // [1,2,3,4]
console.log(newArray); // [3,4,]
```

- __The original array stays the same when the immutable methods are called on it.__

### Pure functions
- A pure function is a function which as well as returning the same result every time it is executed with the same inputs also does not have any other side affects.
- Using immutable array methods in functions contributes to this as they do not produce the side effect of changing the array.

[< JS Advent #1](https://dev.to/jtlavs/js-advent-1-truthy-and-falsy-values-41b1)
Introduction to Numpy
Tags: machinelearning, learnwithmitul, ai, beginners
Published: 2022-12-02 12:37:33

Numpy is a general-purpose array-processing package. It's a linear algebra of python. It's important for data science it's because all py data ecosystem rely on numpy as their building blocks. Numpy is incredibly fast and has bindings to see library.

**Arrays**
Array in Numpy is a table of elements (usually numbers), all of the same type, indexed by a tuple of positive integers. In Numpy, number of dimensions of the array is called rank of the array.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lbdssg76i86mo16tj54i.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1ve1x24aofid1wx54dyz.png)
**Methods in Numpy**
- array()
- arange()                                                      
- zeros()
- argmin()
- argmin()
- random.rand() (from random library)
- amax()
- amin()
- asmatrix()
- arr.shape
- arr.arrange()
- arr.dtype
- randint() (from random library)
- linspace()
- ones()
- eye()
- reshape()

These are just some common ones there are more methods.

**Indexing and slicing**
**One-Dimensional NumPy Array Elements with Indexing**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/c66l7yxmlinh1enpx7ax.png)
**Two-Dimensional NumPy Array Elements with Indexing**





![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3wkicz612srycrar15ot.png)




**Slicing and Striding NumPy Arrays**


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6cm3t5lngis6ghrhf9c3.png)

**Integer Array Indexing in NumPy to Access Multiple Elements**

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b9q4nhys2u3aidzelg0p.png)


**Boolean Indexing in NumPy Arrays for Conditional Slicing**
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hsidkieutoauedpqy557.png)

**Array Operations**
You can use addition, subtract, multiply, divide an array by an array. But when dividing look out if the array has zero inside it or when dividing the whole array by one if the array contains zero it will run but won't give you an error but will give you a warning.

You can also use trigonometric functions, log and other operations. Do look at this website if the functions you are trying to use are available at NumPy library or not. 
https://docs.scipy.org/doc/numpy-1.15.1/reference/ufuncs.html
5 Top Node.js Streams Resources You Should Learn to Level Up Your Node.js Skills 🚀💯
Tags: javascript, node, beginners, webdev
Published: 2022-12-02 12:27:31

If you want to master Node.js, stop screwing around 🛑.

Learn **Streams.**

Streams are the fundamental block of the essential Node.js features:

- HTTP server
- File management
- Encryption and Decryption

This means you can build (TODAY) a video streaming service or ingest and transform terabytes of data just by learning streams.

You can use these resources and unlock all the Node.js power 🔐🔥.

## 1. Streams workshop 📒

Don’t you know anything about theory and practice about streams? This workshop is the gold standard and the perfect place to get started with streams—an ideal combination of theory and practical exercises.

Author ✍🏻: Luciano Mammino

Link 🔗: [https://github.com/lmammino/streams-workshop](https://github.com/lmammino/streams-workshop)

## 2. **Everything you should know about Node.js Streams 📽️**

If you are more of a “video guy,” these are fantastic walkthrough about how streams work and how you can use them in your project.

PLUS 🆕: the first video contains the new stream features introduced in Node.js v18.

Author ✍🏻: Erik Wendell

Link 🔗: [https://youtu.be/BdePYKgrMh0](https://youtu.be/BdePYKgrMh0)

**_EXTRA_**:
Author ✍🏻: Luciano Mammino

Link 🔗: [https://youtu.be/ldcfYB_mo6Q](https://youtu.be/ldcfYB_mo6Q)


## 3. **Understanding Streams in Node.js 🔁**

Modern Node.js streams with async iterator.

Author ✍🏻: Liz Parody

Link 🔗:  [https://nodesource.com/blog/understanding-streams-in-nodejs/](https://nodesource.com/blog/understanding-streams-in-nodejs/)

## 4. **NodeJS Backpressuring in Streams 🚱**

Backpressure is an important technique to prevent your application from being overwhelmed by data. It is essential to understand and master backpressure.

Author ✍🏻: Vladimir Topolev

Link 🔗: [https://enlear.academy/nodejs-backpressuring-in-streams-52638f505e1b](https://enlear.academy/nodejs-backpressuring-in-streams-52638f505e1b)

## 5. **How To Code A Video Streaming Server in NodeJS 🕹️**

All Node.js stream knowledge is put into practice to build a real-world example: a video streaming service.

Author ✍🏻: Abdisalan Mohamud

Link 🔗: [https://youtu.be/ZjBLbXUuyWg](https://youtu.be/ZjBLbXUuyWg)


_If you liked the article, follow us on Twitter [@fabrizio.lallo](https://twitter.com/FabriLallo) and [@AndrewHu368](https://twitter.com/AndrewHu368)_


_Credits to Conny Schneider_
https://unsplash.com/photos/pREq0ns_p_E
How to Setup and Customize Tailwind in Nuxt.js
Tags: frontend, nuxt, tailwindcss, webdev
Published: 2022-12-02 12:14:30

CSS frameworks like Bootstrap, Bulma, and Materialize are hugely popular among front-end developers. They are a great way to quickly style an application on the set of standard guidelines. However, they are a little difficult to customize and bloated for small applications.

Tailwind CSS is a utility-first CSS framework. This means that instead of providing you with ready-made components, Tailwind provides utility-based CSS classes that you can use to style your components. This gives you more flexibility over your design and you can build your own UI framework on top of Tailwind CSS by extending the Tailwind classes.

In this tutorial, you’ll build a portfolio landing page that will have the author’s information and a form to subscribe to a newsletter. You’ll build the front end with Nuxt.js and style it using Tailwind CSS.

Read the full blog on [Mattermost](https://mattermost.com/blog/how-to-set-up-and-customize-tailwind-in-nuxt-js/).

Thanks for reading 💜

---

I publish a [monthly newsletter](https://www.ravsam.in/newsletter/) in which I share personal stories, things that I am working on, what is happening in the world of tech, and some interesting dev-related posts which I come across while surfing the web.

Connect with me through [Twitter](https://twitter.com/ravgeetdhillon) • [LinkedIn](https://linkedin.com/in/ravgeetdhillon) • [GitHub](https://github.com/ravgeetdhillon) or send me an [Email](mailto:ravgeetdhillon@gmail.com).

— [Ravgeet](https://www.ravgeet.in/), *Full Stack Developer and Technical Content Writer*
AWS S3 - Deixando seu bucket público + EXEMPLO: site estático
Tags: aws, s3, staticwebsite
Published: 2022-12-02 11:59:51


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/11jfav7id03lvkdtpu8t.png)

**Simple Storage Service**, o **S3** é um serviço de armazenamento, cujo podemos obter maiores informações na documentação oficial do mesmo.O acesso a ele é controlado pelo usuário garantindo sua segurança e para testes,estudo o deixar público acaba sendo utilizado.
[](https://docs.aws.amazon.com/s3/index.html)

No exemplo abaixo irei explicar um passo a passo(supondo que tenha lido a documentação) de uma utilização em que ele nos possibilita hosperdar um site estática(sem sistemas de gerenciamento de conteúdo,mudanças) na **AWS**.


1. Através do seu painel na console da **AWS** acesse o serviço e crie o seu bucket,definindo o nome e região desejados.
2. Após a criação e clicando no bucket,acessando a aba de 'Permissions'(Permissões), você edita e desmarca a opção 'Block all public access'(bloquear o acesso público).

3. Ainda não está totalmente habilitado o acesso público e necessitamos criar a regra do bucket com as políticas, conforme a necessárias e exemplificadas pela **Amazon** no site oficial[](https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html).Recomendo selecionar para esse exemplo o código que contenha a política de GetObject(Pegar objeto) e no campo resource, em 'example bucket' você adiciona o nome do criado anteriormente.

4. Depois é só testar o endpoint do bucket,apontando para o bobjeto desejado dentro do mesmo.
How to Install Symfony 6 (PHP Framework)
Tags: php, symfony, composer, phpframework
Published: 2022-12-02 11:18:00

---
title: How to Install Symfony 6 (PHP Framework)
published: true
description: A quick guide to Install Symfony 6 with Composer and not
tags: php, symfony, composer, phpFramework
cover_image: https://anastasionico.uk/img/blog/how-to-install-symfony-6.jpeg
# Use a ratio of 100:42 for best results.
published_at: 2022-12-02 11:18 +0000
---

Johnny just finished reading the newspaper.

 

He read that a man had made a few millions dollar during the last year.

 

He found very cheap houses in the county and resold them at a higher price quickly.

 

Johnny is a beginner developer and has some money saved up.

 

So decided to create a program that checks online real estate portals to find the ones with the best price.

 

He only knows vanilla PHP at the moment and so starts doing everything from scratch.

 

A month passed by and he only finished the authentication and authorization functionality.

 

He realizes that the task he set for himself will take too long to be achieved with success and gave up.

 

He would have been much better off learning a PHP framework and letting it do all the hard lifting.

[Let's learn about Symfony](https://anastasionico.uk/blog/how-to-install-symfony-6)
Do not use append/extend in Python Comprehension expression
Tags: python
Published: 2022-12-02 11:17:36

One mistake which for me almost one hour to debug is that I used list method append in the python comprehension expression.

Take a look at belows to assignment, check which one will have output of `[[1, 4], [2, 4]]`

```python
# first 
[subset.append(4) for subset in [[1], [2]] 
# second
[subset + [4] for subset in [[1], [2]]
```
Actually the second one is correct, and with the first one, you will get `[None]`. Why? Because list method like `append` or `extend` does not have return value, they just modify the original list. For that reason you got `None` when you tried to do some operation on a list.

Be careful of those methods which don't have return value when you are trying to use them in comprehension expression in Python.
INTRODUCTION PART 2
Published: 2022-12-02 11:12:09

Today we are going to learn about some basics of python like indentation and branching.

## Indentation
So, first of all indentation refers to the spaces at the beginning of a code line. It mainly indicates a block of code. A code block (body of a condition, loop, function etc) starts with indentation and ends with the first unintended line. In python indentation is very important as it will give you an error if you skip the indentation.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w9t8fczpqe7pdjabf3cz.png)

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/osq4wcnyb8iaxx8on3j7.png)

## Branching

Python programming language provides following types of decision making or branching statements in Python. Python programming language provided decision making statements or branching statements as follows.

- if statements

- if...else statements

- if....elif....else statements

- nested if statements

**if statements**
It is used to decide whether a certain statement or block of statements will be executed or not. If a certain condition is true, then a block of statement is executed otherwise not.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0ibcn6t34b4o8e7ub0sk.png)
**if...else statements**
The else statement is always used with if statement. It is used to execute block of codes whenever if condition gives false result.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0x0o3huev824caahona4.png)
**if...elif...else statements**
For adding more conditions, elif statement in used. The program first checks if condition. When found false, it checks elif conditions. If all elif conditions are found false, then else code block is executed.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qy11jry58712qwqqsslb.png)

**nested if statements**
A nested if statement is an if statement placed inside another if statement. In a nested if construct, you can have an if...elif...else construct inside another if...elif...else construct.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zzfskn9nf3dp8w6ndyd9.png)
## print() function example
There are some functions which are probably used in print().

1. sep="seperator"
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b4d7vsxhs2if94zk7swl.png)
2. end="end"
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1v1fwobyanjqqf0cuvzc.png)
Analyzing the Role of NFTs in the Gaming Industry
Tags: javascript, gamedev, blockchain, programming
Published: 2022-12-02 11:06:02

## Introduction

Non-fungible tokens (NFTs) have become increasingly popular in the gaming industry, offering players a new way to acquire, store, and trade in-game items. 

NFTs are digital assets that are unique and can be used to represent ownership of virtual items, such as weapons, land, and even in-game currency. 

NFTs offer gamers a level of ownership and control over their virtual assets that was not possible before. This has caused a surge of interest in the gaming industry, as NFTs provide a way for gamers to monetize their in-game items and create a marketplace for virtual goods.

## What are NFTs?

NFTs are digital tokens that are created on a blockchain, such as Ethereum. These tokens are unique and cannot be replicated, meaning that each token represents a single asset. 

NFTs come in many forms, such as in-game items, characters, and even virtual real estate. The assets that are represented by these tokens are stored on the blockchain and are owned by the token holder. 

This allows gamers to have full control of their assets, as they can keep track of them and ensure that they cannot be stolen or counterfeited.

## Benefits of NFTs

The use of NFTs in the gaming industry has many benefits for both gamers and developers. For gamers, NFTs offer a new way to acquire, store, and trade in-game items. 

This allows gamers to have full control and ownership of their assets and provides them with a way to monetize their in-game items. 

NFTs also enables developers to create a marketplace for virtual goods, as NFTs can be used to transfer ownership of virtual assets from one person to another. This creates a new economy for virtual goods, which can be used to generate revenue for developers.

## Challenges of NFTs

Despite the benefits of NFTs, there are some challenges that come with using them in the gaming industry. 

One of the main challenges is scalability, as the blockchain networks on which NFTs are created are not designed to support large amounts of transactions. 

This can lead to slow transaction times and increased fees, making it difficult to use NFTs in large-scale applications. 

Additionally, NFTs are still relatively new, meaning that there is a lack of regulation and security measures in place. This can make it difficult to ensure that NFTs are secure and that they are not being misused.

## Conclusion

NFTs have the potential to revolutionize the gaming industry, as they offer a new way for gamers to acquire, store, and trade in-game items. 

NFTs provide gamers with full control over their virtual assets and create a new economy for virtual goods. However, there are still some challenges associated with using NFTs in the gaming industry, such as scalability and lack of regulation.

Despite these challenges, NFTs have the potential to create a new way for gamers and developers to interact and generate revenue from virtual goods.

---

Are you interested in the NFT world?

This Community is for you: [https://discord.gg/P8bjJ7CzBV](https://discord.gg/P8bjJ7CzBV)
What is ICM?
Tags: programming, culture, productivity, management
Published: 2022-12-02 11:05:41

ICM stands for **I**ntensive **C**oding **M**ode.

## Description:
It's the desired mental mood of a programmer. It's also called flow but I think this term is more suitable from the perspective of a programmer.

## Code of Ethics:
If one say or set her/his status to "I'm in the ICM.", please 
- don't look at,
- don't message,
- don't get close to
- don't in any way bother

him/her...

That's the ICM, folks :).
YugabyteDB Yedis
Tags: yugabytedb, redis, deprecated
Published: 2022-12-02 11:02:57

YugabyteDB YEDIS is a key-value database that is compatible with the redis commands library. 

**Important: YEDIS is deprecated!**

There are other limitations for YEDIS see the [description in the YugabyteDB documentation](https://docs.yugabyte.com/preview/yedis/).

The most important part of this blogpost is how to remove the YEDIS, see: 'YEDIS can be removed'.

# YEDIS requires setup 
With current versions of YugabyteDB, YEDIS cannot be used without preparation of the YugabyteDB cluster. This does not mean you cannot connect: that works, but the typical `set` set command to store a key with a value will throw:
```
(error) ERR set: Could not open YBTable
```

If you must use YEDIS, the command to create the YBTable is:
```
yb-admin -init_master_addrs localhost:7100 setup_redis_table
```
This is an operating system level cluster administration utility command.

# How to recognise a YugabyteDB cluster with YEDIS setup
Once the `setup_redis_table` command is executed, a few additional "entities" are created. These can be seen with `yb_stats --print_entities`:
```
% yb_stats --print-entities
...
Keyspace: yedis.system_redis id: fda8f8252cfe4ce4b69f604882dc8c20
Object:   yedis.system_redis.redis, state: RUNNING, id: b495da531ed24edbbd6c777dac4a2b41
Tablet:   yedis.system_redis.redis.24b8acf6c2b341aaa186917b47ac2467 state: RUNNING
            ( VOTER,yb-3.local:9100,FOLLOWER VOTER,yb-2.local:9100,LEADER VOTER,yb-1.local:9100,FOLLOWER )
Tablet:   yedis.system_redis.redis.2bcbda4e8bea4ea0ba810ab003d7264e state: RUNNING
            ( VOTER,yb-3.local:9100,LEADER VOTER,yb-2.local:9100,FOLLOWER VOTER,yb-1.local:9100,FOLLOWER )
Tablet:   yedis.system_redis.redis.2da0cfc6220f47a8896c2f7593500819 state: RUNNING
            ( VOTER,yb-3.local:9100,FOLLOWER VOTER,yb-2.local:9100,FOLLOWER VOTER,yb-1.local:9100,LEADER )
```
The YEDIS setup creates:
- A keyspace named 'system_redis' of the type 'yedis'.
- An object (table) named 'redis'.
- The tablets that represent the 'shards' for the 'redis' table.
- For each tablet the replicas. With RF3, you get 3 replicas.

# YEDIS can be removed
If YEDIS was setup, and is not in use anymore, then the redis table that holds the key-value pairs can be removed with the following operating system cluster command:
```
yb-admin -init_master_addrs localhost:7100 drop_redis_table
``` 
Warning:
- This will drop the redis table, even if it (still) contains key-value pairs. 
- This will leave the `yedis.system_redis` keyspace.