teleren/devto huggingface dataset:
Error when Run pyspark on cluster windows
Published: 2023-07-15 20:37:16

when run code on local work fine but if run it on master and worker return error Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task ١ in stage 0.0 failed ٤ times, most recent failure: Lost task 1.3 in stage 0.0 (TID 5) (192.168.1.7 executor 0): java.io.IOException: Failed to delete original file 'C:\Users\me\AppData\Local\Temp\spark-a1d455a2-235e-4246-aebe-a5a6c65e3366\executor-82f10b93-6cd7-4046-b825-68015156deb2\spark-be325be5-2631-4e3b-aa8d-18a13f2d58bd\broadcast1438069735290788560' after copy to 'C:\Users\me\AppData\Local\Temp\spark-a1d455a2-235e-4246-aebe-a5a6c65e3366\executor-82f10b93-6cd7-4046-b825-68015156deb2\blockmgr-08f31c2a-5b65-413b-a111-ca1f9fa90aea\37\broadcast_0_python'
and this is code 
from pyspark import SparkContext, SparkConf, broadcast
from typing import List
import findspark

if __name__ == "__main__":
    findspark.init("c:\spark-3.4.1")
    conf = SparkConf().setAppName("MyApp").setMaster("spark://ipaddress:7077")
    conf.set("spark.executor.memory", "4g")
    conf.set("spark.deployMode", "cluster")
    #conf.set("spark.task.partitioner", "random")
    conf.set("spark.executor.memory", "4g")
    conf.set("spark.driver.memory", "4g")
    conf.set("spark.executor.instances", "2")
    conf.set("spark.worker.cleanup.interval", "600")
    #conf.set("spark.local.dir", "E:\\aaa")
    conf.set("spark.worker.cleanup.enabled",True)
    conf.set("spark.broadcast.reuse",False)
    
    
    sc = SparkContext(conf=conf)

    # Define the array to be shared
    my_array = [1, 2, 3, 4, 5]

    # Broadcast the array to all worker nodes
    broadcast_array = sc.broadcast(my_array)

    # Define the function to be run on the array
    def my_function(x):
        # Access the broadcasted array
        arr = broadcast_array.value
        return [i * x for i in arr]

    # Create RDD and run the function multiple times on the shared array
    rdd = sc.parallelize([1, 2, 3, 4])
    results = rdd.flatMap(my_function).collect()

    # Stop the Spark context
    sc.stop()

    # Print the results
    print(results)
One feature closer to MVP
Tags: opensource, node, react, contributorswanted
Published: 2023-07-15 20:30:39

Sharkio now has an overview page for a specific service.

This is the code if anyone is interested:
https://github.com/idodav/sharkio/pull/114

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8kz91qhw67f7zfdd9xi8.png)
Postgres Enterprise Manager and its Utilities.
Published: 2023-07-15 20:13:18

Postgres Enterprise Manager (PEM) is a monitoring tool used for managing multiple instances of Postgres databases.  In this Article, I will list out various instances where it is quite useful.

- It can monitor multiple instances of Postgres in one server.

- It provides a graphical web interface where one can perform all database management activities, which makes managing instances easy.

- PEM contains an integrated SQL environment which is quite useful in writing raw SQL queries.

- PEM provides an alert management system.

- It also boasts of REST APIs which can easily be integrated in an application obtaining monitored data via this route.

- PEM supports different operating systems.

**Conclusion.**
PEM as a tool for managing databases comes with so many advantages.
**References**

https://www.enterprisedb.com/docs/pem/latest/monitoring_performance/
Java 02 - Debunking the Myth of Complete Security: Understanding the Limitations of Encapsulation in Java
Tags: java, programming, beginners, tutorial
Published: 2023-07-15 19:56:51

Encapsulation in Java is not primarily focused on security but rather on the concept of data hiding and abstraction. It is one of the four fundamental principles of object-oriented programming (OOP), along with inheritance, polymorphism, and abstraction.



## What is Encapsulation in Java ?


Encapsulation is the practice of bundling data (attributes) and methods that operate on that data within a single unit called a class. The data is kept private (usually achieved using access modifiers like private), and external entities can only access or modify it through the public methods provided by the class. This approach aims to hide the implementation details of the class, making it easier to maintain and understand.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/silv3806i7y3ry8woc76.png)
## Does encapsulation improve security?

While encapsulation can improve code organization and maintainability, it does not directly address security concerns. However, it can indirectly contribute to security in the following ways:

**Controlled Access:** By hiding the internal data and exposing only necessary methods, encapsulation helps control access to sensitive information, reducing the risk of unauthorized modification or misuse.

**Preventing Accidental Modifications:** Encapsulation prevents accidental modifications of data by ensuring that only specific methods can alter the internal state of an object. This can help prevent unintended side effects that might otherwise compromise security.

**Abstraction and Complexity Reduction:** Encapsulation encourages creating well-defined interfaces for classes, reducing the complexity of interactions with other classes. This, in turn, can help minimize potential security vulnerabilities that may arise from complex, tightly coupled code.


## Conclusion

While encapsulation has some indirect benefits for security, it's essential to note that it is not a standalone security mechanism. Proper security measures should be implemented at various levels, such as input validation, authentication, authorization, and other security best practices, to ensure a system is robust and secure.

In summary, while encapsulation is not inherently focused on security, it contributes to better code organization, maintainability, and controlled access to data, which can indirectly help in building more secure Java applications when used in conjunction with other security practices.
Instalando e configurando o Java JDK no Ubuntu
Tags: linux, ubuntu, java, tutorial
Published: 2023-07-15 19:49:27

- Abra seu terminal (Ctrl + alt + t) e verifique se há alguma versão do Java instalada.

```
java --version
```

- Se o retorno for: "Comando 'java' não encontrado", como na imagem a seguir, partimos para a etapa de instalação.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/m2pik0mznu2cq6j4dsze.png)

- Execute no terminal o seguinte comando para a instalação da versão 11:

```
sudo apt-get install openjdk-11-jdk
```

O Ubuntu irá solicitar a senha do root para continuar a instalação, insira e aguarde o fim do processo.

- Após o fim da instalação execute o comando 'Java --version' novamente para verificar se a versão do Java JDK foi instalada corretamente.

```
Java --version
```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/9nl4wvk422r05wygshti.png)

---
**Configurando o Java JDK**

- Para descobrir o caminho onde o Java JDK foi instalado:

```
sudo update-alternatives --config java
```
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7y53ryxioyi1mid696wr.png)

- Copie o caminho removendo o: /bin/java
- Execute o comando para abrir o arquivo bash e alterá-lo:

```
sudo gedit ~/.bashrc
```
- Vá para o final do arquivo e insira:
JAVA_HOME= /usr/lib/jvm/java-11-openjdk-amd64
export JAVA_HOME
export PATH=$PATH:$JAVA_HOME
Lembre-se de substituir o caminho de JAVA_HOME para o seu.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/00o23dn5aq1ksjjmdbuj.png)

- Salve e feche o arquivo, vá para o terminal e execute o seguinte comando para verificar se a alteração foi salva corretamente:

```
cat ~/.bashrc
```
Se o JAVA_HOME estiver presente então tudo ocorreu corretamente.
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hxe0nn05cq5vz37v0mvi.png)

---
✨ Pronto! ✨
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6ghlknzs70kef3if7puc.png)
move all instances of that target value to the end end of the array
Tags: javascript, programming, beginners, interview
Published: 2023-07-15 19:43:29

**This problem was asked by Intel.**

You are given an array and a target value, move all instances of that target value to the end end of the array

Example

```
function moveElement(arr, val){
 
};

// testcase
const arr = [2, 1, 2, 2, 2, 3, 4, 2];
console.log(moveElement(arr, 2)); // [4, 1, 3, 2, 2, 2, 2, 2]
```
**Explanation : All the occurences of 2 are moved at the end of the array.**


**Solution**

```
function moveElement(arr, val) {
  let i = 0,
    j = arr.length - 1;

  while (i < j) {
    while (i < j && arr[j] === val) {
      j--;
    }
    if (arr[i] === val) {
      [arr[i], arr[j]] = [arr[j], arr[i]];
    }
    i++;
  }
  return arr
}
```

**Explanation**

- Initialize i and j pointers at both ends of the array.
- Loop while i is less than j:
- Shift j leftwards until we find a non-target value.
- If arr[i] is the target, swap it with arr[j].
- Increment i towards right.
- This continues until i crosses j, which means all target values are at the end.
- So this approach efficiently moves the target values to the end in a single pass. The swap operation keeps the original order intact.
Hi I am a ... class
Published: 2023-07-15 19:41:03

"Hi, I am umm ... considered as .. umm one of the most important class type zero in any Unity project, I control game flow, and manager and connect a lot of classes together !!! Have you ever used me in your project?"
...
#Tonicat #tonicatfeldiae #taniafeldiae #unity #unity3d #dev #csharp #anime #meme
Ad voluptate pariatur sit velit commodo ex
Published: 2023-07-15 19:34:04

Non aliquip labore id ad excepteur incididunt minim aliqua voluptate non sint culpa dolore tempor. Nulla anim consequat anim ut voluptate cillum ut officia. Excepteur deserunt ea sunt esse laborum dolore sunt.

Ad voluptate pariatur sit velit commodo ex. Aliqua dolore magna minim duis velit. Dolor fugiat labore esse laborum magna eiusmod commodo voluptate ipsum anim.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/uvjr61lv50lwy6sbjf3t.jpg)

Non dolore veniam aliquip laborum ullamco cillum minim tempor quis velit adipisicing officia elit sit. Excepteur Lorem elit ex commodo. Nostrud nulla est commodo culpa Lorem consectetur proident.

Nostrud cillum culpa dolore consequat. Lorem do eiusmod deserunt magna adipisicing labore ad ad id consectetur adipisicing qui. Et non culpa proident adipisicing tempor id. Eu sint deserunt ipsum labore aliqua tempor quis. Incididunt velit ipsum culpa ad reprehenderit consequat fugiat excepteur. Et ut fugiat nostrud cupidatat nulla ullamco tempor pariatur aute. Esse sunt culpa amet qui sit.

Nisi dolore excepteur dolor sint exercitation do proident dolor sit duis mollit do pariatur. Magna cupidatat non irure anim ad reprehenderit cillum sunt elit nisi aliquip et veniam fugiat. Ea duis minim laboris proident cillum minim fugiat voluptate eu culpa eiusmod mollit do commodo. Aute irure adipisicing id elit cupidatat dolore consectetur excepteur sit labore. Reprehenderit irure voluptate culpa reprehenderit non duis pariatur ad cupidatat in id fugiat pariatur esse. Tempor aliquip minim dolor ut voluptate veniam nostrud fugiat nostrud reprehenderit do voluptate.

Sunt in dolore ut id ex deserunt aute tempor anim eu. Exercitation exercitation non voluptate sit pariatur laboris eu exercitation Lorem nostrud voluptate enim qui. Sint veniam aute voluptate incididunt. Eiusmod incididunt magna aliquip adipisicing.

Mollit labore eu deserunt sit aliquip et. Lorem in mollit dolor laborum pariatur Lorem elit. Consequat nulla culpa do proident do aliquip proident enim voluptate incididunt. Nostrud ipsum est non ad est velit laboris anim incididunt non. Minim ullamco dolore tempor occaecat commodo amet ad.

Aute culpa deserunt consequat ullamco sit aliqua anim incididunt aliqua duis enim reprehenderit. Anim reprehenderit amet fugiat nulla deserunt adipisicing. Voluptate eu ex duis ea sunt aute minim labore esse ipsum. Pariatur quis mollit proident ea aliqua. In dolor duis nostrud est tempor aliquip labore nostrud irure Lorem anim sit. Occaecat enim duis officia cupidatat laborum aliquip pariatur.
Mollit in excepteur veniam non eiusmod sint
Published: 2023-07-15 19:33:20

Dolore incididunt consequat laborum sunt consectetur duis anim consectetur sunt ad laborum. Incididunt aliquip reprehenderit laboris sunt mollit occaecat duis sint ex anim aute sit commodo. Velit dolor velit laborum do magna et exercitation.

Reprehenderit non sunt fugiat fugiat. Lorem labore ea duis occaecat adipisicing et. Incididunt enim eu nulla cupidatat tempor do velit consectetur in tempor.

Duis ut aute eiusmod velit adipisicing proident aliqua mollit nulla. Minim reprehenderit eu tempor ullamco irure proident proident officia fugiat ea irure magna duis velit. Reprehenderit enim magna voluptate esse qui ut esse Lorem reprehenderit. Ea non cillum laborum consequat sit occaecat. Officia sit non nisi sit anim Lorem esse quis officia fugiat amet cillum.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vpv2bw4ruixzh5ns0fxy.jpg)

Consectetur sit pariatur cupidatat fugiat aute consectetur tempor voluptate labore laborum ea proident. Tempor sunt deserunt mollit pariatur Lorem incididunt. Nisi officia deserunt sint labore aute esse ea eiusmod culpa. Elit ullamco culpa nisi cillum Lorem eu.

Do consequat officia ad aute excepteur anim quis reprehenderit sint irure commodo tempor nisi. Eiusmod cillum et excepteur in Lorem qui. In ipsum ullamco laboris consequat in dolore dolor laboris. Mollit exercitation laboris do veniam cillum deserunt consequat deserunt eu ad nostrud adipisicing ex veniam. Quis fugiat aute excepteur nulla elit dolore incididunt exercitation. Occaecat veniam irure laboris deserunt reprehenderit incididunt laborum velit.

Laboris voluptate deserunt commodo voluptate fugiat laborum non id nostrud laboris excepteur. Non eiusmod esse voluptate deserunt cillum ad pariatur proident nisi quis in. Aute velit do voluptate pariatur aliquip ut anim ipsum sunt do est enim. Nulla aliquip aliqua cupidatat sunt nostrud. Ullamco laboris irure in consequat duis tempor commodo proident irure labore nisi. Voluptate ea in ut sint. Voluptate ex velit aliquip id officia esse voluptate sit aute.

Dolor aliqua amet qui sit velit. Proident pariatur culpa do consectetur est laborum ea mollit ad enim do tempor. Laborum labore qui sint minim veniam ipsum mollit elit tempor magna. Aliquip adipisicing magna qui amet quis amet laborum eiusmod non reprehenderit. Officia est ea cupidatat laboris esse nostrud eiusmod ullamco et consectetur veniam. Officia enim deserunt ut cillum ea nulla do commodo.

Mollit in excepteur veniam non eiusmod sint Lorem anim adipisicing adipisicing incididunt amet proident. Aute id labore enim pariatur esse velit duis veniam duis aliquip proident commodo. Veniam tempor incididunt cillum adipisicing ullamco. Et ut ex esse sunt ea non fugiat incididunt exercitation adipisicing laborum velit dolore qui. Occaecat consequat culpa ea minim cillum reprehenderit irure deserunt.
Confiabilidade: um dos recursos mais importantes de um sistema
Tags: sre, devops
Published: 2023-07-15 19:28:02

## Definição de confiabilidade

> qualidade do que é confiável;
> capacidade de uma unidade funcional desempenhar, sem falhas ou avarias, dada tarefa sob certas condições e dentro de um período determinado.
> (Definições de Oxford Languages)

## A confiabilidade é a característica mais importante de qualquer sistema

Desde que tenhamos cuidado para mostrar que a confiabilidade abrange uma grande área, as pessoas não discordam muito desta frase.

Alguns argumentos simples:

- Se um sistema não for confiável, os usuários não confiarão nele (simples assim).
- Se os usuários não confiarem em um sistema, quando tiverem escolha, eles não o usarão.
- Se um sistema não tem usuários, esse sistema não vale de nada.

## Os usuários, não seu monitoramento, decidem sua confiabilidade

Você é o que você mede, então escolha suas métricas com cuidado.

Seu monitoramento, registros e alertas são valiosos apenas na medida em que ajudam você a perceber problemas antes que seus clientes o façam.

Dizer que tudo parece bem não fará seus usuários mais felizes. Se eles estão experimentando seu sistema como instável, será disso que eles lembrarão na a hora de escolher entre você e seu concorrente, usar ou não seu sistema.

O valor de qualquer sistema está relacionado com seus usuários. Sendo assim, no fim, a medida de confiabilidade que importa é como os usuários estão experimentando a confiabilidade.
Elit nulla amet amet do consectetur anim velit consectetur
Published: 2023-07-15 19:22:53

Id elit proident tempor proident labore ea labore sit in ad et. Qui nostrud commodo ut dolor quis do velit consequat consectetur eu magna. Dolor nisi exercitation sunt sint eiusmod est mollit tempor enim ut sit. Adipisicing Lorem eu ipsum eu quis nisi.

Proident cillum irure aliqua sit. Eu consequat consequat consectetur non. Aliqua excepteur do incididunt occaecat eu sunt deserunt non duis culpa incididunt.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zfo53zmq2w8usl4ku06g.jpg)

Dolor anim enim exercitation cupidatat ipsum sit amet aliquip excepteur ullamco nostrud anim eu. Tempor excepteur ea magna culpa non aute minim ad dolore ad deserunt. Et ipsum ad proident aliqua mollit ex sint officia laborum esse cupidatat.

Elit nulla amet amet do consectetur anim velit consectetur. Deserunt qui cupidatat mollit et. Dolor incididunt proident reprehenderit culpa ex deserunt sunt nisi consequat excepteur do et mollit. Nostrud officia velit ut ex cupidatat mollit. Commodo in est ut occaecat minim.

Anim ut ea quis id. Ea labore enim deserunt velit tempor. Mollit sint fugiat cillum sit ex ex non occaecat deserunt qui irure anim sit laboris.

Exercitation non sit esse tempor enim. Elit ut laborum est consequat nostrud laboris culpa consequat. Ut ea laboris ipsum commodo deserunt. Proident nostrud ea id eu proident quis aute aute.

Anim elit dolor dolor ipsum incididunt nisi mollit. Proident nisi enim exercitation officia tempor anim incididunt nisi voluptate. In velit deserunt incididunt reprehenderit. Eu reprehenderit ea ullamco veniam. Qui minim minim voluptate ut ex in nostrud in veniam ut anim adipisicing. Eu cillum velit do veniam pariatur Lorem enim minim. Eiusmod nulla commodo consectetur quis proident nisi incididunt adipisicing duis mollit.

Incididunt irure mollit pariatur mollit amet sunt sint do irure elit anim. Veniam ut id commodo cillum ad magna ullamco. Exercitation dolore exercitation ex commodo nostrud. Nulla velit sint occaecat ut.
Officia esse in consequat labore et aute
Published: 2023-07-15 19:21:58

Minim deserunt incididunt pariatur in. Proident laboris eiusmod qui adipisicing amet ipsum aliqua excepteur. Incididunt amet consequat nisi id sunt laboris eiusmod culpa. Pariatur pariatur id ullamco incididunt qui incididunt amet culpa sint eiusmod. Eiusmod sit occaecat sit ea quis ea commodo magna laborum aliquip dolore. Dolore culpa do incididunt et tempor aliquip. Incididunt anim tempor sit magna sunt non do dolore.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/z7dk4lb2rgu41b2sa280.jpg)

Officia esse in consequat labore et aute. Sunt sit culpa consequat qui pariatur. Non magna quis magna reprehenderit do nostrud in nostrud veniam ut mollit officia dolor.

Laboris nisi eiusmod eu qui eu elit est anim fugiat velit enim laboris. Commodo commodo qui do deserunt non eiusmod fugiat velit laborum velit dolore fugiat do est. Et est sunt ea labore quis. Non est ad et consectetur. Elit excepteur consectetur magna amet proident. Incididunt minim culpa sit sint aliqua. Reprehenderit consequat Lorem duis ad eiusmod tempor aute incididunt irure.

Dolore commodo labore ea aliqua duis est duis deserunt qui magna proident ad aliquip nostrud. Lorem nulla cupidatat eu proident anim mollit esse. Sit veniam labore labore officia velit sint aliqua.

Veniam aute excepteur ea voluptate amet non. Ad sunt dolor mollit nostrud ad nisi. Reprehenderit adipisicing cupidatat cillum ea occaecat duis do. Exercitation duis voluptate consequat labore dolore mollit deserunt occaecat consectetur. Id officia et eiusmod sit anim elit. Ullamco ad in mollit nisi ut exercitation aliquip nisi ullamco proident enim elit esse et. Sint qui adipisicing qui consequat velit sint mollit culpa.

In amet non tempor eiusmod minim adipisicing nisi consequat duis incididunt id veniam occaecat. Dolor ad qui proident aute sit incididunt esse commodo ea et qui fugiat commodo. Minim ipsum proident proident pariatur occaecat anim sint laboris in. Anim excepteur consequat ex adipisicing culpa dolore tempor tempor eu aliquip cillum culpa labore nostrud.

Et anim aliqua incididunt nulla exercitation nostrud eu cillum commodo. Et aliqua sunt dolor qui exercitation fugiat aliquip cupidatat amet ex et irure labore. Cupidatat eiusmod fugiat Lorem reprehenderit ad.

Nulla labore ad irure laboris aliquip veniam sit tempor. Officia nostrud elit occaecat consequat eiusmod reprehenderit. Enim occaecat nostrud ea occaecat sint aliqua et velit. Dolore dolore cupidatat in minim veniam est consequat mollit tempor cupidatat irure ea. Proident culpa sit dolore labore sint.
PostgreSQL: An Introduction to Indexes
Tags: postgres, postgressql, database, apach
Published: 2023-07-15 19:03:20

Indexes serve as auxiliary structures within a database, serving two main purposes: enhancing data retrieval speed and enforcing integrity rules. Lets explore indexes in PostgreSQL in detail, look into the different types of indexes available, explain why there is such diversity, and see how they can be utilized to optimize query performance. 

## Index Types

As of version 9.6, PostgreSQL offers six built-in types of indexes. All index types associate a specific key with the corresponding table rows that contain that key. Each row is uniquely identified by a tuple id (TID), which comprises the block number within the file and the row's position within that block.

Any operation performed on indexed data, such as inserting, deleting, or updating table rows, necessitates updating the indexes for that particular table within the same transaction.

## Extensibility of Indexes

PostgreSQL has implemented an interface in its general indexing engine to facilitate the easy addition of new access methods to the system. The primary purpose of this interface is to obtain tuple ids (TIDs) from the access method and perform some tasks. It reads data from the appropriate versions of table rows and retrieve row versions using individual TIDs or in batches using a prebuilt bitmap. It determines the visibility of row versions for the current transaction, considering its isolation level.

The indexing engine is involved in query execution and is invoked based on a plan generated during the optimization stage. The optimizer evaluates various ways to execute the query, taking into account the capabilities of all potential access methods.

By using the indexing engine, PostgreSQL ensures uniform handling of different access methods while considering their specific characteristics. The primary scanning techniques are index scan and bitmap scan. In an index scan, TID values are sequentially returned until the last matching row is reached, and the indexing engine accesses the table rows indicated by these TIDs. On the other hand, in a bitmap scan, all TIDs matching the condition are initially returned, and a bitmap of row versions is constructed based on these TIDs before reading the corresponding row versions from the table. The choice between these scanning techniques depends on factors such as the number of retrieved rows and is determined by the optimizer.