teleren/devto huggingface dataset:
Building a web scraper with Go 💻👨‍💻
Tags: go, programming, building, speed
Published: 2023-01-23 20:56:57

Web scraping is a powerful tool for extracting information from websites. With the rise of big data and machine learning, web scraping has become increasingly important for data analysis and research. In this post, we will explore how to build a web scraper that scrapes the entire content of a webpage using the Go programming language and the colly package.

##Step 1: Setting Up the Project

The first step in building a web scraper is to set up the project. This includes creating a new project directory, initialising the Go module, and installing any necessary dependencies.

To create a new project directory, use the following command:

```
mkdir my-web-scraper

```
Next, navigate to the project directory:

```
cd my-web-scraper

```
To initialize the Go module, use the following command:

```
go mod init

```
You will need to install the colly package by running the following command:

```
go get -u github.com/gocolly/colly

```
##Step 2: Writing the Code

The next step is to write the code for the web scraper. We will start by importing the necessary libraries, and then writing the main function.

First, import the libraries:

```
import (
    "fmt"
    "github.com/gocolly/colly"
)

```
Next, write the main function:

```
func main() {
    c := colly.NewCollector()
    c.OnHTML("html", func(e *colly.HTMLElement) {
        fmt.Println("HTML: ", e.Text)
    })
    c.Visit("https://www.example.com")
}

```
In the above code, we first create a new collector instance with colly.NewCollector(). Then we define an OnHTML callback function that will be called every time an HTML element with the tag "html" is encountered. The function takes an instance of colly.HTMLElement as an argument and prints the entire text of the HTML document. Finally, we visit the website we want to scrape.

##Step 3: Testing the App

The final step is to test the app to ensure that it is functioning as expected. To do this, simply run the following command:

```
go run main.go

```
This will execute the main function, which will scrape the specified website and print the entire HTML content to the console.

In conclusion, building a web scraper that scrapes the entire content of a webpage using the Go programming language and the colly package is a simple process that can be broken down into three key steps: setting up the project, writing the code, and testing the app. By following these steps and using the colly package, you can easily build a web scraper that extracts the entire content of a webpage. The colly package provides a simple and flexible API that makes it easy to extract data from websites quickly and easily.

</>codeDaily💻
Latin x Python - The Start
Tags: programming, python, productivity, latin
Published: 2023-01-23 20:54:58

I'm a Python developer who is really interested in learning Latin.

So while I learn Latin, I've decided to start a project where it can easily generate questions like English-Latin, Latin-English or grammar questions (eg. what case a word is in).
This should be a better alternative to websites like Quizlet or Blooket, where you have to enter your own questions and set your own answers.

I started on the project a few days ago and have decided to record my progress.

---

## Features I have added
- A function to convert a english noun to latin, with the arguments case, word and number
- This function uses words from the word list, and the words are in the 1st, 2nd or 4th declension
- A function to convert an english verb to latin, with the arguments word, person, number and tense
- This function also only uses words from the word list, and the words can be any tense and also any conjugation

---

## Demos

Here is a demo of the noun function so far:
![noun demo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/o8znew4zcbs4te36gen2.png)

And here is a demo of the verb function:
![verb demo](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/n5npyircm3lzcn2d19on.png)

---

Thanks for reading.
If you want to support me with this project, make sure to follow me and like this post!
Require type equality of methods and their decorator interceptors?
Tags: help
Published: 2023-01-23 20:46:46

Is there a way to ensure the type passed into a decorator matches the return type of a method it is acting on, and if not, highlight it as an error to be resolved before allowing the app to compile?

For example, if the return type of my method is Promise<Entity> and I add a decorator to that method via @UseInterceptors(new Interceptor<NotTheSameEntity>()) I want VSCode to highlight this as an error.
10- ANOVA test using R Programming
Tags: anova, programming, rproframming, statistics
Published: 2023-01-23 20:28:39

--- series: R Programming for Scientific Research ---

Create some data
```
group <- rep(c('A1','A2', 'A3'), times=4)
value <- runif(12, 10, 20)
df1 <- data.frame(group, value)
df1

group <- rep(c('A1','A2', 'A3'), times=4)
value <- runif(12, 10, 20)
year <- as.factor(c(rep("2010",6), rep("2020",6)))
df2 <- data.frame(value,group,year) # dataframe
```
**Visualize data**
```
library(ggplot2)
ggplot(df1, aes(x = group, y = value)) +
  geom_boxplot()
```

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1626778225836/USewmMMEW.png)
#### One way ANOVA
It uses to examine statistically significant differences between two samples means.
```
df1.anova <- aov(value ~ group, data=df1)
df1.anova
summary(df1.anova)

# extract the p-value and F value
summary(df1.anova)[[1]][1,4:5]
```
#### Interpret the result:
P-value ≤ α: The differences between some of the means are statistically significant.

P-value > α: The differences between the means are not statistically significant.

p-value of df1 = 0.9358 > 0.05, so we can not reject the null hypothesis (means are equal) and conclude that means are equal.

#### Two way ANOVA
A two-way ANOVA, has two independents. The style of two way ANOVA is:

aov(dependent~as.factor(independent1)*as.factor(indepndent2),data= filename)

```
df2.anova <- aov(value ~ group*year, data=df2)
summary(df2.anova)

# extract the p-value and F value:
summary(df2.anova)[[1]][1,4:5]
summary(df2.anova)[[1]][2,4:5]
summary(df2.anova)[[1]][3,4:5]
```

#### Analysis of covariance
```
df2.anova <- aov(value ~ group+year, data=df2)
summary(df2.anova)

# extract the p-value and F value:
summary(df2.anova)[[1]][1,4:5]
```

```
TukeyHSD(df1.anova) #Tukey Honest Significant Differences
TukeyHSD(df2.anova)
```
<p>If you like the content, please <a target="_blank" href="https://www.youtube.com/channel/UCpbWlHEqBSnJb6i4UemXQpA?sub_confirmation=1">SUBSCRIBE</a> to my channel for the future content</p>

To get full video tutorial and certificate, please, enroll in the course through this link:
https://www.udemy.com/course/r-for-research/?referralCode=B6DCFDE343F0592EA61A
14- Creat basic plots in R Programming
Tags: rprogramming, programming, graphs, tutorial
Published: 2023-01-23 20:18:16

--- series: R Programming for Scientific Research ---

#### Load trees data from datasets library
```
library(datasets)
tr<-trees 
head(tr)
```
#### Histogram
```
hist(tr$Volume, col = 'darkred',main="Your title", xlab="Volume")
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h18ud95dsx7ueijvzexb.png)


#### Barplot
```
barplot(tr$Height, names.arg=c(1:31), col='darkgreen')
```


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/shhfm32jma3vwlg7vgmd.png)




#### Boxplot
```
boxplot(tr$Height, col = 'darkblue', xlab='Height',ylab='Height (m)')
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dahgmmwj1z04sjka7ffo.png)


```
boxplot(tr, col='darkorange')
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/0ucplk4ntqwznpat8ge6.png)


#### Scatter plot
```
plot(tr$Height, tr$Volume, col='#FDAE61', pch=19, xlab='Height', ylab=
      'Volume', main='Your title')
```


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xsdrbjlufupfbcw3xj5r.png)


```
plot(tr$Volume,tr$Height, col = factor(tr$Girth[1:6]), pch=19,
     xlab='Volume', ylab='Height', main='Your title')

# Legend
legend("topleft",
       legend = levels(factor(tr$Girth[1:6])),
       pch = 19,
       col = factor(levels(factor(tr$Girth[1:6]))))
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/w6z81dum46af2n7ttdk5.png)


#### Line graph
```
plot(tr$Height,type = "o",col = "darkgreen", xlab = "Day", ylab = "Value", 
     main = "Your title", ylim = c(0, 85), pch=18, lty=6)
lines(tr$Volume, type = "o", col = "darkblue", pch=19, lty=2)
```


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1ipgtcachbvn44gyyzsf.png)


#### Piechart
```
piedata <-c(79.814, 0.023, 16.21, 1.636, 2.318)
class <- c("Barren Land", "Water", "Built-up Area", "Tree", "Grass")
pct <- round(piedata/sum(piedata)*100, 2)
class <- paste(class, pct)
class <- paste(class,"%",sep="")
pie(piedata,labels = class, col=rainbow(5), main="LULC Types in 2013")
```

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8n2u5ei1lregfuwkbozr.png)



<p>If you like the content, please <a target="_blank" href="https://www.youtube.com/channel/UCpbWlHEqBSnJb6i4UemXQpA?sub_confirmation=1">SUBSCRIBE</a> to my channel for the future content</p>

To get full video tutorial and certificate, please, enroll in the course through this link:
https://www.udemy.com/course/r-for-research/?referralCode=B6DCFDE343F0592EA61A
Java Efetivo Essencial - Interface
Tags: java, programming, oop, backend
Published: 2023-01-23 20:11:41

## Prefira as interfaces em vez das classes abstratas

_Esta é uma série baseada no entendimento de tópicos relacionados ao livro com foco no resumo._


**Interfaces**

As interfaces são tipos abstratos, que utilizam a palavra reservada interface. Utilizadas para criar comportamentos que uma classe pode implementar. Antes apenas criávamos as assinaturas dos métodos nas interfaces, após o Java 8 houveram mudanças, como a implementação de métodos e atributos.
Interfaces permitem trabalharmos com polimorfismo no Java. Por regra uma classe pode implementar várias interfaces, caso o método não tenha implementação na interface é necessário fazer a implementação na classe que implementou a interface.

**Classes Abstratas**

É um tipo de classe no Java, no qual não será possível criar uma instância direta da mesma. Utilizamos a palavra reservada abstract para criarmos esse tipo de classe.
Tem o propósito apenas de criarmos uma estrutura de classe para que ela seja super classe de outras classes através herança.
Assim como regra de herança no Java, uma classe consegue apenas ter uma única super classe.

**Vantagens**

Existe uma grande vantagem em trabalharmos com interface, pois ao adicionar novas interfaces a uma classe já existente não impactamos no funcionamento e estrutura existentes.
Mas não acontece o mesmo quando tentamos adicionar uma classe abstrata nova em uma estrutura de classes existente.
Como uma classe pode implementar várias interfaces, mas não pode ter mais que uma super classe, trabalhar com interfaces acaba sendo mais fácil e maleável.

**Misture**

Antes do Java 8 não havia implementações de métodos, o que sugeria o uso de interfaces com classes abstratas para que fosse possível utilizar o mesmo dos dois mundos. Ainda assim o que me intrigou no livro é que utilizando de exemplos do Java manteve-se essa abordagem para implementação de métodos padrões, utilizando classes abstradas. 
Como por exemplo a classe AbstractMapEntry que implementa a interface Map.Entry.

**Resumo do Resumo**

Trabalhar com interfaces é garantir flexibilidade no design do seu código.
Ainda sim é a melhor forma de trabalharmos com tipagem múltipla.
E se desejar implementar modelos prontos de implementação, utilize os recursos "novos" ( nem tão novos assim) que possibilitam implementação diretamente na interface.
Open-sourcing my project that helps you remember the Kindle books you read
Tags: opensource, webdev
Published: 2023-01-23 20:08:47

I have been working during the past 8 months on a side project called [Just Remind](https://justremind.app/). it's a free tool that helps you remember the Kindle books you read by sending you a daily email with your highlights.

Now, I reached a point where I want to share the projects with the community so it grows better! 

Being new to this part of the open-source community (I always use them, but that's my first time contributing to it), I welcome any advice!

If you are interested in the tool, give it a try! If you actually enjoy it, feel free to visit the [Github repo](https://github.com/Just-Remind/app) and start participating. It's coded in React, with Next. Any and all contributions, whether it be in the form of code, bug reports, or feature requests, are very much welcomed!

I have a few years of web development experience, so I am also eager to help junior developers who are interested in contributing to the project!

Feel free to reach out if you have any questions!

Cheers 👋
13- Create a quick report from data using R programming
Tags: rprogramming, programming, datascience, research
Published: 2023-01-23 20:08:26

--- series: R Programming for Scientific Research ---

#### Install and load package (DataExplorer):
```
#install.packages("DataExplorer")
# Load library
library(DataExplorer) # load DataExplorer
library(datasets)
library(ggplot2) 
```
Download used data [LAI_factors.csv](https://github.com/Azad77/py4researchers/blob/main/data/LAI_factors.csv)  and 
[Countries_LAI_and_LST.csv](https://github.com/Azad77/py4researchers/blob/main/data/Countries_LAI_and_LST.csv) 
#### Read in dataset
```
dt1<-read.csv(file.path('D:', 'R4Researchers', 'Countries_LAI_and_LST.csv'))
dt2<-read.csv(file.path('D:', 'R4Researchers', 'LAI_factors.csv'))
```

```
introduce(airquality) # to describe basic information
introduce(dt1)

plot_bar(mtcars)
plot_boxplot(iris, by = "Species", ncol = 2L)
plot_correlation(iris)
plot_histogram(iris, ncol = 2L)
plot_prcomp(na.omit(airquality), nrow = 2L, ncol = 2L) # Visualize principal component analysis
plot_qq(iris) # plot quantile-quantile for each continuous feature
plot_scatterplot(iris, by = "Species") # create scatterplot for all features
plot_str(iris) # visualize data structure
```
#### Create a report 
```
create_report(iris)
create_report(airquality, y = "Ozone")

create_report(dt1)
plot_histogram(dt1)

create_report(dt2)
plot_histogram(dt2)
```
#### Create customized report
```
create_report(
  data = dt2,
  output_format = html_document(toc = TRUE, toc_depth = 6, theme = "flatly"),
  output_file = "report_LAI_factors.html",
  output_dir = getwd(),
  y = "Year",
  config = configure_report(
    add_plot_prcomp = TRUE,
    plot_qq_args = list("by" = "Year", sampled_rows = 1000L),
    plot_bar_args = list("with" = "LAI_India"),
    plot_correlation_args = list("cor_args" = list("use" = "pairwise.complete.obs")),
    plot_boxplot_args = list("by" = "LST_India"),
    global_ggtheme = quote(theme_light())
  )
)
```
The output will be saved in the directory and opened as html file.


![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hn3d7e6mg4wq4vllv0rb.png)



<p>If you like the content, please <a target="_blank" href="https://www.youtube.com/channel/UCpbWlHEqBSnJb6i4UemXQpA?sub_confirmation=1">SUBSCRIBE</a> to my channel for the future content</p>

To get full video tutorial and certificate, please, enroll in the course through this link:
https://www.udemy.com/course/r-for-research/?referralCode=B6DCFDE343F0592EA61A
12- How to perform multiple linear regression in R programming
Tags: rprogramming, research, programming, tutorial
Published: 2023-01-23 20:03:44

--- series: R Programming for Scientific Research ---

Multiple linear regression (mlr) example:

fit <- lm(y ~ x1 + x2 + x3, data=mydata)

Download  [data](https://github.com/Azad77/py4researchers/blob/main/data/LAI_factors.csv)  used in this tutorial.

Load data:
```
df<- read.csv("D:\\R4Researchers\\LAI_factors.csv")
View(df)
head(df)
```
Rename some long names of columns:
```
library(tidyverse)
df<- df %>% 
  rename(
    lst = LST_India,
    temp = Mean_air_tem_India,
    precip = Total_precipitation_India,
    humid = Hum_India
  )
head(df)
```
#### MLR
```
fit1 <- lm(LAI_China ~ LST_China + Tem_China + Precipitation_China + Hum_China, data = df) #building model
fit2 <- lm(LAI_India ~ lst+ temp + precip + humid, data = df)

summary(fit1) # show results
coefficients(fit1) # model coefficients
confint(fit1, level=0.95) # CIs for model parameters
fitted(fit1) # predicted values
residuals(fit1) # residuals
anova(fit1) # anova table
vcov(fit1) # covariance matrix for model parameters
influence(fit1) # regression diagnostics

summary(fit2)
coefficients(fit2)
confint(fit2, level=0.95)
fitted(fit2)
residuals(fit2) 
anova(fit2) 
vcov(fit2) 
influence(fit2) 
```
#### Diagnostic plots:  
Checks for heteroscedasticity, normality, and influential observations
```
layout(matrix(c(1,2,3,4),2,2)) # optional 4 graphs/page
plot(fit1)
```

![image.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1626822663740/a7MjGZ7lg.png)

#### Remove not significant variables from model:
```
fit3 <- lm(LAI_India ~ lst+ temp , data = df)
coefficients(fit3)
#(Intercept)         lst        temp 
# 1.2433245  -0.1124670   0.1154845
```
#### Interpret the results
In our example, it can be seen that p-value of the F-statistic (fit2) is 0.004692, which is significant. This means that, at least, one of the predictor variables is significantly related to the outcome variable.

LAI in India is significantly associated with lst and air temperature. Decreasing one degree of lst and increasing air temperature leads to an increase of 0.12 unite in LAI.

#### Model accuracy assessment
We can use R2 and Residual Standard Error (RSE) in summary. An R2 value close to 1 displays that the model interprets a large portion of the variance in the outcome variable. While the RSE provides a measure of error of prediction.


#### Extracting R-squared 
```
summary(fit3)$r.squared  
# 0.6575707
summary(fit3)$adj.r.squared
# 0.6048892
```
<p>If you like the content, please <a target="_blank" href="https://www.youtube.com/channel/UCpbWlHEqBSnJb6i4UemXQpA?sub_confirmation=1">SUBSCRIBE</a> to my channel for the future content</p>

To get full video tutorial and certificate, please, enroll in the course through this link:
https://www.udemy.com/course/r-for-research/?referralCode=B6DCFDE343F0592EA61A
11- How to do Student's t.test in R programming
Tags: tutorial, programming, beginners, rprogrammin
Published: 2023-01-23 19:56:07

--- series: R Programming for Scientific Research ---

The t-test uses to examine the difference of means of two groups of data.

Default in t-test assumes unequal variance and applies the Welsh df modification

Create data:
```
group_a<- c(86, 97, 99, 100, 101, 103, 106, 110, 112, 113)
group_b<- c(50, 100, 98, 95, 90, 105, 110, 95, 100, 112)
```
#### One sample t-test
```
t.test(group_a) 
```
#### Independent (unpaired) t-test
Used for an independent data set that is distributed identically. For example, randomly divided 200 samples of treatment cases to 100 as treatment group and 100 as a control group.
```
tt<-t.test(group_a, group_b)
tt
t.test(x=c(2, 4, 6), y = c(2, 4, 6))
t.test(x=c(-2, -4, -6), y = c(2,4,6))
t.test(x=c(2, 4, 6), y = c(4, 8, 12))
```

#### Paired t-test
Used for equal pairs of similar units, or one group of units that has been measured twice, for example, blood pressure before and after treatment.
```
df <- data.frame(Name = c("Jon", "Ali", "Aviar", "Didar", "Shilan", "Tom"),
                 first_exam = c(70, 41, 85, 58, 46, 90),
                 second_exam = c(45, 80, 55, 95, 85, 80)
)

t.test(df$first_exam, df$second_exam, paired=TRUE)
```
#### Extract p-value and the value of the t-statistic
```
names(tt)
t.test(group_a, group_b)$p.value 

tt$p.value
tt$statistic 
# or 
tt[['statistic']]
```
#### Interpret the result
In the output:

- t is the t-test statistic value,
- df is the degrees of freedom 
- p-value is the significance level of the t-test 
- confidence interval is the confidence interval of the mean at 95% 
- sample estimates is the mean value of the sample

 
If p-value >0.05, indicates that means of groups of data are not significantly different.

<p>If you like the content, please <a target="_blank" href="https://www.youtube.com/channel/UCpbWlHEqBSnJb6i4UemXQpA?sub_confirmation=1">SUBSCRIBE</a> to my channel for the future content</p>

To get full video tutorial and certificate, please, enroll in the course through this link:
https://www.udemy.com/course/r-for-research/?referralCode=B6DCFDE343F0592EA61A
Connecting to a Windows server process from WSL
Published: 2023-01-23 19:38:08

No big revelation here, just writing this as a reminder to myself next time I try `curl localhost` from WSL and it doesn't work:

1. Don't use `localhost` or `127.0.0.1`. Use your machine IP. But you shouldn't even need to figure this out. Using `your-computer-name>.local` should work \(eg `curl Joshs-PC.local`. Even better: `curl $(hostname).local`.
2. Make sure your server is listening on `0.0.0.0`, not `127.0.0.1`.

Source: [this excellent StackOverflow answer](https://stackoverflow.com/a/69407064/7370522).
Debugar app Android via WSL
Tags: braziliandevs, android, wsl
Published: 2023-01-23 19:23:12

A lógica que vou seguir é conectar o android usando o ADB pelo USB normalmente. Na sequência configurar um acesso pela rede (tcpip) e com isso permitir que se conecte pelo WSL. **Para isso funcionar, você precisa ter uma conexão de rede entre o smartphone e o computador (provavelmente via WiFi).**

Como vamos precisar conectar via USB pelo windows, e depois acessar pelo WSL, você vai precisar do ADB instalado em ambos os sistemas. O jeito mais prático de conseguir isso é instalando o Android Studio. Mas se você, assim como eu prefere não instalar um IDE PESADONA que nunca vai usar, pode instalar o SDK Platform-Tools e fazer o resto tudo via linha de comando.

[Baixar o SDK Platform-Tools](https://developer.android.com/studio/releases/platform-tools?hl=pt-br)

O conteúdo baixado será um pacote compactado (5MB), que contém todos os executáves que vamos usar. Decompacte em um local de fácil acesso e se pretende usar com frequência, adicione ao PATH do sistema. (Isso é válido tanto para o windwos, quanto para o WSL)

Agora é simples, pelo terminal do windows conecte seu dispositivo usando o ADB:

```pws
adb start-server
# certifique-se que seu device se conectou (aprove se necessário)
adb tcpip 5555
# veja o ip do wifi obtido pelo seu smartphone
adb connect <seu-ip>
```

Agora, é só ir ao WSL (com adb já instalado) e conectar por lá!

```bash
adb connect <seu-ip>
```

É isso! E você ainda pode remover o cabo USB e seguir só com o WiFi.

=)
Fear Level: 0.1
Tags: beginners
Published: 2023-01-23 19:16:38

What if I study a course and can't complete challenges at the end? Not because I'm **ill-prepared**, but because I'm just **inadequate**.   

When I'm staring at a blank page in VSCode, is it just **writer's block** or something a bit more **chronic** and **permanent**?

What if the bar is so high when I graduate that a saturated job market leaves me with zero opportunities?

These thoughts tend to **paralyze** me, but I've found a new coping mechanism: watching skateboard videos. 

These lunatics risk **hospitalization** for their craft. If they're dedicated and skilled and lucky enough to make it into the **top 0.1%** then they earn a living comparable to a shift-manager at Panda Express. 

{% embed https://www.youtube.com/watch?v=Y8dIIRCBuOI %}

There won't be any broken bones when I fall, so I'm jumping back in right about now.