Hello Telescope
Tags: opensource
Published: 2020-11-07 04:13:23


![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/14jzosuo9axnvvl6fzpm.gif)

## What is Telescope

[Telescope](https://github.com/Seneca-CDOT/telescope) is an example of [planet](https://en.wikipedia.org/wiki/Planet_(software)): a tool for
aggregating blog feeds. It reminds me of google reader which I used for a long time to read the news.

## Setup Environment

I am working on a fedora33 with KDE Plasma Desktop. Docker still not support CgroupsV2. It takes a little time to install docker and I have written a blog post for the installation [https://dev.to/yzwdroid/fedora-33-docker-445k](https://dev.to/yzwdroid/fedora-33-docker-445k).

For some reason, redis has a permission error when `docker-compose`

```bash
Starting telescope_redis_1        ... done 
Starting telescope_elasticsearch_1 ... done 
Attaching to telescope_redis_1, telescope_elasticsearch_1 
redis_1         | find: '.': Permission denied 
redis_1         | chown: changing ownership of '.': Permission denied 
telescope_redis_1 exited with code 1 
```

So I installed Redis by `sudo dnf install redis`, and start it with
`redis-server`

All other part went well, I just follow the [environment-setup.md](https://github.com/Seneca-CDOT/telescope/blob/master/docs/environment-setup.md)

## Modify goURL

To work with the REST API of telescope, I added a new flag `-u --url`, fetch
the json data, convert the data to a slice of post, get each post's contents by
the url. I just added one new function to make this work. Below is the diff I
made.

```bash
git diff HEAD main.go utils.go > lab6.diff
```
The diff 

```bash
diff --git c/main.go w/main.go
index 4679a1a..fd3d249 100644
--- c/main.go
+++ w/main.go
@@ -25,6 +25,7 @@ var js = flag.BoolP("json", "j", false, "output json format to stdout")
 var fp = flag.StringP("file", "f", "", "file name to check")
 var ignore = flag.BoolP("ignore", "i", false, "ignore url patterns")
 var failOnly = flag.Bool("fails", false, "show only urls that failed")
+var urlFlag = flag.BoolP("url", "u", false, "read telescope restful API")
 
 func main() {
 	flag.Parse()
@@ -43,14 +44,20 @@ go run main.go -v or --version check version.
 		os.Exit(-1)
 	}
 
-	dat, err := ioutil.ReadFile(*fp)
-	check(err)
-
+	var urls []string
+	var dat []byte
+	if *urlFlag {
+		dat = dataTelscope()
+	} else {
+		var err error
+		dat, err = ioutil.ReadFile(*fp)
+		check(err)
+	}
 	// use xurls tool to exact links from file. Strict mod only match http://
 	// and https:// schema
 	rxStrict := xurls.Strict()
 	// urls is a slice of strings
-	urls := rxStrict.FindAllString(string(dat), -1)
+	urls = rxStrict.FindAllString(string(dat), -1)
 	urls = removeDuplicate(urls)
 
 	if *ignore {
diff --git c/utils.go w/utils.go
index 5c27fc7..aee4859 100644
--- c/utils.go
+++ w/utils.go
@@ -2,7 +2,9 @@ package main
 
 import (
 	"bufio"
+	"encoding/json"
 	"fmt"
+	"io/ioutil"
 	"log"
 	"net/http"
 	"os"
@@ -18,6 +20,43 @@ type urlStatus struct {
 	Status int
 }
 
+type post struct {
+	ID  string
+	URL string
+}
+
+func dataTelscope() []byte {
+	var data []byte
+	var posts []post
+	resp, err := http.Get("http://localhost:3000/posts")
+	check(err)
+	defer resp.Body.Close()
+
+	if resp.StatusCode == http.StatusOK {
+		body, err := ioutil.ReadAll(resp.Body)
+		check(err)
+		if err := json.Unmarshal(body, &posts); err != nil {
+			panic(err)
+		}
+		for _, p := range posts {
+			resp, err := http.Get("http://localhost:3000" + p.URL)
+			if err != nil {
+				fmt.Println(err)
+			}
+			defer resp.Body.Close()
+			if resp.StatusCode == http.StatusOK {
+				bodyData, err := ioutil.ReadAll(resp.Body)
+				if err != nil {
+					fmt.Println(err)
+				}
+				data = append(data, bodyData...)
+			}
+		}
+	}
+
+	return data
+}
+
 func removeDuplicate(urls []string) []string {
 	result := make([]string, 0, len(urls))
 	temp := map[string]struct{}{}
```

The gist version is here [https://gist.github.com/yzwdroid/26b6cb71f75f3799dfccd5fb0ed9ec79](https://gist.github.com/yzwdroid/26b6cb71f75f3799dfccd5fb0ed9ec79)