neon_arch commited on
Commit
a412a75
2 Parent(s): cb1edad dcfec51

Merge pull request #1 from alamin655/rolling

Browse files

Improve documentation clarity and completeness

Files changed (3) hide show
  1. CONTRIBUTING.org +15 -13
  2. README.org +18 -18
  3. goals.org +26 -28
CONTRIBUTING.org CHANGED
@@ -1,24 +1,26 @@
1
- * Things to consider before contributing.
2
 
3
  ** Knowledge Required
4
- - Rust basics.
5
- - Actix-web crate basics.
6
- - Tokio crate and async/await.
7
- - Reqwest crate basics.
8
- - Serde and serde_json crate basics.
9
- - fake_useragent crate basics.
10
- - pyo3/hlua/rlua crates basics.
11
 
12
  ** Guidelines
13
 
14
  - Please be patient.
15
 
16
- - Treat everyone with respect --- ("give respect and take respect").
17
 
18
- - Document your code properly with rust coding conventions in mind.
19
 
20
- - Provide a brief description of what changes you made in the pull request.
21
 
22
- - Provide an apropriate header to pull request.
23
 
24
- *NOTE:* The rolling branch is where all the contributions should go. In simple terms, it is the working branch for this project.
 
 
 
1
+ * Things to Consider Before Contributing
2
 
3
  ** Knowledge Required
4
+ - Rust basics
5
+ - Actix-web crate basics
6
+ - Tokio crate and async/await
7
+ - Reqwest crate basics
8
+ - Serde and serde_json crate basics
9
+ - fake_useragent crate basics
10
+ - pyo3/hlua/rlua crates basics
11
 
12
  ** Guidelines
13
 
14
  - Please be patient.
15
 
16
+ - Treat everyone with respect -- "give respect and take respect."
17
 
18
+ - Document your code properly with Rust coding conventions in mind.
19
 
20
+ - Provide a brief description of the changes you made in the pull request.
21
 
22
+ - Provide an appropriate header for the pull request.
23
 
24
+ *NOTE:* The rolling branch is where all contributions should go. In other words, it is the working branch for this project.
25
+
26
+ We appreciate any contributions and suggestions to help improve the Websurfx project. Please keep in mind the above requirements and guidelines before submitting a pull request.
README.org CHANGED
@@ -1,6 +1,6 @@
1
  * Websurfx
2
 
3
- A lightening fast, privacy respecting, secure [[https://en.wikipedia.org/wiki/Metasearch_engine][meta search engine]]. (pronounced as websurface or web-surface /wɛbˈsɜːrfəs/.)
4
 
5
  * Preview
6
 
@@ -18,66 +18,66 @@ A lightening fast, privacy respecting, secure [[https://en.wikipedia.org/wiki/Me
18
 
19
  * Installation and Testing
20
 
21
- To start installing, testing and playing around with search engine. Clone the repo and then *cargo run* as shown below:
22
 
23
  #+begin_src shell
24
- git clone https://gitlab.com/NEON-MMD/websurfx.git
25
  cd websurfx
26
  cargo build
27
  ./target/debug/websurfx --port 8080
28
  #+end_src
29
 
30
- If you want to be on the rolling/edge branch then do the following steps:
31
 
32
  #+begin_src shell
33
- git clone https://gitlab.com/NEON-MMD/websurfx.git
34
  cd websurfx
35
  git checkout rolling
36
  cargo build
37
  ./target/debug/websurfx --port 8080
38
  #+end_src
39
 
40
- and then open your browser of choice and visit [[http://127.0.0.1:8080]] and then you can start playing with it right away.
41
 
42
- *Important Note:* Do not over refresh your page or put too many requests or start hosting on production server as there are no behaviours right now in the code to prevent ip blocking (it is still work under process) and you can potential can get your ip blocked/banned and also the project is still under testing phase and far from being complete and ready to be hosted on production machines.
43
 
44
  * More Contributers Wanted
45
 
46
- I am looking for more willing contributors for this project so that it helps grow this project. For more information on what you can contribute this project check out the [[file:goals.org][goals.org]] and also check out the [[file:CONTRIBUTING.org][CONTRIBUTING.org]] for guidelines and rules for making the contribution.
47
 
48
  * FAQ (Frequently Asked Questions)
49
 
50
  ** Why Websurfx?
51
 
52
- The main goal of the project is to provide a fast, secure and privacy focused [[https://en.wikipedia.org/wiki/Metasearch_engine][meta search engine]]. Though there are many meta search engine out there but they don't provide gaurantee security of the their search engine which is essential because sometimes privacy is related to security like for example some memory vulnerabilities can leak private or sensitive information which is never good so the project being written in rust gaurantees memory safety and thus eliminating such problems and also many meta search engines lack many features like advanced image search *(which is required by may graphics designers, content creators, etc), proper nsfw blocking (many links are still visible even on strict safe search), etc which *websurfx* aims to provide.
53
 
54
  ** Why AGPLv3?
55
 
56
- The reason why I am releasing my meta search engine under *AGPLv3* is because my main motive of this meta search is to provide privacy, speed and security to the user and *GPL licensing or closed sourcing or any other licensing* makes it loose for the user/server admin to not release the source code which is shady because you never know what he is putting in the source code lets say he is adding telemetry, spyware stuff, logging and what not which gets shady and is against privacy.
57
 
58
  ** Why Rust?
59
 
60
- Rust provides memory safe code which eliminates issues relating memory based vulnerabilites which makes the project more secure and less vulnerable to memory based bugs and issues and also rust being a faster programming language than c++ provides a much faster, smooth and secure meta search engine :).
61
 
62
  * Contributing
63
 
64
- Contributions are welcome. It does not matter who you are you can still contribute to the project in your way :).
65
 
66
- ** Not a developer but still want to contribute
67
 
68
- Here is [[https://youtu.be/FccdqCucVSI][video]] by Mr. Nick on how to contribute and credit to him as well
69
 
70
  ** Developer
71
 
72
- If you are developer, have a look at the [[file:CONTRIBUTING.org][CONTRIBUTING.org]] document for more information.
73
 
74
  * License
75
 
76
- The project is available under the [[file:LICENSE][GPLv3]] license.
77
 
78
  * Credits
79
 
80
- Thanks to this wonderful people for *contributing* and *supporting* this project:
81
 
82
- - [[https://gitlab.com/XFFXFF][zhou fan]]
83
  - [[https://gitlab.com/johannesrexx][johannes rex]]
 
1
  * Websurfx
2
 
3
+ Websurfx is a lightning-fast, privacy-respecting, secure [[https://en.wikipedia.org/wiki/Metasearch_engine][meta search engine]] (pronounced as websurface or web-surface /wɛbˈsɜːrfəs/.) written in Rust. It provides a fast and secure search experience while respecting user privacy.
4
 
5
  * Preview
6
 
 
18
 
19
  * Installation and Testing
20
 
21
+ To get started with Websurfx, clone the repository and run it using the following commands:
22
 
23
  #+begin_src shell
24
+ git clone https://github.com/neon-mmd/websurfx.git
25
  cd websurfx
26
  cargo build
27
  ./target/debug/websurfx --port 8080
28
  #+end_src
29
 
30
+ If you want to use the rolling/edge branch, run the following commands instead:
31
 
32
  #+begin_src shell
33
+ git clone https://github.com/neon-mmd/websurfx.git
34
  cd websurfx
35
  git checkout rolling
36
  cargo build
37
  ./target/debug/websurfx --port 8080
38
  #+end_src
39
 
40
+ Once you have started the server, open your preferred web browser and navigate to http://127.0.0.1:8080 to start using Websurfx.
41
 
42
+ *Important Note:* Please be aware that the project is still in the testing phase and is not ready for production use. Do not refresh the page excessively or make too many requests, as this could result in IP blocking.
43
 
44
  * More Contributers Wanted
45
 
46
+ We are looking for more willing contributors to help grow this project. For more information on how you can contribute, check out the [[file:goals.org][goals.org]] file and the [[file:CONTRIBUTING.org][CONTRIBUTING.org]] file for guidelines and rules for making contributions.
47
 
48
  * FAQ (Frequently Asked Questions)
49
 
50
  ** Why Websurfx?
51
 
52
+ The main goal of the Websurfx project is to provide a fast, secure, and privacy-focused [[https://en.wikipedia.org/wiki/Metasearch_engine][meta search engine]]. While there are many meta search engines available, they do not always guarantee the security of their search engine, which is essential for ensuring privacy. For example, memory vulnerabilities can leak private or sensitive information, which is never good. Websurfx is written in Rust, which guarantees memory safety and eliminates such problems. Many meta search engines also lack key features such as advanced image search, which is required by many graphic designers, content creators, and others. Websurfx aims to provide these features and others, such as proper NSFW blocking, to improve the user experience.
53
 
54
  ** Why AGPLv3?
55
 
56
+ Websurfx is released under the *AGPLv3* license to ensure that the source code remains open and transparent. This helps to prevent the inclusion of spyware, telemetry, or other malicious code in the project. *AGPLv3* is a strong copyleft license that ensures the source code of the software remains open and available to everyone, including any modifications or improvements made to the code.
57
 
58
  ** Why Rust?
59
 
60
+ Rust was chosen as the programming language for Websurfx due to its memory safety features, which can help prevent vulnerabilities and make the codebase more secure. Rust is also faster than C++, which helps to make Websurfx fast and responsive. In addition, Rust's ownership and borrowing system allows for safe concurrency and thread safety in the codebase.
61
 
62
  * Contributing
63
 
64
+ Contributions are welcome from anyone. It doesn't matter who you are; you can still contribute to the project in your way.
65
 
66
+ ** Not a developer but still want to contribute?
67
 
68
+ Check out this [[https://youtu.be/FccdqCucVSI][video]] by Mr. Nick on how to contribute.
69
 
70
  ** Developer
71
 
72
+ If you are a developer, have a look at the [[file:CONTRIBUTING.org][CONTRIBUTING.org]] document for more information.
73
 
74
  * License
75
 
76
+ Websurfx is available under the [[file:LICENSE][AGPLv3]] license.
77
 
78
  * Credits
79
 
80
+ We would like to thank the following people for their contributions and support:
81
 
82
+ - [[https://github.com/XFFXFF][zhou fan]]
83
  - [[https://gitlab.com/johannesrexx][johannes rex]]
goals.org CHANGED
@@ -1,57 +1,55 @@
1
  * TODO Goals for v0.1.0
2
 
3
- - [ ] Add unit tests, integration test, doc tests in the project source code.
4
 
5
- - [ ] Add code to remove nsfw content from search results using a blocklist.
6
 
7
- - [ ] Add code to disallow user to search sensitive content (similar functionality to swisscows search engine) if strict safe search is turned on.
8
 
9
- - [ ] Add better error handling code to handle scraping error, reqwest error, etc.
10
 
11
- - [ ] Add ability to change colorschemes with the theme style of the page.
12
 
13
  =For example:=
14
 
15
- If there is simple theme is there then there is option of 9 different colorschemes to choose from such as catppuccin-mocha, solarized dark, nord, etc.
16
 
17
- - [ ] Add random delays and behaviours to emulate human behaviour if needed to evade ip blocking.
18
 
19
- - [ ] Add python/lua config to the search engine and also ensuring and giving more control to the user (server maintainer/administrator).
20
 
21
- - [ ] Add settings page to configure search engine on the fly by the user and save his preferences using cookies.
22
 
23
- - [ ] Add search engine logo to index page and to the navbar (on the right hand side).
24
 
25
- - [X] Write rust.yml for automating testing of the project.
26
 
27
- - [X] Add better documentation to source code.
28
 
29
- - [X] Add FAQ section in readme.org for answering commonly asked questions like why websurfx?, why rust?, why GPLv3?, etc.
30
 
31
- - [X] Add logging to websurfx project.
32
 
33
- - [X] Add code to generate random user agent to protect user's privacy.
34
 
35
- - [X] Add duckduckgo engine as an upstream.
36
 
37
- - [X] Add atleast one searx engine instance as an upstream engine.
38
 
39
- - [X] Add pagination support.
40
-
41
- - [X] Add basic handlebars pages and theme with catppuccin colorscheme.
42
 
43
- * Goals for the future
44
 
45
- - Move from handlebars to faster templating engine /Tera/.
46
 
47
- - Adding more upstream search engines.
48
 
49
- - Add dorking support (like google).
50
 
51
- - Add advanced search functionality and dropdown menu for it.
52
 
53
- - Add more categories to search engine's search page like images, files, all, news, maps, etc.
54
 
55
- - Add advanced image functionality to images category (which will be a great aid for content creators, video editors, etc.).
56
 
57
- - Add GPT integration (taking insipiration from langchain module of python or incorporating it using pyo3) and give user the choice to add api key (giving user the choice is important as there are many people who are against ai and so this will ensure that those people also get satisfied like if they don't gpt so they will not their api key and so it will be disabled).
 
1
  * TODO Goals for v0.1.0
2
 
3
+ - [ ] Add unit tests, integration test, and doc tests to the project source code.
4
 
5
+ - [ ] Add code to remove NSFW content from search results using a blocklist.
6
 
7
+ - [ ] Add code to disallow user from searching for sensitive content (similar functionality to swisscows search engine) if strict safe search is turned on.
8
 
9
+ - [ ] Improve error handling to handle scraping errors, reqwest errors, and other issues.
10
 
11
+ - [ ] Add the ability for users to change the colorscheme of the page.
12
 
13
  =For example:=
14
 
15
+ If a simple theme is used, then there should be the option of 9 different colorschemes to choose from, such as Catppuccino-Mocha, Solarized Dark, Nord, etc.
16
 
17
+ - [ ] Implement random delays and behaviors to emulate human behavior to evade IP blocking.
18
 
19
+ - [ ] Add Python/Lua configuration options to give more control to the user (server maintainer/administrator).
20
 
21
+ - [ ] Create a settings page to allow users to configure the search engine and save their preferences using cookies.
22
 
23
+ - [ ] Add a search engine logo to the index page and the navbar.
24
 
25
+ - [X] Write a rust.yml file for automated testing of the project.
26
 
27
+ - [X] Improve documentation for the project source code.
28
 
29
+ - [X] Add a FAQ section to the README.org file to answer common questions about the project.
30
 
31
+ - [X] Add logging to the websurfx project.
32
 
33
+ - [X] Generate random user agents to protect user privacy.
34
 
35
+ - [X] Add DuckDuckGo and at least one Searx engine instance as upstream providers.
36
 
37
+ - [X] Add pagination support for search results.
38
 
39
+ - [X] Create basic Handlebars pages and theme with Catppuccino colorscheme.
 
 
40
 
41
+ * Goals for future development
42
 
43
+ - Replace Handlebars with the faster templating engine /Tera/.
44
 
45
+ - Add more upstream search engines.
46
 
47
+ - Implement dorking support (like Google).
48
 
49
+ - Add advanced search functionality and a dropdown menu for it.
50
 
51
+ - Add more categories to the search engine's search page, such as images, files, news, and maps.
52
 
53
+ - Add advanced image functionality for content creators, video editors, etc.
54
 
55
+ - Add GPT integration, taking inspiration from the Langchain module of Python or incorporating it using pyo3, and give users the choice to add an API key. Giving users the choice is important, as some people may be against AI, and this will ensure that those users who don't want to use GPT can disable it by not adding an API key.