Create your own ChatGPT by scraping your website
Tags: chatgpt, machinelearning, ai, webdev
Published: 2023-01-18 18:06:13

I've just released a new version of [Magic](https://github.com/polterguy/magic). However, I think an image says more than a thousand words, so let me show you a screenshot from it.

![Generate a ChatGPT bot by scraping your website](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jw4n9kqr2q9cmhkbrqph.jpg)

It's able to scrape our [documentation website](https://docs.aista.com) in some 2 to 3 minutes, at which point we've got 690 training snippets for creating our own ChatGPT fine-tuning model, serving as a _"domain expert"_ on whatever subject our website has knowledge about. For me personally, this increases the value of [OpenAI and ChatGPT](https://chat.openai.com) by at least some 10x to 100x, since it allows me to create a ChatGPT based bot that answers questions _exactly_ how I _want_ it to answer questions. Basically giving me my own _"expert AI brain"_ for whatever tasks I happen to need it to help me with. I've listed some example use case below.

* AI based documentation for your software project
* Indian law expert system based upon legal data from your government
* AI based secretary answering questions related to for instance coal mines in South Africa
* Medical expert with expertise in cardiology
* Support chatbot answering questions related to _your_ company
* Etc, etc, etc - I suspect only your imagination sets the limits here.

The thing is entirely Open Source, and I would appreciate a star on our [GitHub profile](https://github.com/polterguy/magic). If you're too lazy to download the thing, you can also [signup at Aista.com](https://aista.com), and have your own cloudlet in some 30 seconds. The latter is what I am using in the following YouTube video where I am illustrating how to use it and its features.

{% embed https://www.youtube.com/watch?v=_Im3cwrl36k %}

You trigger the thing by supplying it with the root domain to some website, at which point it will scrape your website in some few minutes, creating thousands of _"training snippets"_ for you, which you can later submit to OpenAI's API to create your own fine-tuned AI model. Below is an example of how it answered a couple of questions related to my own company after having trained it on our documentation, which is a website with some 100 pages, with fairly good structure, and medium quality data.

![How ChatGPT performs when fine-tuned](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wr1vn6gfdlacwdbjjqv0.jpg)

The above questions of course would give me 100% perfectly irrelevant answers if I asked ChatGPT directly, without training my own model first. Notice, the above HTML chat interface was 99% automatically generated by ChatGPT and its _"generic version"_. Maybe somebody out there wants to create a version of it that generates beautiful designs. The generic version of ChatGPT is quite frankly _terrible_ at HTML designs, and tends to mess up margins, CSS, etc ... :/

All in all, the process took me 2 hours, however for 90 minutes I was waiting for my training session to go through, due to an insane amount of interest in ChatGPT these days, resulting in that OpenAI's API sometimes literally is _down_ due to too much pressure. To play around with it for yourselves, the easiest is to [register at our website](https://aista.com), create a cloudlet, and click the Manage/Machine Learning menu item once you're inside your cloudlet.

Every single time I have heard somebody say something negative about ChatGPT, it's always been a variation of the same; _"It didn't solve my problem"_ - Well, my reply is because you haven't taught it about your problem. With Aista Magic Cloud, you can now teach it how to solve your problem in some 5 minutes, by simply pointing it at some website that has knowledge about your problem domain, and 1 hour later it's an expert in your domain. Regardless of what your domain happens to be - And yes, before you ask, scraping websites is 100% perfectly legal :)

* [Sign up for Aista Magic Cloud](https://aista.com) and create your cloudlet here
* [Give us a star please](https://github.com/polterguy/magic)