Chainsight Hands-on: Collecting Data from Web
Tags: blockchain, smartcontract, tutorial, oracle
Published: 2024-02-08 07:31:47

So far, in [medium.com](https://medium.com/@Chainsight_Network) we have delved deeply into the behind-the-scenes of components and SDKs, as well as showcases that combine various components. This time, we will actually go hands-on to create our own component. In this article, we will try to create a component that actually collects data from the Web using Snapshot Indexer HTTPS.

**Completed Component Definition**

Completed Component DefinitionThis time, let's create a Snapshot Indexer HTTPS that collects bitcoin prices. There are several APIs that provide bitcoin prices, but we will use coingecho's API to collect the current price. Let's construct Snapshot Indexer with a collection interval of one hour.

![chainsight_deepdive_into_showcase-handson-snapshot_indexer_https.png](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ri4u3jsnq7xq3imdxslq.png)

**Prerequisite**

The Chainsight CLI requires several tools and this CLI installation. Please refer to the following article for this preliminary preparation.

[Step-by-step creating an EVM Price Oracle with Chainsight - DEV Community](https://dev.to/hide_yoshi/step-by-step-creating-an-evm-price-oracle-with-chainsight-469g)

**Create a project**

The first step is to build a project template. In this case, we only need one manifest, so let's proceed by adding a manifest of the type we want to create from an empty project. The command to create a project is `csx new`.  Give the `-no-sample` option to make the project empty.

![chainsight-handson-web-1](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/v8ftpht5hy0wbawenwkq.png)

Looking at the created project, the manifest folder, components, is empty.

![chainsight-handson-web-2](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p8m5u7pawy7uh8pm74yb.png)


Now let's run csx add to add the manifest as instructed in the log.

**Add a manifest**

You can use `csx add` to interactively add a manifest of the type you wish to add. Let's actually run `csx add` with the name `btc_snapshot_indexer` and the type `snapshot_indexer_https`.

![chainsight-handson-web-3](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/klcnzp20et44kzbb3rxq.png)

This will generate a manifest (.yaml) for `btc_snapshot_indexer` and update `project.yaml` to include `btc_snapshot_indexer` in the project itself.

![chainsight-handson-web-4](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3dm2pxho4pfrmsyk6fl8.png)

Then update this generated manifest so that the bitcoin price can be retrieved. Get the bitcoin price by calling the /simple/price endpoint of the coingecko API.

[Crypto API Documentation | CoinGecko](https://www.coingecko.com/api/documentation)

The parameters for actually taking this data are as follows. The id to retrieve bitcoin data is `bitcoin`. Let's also specify some parameters to extend the data available at this endpoint.

```txt
https://api.coingecko.com/api/v3/simple/price
  ?ids=bitcoin
  &vs_currencies=usd,eur,btc,eth
  &include_market_cap=true
  &include_24hr_vol=true
  &include_24hr_change=true
  &include_last_updated_at=true
  &precision=18
```

Modify `datasource` field to specify this parameter in the `snapshot_indexer_https` manifest. Specify the endpoint in the `url` field. Since the query parameters are static and immutable this time, set `queries` type to `static` and `value` field to the respective key and value.

```yaml
datasource:
  url: https://api.coingecko.com/api/v3/simple/price
  headers:
    Content-Type: application/json
  queries:
    type: static
    value:
      ids: bitcoin
      vs_currencies: usd,eur,btc,eth
      include_market_cap: true
      include_24hr_vol: true
      include_24hr_change: true
      include_last_updated_at: true
      precision: 18
```

The difference between the template generated by `csx add` and the updated manifest is as follows

```bash
9,10c9,11
<   - DAI
<   - USD
---
>   - BTC
>   - bitcoin
>   - "Hands-on Sample"
18,19c19,25
<       ids: dai
<       vs_currencies: usd
---
>       ids: bitcoin
>       vs_currencies: usd,eur,btc,eth
>       include_market_cap: true
>       include_24hr_vol: true
>       include_24hr_change: true
>       include_last_updated_at: true
>       precision: 18
```

Use this manifest to move on to the next step.

**Generate a code**

To generate the canister code for a component from its manifest, run `csx generate` with the `project.yaml` declaring the path to its manifest.

![chainsight-handson-web-5](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fhfloy6qeakb8z9m05cb.png)

`csx generate` generates a project to generate the canister module under the src folder. If no customization is required, module generation can be performed immediately by running `csx build` in this state.

Since this Snapshot Indexer is intended to collect only bitcoin prices, not all data that can be retrieved from the API is required. Also, the more data content, the more expensive it is to retain, and the higher the probability of consensus generation failure when retrieving data via HTTPS outcalls. For more information, please refer to the following article.

[Behind Chainsight: How Modules are Created Pt. 1 | by Chainsight | Jan, 2024 | Medium](https://medium.com/@Chainsight_Network/behind-chainsight-a-look-into-snapshot-indexer-component-d8a6074237e2)

Therefore, in order to avoid storing unnecessary data, modifications are made to narrow down the data to be collected.
To make customizations to the generated code, modify `logics/(canister-name)/src/lib.rs` in the `src` directory. For any type of canister, you can add your own customizations by modifying the file replacing `canister-name` with the name of the component of interest. If you want to know more about the project generated by `csx generate`, please refer to this article.

[Behind Chainsight: How Modules are Created Pt. 1 | by Chainsight | Jan, 2024 | Medium](https://medium.com/@Chainsight_Network/behind-chainsight-how-modules-are-created-pt-1-32b13aa7f696)

Now let's get back to the main issue. Let's open `logics/btc_snapshot_indexer/src/lib.rs` in your project. The code should look something like this

```rust
// Auto-generated code from manifest.
// You update the structure as needed.
// The existence of the SnapshotValue structure must be maintained.
use candid::{Decode, Encode};
#[derive(Debug, Clone, candid::CandidType, candid::Deserialize, serde::Serialize, chainsight_cdk_macros::StableMemoryStorable)]
pub struct SnapshotValue {
    pub bitcoin: Bitcoin,
}

#[derive(Debug, Clone, candid::CandidType, candid::Deserialize, serde::Serialize, chainsight_cdk_macros::StableMemoryStorable)]
pub struct Bitcoin {
    pub usd: f64,
    pub usd_market_cap: f64,
    pub usd_24h_vol: f64,
    pub usd_24h_change: f64,
    pub eur: f64,
    pub eur_market_cap: f64,
    pub eur_24h_vol: f64,
    pub eur_24h_change: f64,
    pub btc: f64,
    pub btc_market_cap: f64,
    pub btc_24h_vol: f64,
    pub btc_24h_change: f64,
    pub eth: f64,
    pub eth_market_cap: f64,
    pub eth_24h_vol: f64,
    pub eth_24h_change: f64,
    pub last_updated_at: i64,
}
```

This is an implementation for mapping the data obtained by calling the endpoints you specify. The trait you are using has the necessary functionality for the structure, so understand that that is the way it is and don't worry about it. The point of concern is the fields of the structure. Fields for a number of elements have already been declared. Where this code, which is implemented from the beginning, comes from is that the CLI automatically generates an implementation that matches the response from the endpoint you have set up. The actual response data will give you a sense of the implementation for mapping.

```json
{
  "bitcoin": {
    "usd": 42451.59160065723,
    "usd_market_cap": 832331572531.3098,
    "usd_24h_vol": 12740914009.764158,
    "usd_24h_change": 0.7450874696750942,
    "btc": 1.0,
    "btc_market_cap": 19611887.0,
    "btc_24h_vol": 300160.8233960758,
    "btc_24h_change": 0.0,
    "eth": 18.650627839906598,
    "eth_market_cap": 365799851.323311,
    "eth_24h_vol": 5597576.829903464,
    "eth_24h_change": 0.9213166048247731,
    "last_updated_at": 1706500074
  }
}
```

In this case, we only need to collect bitcoin price information. In other words, only `bitcoin.usd` data needs to be collected. Let's remove all but the `usd` field of the Bitcoin structure.

```diff
pub struct Bitcoin {
    pub usd: f64,
-    pub usd_market_cap: f64,
-    pub usd_24h_vol: f64,
-    pub usd_24h_change: f64,
-    pub eur: f64,
-    pub eur_market_cap: f64,
-    pub eur_24h_vol: f64,
-    pub eur_24h_change: f64,
-    pub btc: f64,
-    pub btc_market_cap: f64,
-    pub btc_24h_vol: f64,
-    pub btc_24h_change: f64,
-    pub eth: f64,
-    pub eth_market_cap: f64,
-    pub eth_24h_vol: f64,
-    pub eth_24h_change: f64,
-    pub last_updated_at: i64,
}
```

Let's update the logic in this way and run csx generate again. You have the template logic generated once you run generate, and you have placed code that modifies it. Therefore, if you run `csx generate` again, you will see the log output that the execution has completed, skipping the logic generation.

![chainsight-handson-web-6](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nobp1o2x3a4nd0tvzkuy.png)

**Build a module**

Generate a module from the code created in the previous processes. Let's run `csx build` immediately.

![chainsight-handson-web-7](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7pvx7bf4qzjj6wd4yjgu.png)

That's all it took to generate a module for Chainsight from your code! Let's actually check out the artifact.

![chainsight-handson-web-8](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/vycw2lm4zvo4a4fjyk9y.png)

The artifacts generated by `csx build` are placed in the `artifacts` folder. If you look in that folder, you will see WASM modules and candid interface files with the actual component names as their file names. For a more in-depth understanding, please refer to the following articles

[Behind Chainsight: How Modules are Created Pt. 2 | by Chainsight | Feb, 2024 | Medium](https://medium.com/@Chainsight_Network/behind-chainsight-how-modules-are-created-pt-2-4cf29234b98b)

That's all there is to the build. Now all you have to do is deploy and get the component up and running.

**Deploy a component**

Let's actually deploy the generated module in Chainsight. Components are deployed with `csx deploy`, and after deployment is complete, `csx exec` is executed to send a command to the component to start execution. Let's run through some options. Let's deploy to Internet Computer using `--component` to explicitly specify the component to process and `--network` to specify the network to run on.

```bash
csx deploy --network ic --component btc_snapshot_indexer
csx exec --network ic --component btc_snapshot_indexer
```

![chainsight-handson-web-9](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/b1vuownqf4pcppz5sigl.png)

![chainsight-handson-web-10](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3dypqr3cq1aapbsu5ck0.png)

The deployment and operation start instruction is now complete! Now all you have to do is wait for the interval of the automatic execution and you can refer to the data.

***

Please take a look at the component that can be created in this hands-on session, which is actually running as a sample. (The description and execution interval have been slightly changed.)
Using the Chainsight UI, you can check the acquired data, the execution cycle, and the state of the cycles that fuel it!

![chainsight-handson-web-11](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rarnu8ohsdpsg32727st.png)

Source: https://beta.chainsight.network/explore/components/wqsbs-fyaaa-aaaal-qdezq-cai

This concludes our hands-on with Snapshot Indexer HTTPS. We will continue to create and publish new hands-on sessions in the future, so please look forward to them!

Written by [Megared@Chainsight](https://twitter.com/megared05)