Advancing dApp development with Hardhat Indexing: A Game-Changer for Ethereum Devs
Tags: tutorial, blockchain, graphql, typescript
Published: 2023-03-24 08:52:24

### Summary

Blockchain data indexing helps dApp developers to build better, more performant decentralized applications, ultimately improving the user experience.

One of the most important ways they do this is by decoupling of smart contract development from frontend and, when needed, backend development.

But traditionally, there has been a lack of solutions to help local development in this sense, **no one thought about indexing a local EVM node…**

> But Subsquid did think of it

And this article explains how.

Having an indexing middleware from the start of a project, when no contracts have been deployed on-chain yet, means developers can build the smart contract locally (just as they do now), and index the local node as well. This enables early development of frontend as well, with reduced coupling to the smart contract, or its blockchain.

A smoother, better development experience leads to better dApps, which can be tested earlier and better, before deploying the smart contract on-chain (not even on testnet!).

### Introduction

{% embed https://www.youtube.com/watch?v=KDPPJYOfgfc %}

A little over a month ago, I have written [an article on how to index a development Ethereum node, using Subsquid and the Ganache/Truffle suite](https://medium.com/subsquid/boost-your-dapp-development-productivity-with-local-indexing-3936ba7a8cec).

I have been positively surprised by the appreciation of the developer community over this very useful feature offered by Subsquid's framework, and over the article itself.

The inspiration for the article came from a project I had built a little over a month earlier, [for which I wrote a different article](https://medium.com/subsquid/how-to-build-a-performant-and-scalable-full-stack-nft-marketplace-63c12466b959). In this initial project, I was actually using a Hardhat local node for my smart contract development. I decided to switch to [Truffle](https://trufflesuite.com/) for the final article, because it had a more intuitive demo contract, and because the command line tool made it easier to create manual transactions.

In a quest to provide developers with as many options possible, as much material possible and the maximum number of examples, I decided to revisit this material, and write a guide on how to index a Hardhat node.

The project discussed in this article is available in this repository:

%[https://github.com/RaekwonIII/local-evm-indexing/] 

### The project

If you haven’t already, you need to install Subsquid CLI first:

```bash
npm i -g @subsquid/cli@latest
```

In order to create a new squid project, simply open a terminal and launch the command:

```bash
sqd init local-evm-indexing -t evm
```

Where `local-evm-indexing` is the name we decided to give to our project, you can change it to anything you like. And `-t evm` specifies what template should be used and makes sure the setup will create a project from the EVM-indexing template.

#### 1\. Create a Hardhat project

In order to start working with Hardhat, it’s necessary to [install the package](https://hardhat.org/hardhat-runner/docs/getting-started#installation).

In a console window, from the squid project’s root folder, launch:

```bash
npx hardhat
```

> ***Note:*** *If you get an error referencing your* [`README.md`](http://README.md) *or other files, delete them and run* `npx hardhat` *again. Alternatively, choose a different folder, but doing so will slightly diverge from this tutorial.*

And follow the guided initialization process. For the purpose of this tutorial, I have chosen a **TypeScript project** and left other choices as default.

#### Configure Hardhat automining (optional)

Then, open the `hardhat.config.ts` (the file extension will be `.js` if you selected a JavaScript file in the previous step), and add this in the `HardhatUserConfig` object (or the `module.exports` for a JavaScript file):

```typescript
const config: HardhatUserConfig = {
  defaultNetwork: "hardhat",
  networks: {
    hardhat: {
      chainId: 1337,
      mining: {
        auto: false,
        interval: [4800, 5200]
      }
    },
  },
  solidity: "0.8.17",
};
```

The `mining` configuration will continuously mine blocks, even if no events or transactions are executed, it can be useful for debugging. You can read more about it on hardhat's official documentation.

Finally, launch the node with this command:

```bash
npx hardhat node
```

#### 2\. Deploy the contract (and generate transactions)

Hardhat ships with its own [demo contract, called `Lock`](https://hardhat.org/hardhat-runner/docs/getting-started#compiling-your-contracts), but I found myself more at ease with the `MetaCoin` contract from Truffle, which part of the reason why the first article uses Ganache and Truffle as an example.

In order to better compare the setup between the two articles and the two options (Ganache and Hardhat), I am going to continue using the `MetaCoin` contract in this article. 

My suggestion is to just copy the [Solidity source code from the repository](https://github.com/RaekwonIII/local-evm-indexing/tree/main/contracts) and paste the two files in the `contracts` folder of the Hardhat project.

The next thing to do is change the `deploy.ts` (or `.js`, depending on how you initialized the project) script, located in the `scripts` folder. It needs to be edited in order to deploy our the `ConvertLib` library, as well as our ne. `MetaCoin` contract.

The [hardhat-ethers plugin](https://github.com/NomicFoundation/hardhat/tree/main/packages/hardhat-ethers) adds the `ethers.js` library to the Hardhat node, making it possible to use its functionalities. It's possible to use it to deploy the contract, but since our goal is to create transactions, we can also add some code to do that as well.

It should look like this:

```typescript
import { ethers } from "hardhat";

async function main() {

  const ConvertLib = await ethers.getContractFactory("ConvertLib");
  const convertLib = await ConvertLib.deploy();

  await convertLib.deployed();

  const MetaCoin = await ethers.getContractFactory("MetaCoin", {
    libraries: {
      ConvertLib: convertLib.address,
    },
  });
  const metaCoin = await MetaCoin.deploy();

  let instance = await metaCoin.deployed();

  console.log(`MetaCoin  deployed to ${metaCoin.address}`);
  let accounts = await ethers.getSigners()
  console.log(`There are a total of ${accounts.length} accounts`)
  await instance.sendCoin(accounts[1].address, 10, {from: accounts[0].address})
  console.log(`Sent 10 coin from ${accounts[0].address} to ${accounts[1].address}`)
  await instance.sendCoin(accounts[2].address, 10, {from: accounts[0].address})
  console.log(`Sent 10 coin from ${accounts[1].address} to ${accounts[2].address}`)
  await instance.sendCoin(accounts[3].address, 10, {from: accounts[0].address})
  console.log(`Sent 10 coin from ${accounts[1].address} to ${accounts[3].address}`)
  await instance.sendCoin(accounts[4].address, 10, {from: accounts[0].address})
  console.log(`Sent 10 coin from ${accounts[1].address} to ${accounts[3].address}`)
}

// We recommend this pattern to be able to use async/await everywhere
// and properly handle errors.
main().catch((error) => {
  console.error(error);
  process.exitCode = 1;
});
```

Then open a different terminal (the first one is occupied by the running node), and run this command to deploy the contract:

```bash
npx hardhat run scripts/deploy.ts --network localhost
```

> **Note:** please pay attention to the address of the deployed`MetaCoin` smart contract, as it will be used in the indexer logic section below, and it may vary from the one shown there.

The next sections about the Subsquid Archive and the development of the Squid indexer, are actually [unchanged from the original article](https://medium.com/subsquid/boost-your-dapp-development-productivity-with-local-indexing-3936ba7a8cec), and this is not only on purpose, but it's actually the point:

> We have changed the tool running the local Ethereum node, but the indexing part stays the same
> 
> Hardhat offers the ability to [interact with the running blockchain node through a console](https://hardhat.org/hardhat-runner/docs/guides/hardhat-console). If you don’t want to generate transactions on deployment, you can open the console and run the necessary code later on.
> 
> Using it is as easy as running the command:
> 
> ```bash
> npx hardhat console
> ```

### Archive Docker image

In order to index the local Ethereum node, we need to use a pre-built Docker image of Subsquid’s Ethereum Archive. You can do it, by creating a `docker-compose.archive.yml` file (I have added one in the `archive` folder) and paste this:

```yaml
version: "3"

services:
  worker:
    image: subsquid/eth-archive-worker:latest
    environment:
      RUST_LOG: "info"
    ports:
      - 8080:8080
    command: [
            "/eth/eth-archive-worker",
            "--server-addr", "0.0.0.0:8080",
            "--db-path", "/data/db",
            "--data-path", "/data/parquet/files",
            "--request-timeout-secs", "300",
            "--connect-timeout-ms", "1000",
            "--block-batch-size", "10",
            "--http-req-concurrency", "10",
            "--best-block-offset", "10",
            "--rpc-urls", "http://host.docker.internal:8545",
            "--max-resp-body-size", "30",
            "--resp-time-limit", "5000",
    ]
    # Uncomment this section on Linux machines.
    # The connection to local RPC node will not work otherwise.
    # extra_hosts:
    #   - "host.docker.internal:host-gateway"
    volumes:
      - database:/data/db
volumes:
  database:
```

Now, let’s open the `commands.json` file, and add `archive-up` and `archive-down` to the list of commands. It's not mandatory, but these shortcuts are quicker to type, and easier to remember! 😅

```json
{
    "$schema": "https://cdn.subsquid.io/schemas/commands.json",
    "commands": {
      // ...
      "archive-up": {
        "description": "Start a local EVM Archive",
        "cmd": ["docker-compose", "-f", "archive/docker-compose.archive.yml", "up", "-d"]
      },
      "archive-down": {
        "description": "Stop a local EVM Archive",
        "cmd": ["docker-compose", "-f", "archive/docker-compose.archive.yml", "down"]
      },
      // ...
    }
  }
```

Then start the service by opening the terminal and launching the command:

```bash
sqd archive-up
```

### Squid development

It’s finally time to index the smart contract events we generated earlier, with Subsquid’s SDK. Using the **contract’s ABI** and contract **address** from previous steps.

If you want to know more about how to develop your squid ETL, please head over to the [dedicated tutorial](https://docs.subsquid.io/tutorials/create-an-ethereum-processing-squid/) on Subsquid’s official documentation.

#### Schema

Since we are using `MetaCoin` contract as a reference, so we'll build a schema that makes sense with respect to the data it produces.

```graphql
type Transfer @entity {
  id: ID!
  block: Int!
  from: String! @index
  to: String! @index
  value: BigInt!
  txHash: String!
  timestamp: BigInt!
}
```

And now let’s generate the TypeScript model classes for this schema. In a console window, from the project’s root folder, type this command:

```bash
sqd codegen
```

#### ABI facade

We also need to generate TypeScript code to interact with the contract, and for that, we need the contract’s ABI.

Similarly to Truffle, Hardhat generates that when it compiles the Solidity code, and puts the ABI under `artifacts/contracts`(while Truffle does the same under `build/contracts/`). So let's use Subsquid CLI tools and in a console window, from the project’s root folder, type this command:

```bash
sqd typegen artifacts/contracts/MetaCoin.sol/MetaCoin.json
```

#### Processor configuration

Let’s open the `processor.ts` file, and make sure we import the new `Transfer` model, as well as the ABI facade. From this, we are only going to import the `events` object.

```typescript
import { Transfer } from './model';
import { events } from './abi/MetaCoin';
```

Because we want to index our local Archive, the data source of the processor class needs to be set to the local environment.

I have also created a constant for the address of the contract we deployed, and always remember to `toLowerCase()` your contract addresses, to avoid issues.

> **Note:** This is where you need to use the contract address that has been logged by the `deploy.ts` script.

And finally, we want to configure the processor to request data about the `Transfer` event:

```typescript
// ...
const contractAddress = "0x9fE46736679d2D9a65F0992F2272dE9f3c7fa6e0".toLowerCase()
const processor = new EvmBatchProcessor()
  .setDataSource({
    chain: "http://localhost:8545",
    archive: "http://localhost:8080",
  })
  .addLog(contractAddress, {
    filter: [[events.Transfer.topic]],
    data: {
      evmLog: {
        topics: true,
        data: true,
      },
      transaction: {
        hash: true,
      },
    },
  });
// ...
```

You can also use environment variables, just like shown in [this complete end-to-end project example](https://medium.com/subsquid/how-to-build-a-performant-and-scalable-full-stack-nft-marketplace-63c12466b959). This will make sure the project will stay the same, and only configuration files containing environment variables will be changed, depending on the deployment.

#### Process events

We now need to change the logic of the [`processor.run`](http://processor.run)`()` function, to apply our custom logic.

In order to process `Transfer` events, we need to "unbundle" all the events our processor receives from the Archive in a single batch, and decode each one of them.

The best practice in this scenario is to use an array as a temporary data storage and save all the processed transfers at the end, in another batch operation. This will yield better performance, as data storages such as the Postgres database we are using at the moment, are optimized for this. The code should look like this:

```typescript
processor.run(new TypeormDatabase(), async (ctx) => {
  const transfers: Transfer[] = []
  for (let c of ctx.blocks) {
    for (let i of c.items) {
      if (i.address === contractAddress && i.kind === "evmLog"){
        if (i.evmLog.topics[0] === events.Transfer.topic) {
          const { _from, _to, _value } = events.Transfer.decode(i.evmLog)
          transfers.push(new Transfer({
            id: `${String(c.header.height).padStart(10, '0')}-${i.transaction.hash.slice(3,8)}`,
            block: c.header.height,
            from: _from,
            to: _to,
            value: _value.toBigInt(),
            timestamp: BigInt(c.header.timestamp),
            txHash: i.transaction.hash
          }))
        }
      }
    }
   }
   await ctx.store.save(transfers)
});
```

#### Run and test

It’s finally time to perform a full test. Let’s build the project first:

```bash
sqd build
```

Then, we need to launch the database Docker container, by running the command:

```bash
sqd up
```

Next, we need to remove the database migration file that comes with the template, and generate a new one, for our new database schema:

```bash
sqd migration:clean
sqd migration:generate
```

And we can finally launch the processor:

```bash
sqd process
```

In order to test our data, we can launch the GraphQL server as well. In a different console window, run:

```bash
sqd serve
```

And perform a test query:

```graphql
query MyQuery {
  transfers {
    block
    from
    id
    timestamp
    to
    txHash
    value
  }
}
```

![](https://cdn-images-1.medium.com/max/800/1*PZD6_viW_y2VFsTGHxDRqg.png align="left")

### Conclusions

What I wanted to show with this article is that thanks to Subsquid, it is possible to have an indexing middleware at the same time as developing smart contract(s) for a project.

Developers can build their dApp by building their smart contract locally, deploying it on a local node, and indexing that local node with a squid. This leads to a development and testing cycle with much tighter iterations.

Furthermore, since the indexing middleware is present from the start, fewer changes have to be made (just changing the `chain`, `address` and the contract address itself).

If you found this article interesting, and you want to read more, please [follow me](https://linktr.ee/raekwoniii) and most importantly, Subsquid.

**Subsquid socials:**

[Website](https://subsquid.io/) | [Twitter](https://www.twitter.com/subsquid) | [Discord](https://discord.gg/subsquid) | [LinkedIn](https://www.linkedin.com/company/subsquid) | [Telegram](https://t.me/subsquid) | [GitHub](https://github.com/subsquid) | [YouTube](https://www.youtube.com/@subsquid)