Building a Real-Time Analytics Dashboard with Next.js, Tinybird, and Tremor: A Comprehensive Guide
Tags: nextjs, opensource, analytics, tutorial
Published: 2023-07-06 20:08:05

# What you will find in this article?
Real-time analytics have become a vital part of today's applications, especially when it comes to understanding user behavior and improving your platform based on data-driven insights. Even more so, if your users rely on your product to deliver them with insights.

In this post, we are going to explore how we can create a real-time analytics dashboard for page views using Tinybird, Tremor.so, and Next.js.

We will be using Tinybird for data ingestion and real-time data analysis, Tremor for data visualization, and Next.js for server-side rendering.

![Analytics GIF](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/52m2fwocyft5izpp0qe3.gif)

# Papermark - the open-source DocSend alternative.
Before we kick off, let me introduce you to Papermark. It's an open-source project for securely sharing documents with built-in real-time, page-by-page analytics, powered by Tinybird and visualized by Tremor.

I would be absolutely thrilled if you could give us a star! Don't forget to share your thoughts in the comments section ❤️

https://github.com/mfts/papermark

![Papermark Analytics](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1gzfhi5hp28ckx7tpday.gif)

# Setup the project
Let's set up our project environment. We will be setting up a Next.js app, installing Tinybird CLI and configuring the needed services and tools.

## Set up tea
It's a good idea to have a package manager handy, like [`tea`](https://github.com/teaxyz/cli). It'll handle your development environment and simplify your (programming) life!

```sh
sh <(curl https://tea.xyz)

# --- OR ---
# using brew
brew install teaxyz/pkgs/tea-cli
```

`tea` frees you to focus on your code, as it takes care of installing `python` and `pipenv` (which I'm using to run `tinybird-cli`), `node`, `npm` and any other packages you may need. The best part is, `tea` installs all packages in a dedicated directory (_default: `~/.tea`_), keeping your system files neat and tidy.

## Set up Next.js with TypeScript and Tailwindcss
We will use create-next-app to generate a new Next.js project. We will also be using TypeScript and Tailwind CSS, so make sure to select those options when prompted.

```bash
npx create-next-app

# ---
# you'll be asked the following prompts
What is your project named?  my-app
Would you like to add TypeScript with this project?  Y/N
# select `Y` for typescript
Would you like to use ESLint with this project?  Y/N
# select `Y` for ESLint
Would you like to use Tailwind CSS with this project? Y/N
# select `Y` for Tailwind CSS
Would you like to use the `src/ directory` with this project? Y/N
# select `N` for `src/` directory
What import alias would you like configured? `@/*`
# enter `@/*` for import alias
```

## Install Tinybird
Tinybird's command line interface (CLI) helps us to manage data sources and pipes. I'm installing `tinybird-cli` using `pipenv`, which manages a virtual environment for my pip packages, in our local environment.

```bash
# Navigate to your Next.js repo
cd my-app

# Create a new virtual environment and install Tinybird CLI
# if you install `tea` in the previous step, then tea will take care of installing pipenv and its dependencies
pipenv install tinybird-cli

# Activate the virtual environment
pipenv shell
```

### Configuring Tinybird
Head over to [tinybird.co](https://www.tinybird.co/) and create a free account. You need to have a Tinybird account and an associated token. You can get the token from your Tinybird's dashboard.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7tpebc5w57f1471qlh1d.png)

```sh
# Authenticate with tinybird-cli using your auth token when prompted
tb auth
```

A `.tinyb` file will be added to your repo's root. No need to modify it. However add it it to your `.gitignore` file to avoid exposing your token.

```sh
echo ".tinyb" >> .gitignore
```

# Building the application
Now that we have our setup in place, we are ready to start building our application. The main features we'll cover are:

- Tinybird Pipes and Datasources
- Page View Recording
- Tremor Bar Chart
  
## #1 Tinybird Pipes and Datasource
The ability to programmatically configure pipes and datasources for Tinybird offers a significant advantage. This flexibility enables us to treat our data infrastructure as code, meaning that the entire configuration can be committed into a version control system. For an open-source project like Papermark, this capability is highly beneficial. It fosters transparency and collaboration, as contributors can readily understand the data structure without any ambiguity.

We set up Tinybird pipes and datasource as follows:

```sh
mkdir -p lib/tinybird/{datasources,endpoints}
```

### 1. Datasources
This is basically a versioned schema for `page_views`. This the only datasource we need for ingesting, storing and reading analytics about page views. Feel free to add/remove fields. 

```sh
# lib/tinybird/datasources/page_views.datasource
VERSION 1

DESCRIPTION >
  Page views are events when a user views a document

SCHEMA >
  `id` String `json:$.id`,
  `linkId` String `json:$.linkId`,
  `documentId` String `json:$.documentId`,
  `viewId` String `json:$.viewId`,
  # Unix timestamp
  `time` Int64 `json:$.time`,
  `duration` UInt32 `json:$.duration`,
  # The page number
  `pageNumber` LowCardinality(String) `json:$.pageNumber`,
  `country` String `json:$.country`,
  `city` String `json:$.city`,
  `region` String `json:$.region`,
  `latitude` String `json:$.latitude`,
  `longitude` String `json:$.longitude`,
  `ua` String `json:$.ua`,
  `browser` String `json:$.browser`,
  `browser_version` String `json:$.browser_version`,
  `engine` String `json:$.engine`,
  `engine_version` String `json:$.engine_version`,
  `os` String `json:$.os`,
  `os_version` String `json:$.os_version`,
  `device` String `json:$.device`,
  `device_vendor` String `json:$.device_vendor`,
  `device_model` String `json:$.device_model`,
  `cpu_architecture` String `json:$.cpu_architecture`,
  `bot` UInt8 `json:$.bot`,
  `referer` String `json:$.referer`,
  `referer_url` String `json:$.referer_url`


ENGINE "MergeTree"
ENGINE_SORTING_KEY "linkId,documentId,viewId,pageNumber,time,id"
```

### 2. Pipes
In Tinybird, a pipe is a series of transformations that are applied to your data. These transformations could include anything from simple data cleaning operations to complex aggregations and analytics.

We have one versioned pipe (also sometimes called endpoint) to retrieve page_view data. This Tinybird pipe, named `endpoint`, calculates the average duration users spend on each page of a specific document within a defined timeframe, and presents the results in ascending order by page number.

```sh
# lib/endpoints/get_average_page_duration.pipe
VERSION 1

NODE endpoint
SQL >
    %
    SELECT
        pageNumber,
        AVG(duration) AS avg_duration
    FROM
        page_views__v1
    WHERE
        documentId = {{ String(documentId, required=True) }}
        AND time >= {{ Int64(since, required=True) }}
    GROUP BY
        pageNumber
    ORDER BY
        pageNumber ASC
```

Now that we have Tinybird datasources and pipes set up, you need to push them to your Tinybird's account using the CLI.

```bash
# Navigate to the directory containing your datasource and pipe files
cd lib/tinybird

# Push your files to Tinybird
tb push datasources/*.datasource pipes/*.pipe
```

### 3. Typescript functions 
Let's set up the appropriate typescript functions to actually send and retrieve data from Tinybird. We are using `zod` and chronark's `zod-bird` library

This function is for **retrieving** data from Tinybird:
```ts
// lib/tinybird/pipes.ts
import { z } from "zod";
import { Tinybird } from "@chronark/zod-bird";

const tb = new Tinybird({ token: process.env.TINYBIRD_TOKEN! });

export const getTotalAvgPageDuration = tb.buildPipe({
  pipe: "get_total_average_page_duration__v1",
  parameters: z.object({
    documentId: z.string(),
    since: z.number(),
  }),
  data: z.object({
    pageNumber: z.string(),
    avg_duration: z.number(),
  }),
});
```

This function is for **sending** data to Tinybird:
```ts
// lib/tinybird/publish.ts
import { z } from "zod";
import { Tinybird } from "@chronark/zod-bird";

const tb = new Tinybird({ token: process.env.TINYBIRD_TOKEN! });

export const publishPageView = tb.buildIngestEndpoint({
  datasource: "page_views__v1",
  event: z.object({
    id: z.string(),
    linkId: z.string(),
    documentId: z.string(),
    viewId: z.string(),
    time: z.number().int(),
    duration: z.number().int(),
    pageNumber: z.string(),
    country: z.string().optional().default("Unknown"),
    city: z.string().optional().default("Unknown"),
    region: z.string().optional().default("Unknown"),
    latitude: z.string().optional().default("Unknown"),
    longitude: z.string().optional().default("Unknown"),
    ua: z.string().optional().default("Unknown"),
    browser: z.string().optional().default("Unknown"),
    browser_version: z.string().optional().default("Unknown"),
    engine: z.string().optional().default("Unknown"),
    engine_version: z.string().optional().default("Unknown"),
    os: z.string().optional().default("Unknown"),
    os_version: z.string().optional().default("Unknown"),
    device: z.string().optional().default("Desktop"),
    device_vendor: z.string().optional().default("Unknown"),
    device_model: z.string().optional().default("Unknown"),
    cpu_architecture: z.string().optional().default("Unknown"),
    bot: z.boolean().optional(),
    referer: z.string().optional().default("(direct)"),
    referer_url: z.string().optional().default("(direct)"),
  }),
});
```

### 4. Configure Auth Token for production
Don't forget to add `TINYBIRD_TOKEN` to your `.env` file. It's advised that you create a token that has the minimum scope for your operations:

- Read from specific pipe
- Append to specific datasource

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wp1mue6v85ro6qhkvv2t.png)

Congrats! 🎉 You successfully configured Tinybird and ready to move to the next


## #2 Page View Recording
We will capture the page view event in our PDFviewer component and publish it to our Tinybird's datasource.

Let's build an API function to send data to Tinybird every time a page is viewed:

```ts
// pages/api/record_view.ts
import { NextApiRequest, NextApiResponse } from "next";
import { publishPageView } from "@/lib/tinybird";
import { z } from "zod";
import { v4 as uuidv4 } from 'uuid';

// Define the validation schema
const bodyValidation = z.object({
  id: z.string(),
  linkId: z.string(),
  documentId: z.string(),
  viewId: z.string(),
  time: z.number().int(),
  duration: z.number().int(),
  pageNumber: z.string(),
  ...
});

export default async function handle(
  req: NextApiRequest,
  res: NextApiResponse
) {
  // We only allow POST requests
  if (req.method !== "POST") {
    res.status(405).json({ message: "Method Not Allowed" });
    return;
  }

  const { linkId, documentId, viewId, duration, pageNumber } = req.body;
  const time = Date.now(); // in milliseconds
  const pageViewId = uuidv4();

  const pageViewObject = {
    id: pageViewId,
    linkId,
    documentId,
    viewId,
    time,
    duration,
    pageNumber: pageNumber.toString(),
    ...
  };

  const result = bodyValidation.safeParse(pageViewObject);
  if (!result.success) {
    return res.status(400).json(
      { error: `Invalid body: ${result.error.message}` }
    );
  }

  try {
    await publishPageView(result.data);
    res.status(200).json({ message: "View recorded" });
  } catch (error) {
    res.status(500).json({ message: (error as Error).message });
  }
}
```

And finally the PDFViewer component that sends the request:
```tsx
// components/PDFViewer.tsx
import { useState, useEffect } from 'react';

const PDFViewer = () => {
  const [pageNumber, setPageNumber] = useState<number>(1)

  useEffect(() => {
    startTime = Date.now(); // update the start time for the new page

    // when component unmounts, calculate duration and track page view
    return () => {
      const endTime = Date.now();
      const duration = Math.round(endTime - startTime);
      trackPageView(duration);
    };
  }, [pageNumber]); // monitor pageNumber for changes
  
  async function trackPageView(duration: number = 0) {
    await fetch("/api/record_view", {
      method: "POST",
      body: JSON.stringify({
        linkId: props.linkId,
        documentId: props.documentId,
        viewId: props.viewId,
        duration: duration,
        pageNumber: pageNumber,
      }),
      headers: {
        "Content-Type": "application/json",
      },
    });
  }

  return (
    // Your PDF Viewer implementation
  );
}

export default PDFViewer;
```

## #3 Tremor Bar Chart
Let's now create a bar chart to display the page views for each document. We are using [tremor.so](https://www.tremor.so/) to build our beautiful dashboard. 

```sh
# Install tremor with their CLI 
npx @tremor/cli@latest init
```

```tsx
// components/bar-chart.tsx
import { BarChart } from "@tremor/react";

const timeFormatter = (number) => {
  const totalSeconds = Math.floor(number / 1000);
  const minutes = Math.floor(totalSeconds / 60);
  const seconds = Math.round(totalSeconds % 60);

  // Adding zero padding if seconds less than 10
  const secondsFormatted = seconds < 10 ? `0${seconds}` : `${seconds}`;

  return `${minutes}:${secondsFormatted}`;
};

export default function BarChartComponent({data}) {
  return (
    <BarChart
      className="mt-6 rounded-tremor-small"
      data={data}
      index="pageNumber"
      categories={["Time spent per page"]}
      colors={["gray"]}
      valueFormatter={timeFormatter}
      yAxisWidth={50}
      showGridLines={false}
    />
  );
}
```

```ts
// lib/swr/use-stats.ts
import { useRouter } from "next/router";
import useSWR from "swr";
import { getTotalAvgPageDuration } from "@/lib/tinybird/pipes";

export function useStats() {
  const router = useRouter();
  const { id } = router.query as { id: string };

  const { data, error } = useSWR(
    id,
    () => getTotalAvgPageDuration({ documentId: id, since: 0 
}),
    {
      dedupingInterval: 10000,
    }
  );

  return {
    durationData: data,
    isLoading: !error && !data,
    error,
  };
}
```

```tsx
// pages/document/[id].tsx
import { useDocument } from "@/lib/swr/use-document";
import { useStats } from "@/lib/swr/use-stats";
import BarChartComponent from "@/components/bar-chart";

export default function DocumentPage() {
  const { document, error: documentError } = useDocument();
  const { stats, error: statsError } = useStats();

  if (documentError) {
    // handle document error
  }

  if (statsError) {
    // handle stats error
  }

  return (
    <>
      <main>
        {document && (
          <header>
            <h1>{document.name}</h1>
            {stats && <BarChartComponent data={stats.durationData} />}
          </header>
        )}
      </main>
    </>
  );
}
```

Voila 🎉 The Bar Chart with accurate page-by-page time! 

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fsrkvu1y1m7fn6zmvvej.png)

# Conclusion
That's it! We've built a real-time analytics dashboard for page views using Tinybird, Tremor, and Next.js. While the example here is simple, the same concepts can be expanded to handle any kind of analytics your app might need to perform.

Thank you for reading. I'm Marc, an open-source advocate. I am building [papermark.io](https://www.papermark.io/) - the open-source alternative to DocSend with millisecond-accurate page analytics.

# Help me out!
If you found this article helpful and got to understand Tinybird and dashboarding with Tremor better, I would be eternally grateful if you could give us a star! And don't forget to share your thoughts in the comments ❤️

https://github.com/mfts/papermark

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/wr4kd6fhpuaueax2kmih.gif)