add duplicate checker

src/snippets/checking-duplicate-dataset.md

@@ -0,0 +1,29 @@
+---
+id: "duckdb-check-duplicate-rows"
+title: "Check Duplicate Rows"
+slug: "duckdb-check-duplicate-rows-query"
+description: "Count the number of duplicate rows in a table using DuckDB."
+code: |
+    -- Count duplicate rows in the 'train' table
+    SELECT COUNT(*) - COUNT(DISTINCT columns(*))
+    FROM train;
+---
+
+# DuckDB Check Duplicate Rows Query
+
+This snippet demonstrates how to count the number of duplicate rows in a DuckDB table using a SQL query.
+
+```sql
+-- Count duplicate rows in the 'train' table
+SELECT COUNT(*) - COUNT(DISTINCT columns(*))
+FROM train;
+```
+
+This query works by:
+1. Counting all rows using `COUNT(*)`
+2. Subtracting the count of distinct rows using `COUNT(DISTINCT *)`
+3. The result is the number of duplicate rows
+
+You can replace 'train' with the name of your specific table to check for duplicates in other tables.
+
+Note: This query can be computationally expensive for large tables, as it needs to check all columns for uniqueness.

src/snippets/unnest.md

@@ -0,0 +1,47 @@
+---
+id: "duckdb-table-info"
+title: "Table Info"
+slug: "duckdb-table-info-query"
+description: "Get detailed information about a table's structure"
+code: |
+    with nested as 
+    (
+    select unnest(reasoning_chains) as reasoning_chains
+    from train limit 100
+    )
+
+    select reasoning_chains.* from nested;
+---
+
+# DuckDB Table Info Query
+
+This snippet demonstrates how to use the `PRAGMA table_info` function in DuckDB to get detailedinformation about a specific table's structure.
+
+Try out this query [here](https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1?sql_console=true&sql=pragma+table_info%28%27func_calling_singleturn%27%29)
+
+```sql
+-- Returns detailed information about a table's structure
+-- Use this to get the column names, types, and other details
+PRAGMA table_info('table_name');
+```
+
+# Example Output
+
+| cid | name          | type                                     | notnull | dflt_value | pk    |
+|-----|---------------|------------------------------------------|---------|------------|-------|
+| 0   | id            | VARCHAR                                  | false   | null       | false |
+| 1   | conversations | STRUCT("from" VARCHAR, "value" VARCHAR)[] | false   | null       | false |
+| 2   | category      | VARCHAR                                  | false   | null       | false |
+| 3   | subcategory   | VARCHAR                                  | false   | null       | false |
+| 4   | task          | VARCHAR                                  | false   | null       | false |
+
+You can read more about this [here](https://duckdb.org/docs/sql/query_syntax/unnest.html).
+
+# Example Usage
+
+<iframe
+		src="https://huggingface.co/datasets/SkunkworksAI/reasoning-0.01/embed/viewer/default/train?sql_console=true&sql=with+nested+as+%0A++%28%0A++select+unnest%28reasoning_chains%29+as+reasoning_chains%0A++from+train+limit+100%0A%29%0A%0Aselect+reasoning_chains.*+from+nested%3B"
+		frameborder="0"
+		width="100%"
+	height="560px"
+></iframe>