File Database in Node Js from scratch part 3: introducing types
Tags: javascript, node, programming, database
Published: 2022-07-11 13:30:01



## Introduction 

At the end of this tut or article the aim is simple, having types for each column and a type checking system on insert, for now we will support three types and add more as they are needed.

In this tut, I've experimented a bit with a new type of writing as I am trying to improve my technical writing skills, I hope there is a notable difference, your feedback will be highly appreciated if you have any.

## Add types in utils.js 

```js
export const types = {
     "String": 1,
     "Number": 2,
     "Object": 3,

}

// you will see how this object is useful in a sec


```

## database.js 

now for types to work we have to enforce them early on, meaning a `types option` when creating a new db is not optional anymore, if it is not present we throw an error

```js


let store = new db("test db", {timeStamp: true,
                   // mandatory
                   types: {name:"String", surname:"String", age:"Number"}}

                  )

      

```

I could have used an array to manage types: `["String", "String", "Number"]` which would be simpler: an index corresponds with a column, the problem is a document/object `{name: "sk", surname: "mhlungu", age: 23}` cannot be really trusted to maintain the order of columns, as objects are not by index but keys(for a large enough object the values MAY(i am not sure either and don't want to find out) change positions even if we use object.keys. 

so that is why I am mapping column names to their types, which consequently adds a new feature: you cannot add a document with a column that does not exist

e.g `{name: "sk", surname: "mhlungu", age: 23}` correct 
    `{name: "sk", surname: "mhlungu", age: 23, stack: "React, Ionic"}` wrong: must throw an error
	
let's update `database.js to handle this`

```js
 ... // three dots represent previous/existing code
 import {types} from "./utils.js"



function db(name,options) {
   	 // if options does not have the types property
    if(!options.types){

        throw new Error("db needs a types option to work")
	}
	
   // checking if the types are supported by the database
  const n = Object.keys(options.types).map((val, i)=> {

      return types[options.types[val]]

  })
	
	
	
	...
}




```


type support check breakdown 


```js

 const n = Object.keys(options.types).map((val, i)=> {

      return types[options.types[val]]

  })
 
 // this code is simple really 
 // for  {name:"String", surname:"String", age:"Number"}
 // the above loops over the object
 // the magic happens in the return, remember our types util:
  export const types = {
     "String": 1,
     "Number": 2,
     "Object": 3,

}

 // if the type in options.types is present in utils types
 // a number corresponding to that type is returned else undefined is returned
 // for {name:"String", surname:"String", age:"Number"} 
  // [1, 1, 2] will be returned 
  // for {name:"String", surname:"String", age:"Number", stack: "Array"}
  // [1, 1, 2, undefined] will be returned 
  // all we need to do now is check for undefined in the array if we find one
  // then we throw an error of unsupported type else we continue and create the db
  




```

checking for undefined in database.js 



```js

function db(name,options) {
	
	
	...
	
  if(n.indexOf(undefined) !== -1){ // if we have undefined 

     const m = Object.keys(options.types)[n.indexOf(undefined)]
     // show which column and type is unsupported
     throw new Error(`type of ${options.types[m]} for column ${m} does not exist`)

  }
	
	// if the above two if's are cleared then we can create the db 
	
	
	   this.store = new Store(name, options)
}


```

Goal one complete we have successfully introduced types, now we need to make sure on insert every document follows the same rules, insert a required type for a column, a column of type string cannot hold a number, that's an error

## Store.js - enforcing types on insert

in store's setData we want to end up with something of sort 

```js

   set setData(data){
          // new code
	   // check if the document has required columns
         if(!checkColumns(data, this.#meta.options.types)){

               throw new Error(`db expected a document with these columns: ${Object.keys(this.#meta.options.types)},

                                          but got ${Object.keys(data)} for this document ${JSON.stringify(data)}`)

         }

	     // check if the document has correct types
         if(!checkTypes(data, this.#meta.options.types)){

         throw new Error(`db expected a document with these types: ${Object.values(this.#meta.options.types)},

                                          but got ${Object.values(data)} for this document ${JSON.stringify(data)}`)

         }

	   
	   // new code ends
      data._id = this.#meta.length

      if(this.#meta.options && this.#meta.options.timeStamp && this.#meta.options.timeStamp){

           data.timeStamp = Date.now()

  

      }

      this.#data[this.#meta.length] = data

      this.#meta.length++

      // console.log('data', this.#data)

   }




```

before we write checkColumns and types we need a few utils 

in utils.js add :


```js

// return booleans        
// () =>  👈 functions of sort are called immediate return functions
// they have no {}, they return their values after runnig
export const isStr = (val) => typeof val === "string"

export const isNumb = (val) => typeof val === "number"

export const isObj = (val) => typeof val === "object"

```

back to Store.js 


### CheckColumns function 

place these func's on top of the class

```js


function checkColumns(doc, types){

  let checkOut = true  // state -> the most important value here 
                       // if true everything is correct else not

   // yes you can definetley use forEach below instead of map(will change it too) 
  // react.js habits cause me to use map everywhere 😂😂 i just noticed writing the article 
   Object.keys(types).map((key, i)=> {

      if(!checkOut) return checkOut;

      if(doc[key] === undefined){

        console.log(key, "is missing in this document")

        checkOut = false

      }

   })

   if(Object.keys(types).length !== Object.keys(doc).length) checkOut = false

   return checkOut

  
  

}


```

explanation:

```js


 Object.keys(types).map((key, i)=> {

      if(!checkOut) return checkOut;  // break out of map early if we already have a
	                                  // a column problem

      if(doc[key] === undefined){ // if the document has a missing column

        console.log(key, "is missing in this document")

        checkOut = false

      }

   })


```

to notice in the above is that the code will pass even if we have an extra column that does not exist in types `Object.keys(types)` as we checking columns in types against  doc 

example: 

```js
{name:"String", surname:"String", age:"Number"}
{name: "sk", surname: "mhlungu", age: 23, stack: "React"}

// stack is extra
// the above map will pass cause doc has all types column, the extra will be ignored 
// which is wrong, hence the below code to handle this and make sure 
// columns are of the same size and we have no extra column


```

checking for extra columns

```js

 if(Object.keys(types).length !== Object.keys(doc).length) checkOut = false

```

if we found an extra column we return false then insert won't run but throw an error 

```js

 if(!checkColumns(data, this.#meta.options.types)){

               throw new Error(`db expected a document with these columns: ${Object.keys(this.#meta.options.types)},

                                          but got ${Object.keys(data)} for this document ${JSON.stringify(data)}`)

         }


```

if the column check passes then we can check for types 

### CheckTypes function


```js
import {isStr, isNumb, isObj} from "./utils.js" // typecheck helpers 



// basically this function is the same as columns check 


function checkTypes(doc, types){

    let checkOut = true  // state

  
     // map again 🤦‍♂️, change to forEach please
    Object.keys(doc).map((key,i)=> { // looping over the doc keys {name: "sk", surname: "mhlungu", age: 23}

        if(!checkOut) return checkOut; // early break

       if(types[key] === "String"){ // if the column in question expects a string

           if(!isStr(doc[key])) checkOut = false // and the value in doc is not a string throw an error(checkout = false)

       }else if(types[key] === "Number"){

          if(!isNumb(doc[key])) checkOut = false

       }else if(types[key] === "Object"){

          if(!isObj(doc[key])) checkOut = false

       }

    })

  
  

    return checkOut

}




```

same thing happens here also if the check types fail insert breaks without inserting, I am one to admit for now error handling is horrible , we cannot just break(which is an assumption the developer is using try catch, which is very rare), I am thinking of a dedicated article to handle errors better maybe returning an object with status, and what happened etc

this will checktypes before running insert code

```js

 if(!checkTypes(data, this.#meta.options.types)){

         throw new Error(`db expected a document with these types: ${Object.values(this.#meta.options.types)},

                                          but got ${Object.values(data)} for this document ${JSON.stringify(data)}`)

         }


```

what I am noticing so far in these three articles is the vortex abstract API thing we been following is kinda working, look we added a bunch of code, done a lot of refactoring without touching the end point and changing much of previous code, that is indeed a victory 🍾👌🎉, our end point is still clean in index.js no plumbing yet:


```js

import db from "./database.js"

  
  
  

export default db


```

no shade to plumbers by the way, plumbing or plumber is a football(soccer) slang from my country, meaning a coach who looks promising but is doing absolute rubbish in tactics and formation while having a quality team, which is losing by the way, by plumbing code I mean something similar.

we have basically achieved both goals we set out in the beginning, but remember the main goal was to assist the `where` function from the previous article with transforming `age > 23` string commands to proper values without trying  much

let's do that now, 

## Select.js

Remember our vortex analogy, code that does not concern itself with certain data or state or does not need or require directly must ask the responsible end point for it, so here Select will need types so select must ask Store for them meaning we need a function to return types from the store.

in store.js

```js

// add under set setData inside the class
  get getTypes(){

         return this.#meta.options.types

      }



```

our proto to get types 

```js

  

Store.prototype.types = function(){

    return this.getTypes

  

}

```


back to select, because types will be used by the entire channel(possibly in the future), we can add them in the tracker for each channel, this will make it so on channel destruction the types are destroyed also(saving memory)

update beginQuery  with code followed by `new code` comment

```js

 this.beginQuery = (channelName = "") => {

                   // prepare

                   console.log("creating channel", channelName)

                     if(tracker[this.id] && tracker[this.id].beganQ){

                                    console.warn('please close the previous query');

                                    return

                      }

                     // keys = this.store.allKeys()

              this.id = tracker.id

               tracker[this.id] = {

              filtered: [],

              types: {}, // new code

              beganQ: false,

              cName : channelName === "" ? this.id : channelName

             }

            tracker.id++

                    tracker[this.id].filtered = Object.values(store.getAll())

                    tracker[this.id].types = store.types() // new code

                    tracker[this.id].beganQ = true

                    console.log('opening channel: ', tracker[this.id].cName)

                    // console.log('tracker obj', tracker)

                   };




```


update `where` also to pass the types to search, we can pass the id but it's not that necessary if we can pass the types directly

```js



//update where 
     // now search takes three arguments
     // command, data and types
    let f = search(str, tracker[this.id].filtered, tracker[this.id].types)

```

next we need to update search, for now all we need to know in search is does the command have a number and convert that str number to an actual number, solving our previous problems `23 === "23" // returning false`

```js

const search = function(comm, data, types){
	
  let split = comm.split(" ")

    
     // new
     if(types[split[0]] === 'Number'){

        split[2] = +split[2]  // converting "23" to 23 (number)

     }
	  // Problems split[0] is a column
	  // if the column does not exist cause in where you can pass 
	  // a column that does not exist e.g "aged > 23" 
	  // we need to handle that gracefully
	  // for now it will fail silently because of the way `where` works
	  // and return unfiltered data
	
	...
	
	
	
}

```


that is it for this article you can experiment with test.js, with that we have types finally, and things are getting exciting honestly, I am thinking on moving to dumping the data to a file next. to fulfill the file part in file database, we will deal with other CRUD functions later


## conclusion

If you want a programming buddy I will be happy to connect on [twitter](https://twitter.com/MhlunguSfundo) , or you or you know someone who is hiring for a front-end(react or ionic) developer or just a JS developer(modules, scripting etc) **I am looking for a job or gig** please contact me: [mhlungusk@gmail.com](mailto:mhlungusk@gmail.com), twitter will also do

Thank you for your time, enjoy your day or night. until next time