File Database in Node Js from scratch part 2: Select function & more
Tags: javascript, node, webdev, programming
Published: 2022-07-05 19:48:43




## introduction 

Welcome to part 2 of trying to build from scratch and becoming a better programmer, if you just stumbled upon this post and have no idea what's going you can find part 1 [here](https://dev.to/sfundomhlungu/-i-dont-know-what-i-am-doing-file-database-from-scratch-in-node-js-part-1-9f8), else welcome back and thank you for your time again.

Part 1 was just a setup, nothing interesting happened really and since then I had some time to think about stuff, hence some refactor and lot of code in this part.


## Database.js 

Soon to be previous db function:

```js


function db(options) {
	
	this.meta = {
           length: 0,
           types: {},   
           options 
		   

	}
	this.store = {} 
   

}


```

The first problem I noticed here is `this.store` is referenced a lot in operations.js by different functions, initially it may not seem like a big deal, but if you think for a minute as object are values by reference, meaning allowing access to a single object by multiple functions can cause a huge problem, like receiving an outdated object, trying to access a deleted value etc, 

The functions(select, insert, delete_, update) themselves need to do heavy lifting making sure they are receiving the correct state, checking for correct values and so on, this leads to code duplication and spaghetti code. 

I came up with a solution inspired by state managers, having a single store which exposes it's own API, and no function outside can access it without the API. 

The API is responsible for updating the state, returning the state and maintaining the state, any function outside can request the store to do something and wait, code speaks louder than words, here is a refactored db function

```js

import Store from "./Store.js"
 


function db(options) {
	this.store = new Store("Test db", options) // single endpoint to the database 
     

}





```

I guess the lesson here is once everything starts getting out of hand and spiraling, going back to the vortex abstraction and creating a single endpoint to consolidate everything can be a solution. This will be apparent once we work on the select function. 

one last thing we need  is to remove select from operators to it's own file, select has a lot of code

## updated Database.js 


```js


import {insert, update, delete_} from './operators.js' // remove select
import Store from "./Store.js"
import select from "./select.js" // new select

function db(options) {
	// minor problem: store can be accessed from db object in index.js
	// not a problem thou cause #data is private
	this.store = new Store("Test db", options)
   

}



db.prototype.insert = insert
db.prototype.update = update
db.prototype.select = select
db.prototype.delete_ = delete_


export default db 



```



## Store.js   (new file) 


I chose to use a class for store, you can definitely use a function, my reason for a class is, it is intuitive and visually simple for me to traverse, and easy to declare private variables 

If you are unfamiliar with OOJS(Object Oriented JS) I do have two short articles [here](https://dev.to/sfundomhlungu/unpacking-javascript-01-prototypes-emulating-proto-runtime--1m68), and for this article you need to be familiar with the `this` keyword


```js



export default class Store{
  // private variables start with a "#"
		#data = {}
	    #meta = {
           length: 0,
		  }
      
	  // runs immediatley on class Instantiation
      constructor(name, options){
	  
             this.#meta.name = name;
		     this.#meta.options = options


	  }

	
	// API 
	
	// getters and setters(covered in OOJS)
      //simply returns data  
	 get getData(){

          return this.#data
	  }
	
	// sets data 
	// data is type Object
	set setData(data){

		data._id = this.#meta.length
		
	if(this.#meta.options && this.#meta.options.timeStamp &&   this.#meta.options.timeStamp){
           data.timeStamp = Date.now()

		}
		this.#data[this.#meta.length] = data
		this.#meta.length++
	
	}
	
}


```

Explaining setData 

```js
data._id = this.#meta.length  // _id is reserved so the documents(rows) can actually know their id's

```

adding timeStamp 

```js

	if(this.#meta.options &&        this.#meta.options.timeStamp &&  this.#meta.options.timeStamp){
           data.timeStamp = Date.now()

		}

// this lines reads 
// if meta has the options object 
// and the options object has timeStamp
// and timeStamp(which is a boolean) is true 
  // add datetime to the data before commiting to the db 
// this check is necessary to avoid cannot convert null Object thing error


```

```js
// putting the document or row
this.#data[this.#meta.length] = data
// incrementing the id(pointer) for the next row
this.#meta.length++


```

Now we can safely say we have a single endpoint to the database(#data)  outside access should consult the API, and not worry about how it gets or sets the data

However using setData and getData sounds weird, we can wrap these in familiar functions and not access them directly 

classes also have a proto object covered [here](https://dev.to/sfundomhlungu/unpacking-javascript-01-prototypes-emulating-proto-runtime--1m68)

```js

Store.prototype.insert = function(data){
	 // invoking the setter 
	// this keyword points to the class(instantiated object)
	this.setData  = data
}



```

now we can update operators.js insert


## operators.js 

```js
// updating insert(letting the store handle everything)
export function insert(row){
            this.store.insert(row)
}

```



## Select.js

I had many ideas for select, mostly inspired by other db's, but I settled on a simple and I believe  powerful enough API, for now I want select to just do two things select by ID and query the db based on certain filters.

let's start with select by id as it is simple

```js


export default function select(option = "*"){
	    // checking if option is a number
        if(Number(option) !== NaN){
              
             // return prevents select from running code below this if statement()
			// think of it as an early break
             return this.store.getByid(+option)
			// the +option converts option to a number just to make sure it is
		}
	
	 // query mode code will be here
	
}


```

based on the value of option we select to do one of two select by ID or enter what i call a query mode, to select by id all we need to check is if option is a number, if not we enter query mode

## Store.js 

we need to add the select by id function to the store

```js

... 


Store.prototype.getByid = function(id){
	const data = this.getData  // get the pointer the data(cause it's private we cannot access it directly) 
	//object(remember the value by reference concept)
	
    if(data[id]){  // checking if id exists

          return data[id]  // returning the document
		}else{

          return "noDoc" // for now a str will do
			// but an error object is more appropriate(future worry)
		}

}



```


Simple and now we can get a row by id, query mode is a little bit involved, more code and some helpers

## Select.js  query mode

The core idea is simple really, I thought of the db as a huge hub, a central node of sort, and a query is a small node/channel connected to the center, such that each query node is self contained, meaning it contains it's own state until it is closed.

example

```js


let a = store.select() // returns a query chanel/node
let b = store.select() 

// a is not aware of b, vice versa, 
//whatever happens in each node the other is not concerned



```

for this to work we need to track open channels and their state as the querying continues, an object is a simple way to do just that.

```js

const tracker = {
     id: 0, // needed to ID each channel and retrieve or update it's state
 }


function functionalObj(store){
    this.id = NaN  // to give to tracker.id(self identity)

}


export default function select(option = "*"){
...
   
   	 // query mode code will be here
    
   // functionalObj will return the node/channel
	return  new functionalObj(this.store)
}




```


functionalObj will have four functions:

beginQuery - will perform the necessary setup to open an independent channel/node to the db

Where - will take a string(boolean operators) to query the db e.g `Where('age > 23')` return all docs where the age is bigger than 23

endQuery - returns the queried data 

close - destroys the channel completely with all it's data

### beginQuery

```js

...


function functionalObj(store){
	...
    // channelName will help with Identifying and dubugging for the developer using our db
	this.beginQuery = (channelName = "") => {
		  // safeguard not to open the same query/channel twice 
	      if(tracker[this.id] && tracker[this.id].beganQ){ // checking if the channel already exists(when this.id !== NaN)
			      console.warn('please close the previous query');
					return 
		  }
          
		
		
		 // opening a node/channel
		     this.id = tracker.id
             tracker[this.id] = {
              filtered: [], // holds filtered data
              beganQ: false,  // initial status of the channel(began Query)
              cName : channelName === "" ? this.id : channelName  
             }
                      
            tracker.id++  // for new channels

		
		     // officially opening the channel to be queried
		
		                             // we will define the getAll func later
		                             // it basically does what it's says
		       tracker[this.id].filtered = Object.values(store.getAll()) // to be filtered data
			   tracker[this.id].beganQ = true  // opening the channel
			   console.log('opening channel: ', tracker[this.id].cName) // for debugging 
	
	}
	// end of begin query function

	
	
}




```

update Store.js and put this getAll func 

```js

Store.prototype.getAll = function(){

  return this.getData

}

```

### Where, endQuery, close

```js


function functionalObj(store){
	this.beginQuery = (channelName = "") => {
	...	
	}
	// end of beginQuery 
	
	this.Where = (str) => {
		 // do not allow a query of the channel/node if not opened
	     if(!tracker[this.id] || tracker[this.id] && !tracker[this.id].beganQ){
            console.log('begin query to filter')
			return
		 }
		
		 let f = search(str, tracker[this.id].filtered) // we will define search later(will return filtered data and can handle query strings)
		 
		    // update filtered data for the correct channel
		 	if(f.length > 0){
				
				tracker[this.id].filtered = f
		     }
	 
	
	}
	// end of where
	
	this.endQuery = () => {
			
			    if(!tracker[this.id] || tracker[this.id] && !tracker[this.id].beganQ){
					        console.warn('no query to close')
					       return
					}
				        
				 
			// returns data 	           
         return {data:tracker[this.id].filtered, channel: tracker[this.id].cName}     
			};
		
		// end of endQuery 
		
	   this.close = ()=> {
				 // if a node/channel exist destroy it
	           if(tracker[this.id] && !tracker[this.id].closed){
				          Reflect.deleteProperty(tracker, this.id) // delete 
				          console.log('cleaned up', tracker) 
				    
				    }
				
				
		}
}
	
	
	
	
	


```


### Search 

```js

// comm - stands for commnads e.g "age > 23"
const search = function(comm, data){
	 let split = comm.split(" ") // ['age', '>', 23]  
	 // split[0] property to query 
	 // split[1] operator 
	 // compare against
     let filtered = []	
	 
	  // detecting the operator
	  if(split[1] === "===" || split[1] === "=="){
             
		    data.map((obj, i)=> {
				 // mapSearch maps every operator to a function that can handle it
				// and evalute it 
				// mapSearch returns a boolean saying whether the object fits the query if true we add the object to the filtered
				 if(mapSearch('eq' , obj[split[0]], split[2])){
                      // e.g here mapSearch will map each object with a function
					 // that checks for equality(eq)
					   filtered.push(obj)
				 }
                   
		    }) 
        
	}else if(split[1] === "<"){
             data.map((obj, i)=> {
				  // less than search
					 if(mapSearch('ls' , obj[split[0]], split[2])){

						   filtered.push(obj)
					 }

		    })     

	}else if(split[1] === ">"){


                data.map((obj, i)=> {
					// greater than search
					 if(mapSearch('gt' , obj[split[0]], split[2])){

						   filtered.push(obj)
					 }

		     }) 
	}
	
	
	   return filtered // assigned to f in Where function
}

function functionalObj(store){
...	
}



```


### mapSearch 

```js

// direct can be eq, gt, ls which directs the comparison 
// a is the property  --- age 
// b to compare against --- 23
const mapSearch = function(direct, a, b){
      

	     if(direct === "eq"){
               // comparers defined func below
           return comparers['eq'](a, b) // compare for equality
	 }else if(direct === "gt"){

              return comparers['gt'](a, b) // is a > b

		}else if(direct === "ls"){
            return comparers['ls'](a, b) // is a < b


			}else{

                console.log('Not handled')
			}


}





const search = function(comm, data){
...	
}
...

```


### Comparers 

actually does the comparison and returns appropriate booleans to filter the data

```js
  // return a boolean (true || false)
const comparers = {
  "eq": (a, b) => a === b,
   "gt": (a, b) => a > b,
	"ls": (a, b) => a < b
 

}

```


Select should work now, we can query for data through dedicated channels

## test.js 

testing everything


```js

import db from './index.js'



let store = new db({timeStamp: true})


store.insert({name: "sk", surname: "mhlungu", age: 23})
store.insert({name: "np", surname: "mhlungu", age: 19})
store.insert({name: "jane", surname: "doe", age: 0})


const c = store.select() // return a new node/channel to be opened

c.beginQuery("THIS IS CHANNEL C")  // opening the channel and naming it

c.Where('age < 23') // return all documents where age is smaller than 23


const d = store.select() // return a new node/channel 

d.beginQuery("THIS IS CHANNEL D") // open the channel
d.Where('age > 10') // all documents where age > 10

console.log('===============================================')
console.log(d.endQuery(), 'D RESULT age > 10')  // return d's data 
console.log('===============================================')
console.log(c.endQuery(), "C RESULT age <  23") // return c's data
console.log('===============================================')
c.close() // destroy c 
d.close()  // destroy d


```

```cmd

node test.js
 
```


you can actually chain multiple where's on each node, where for now takes a single command 

example 

```js
const c = store.select() 

c.beginQuery("THIS IS CHANNEL C")

c.Where("age > 23")
c.Where("surname === doe") // will further filter the above returned documents

```

## Problems
the equality sign does not work as expected when comparing numbers, caused by the number being a string 

```js


// "age === 23"
 
comm.split(" ") // ['age', '===', '23'] // 23 becomes a string 

23 === '23' // returns false

// while 'name === sk' will work

comm.split(" ") // ['name', '===', 'sk']

'sk'  === 'sk'


```

a simple solution will be to check if each command is comparing strings or numbers, which in my opinion is very hideous and not fun to code really, so a solution I came up with is to introduce types for the db, meaning our db will be type safe, and we can infer from those types the type of operation/comparisons

for example a new db will be created like this:


```js


let store = new db({
	             timeStamp: true,
	             types: [db.String, db.String, db.Number]  // repres columns
				   
				   })



// if you try to put a number on column 1 an error occurs, because insert expect a string
```

the next tut will focus on just that.


# conclusion


If you want a programming buddy I will be happy to connect on [twitter](https://twitter.com/MhlunguSfundo) , or you or you know someone who is hiring for a front-end(react or ionic) developer or just a JS developer(modules, scripting etc) **I am looking for a job or gig** please contact me: [mhlungusk@gmail.com](mailto:mhlungusk@gmail.com), twitter will also do

Thank you for your time, enjoy your day or night. until next time