qualC 1.0.7
qualC is a Data Quality Check Package


#qualC


qualC is a pyspark library which can perform data quality checks

It takes SparkSession as a parameter and registers UDFs. The UDFs can be used to perform data quality checks.

The functions are:

suppressLeadingZeroesUDF(x):


    Returns the string after removing zeroes from the left of the string.
    Use the suppressLeadingZeroesUDF. 
    
    Parameters:
    String column

    Examples:
    spark.sql("select suppressLeadingZeroesUDF(columnname) from TABLE").show() to use as a query.
    df=spark.sql("select suppressLeadingZeroesUDF(columnname) from TABLE") to store the values in a DataFrame


padZeroUDF(max_length,x):
    

       Adds zeroes to the left of string if the length of string is less than the maximum length.
        Use the padZeroUDF. 
        
       Parameters:
       Max_length
       Column of Numeric Strings or Numbers
       
       Examples:
       spark.sql("select padZeroUDF(Max_Length,columnname) from TABLE").show() to use as a query.
       df=spark.sql("select padZeroUDF(Max_Length,columnname) from TABLE") to store the values in a DataFrame

isNumericUDF(x) :
   
       Checks if the string is numeric.
       Use the isNumericUDF. 
        
       Parameters:
       Column of Strings
       
       Examples:
       spark.sql("select isNumericUDF(columnname) from TABLE").show() to use as a query.
       df=spark.sql("select isNumericUDF(columnname) from TABLE") to store the values in a DataFrame
    

getMantissaValueUDF(x):
    
        Get Mantissa Value of a decimal number or a numeric string.
        Use the getMantissaValueUDF. 
        
        Parameters:
        Column of Numeric Strings or Decimal Numbers
        
        Examples: 
        spark.sql("select ID, getMantissaValueUDF(columnname) from TABLE").show() to use as a query.
        df=spark.sql("select ID, getMantissaValueUDF(columnname) from TABLE") to store the values in a DataFrame
    

getDecimalValueUDF(x):
    
        Get Decimal Value of a decimal number or a numeric string.
        Use the getDecimalValueUDF. 
        
        Parameters:
        Column of Numeric Strings or Decimal Numbers
        
        Examples: 
        spark.sql("select getDecimalValueUDF(columnname) from TABLE").show() to use as a query.
        df=spark.sql("select getDecimalValueUDF(columnname) from TABLE") to store the values in a DataFrame
    

def getPrecisionUDF(x):
    
        Get Precision of a decimal number or a numeric string.
        Use the getPrecisionUDF. 
        
        Parameters:
        Column of Numeric Strings or Decimal Numbers
        
        Examples: 
        spark.sql("select getPrecisionUDF(columnname) from TABLE").show() to use as a query.
        df=spark.sql("select getPrecisionUDF(columnname) from TABLE") to store the values in a DataFrame
   

matchPatternUDF(x,patterns):
    
        Check if the column of the Table contains given patterns.
        
        Parameters:
        Column: Provide Column of the Table
        patterns: Write all patterns with delimeter '**' . Example: "ar**la"
    
        
        Examples:
        spark.sql("select matchPatternUDF(columnname) from TABLE").show() to use as a query.
        df=spark.sql("select matchPatternUDF(columnname) from TABLE") to store the values in a DataFrame.
    

valueCheckUDF(var,listValues):
    
        Check whether values of a column exist in a given list.
        Use the valueCheckUDF. 
        
        Parameters:
        Column: Provide Column of the Table
        List: Provide a list of values. Examples: ('Value1','Value2')
        
        Examples: 
        spark.sql("select valueCheckUDF(columnname,list) from TABLE").show() to use as a query.
        df=spark.sql("select valueCheckUDF(columnname,list) from TABLE") to store the values in a DataFrame.