maadstml 3.66
Multi-Agent Accelerator for Data Science (MAADS): Transactional Machine Learning
License: MIT License
Keywords: genai, multi-agent, transactional machine learning, artificial intelligence, chatGPT, generative AI, privateGPT, data streams, data science, optimization, prescriptive analytics, machine learning, automl, auto-ml, artificial intelligence, predictive analytics, advanced analytics

**Multi-Agent Accelerator for Data Science Using Transactional Machine Learning (MAADSTML)**

*Revolutionizing Data Stream Science with Transactional Machine Learning*

**Overview**

*MAADSTML combines Artificial Intelligence, ChatGPT, PrivateGPT, Auto Machine Learning with Data Streams Integrated with Apache Kafka (or Redpanda) to create frictionless and elastic machine learning solutions.*  

This library allows users to harness the power of agent-based computing using hundreds of advanced linear and non-linear algorithms. Users can easily integrate Predictive Analytics, Prescriptive Analytics, Pre-Processing, and Optimization in any data stream solution by wrapping additional code around the functions below. It connects with **Apache KAFKA brokers** for cloud based computing using Kafka (or Redpanda) as the data backbone. 

If analysing MILLIONS of IoT devices, you can easily deploy thousands of VIPER/HPDE instances in Kubernetes Cluster in AWS/GCP/Azure. 

It uses VIPER as a **KAFKA connector and seamlessly combines Auto Machine Learning, with Real-Time Machine Learning, Real-Time Optimization and Real-Time Predictions** while publishing these insights in to a Kafka cluster in real-time at scale, while allowing users to consume these insights from anywhere, anytime and in any format. 

It also HPDE as the AutoML technology for TML.  Linux/Windows/Mac versions can be downloaded from [Github](https://github.com/smaurice101/transactionalmachinelearning)

It uses VIPERviz to visualize streaming insights over HTTP(S). Linux/Windows/Mac versions can be downloaded from [Github](https://github.com/smaurice101/transactionalmachinelearning)

MAADSTML details can be found in the book: [Transactional Machine Learning with Data Streams and AutoML](https://www.amazon.com/Transactional-Machine-Learning-Streams-AutoML/dp/1484270223)


To install this library a request should be made to **support@otics.ca** for a username and a MAADSTOKEN.  Once you have these credentials then install this Python library.

**Compatibility**
    - Python 3.8 or greater
    - Minimal Python skills needed

**Copyright**
   - Author: Sebastian Maurice, PhD
   
**Installation**
   - At the command prompt write:
     **pip install maadstml**
     - This assumes you have [Downloaded Python](https://www.python.org/downloads/) and installed it on your computer.  

**MAADS-VIPER Connector to Manage Apache KAFKA:** 
  - MAADS-VIPER python library connects to VIPER instances on any servers; VIPER manages Apache Kafka.  VIPER is REST based and cross-platform that can run on windows, linux, MAC, etc.. It also fully supports SSL/TLS encryption in Kafka brokers for producing and consuming.

**TML is integrated with PrivateGPT (https://github.com/imartinez/privateGPT), which is a production ready GPT, that is 100% Local, 100% Secure and 100% FREE GPT Access.
  - Users need to PULL and RUN one of the privateGPT Docker containers:
  - 	1. Docker Hub: maadsdocker/tml-privategpt-no-gpu-amd64 (without NVIDIA GPU for AMD64 Chip)
  -     2. Docker Hub: maadsdocker/tml-privategpt-with-gpu-amd64 (with NVIDIA GPU for AMD64 Chip)
  - 	3. Docker Hub: maadsdocker/tml-privategpt-no-gpu-arm64 (without NVIDIA GPU for ARM64 Chip)
  -     4. Docker Hub: maadsdocker/tml-privategpt-with-gpu-arm64 (with NVIDIA GPU for ARM64 Chip)
  - Additional details are here: https://github.com/smaurice101/raspberrypi/tree/main/privategpt
  - TML accesses privateGPT container using REST API. 
  - For PrivateGPT production deployments it is recommended that machines have the NVIDIA GPU as this will lead to significant performance improvements.

- **pgptingestdocs**
  - Set Context for PrivateGPT by ingesting PDFs or text documents.  All responses will then use these documents for context.  

- **pgptgetingestedembeddings**
  - After documents are ingested, you can retrieve the embeddings for the ingested documents.  These embeddings allow you to filter the documents for specific context.  

- **pgptchat**
  - Send any prompt to privateGPT (with or without context) and get back a response.  

- **pgptdeleteembeddings**
  - Delete embeddings.  

- **pgpthealth**
  - Check the health of the privateGPT http server.  

- **vipermirrorbrokers**
  - Migrate data streams from (mutiple) brokers to (multiple) brokers FAST!  In one simple function you have the 
    power to migrate from hundreds of brokers with hundreds of topics and partitions to any other brokers
	with ease.  Viper ensures no duplication of messages and translates offsets from last committed.  Every transaction 
	is logged, making data validation and auditability a snap.  You can also increase or decrease partitions and 
	apply filter to topics to copy over.  
	
- **viperstreamquery**
  - Query multiple streams with conditional statements.  For example, if you preprocessed multiple streams you can 
    query them in real-time and extract powerful insights.  You can use >, <, =, AND, OR. 

- **viperstreamquerybatch**
  - Query multiple streams with conditional statements.  For example, if you preprocessed multiple streams you can 
    query them in real-time and extract powerful insights.  You can use >, <, =, AND, OR. Batch allows you to query
	multiple IDs at once.

- **viperlisttopics** 
  - List all topics in Kafka brokers
 
- **viperdeactivatetopic**
  - Deactivate topics in kafka brokers and prevent unused algorithms from consuming storage and computing resources that cost money 

- **viperactivatetopic**
  - Activate topics in Kafka brokers 

- **vipercreatetopic**
  - Create topics in Kafka brokers 
  
- **viperstats**
  - List all stats from Kafka brokers allowing VIPER and KAFKA admins with a end-end view of who is producing data to algorithms, and who is consuming the insights from the algorithms including date/time stamp on the last reads/writes to topics, and how many bytes were read and written to topics and a lot more

- **vipersubscribeconsumer**
  - Admins can subscribe consumers to topics and consumers will immediately receive insights from topics.  This also gives admins more control of who is consuming the insights and allows them to ensures any issues are resolved quickly in case something happens to the algorithms.
  
- **viperunsubscribeconsumer**
  - Admins can unsubscribe consumers from receiving insights, this is important to ensure storage and compute resources are always used for active users.  For example, if a business user leaves your company or no longer needs the insights, by unsubscribing the consumer, the algorithm will STOP producing the insights.

- **viperhpdetraining**
  - Users can do real-time machine learning (RTML) on the data in Kafka topics. This is very powerful and useful for "transactional learnings" on the fly using our HPDE technology.  HPDE will find the optimal algorithm for the data in less than 60 seconds.  

- **viperhpdetrainingbatch**
  - Users can do real-time machine learning (RTML) on the data in Kafka topics. This is very powerful and useful for "transactional learnings" on the fly using our HPDE technology. 
    HPDE will find the optimal algorithm for the data in less than 60 seconds.  Batch allows you to perform ML on multiple IDs at once.

- **viperhpdepredict**
  - Using the optimal algorithm - users can do real-time predictions from streaming data into Kafka Topics.

- **viperhpdepredictprocess**
  - Using the optimal algorithm you can determine object ranking based on input data.  For example, if you want to know which human or machine is the 
    best or worst given input data then this function will return the best or worst human or machine.

- **viperhpdepredictbatch**
  - Using the optimal algorithm - users can do real-time predictions from streaming data into Kafka Topics. Batch allows you to perform predictions
    on multiple IDs at once.
  
- **viperhpdeoptimize**
  -  Users can even do optimization to MINIMIZE or MAXIMIZE the optimal algorithm to find the BEST values for the independent variables that will minimize or maximize the dependent variable.

- **viperhpdeoptimizebatch**
  -  Users can even do optimization to MINIMIZE or MAXIMIZE the optimal algorithm to find the BEST values for the independent variables that will minimize or maximize the dependent 
     variable. Batch allows you to optimize multiple IDs at once.

- **viperproducetotopic**
  - Users can produce to any topics by injesting from any data sources.

- **viperproducetotopicbulk**
  - Users can produce to any topics by injesting from any data sources.  Use this function to write bulk transactions at high speeds.  With the right architecture and
  network you can stream 1 million transactions per second (or more).
  
- **viperconsumefromtopic**
  - Users can consume from any topic and graph the data. 

- **viperconsumefromtopicbatch**
  - Users can consume from any topic and graph the data.  Batch allows you to consume from multiple IDs at once.
  
- **viperconsumefromstreamtopic**
  - Users can consume from a multiple stream of topics at once

- **vipercreateconsumergroup**
  - Admins can create a consumer group made up of any number of consumers.  You can add as many partitions for the group in the Kafka broker as well as specify the replication factor to ensure high availaibility and no disruption to users who consume insights from the topics.

- **viperconsumergroupconsumefromtopic**
  - Users who are part of the consumer group can consume from the group topic.

- **viperproducetotopicstream**
  - Users can join multiple topic streams and produce the combined results to another topic.

- **viperpreprocessrtms**
  - Users can use this function to mesh TEXT data with TML machine learning output to cross-reference entities with TEXT files like log files.
    This function is a very powerful function that incorporates "past memory"  of real-time events using sliding time windows. 
    For details see: [How TML incorporates real-time memory using sliding time windows](https://tml.readthedocs.io/en/latest/rtms.html#how-tml-maintains-past-memory-of-events-using-sliding-time-windows)  

- **viperpreprocessproducetotopicstream**
  - Users can pre-process data streams using the following functions: MIN, MAX, AVG, COUNT, COUNTSTR, DIFF, DIFFMARGIN, SUM, MEDIAN, VARIANCE, OUTLIERS, OUTLIERSX-Y,VARIED, 
    ANOMPROB,ANOMPROBX-Y,ENTROPY, AUTOCORR, TREND, CONSISTENCY, IQR (InterQuartileRange), Midhinge, GM (Geometric mean), HM (Harmonic mean), Trimean, 
	CV (coefficient of Variation),Mad (Mean absolute deviation), Skewness, Kurtosis, Spikedetect, Unique, Uniquestr, Timediff: time should be in this 
	layout:2006-01-02T15:04:05, Timediff returns the difference in seconds between the first date/time and last datetime. Avgtimediff returns the 
    average time in seconds between consecutive dates.. Spikedetect uses a Zscore method to detect 
	spikes in the data using lag of 5, StD of 3.5 from mean and influence of 0.5.  Geodiff (returns distance in Kilometers between two lat/long points)
	
    Dataage_[UTC offset]_[timetype], dataage can be used to check the last update time of the data in the data stream from
	current local time.  You can specify the UTC offset to adjust the current time to match the timezone of the data stream.
	You can specify timetype as millisecond, second, minute, hour, day.  For example, if Dataage_1_minute, then this processtype
	will compare the last timestamp in the data stream, to the local UTC time offset +1 and compute the time difference
	between the data stream timestamp and current local time and return the difference in minutes.  This is a very powerful processtype
	for data quality and data assurance programs for any number of data streams.
		
	Unique Checks numeric data for duplication.  Returns 1 if no data duplication (unique), 0 otherwise.
 
    Uniquestr Checks string data for duplication.  Returns 1 if no data duplication (unique), 0 otherwise.

    Uniquecount Checks numeric data for duplication.  Returns count of unique numbers.
 
    Uniquestrcount Checks string data for duplication.  Returns count of unique strings.
	
    CONSISTENCY checks if the data all have consistent data types. Returns 1 for consistent data types, 0 otherwise.
	
	Meanci95 or Meanci99 - returns a 95% or 99% confidence interval: mean, low, high 

    RAW for no processing.
	
    ANOMPROB=Anomaly Probability, it will run several algorithms on the data stream window to determine a probability percentage of 
	anomalous behaviour.  This can be cross-referenced with other process types. This is very useful if you want to extract aggregate 
	values that you can then use to build TML models and/or make decisions to prevent issues.  ENTROPY will compute the amount of information
	in the data stream.  AUTOCORR will run a autocorrelation regression: Y = Y (t-1), to indicate how previous value correlates with future 
    value.  TREND will run a linear regression of Y = f(Time), to determine if the data in the stream are increasing or decreasing.	

    ANOMPROBX-Y (similar to OUTLIERSX-Y), where X and Y are numbers or "n", if "n" means examine all anomalies for recurring patterns.
	They allow you to check if the anomalies in the streams are truly anomalies and not some
    pattern.  For example, if a IoT device shuts off and turns on again routinely, this may be picked up as an anomaly when in fact
    it is normal behaviour.  So, to ignore these cases, if ANOMPROB2-5, this tells Viper, check anomalies with patterns of 2-5 peaks.
    If the stream has two classes and these two classes are like 0 and 1000, and show a pattern, then they should not be considered an anomaly.
    Meaning, class=0, is the device shutting down, class=1000 is the device turning back on.  If ANOMPROB3-10, Viper will check for 
    patterns of classes 3 to 10 to see if they recur routinely.  This is very helpful to reduce false positives and false negatives.

- **viperpreprocessbatch**
  - This function is similar to *viperpreprocessproducetotopicstream* the only difference is you can specify multiple
    tmlids in Topicid field. This allows you to batch process multiple tmlids at once.  This is very useful if using
	kubernetes architecture.

- **vipercreatejointopicstreams**
  - Users can join multiple topic streams
  
- **vipercreatetrainingdata**
  - Users can create a training data set from the topic streams for Real-Time Machine Learning (RTML) on the fly.

- **vipermodifyconsumerdetails**
  - Users can modify consumer details on the topic.  When topics are created an admin must indicate name, email, location and description of the topic.  This helps to better manage the topic and if there are issues, the admin can contact the individual consuming from the topic.
  
- **vipermodifytopicdetails**
  - Users can modify details on the topic.  When topics are created an admin must indicate name, email, location and description of the topic.  This helps to better manage the topic and if there are issues, the admin can contact the developer of the algorithm and resolve issue quickly to ensure disruption to consumers is minimal.
 
- **vipergroupdeactivate**
  - Admins can deactive a consumer group, which will stop all insights being delivered to consumers in the group.
  
- **vipergroupactivate**
  - Admins can activate a group to re-start the insights.
 
- **viperdeletetopics**
  - Admins can delete topics in VIPER database and Kafka clusters.
		
- **viperanomalytrain**
  - Perform anomaly/peer group analysis on text or numeric data stream using advanced unsupervised learning. VIPER automatically joins 
    streams, and determines the peer group of "usual" behaviours using proprietary algorithms, which are then used to predict anomalies with 
	*viperanomalypredict* in real-time.  Users can use several parameters to fine tune the peer groups.  
	
	*VIPER is one of the very few, if not only, technology to do anomaly/peer group analysis using unsupervised learning on data streams 
	with Apache Kafka.*

- **viperanomalytrainbatch**
  - Batch allows you to perform anomaly training on multiple IDs at once.

- **viperanomalypredict**
  - Predicts anomalies for text or numeric data using the peer groups found with *viperanomalytrain*.  VIPER automatically joins streams
  and compares each value with the peer groups and determines if a value is anomalous in real-time.  Users can use several parameters to fine tune
  the analysis. 
  
  *VIPER is one of the very few, if not only, technology to do anomaly detection/predictions using unsupervised learning on data streams
  with Apache Kafka.*
		
- **viperanomalypredictbatch**
  - Batch allows you to perform anomaly prediction on multiple IDs at once.
				
- **viperstreamcorr**
  - Performs streaming correlations by joining multiple data streams with 2 variables.  Also performs cross-correlations with 4 variables.
    This is a powerful function and can offer important correlation signals between variables.   Will also correlate TEXT using 
    natural language processing (NLP).	

- **viperpreprocesscustomjson**
  - Immediately start processing ANY RAW JSON data in minutes.  This is useful if you want to start processing data quickly.  

- **viperstreamcluster**
  - Perform cluster analysis on streaming data.  This uses K-Means clustering with Euclidean or EuclideanSquared algorithms to compute 
    distance.  It is a very useful function if you want to determine common behaviours between devices, patients, or other entities.
	Users can also setup email alerts if specific clusters are found.

- **vipersearchanomaly**
  - Perform advanced analysis for user search.  This function is useful if you want to monitor what people are searching for, and determine
    if the searches are anamolous and differ from the peer group of "normal" search behaviour.

- **vipernlp**
  - Perform advanced natural language summary of PDFs.

- **viperchatgpt**
  - Start a conversation with ChatGPT in real-time and stream responses.

- **viperexractpdffields**
  - Extracts fields from PDF file

- **viperexractpdffieldbylabel**
  - Extracts fields from PDF file by label name.

- **videochatloadresponse**
  - Analyse videos with video chatgpt.  This is a powerful GPT LLM that will understand and reason with videos frame by frame.  
    It will also understand the spatio-temporal frames in the video.  Video gpt runs in a container. 

- **areyoubusy**
  - If deploying thousands of VIPER/HPDE binaries in a Kubernetes cluster - you can broadcast a 'areyoubusy' message to all VIPER and HPDE
    binaries, and they will return back the HOST/PORT if they are NOT busy with other tasks.  This is very convenient for dynamically managing  
	enormous load among VIPER/HPDE and allows you to dynamically assign HOST/PORT to **non-busy** VIPER/HPDE microservices.

**First import the Python library.**

**import maadstml**


**1. maadstml.viperstats(vipertoken,host,port=-999,brokerhost='',brokerport=-999,microserviceid='')**

**Parameters:**	

*VIPERTOKEN* : string, required

- A token given to you by VIPER administrator.

*host* : string, required
       
- Indicates the url where the VIPER instance is located and listening.

*port* : int, required

- Port on which VIPER is listenting.


*brokerhost* : string, optional

- Address where Kafka broker is running - if none is specified, the Kafka broker address in the VIPER.ENV file will be used.


*brokerport* : int, optional

- Port on which Kafka is listenting.

*microserviceid* : string, optional

- If you are routing connections to VIPER through a microservice then indicate it here.

RETURNS: A JSON formatted object of all the Kafka broker information.

**2. maadstml.vipersubscribeconsumer(vipertoken,host,port,topic,companyname,contactname,contactemail,
		location,description,brokerhost='',brokerport=-999,groupid='',microserviceid='')**

**Parameters:**	

*VIPERTOKEN* : string, required

- A token given to you by VIPER administrator.

*host* : string, required
       
- Indicates the url where the VIPER instance is located and listening.

*port* : int, required

- Port on which VIPER is listenting.

*topic* : string, required

- Topic to subscribe to in Kafka broker

*companyname* : string, required

- Company name of consumer

*contactname* : string, required

- Contact name of consumer

*contactemail* : string, required

- Contact email of consumer

*location* : string, required

- Location of consumer

*description* : string, required

- Description of why consumer wants to subscribe to topic

*brokerhost* : string, optional

- Address of Kafka broker - if none is specified it will use broker address in VIPER.ENV file

*brokerport* : int, optional

- Port Kafka is listening on - if none is specified it will use port in the VIPER.ENV file

*groupid* : string, optional

- Subscribe consumer to group

*microserviceid* : string, optional

- If you are routing connections to VIPER through a microservice then indicate it here.

RETURNS: Consumer ID that the user must use to receive insights from topic.


**3. maadstml.viperunsubscribeconsumer(vipertoken,host,port,consumerid,brokerhost='',brokerport=-999,
	microserviceid='')**

**Parameters:**	

*VIPERTOKEN* : string, required

- A token given to you by VIPER administrator.

*host* : string, required
       
- Indicates the url where the VIPER instance is located and listening.

*port* : int, required

- Port on which VIPER is listenting.

*consumerid* : string, required
       
- Consumer id to unsubscribe

*brokerhost* : string, optional

- Address of Kafka broker - if none is specified it will use broker address in VIPER.ENV file

*brokerport* : int, optional

- Port Kafka is listening on - if none is specified it will use port in the VIPER.ENV file

RETURNS: Success/failure 

**4. maadstml.viperproducetotopic(vipertoken,host,port,topic,producerid,enabletls=0,delay=100,inputdata='',maadsalgokey='',
	maadstoken='',getoptimal=0,externalprediction='',subtopics='',topicid=-999,identifier='',array=0,brokerhost='',
	brokerport=-999,microserviceid='')**

**Parameters:**	

*VIPERTOKEN* : string, required

- A token given to you by VIPER administrator.

*host* : string, required
       
- Indicates the url where the VIPER instance is located and listening.

*port* : int, required

- Port on which VIPER is listenting.

*topic* : string, required

- Topic or Topics to produce to.  You can separate multiple topics by a comma.  If using multiple topics, you must 
  have the same number of producer ids (separated by commas), and same number of externalprediction (separated by
  commas).  Producing to multiple topics at once is convenient for synchronizing the timing of 
  streams for machine learning.

*subtopic* : string, optional

- Enter sub-topic streams.  This is useful if you want to reduce the number of topics/partitions in Kafka by adding
  sub-topics in the main topic.  

*topicid* : int, optional

- Topicid represents an id for some entity.  For example, if you have 1000 IoT devices, with 10 subtopic streams 
  you can assign a Topicid to each IoT device and each of the 10 subtopics will be associated to each IoT device.
  This way, you do not create 10,000 streams, but just 1 Main Topic stream, and VIPER will add the 10,000 streams
  in the one topic.  This will also drastically reduce the partition costs.  You can also create custom machine 
  learning models, predictions, and optimization for each 1000 IoT devices quickly: **It is very powerful.**

"array* : int, optional

- You can stream multiple variables at once, and use array=1 to specify that the streams are an array.
  This is similar to streaming 1 ROW in a database, and useful if you want to synchonize variables for machine learning.  
  For example, if a device produces 3 streams: stream A, stream B, stream C, and rather than streaming A, B, C separately
  you can add them to subtopic="A,B,C", and externalprediction="value_FOR_A,value_FOR_B,value_FOR_C", then specify
  array=1, then when you do machine learning on this data, the variables A, B, C are date/time synchronized
  and you can choose which variable is the depdendent variable in viperhpdetraining function.


*identifier* : string, optional

- You can add any string identifier for the device.  For examaple, DSN ID, IoT device id etc.. 

*producerid* : string, required
       
- Producer ID of topic to produce to in the Kafka broker

*enabletls* : int, optional
       
- Set to 1 if Kafka broker is enabled with SSL/TLS encryption, otherwise 0 for plaintext.

*delay*: int, optional

- Time in milliseconds from VIPER backsout from writing messages

*inputdata* : string, optional

- This is the inputdata for the optimal algorithm found by MAADS or HPDE

*maadsalgokey* : string, optional

- This should be the optimal algorithm key returned by maadstml.dotraining function.

*maadstoken* : string, optional
- If the topic is the name of the algorithm from MAADS, then a MAADSTOKEN must be specified to access the algorithm in the MAADS server

*getoptimal*: int, optional
- If you used the maadstml.OPTIMIZE function to optimize a MAADS algorithm, then if this is 1 it will only retrieve the optimal results in JSON format.

*externalprediction* : string, optional
- If you are using your own custom algorithms, then the output of your algorithm can be still used and fed into the Kafka topic.

*brokerhost* : string, optional

- Address of Kafka broker - if none is specified it will use broker address in VIPER.ENV file

*brokerport* : int, optional

- Port Kafka is listening on - if none is specified it will use port in the VIPER.ENV file

*microserviceid* : string, optional

- If you are routing connections to VIPER through a microservice then indicate it here.

RETURNS: Returns the value produced or results retrieved from the optimization.

**4.1. maadstml.viperproducetotopicbulk(vipertoken,host,port,topic,producerid,inputdata,partitionsize=100,enabletls=1,delay=100,
        brokerhost='',brokerport=-999,microserviceid='')**

**Parameters:**	

*VIPERTOKEN* : string, required

- A token given to you by VIPER administrator.

*host* : string, required
       
- Indicates the url where the VIPER instance is located and listening.

*port* : int, required

- Port on which VIPER is listenting.

*topic* : string, required

- Topic or Topics to produce to.  You can separate multiple topics by a comma.  If using multiple topics, you must 
  have the same number of producer ids (separated by commas), and same number of externalprediction (separated by
  commas).  Producing to multiple topics at once is convenient for synchronizing the timing of 
  streams for machine learning.

*producerid* : string, required
       
- Producer ID of topic to produce to in the Kafka broker.  Separate multiple producer ids with comma.

*inputdata* : string, required
       
- You can write multiple transactions to each topic.  Each group of transactions must be separated by a tilde.  
  Each transaction in the group must be separate by a comma.  The number of groups must match the producerids and 
  topics.  For example, if you are writing to two topics: topic1,topic2, then the inputdata should be:
  trans1,transn2,...,transnN~trans1,transn2,...,transnN.  The number of transactions and topics can be any number.
  This function can be very powerful if you need to analyse millions or billions of transactions very quickly.

*partitionsize* : int, optional

- This is the number of partitions of the inputdata.  For example, if your transactions=10000, then VIPER will 
  create partitions of size 100 (if partitionsize=100) resulting in 100 threads for concurrency.  The higher
  the partitionsize, the lower the number of threads.  If you want to streams lots of data fast, then a 
  partitionzie of 1 is the fastest but will come with overhead because more RAM and CPU will be consumed.

*enabletls* : int, optional
       
- Set to 1 if Kafka broker is enabled with SSL/TLS encryption, otherwise 0 for plaintext.

*delay*: int, optional

- Time in milliseconds from VIPER backsout from writing messages

*brokerhost* : string, optional

- Address of Kafka broker - if none is specified it will use broker address in VIPER.ENV file

*brokerport* : int, optional

- Port Kafka is listening on - if none is specified it will use port in the VIPER.ENV file

*microserviceid* : string, optional

- If you are routing connections to VIPER through a microservice then indicate it here.

RETURNS: None

**5. maadstml.viperconsumefromtopic(vipertoken,host,port,topic,consumerid,companyname,partition=-1,enabletls=0,delay=100,offset=0,
	brokerhost='',brokerport=-999,microserviceid='',topicid='-999',rollbackoffsets=0,preprocesstype='')**

**Parameters:**	

*VIPERTOKEN* : string, required

- A token given to you by VIPER administrator.

*host* : string, required
       
- Indicates the url where the VIPER instance is located and listening.

*port* : int, required

- Port on which VIPER is listenting.

*topic* : string, required
       
- Topic to consume from in the Kafka broker

*preprocesstype* : string, optional

- If you only want to search for record that have a particular processtype, you can enter:
  MIN, MAX, AVG, COUNT, COUNTSTR, DIFF, DIFFMARGIN, SUM, MEDIAN, VARIANCE, OUTLIERS, OUTLIERSX-Y, VARIED, ANOMPROB,ANOMPROBX-Y,ENTROPY, 
  AUTOCORR, TREND, CONSISTENCY, Unique, Uniquestr, Geodiff (returns distance in Kilometers between two lat/long points)
  IQR (InterQuartileRange), Midhinge, GM (Geometric mean), HM (Harmonic mean), Trimean, CV (coefficient of Variation), 
  Mad (Mean absolute deviation), Skewness, Kurtosis, Spikedetect, Timediff: time should be in this layout:2006-01-02T15:04:05,
  Timediff returns the difference in seconds between the first date/time and last datetime. Avgtimediff returns the 
  average time in seconds between consecutive dates.
  Spikedetect uses a Zscore method to detect spikes in the data using lag of 5, StD of 3.5 from mean and influence of 0.5.   

  Dataage_[UTC offset]_[timetype], dataage can be used to check the last update time of the data in the data stream from
  current local time.  You can specify the UTC offset to adjust the current time to match the timezone of the data stream.
  You can specify timetype as millisecond, second, minute, hour, day.  For example, if Dataage_1_minute, then this processtype
  will compare the last timestamp in the data stream, to the local UTC time offset +1 and compute the time difference
  between the data stream timestamp and current local time and return the difference in minutes.  This is a very powerful processtype
  for data quality and data assurance programs for any number of data streams.

  Unique Checks numeric data for duplication.  Returns 1 if no data duplication (unique), 0 otherwise.

  Uniquestr Checks string data for duplication.  Returns 1 if no data duplication (unique), 0 otherwise.

  Uniquecount Checks numeric data for duplication.  Returns count of unique numbers.
 
  Uniquestrcount Checks string data for duplication.  Returns count of unique strings.

  CONSISTENCY checks if the data all have consistent data types. Returns 1 for consistent data types, 0 otherwise.
  
  Meanci95 or Meanci99 - returns a 95% or 99% confidence interval: mean, low, high 

  RAW for no processing.
  
  ANOMPROB=Anomaly probability,
  it will run several algorithms on the data stream window to determine a probaility of anomalous
  behaviour.  This can be cross-refenced with OUTLIERS.  It can be very powerful way to detection
  issues with devices.
  
  ANOMPROBX-Y (similar to OUTLIERSX-Y), where X and Y are numbers, or "n".  If "n", means examine all anomalies for patterns.
  They allow you to check if the anomalies in the streams are truly anomalies and not some
  pattern.  For example, if a IoT device shuts off and turns on again routinely, this may be picked up as an anomaly when in fact
  it is normal behaviour.  So, to ignore these cases, if ANOMPROB2-5, this tells Viper, check anomalies with patterns of 2-5 peaks.
  If the stream has two classes and these two classes are like 0 and 1000, and show a pattern, then they should not be considered an anomaly.
  Meaning, class=0, is the device shutting down, class=1000 is the device turning back on.  If ANOMPROB3-10, Viper will check for 
  patterns of classes 3 to 10 to see if they recur routinely.  This is very helpful to reduce false positives and false negatives.

  
*topicid* : string, optional

- Topicid represents an id for some entity.  For example, if you have 1000 IoT devices, you can consume on a per device by entering
  its topicid  that you gave when you produced the topic stream. Or, you can read from multiple topicids at the same time.  
  For example, if you have 10 ids, then you can specify each one separated by a comma: 1,2,3,4,5,6,7,8,9,10
  VIPER will read topicids in parallel.  This can drastically speed up consumption of messages but will require more 
  CPU.

*rollbackoffsets* : int, optional, enter value between 0 and 100

- This will rollback the streams by this percentage.  For example, if using topicid, the main stream is rolled back by this
  percentage amount.

*consumerid* : string, required

- Consumer id associated with the topic

*companyname* : string, required

- Your company name

*partition* : int, optional

- set to Kafka partition number or -1 to autodect

*enabletls*: int, optional

- Set to 1 if Kafka broker is SSL/TLS enabled for encrypted traffic, otherwise set to 0 for plaintext.

*delay*: int, optional

- Time in milliseconds before VIPER backsout from reading messages

*offset*: int, optional

- Offset to start the reading from..if 0 then reading will start from the beginning of the topic. If -1, VIPER will automatically 
  go to the last offset.  Or, you can extract the LastOffet from the returned JSON and use this offset for your next call.  

*brokerhost* : string, optional

- Address of Kafka broker - if none is specified it will use broker address in VIPER.ENV file

*brokerport* : int, optional

- Port Kafka is listening on - if none is specified it will use port in the VIPER.ENV file

*microserviceid* : string, optional

- If you are routing connections to VIPER through a microservice then indicate it here.

RETURNS: Returns a JSON object of the contents read from the topic.

**5.1 maadstml.viperconsumefromtopicbatch(vipertoken,host,port,topic,consumerid,companyname,partition=-1,enabletls=0,delay=100,offset=0,
	brokerhost='',brokerport=-999,microserviceid='',topicid='-999',rollbackoffsets=0,preprocesstype='',timedelay=0,asynctimeout=120)**

**Parameters:**	

*VIPERTOKEN* : string, required

- A token given to you by VIPER administrator.

*host* : string, required
       
- Indicates the url where the VIPER instance is located and listening.

*port* : int, required

- Port on which VIPER is listenting.

*asynctimeout* : int, optional
 
  -This is the timeout in seconds for the Python library async function.

*timedelay* : int, optional

 - Timedelay is in SECONDS. Because batch runs continuously in the background, this will cause Viper to pause 
   *timedelay* seconds when reading and writing to Kafka.  For example, if the raw data is being generated
   every 3600 seconds, it may make sense to set timedelay=3600
 
*topic* : string, required
       
- Topic to consume from in the Kafka broker

*preprocesstype* : string, optional

- If you only want to search for record that have a particular processtype, you can enter:
  MIN, MAX, AVG, COUNT, COUNTSTR, DIFF, DIFFMARGIN, SUM, MEDIAN, VARIANCE, OUTLIERS, OUTLIERSX-Y, VARIED, ANOMPROB,ANOMPROBX-Y,ENTROPY, AUTOCORR, TREND, 
  IQR (InterQuartileRange), Midhinge, CONSISTENCY, GM (Geometric mean), HM (Harmonic mean), Trimean, CV (coefficient of Variation), 
  Mad (Mean absolute deviation), Skewness, Kurtosis, Spikedetect, Unique, Uniquestr, Timediff: time should be in this layout:2006-01-02T15:04:05,
  Timediff returns the difference in seconds between the first date/time and last datetime. Avgtimediff returns the 
  average time in seconds between consecutive dates. 
  Spikedetect uses a Zscore method to detect spikes in the data using lag of 5, StD of 3.5 from mean and influence of 0.5.   
  Geodiff (returns distance in Kilometers between two lat/long points)
  Unique Checks numeric data for duplication.  Returns 1 if no data duplication (unique), 0 otherwise.

  Dataage_[UTC offset]_[timetype], dataage can be used to check the last update time of the data in the data stream from
  current local time.  You can specify the UTC offset to adjust the current time to match the timezone of the data stream.
  You can specify timetype as millisecond, second, minute, hour, day.  For example, if Dataage_1_minute, then this processtype
  will compare the last timestamp in the data stream, to the local UTC time offset +1 and compute the time difference
  between the data stream timestamp and current local time and return the difference in minutes.  This is a very powerful processtype
  for data quality and data assurance programs for any number of data streams.

  Uniquestr Checks string data for duplication.  Returns 1 if no data duplication (unique), 0 otherwise.

  Uniquecount Checks numeric data for duplication.  Returns count of unique numbers.
 
  Uniquestrcount Checks string data for duplication.  Returns count of unique strings.
  
  CONSISTENCY checks if the data all have consistent data types. Returns 1 for consistent data types, 0 otherwise.

  Meanci95 or Meanci99 - returns a 95% or 99% confidence interval: mean, low, high 

  RAW for no processing.

  ANOMPROB=Anomaly probability,
  it will run several algorithms on the data stream window to determine a probaility of anomalous
  behaviour.  This can be cross-refenced with OUTLIERS.  It can be very powerful way to detection
  issues with devices.
  
  ANOMPROBX-Y (similar to OUTLIERSX-Y), where X and Y are numbers, or "n".  If "n", means examine all anomalies for patterns.
  They allow you to check if the anomalies in the streams are truly anomalies and not some
  pattern.  For example, if a IoT device shuts off and turns on again routinely, this may be picked up as an anomaly when in fact
  it is normal behaviour.  So, to ignore these cases, if ANOMPROB2-5, this tells Viper, check anomalies with patterns of 2-5 peaks.
  If the stream has two classes and these two classes are like 0 and 1000, and show a pattern, then they should not be considered an anomaly.
  Meaning, class=0, is the device shutting down, class=1000 is the device turning back on.  If ANOMPROB3-10, Viper will check for 
  patterns of classes 3 to 10 to see if they recur routinely.  This is very helpful to reduce false positives and false negatives.

  
*topicid* : string, required

- Topicid represents an id for some entity.  For example, if you have 1000 IoT devices, you can consume on a per device by entering
  its topicid  that you gave when you produced the topic stream. Or, you can read from multiple topicids at the same time.  
  For example, if you have 10 ids, then you can specify each one separated by a comma: 1,2,3,4,5,6,7,8,9,10
  VIPER will read topicids in parallel.  This can drastically speed up