Data type support & mapping for MariaDB and MySQL sourcesSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideData type support & mapping for schema conversion & evolutionData type support & mapping for MariaDB and MySQL sourcesPrevNextData type support & mapping for MariaDB and MySQL sourcesMariaDB / MySQL source typeBigQueryDatabricksMySQLOraclePostgreSQLSnowflakeSpannerSQL Server / Azure Synapsebigint / bigint unsignedint64 / numericbigintbigint / bigint unsignedintbigInt / double precisionintegerint64 / float64bigInt / numeric(38)binarybytesbinarybinary(s)blobbyteabinarybytesbinary(s)bit / bit(n)not supportednot supportedbit / bit(n)blobbit / bit varying(s)binarystringvarchar(n)blobbytesbinarylongblobblobbyteabinarybytes(65535)bytes(max)varbinary(p) if <= 8000varbinary(max) if > 8000char(s)stringstringcharacter(s) if s<255longtext if s>255character(s) if s<=2000clob if s>2000character(s)character(s)string(s)character(s) if s<=8000varchar(max) if s>8000datedatedatedatedatedatedatedatedatedatetimedatetimetimestampdatetimetimestamptimestamp without time zonedatetimetimestampdatetime2decimalnumericdoubledecimal(10,0)number(10,0)numeric(10,0)numeric(10,0)float64numeric(10,0)decimal(p,s)numeric if p<=38 and s<=8string if p>38 or s>9double if p<=38 string if p>38decimal(p,s) if p<=65 and s<=30text if p>65 or s>30number(p,s) if p<=38 and s<=127varchar2(1000) if p>38 or s>127numeric(p,s)numeric(p,s) if p<=38 snf s<=37varchar if p>38 or s>37float64 if p<=308 and s<=15string(max) if p>308 pr s>15decimal(p,s) if p<=38 and s<=38varchar(8000) if p>38 or s>38decimal(p)numeric if p<=38string if p>38double if p<=38 string if p>38decimal(p) if p<=65text if p>65number(p) if p<=38varchar2(1000) if p>38numeric(p)numeric(p) if p<=38varchar if p>38float64 if p<=308string(max) if p>308decimal(p) if p<=38varchar(8000) if p>38doublefloat64doubledoubledouble precisiondouble precisiondouble precisionfloat64double precisiondouble(p,s)float64doubledoubledouble precisiondouble precisiondouble precisionfloat64 if p<=308 and s<=15string(max) if p>308 or s>15floatenumnot supportednot supportednot supportednot supportednot supportedvarcharnot supportednot supportedfloatfloat64doubledecimalfloatdouble precisionfloatfloat64floatfloat(p,s)float64floatfloat(p) if p<=38double if p>38float(p)double precisionfloat(p) if p<=38float if p>38float64float(p) if p<=53varchar(8000) if p>53int / int unsignedint64bigintint / int unsignedintbigIntintegerint64integerjsonstringstringjsonclobjsonvariantstring(max)varchar if <= 8000varchar(max) > 8000longblobbytesbinarylongblobblobbyteabinarybytes(max)varbinary(p) if <= 8000varbinary(max) for > 8000longtextstringstringtinytext / longtextclobtextvarcharstring(max)character(s) if s<=8000varchar(max) if s>8000mediumblobbytesbinarylongblobblobbyteabinarybytes(max)varbinary(p) if <= 8000varbinary(max) for > 8000mediumInt / mediumInt unsignedint64bigintmediumInt / mediumInt unsignedintintegerintegerint64integermediumtextstringstringtinytext / longtextclobtextvarcharstring(max)character(s) if s<=8000varchar(max) if s>8000smallint / smallint unsignedint64bigintsmallint / smallint unsignedintsmallintintegerint64smallinttextstringstringtextclobtextvarcharstring(s)varchar if s<=8000varchar(max) if s>8000timetimetimestamptimevarchar2(200)timetimestring(max)timetimestamptimestamptimestamptimestamptimestamp with time zonetimestamp without time zonetimestamp with timezonetimestampdatetimeOffsettinyblobbytesbinarylongblobblobbyteabinarybytes(255)bytes(max)varbinary(p) if <= 8000varbinary(max) for > 8000tinyint / tinyint unsignedint64biginttinyint / tinyint unsignedblobsmallintintegerint64tinyInttinyint(1)booleanbiginttinyint(1)not supportedbooleanbinarystringnot supportedtinytextstringstringtinytextclobtextvarcharstring(s)character(s) if s<=8000varchar(max) if s>8000varbinary(s)bytesbinaryvarbinary(s)blobcharacter varying(s)varbinarybytes(s)varbinary(s) if s<=8000varbinary(max) if s>8000varchar(s)stringstringvarchar(s)varchar2(s) if s<=4000clob if s>4000character varying(s)varchar(s)string(s)varchar(s) if s<=8000varchar(max) if s>8000yearnot supportednot supportednot supportednot supportednot supportednumeric(38,0)not supportednot supportedMySQL spatial types are not supported.See Using Data Types from Other Database Engines for a list of supported data type aliases (such as boolean and numeric).In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2023-06-02


Data type support & mapping for Oracle sourcesSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideData type support & mapping for schema conversion & evolutionData type support & mapping for Oracle sourcesPrevNextData type support & mapping for Oracle sourcesOracle source typeBigQueryDatabricksMySQLOraclePostgreSQLSnowflakeSpannerSQL Server / Azure Synapsebinary_doublefloat64doubledoubledouble precisiondouble Precisionfloatfloat64float(s)binary_floatfloat64floatfloat(s)floatrealfloatfloat64float(s)blobbytesbinarylongblobblobbyteabinarybytes(p)bytes(max) if p > 10485760varbinary(p) if <= 8000varbinary(max) if > 8000characterstringstringcharacter(1)character(1)character(1)character(1)string(1)character(1)character(s)stringstringcharacter(s)longtext if (s)>255character(s)character(s)character(s)string(s)character(s)clobstringstringtextclobtextvarcharstring(4000)string(max) if (s) > 2621440varchar(s)varchar(max) if (s) > 8000datedatedatedatetimedatetimestamp without time zonedatedatedatetime2floatfloat64floatdoublefloatdouble precisionfloatfloat64varchar(8000)float(s)float64floatfloat(s)floatdouble precisionfloatfloat64float(s)intnumericdecimal(38)decimal(38,0)number(38,0)numeric(38,0)numeric(38,0)float64numeric(38,0)interval day to secondstringstringvarchar(100)interval day(2) to second(6)character varyingvarcharstring(max)varchar(100)interval year to Monthstringstringvarchar(100)interval year(2) to monthcharacter varyingvarcharstring(max)varchar(100)long*bytesbinarylongbloblongblobbyteabinarybytes(max)varbinary(max)long_raw*bytesbinarylongbloblongblobbyteabinarybytes(max)varbinary(max)nchar(s)stringstringnvarchar(s)nchar(s)character(s)character(s)string(s)nchar(s)nclobstringstringlongtextnclobtextvarcharstring(s)string(max) if > 2621440nvarchar if <= 4000nvarchar(max) if > 4000numbernumericdecimal(38)decimal(65)numbernumericnumericfloat64numeric(38)number(p,s)numericstring if (s)>9decimal(p,s) (if p<=38 and s<=37)string (if p>38 or s>37)decimal(p,s)numeric(p,s)numeric (p,s)numeric(p,s)numericstring(max) if (s)>15numeric(p,s)number(p)numericdecimal(p,0) (if p <= 38)string (if p>38)decimal(p,0)number(p,0)numeric(p,0)numeric(p,0)float64numeric(p,0)nvarchar2(s)stringstringnvarchar(s)nvarchar2(s)character varying(s)varchar(s)string(s)nvarchar(s)Object typenot supportedNot Supportednot supportednot supportednot supportednot supportednot supportednot supportedrawbytesbinarylongblobblobbyteabinarybytes(p)bytes(max) if > 10485760varbinary(p) if <= 8000varbinary(max) if > 8000timestampdatetimetimestampdatetimetimestamptimestamp without timezonetimestamptimestampdatetime2timestamp with local time zonetimestamptimestampdatetimetimestamp with time zonetimestamp with time zonetimestamp with time zonetimestampdatetimeOffsettimestamp with time zonetimestamptimestampdatetimetimestamp with time zonetimestamp with time zonetimestamp with time zonetimestampdatetimeoffsetvarcharstringstringvarchar(s)Mapped with varchar(s)character varying(s)Mapped with varchar(s)string(s)varchar(s)varchar2(s)stringstringvarchar(s)varchar2(s)character Varying(s)varchar(s)string(s)varchar(s)xmlstringstringlongtextxmltypexmlvarcharstring(max)xml*OJet supports these types, but Oracle Reader does not. Oracle suggests using CLOB in place of LONG or LONG RAW.Oracle source types ADT, ARRAY, BFILE, LONG, LONG RAW, NESTED TABLE, REF, ROWID, SD0_GEOMETRY, UDT, and UROWID are not supported.VARCHAR with a JSON constraint is not supported in this release (DEV-26815).In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2023-01-20


Data type support & mapping for PostgreSQL sourcesSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideData type support & mapping for schema conversion & evolutionData type support & mapping for PostgreSQL sourcesPrevNextData type support & mapping for PostgreSQL sourcesPostgreSQL source typeBigQueryDatabricksMySQLOraclePostgreSQLSnowflakeSpannerSQL Server / Azure Synapsebigint / int8int64bigintbigintintbingintintegerint64bigintbigserial / serial8int64bigintbigintintbigserialintegerint64bigintbit varying(s)not supportednot supportednot supportednot supportedbit varying(s)binarynot supportedvarbinary(10)bit(s)not supportednot supportedbit(s) bit(64)not supportedbit varying(s)binarynot supportedvarchar(s)booleanbooleanbooleanboolnot supportedbooleanbooleanboolbitbyteabytesbinarylongblobblobbyteabinarybytes(max)varbinary(max)character varying(s)stringstringvarchar(s)if (s) <65535 longtextvarchar2(s)character varying(s)if (s) <65535 longtextvarchar(s)string(s)varchar(s)character(s)stringstringcharacter(s)char(s)bpchar(s)character(s)string(s)character(s)cidrnot supportednot supportednot supportednot supportedcidrnot supportednot supportednot supporteddatedatedatedatedatedatedatedatedatedouble precisionnumericdoubledoubledouble precisiondouble precisiondouble precisionfloat64float(s)floatfloat64floatfloat(s)floatdouble precisionfloatfloat64float(17)inetnot supportednot supportednot supportednot supportedinetnot supportednot supportednot supportedinteger / int4int64bigintintegerintintegerintegerint64integerintervalstringstringvarchar(100)varchar2(100)intervalvarcharstring(max)varchar(100)jsonstringstringjsonclobjsonvariantstring(max)varchar(max)jsonbstringstringjsonclobjsonbvariantstring(max)varchar(max)macaddrnot supportednot supportednot supportednot supportedmacaddrnot supportednot supportednot supportednumeric / decimalnumericdecimal(38)decimal(65)numbernumericnumericfloat64numeric(38)numeric(p,s) / decimal(p,s)stringdecimal(p,s) if p<=38, s<=37string if p>38 or s>37decimal(p,s)number(p,s)numeric(p,s)numeric(p,s)float64numeric(p,s)numeric(p) / decimal(p)numericdecimal(p,0) if p<=38string if p>38decimal(p,0)numeric(p,0)numeric(p,0)numeric(p,0)float64numeric(p,0)pg_lsnnot supportednot supportednot supportednot supportedpg_lsnnot supportednot supportednot supportedrealfloat64floatfloat(s)floatrealfloatfloat64floatserial / serial4int64bigintintegerintserialintegerint64integersmallInt / int2int64bigintsmallintintsmallintintegerint64smallintsmallserial / serial2int64bigintsmallintintsmallserialintegerint64smallinttextstringstringlongtextclobtextvarcharstring(max)varchar(max)timetimetimestamptimevarchar2(100)time(s)timestring(max)timetime without Time zonetimetimestamptimevarchar2(100)time(s)timestring(max)timetimestampdatetimetimestampdatetimetimestamptimestamp(s) without timezonetimestamptimestampdatetime2timestamp With Timezonetimestamptimestamptimestamptimestamp with timezonetimestamp(s) with timezonetimestamp with time zonetimestampdatetimeoffsettxid_snapshotnot supportednot supportednot supportednot supportedtxid_snapshotnot supportednot supportednot supporteduuidnot supportednot supportednot supportednot supporteduuidnot supportednot supportednot supportedxmlstringstringlongtextxmltypexmlvarcharstring(max)xmlPostgreSQL source types box, cid, circle, daterange, int4range, int8range, line, lseg, money, numrange, oid, path, point, polygon, tsquery, tsrange, tstzrange, tsvector, and xid are not supported.See Data Types for a list of supported data type aliases (such as decimal and varchar).In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2023-04-07


Data type support & mapping for SQL Server sourcesSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideData type support & mapping for schema conversion & evolutionData type support & mapping for SQL Server sourcesPrevNextData type support & mapping for SQL Server sourcesSQL Server source typeBigQueryDatabricksMySQLOraclePostgreSQLSnowflakeSpannerSQL Server / Azure Synapsebigintint64bigintbigintintbigintintegerint64bigintbinarybytesbinarybinary(1)blobbyteabinarybytes(1)binary(1)binary(s)bytesbinarybinary(s)blobbyteabinarybytes(s)binary(s)bitnot supportednot supportedbit(1)not supportedbit varyingbooleannot supportedbitcharstringstringcharacter(1)character(1)character(1)character(1)string(1)character(1)char(s)stringstringcharacter(s)longtext if s>255character(s)clob if s>2000character(s)character(s)string(s)character(s)datedatedatedatedatedatedatedatedatedatetimedatetimetimestampdatetimetimestamptimestamp without timezonetimestamptimestampdatetime2datetime2datetimetimestampdatetimetimestamptimestamp without timezonetimestamptimestampdatetime2datetimeoffsettimestamptimestamptimestamptimestamp with time zonetimestamp with time zonetimestamp with time zonetimestampdatetimeoffsetdecimalstringdecimal(38)decimal(18,0)number(18,0)numeric(18,0)numeric(18,0)float64decimal(18,0)decimal(p,s)numeric(p) if p<=38 and s<=8string if p>38 or s>9decimal(p,s) if p<=38, s<=37string if p>38 or s>37decimal(p,s) if p<=65 and s<=30text if p>65 or s>30number(p,s) if p<=38 ands<=127varchar2(1000) if p>38 or s>127numeric(p,s)numeric(p,s) if p<=38 and s<=37varchar if p>38 or s>37float64 if p<=308 and s<=15string(max) if p>308 pr s>15decimal(p,s) if p<=38 and s<=38varchar(8000) if p>38 or s>38decimal(p)numeric(p) if p<=38string if p>38decimal(p,0) if p<=38string if p>38decimal(p) if p<=65text if p>65number(p,s) if p<=38varchar2(1000) if p>38numeric(p)numeric(p) if p<=38varchar if p>38float64 if p<=308string(max) if p>308decimal(p) if p<=38varchar(8000) if p>38floatfloat64floatdoublefloatdouble precisionfloatfloat64float(s)float(p)float64floatfloat(p) if p<=38double if p>38float(p)double precisionfloat(p) if p<=38float if p>38float64float(p) if p<=53varchar(8000) if p>53imagebytesbinarylongblobblobbyteabinarybytes(max)varbinary(max)intint64bigintintegerintintegerintegerint64integermoneynot supportednot supportednot supportednot supportednot supportednot supportednot supportednot supportednchar(s)stringstringnchar(s) if s<255nvarchar(s) if s>255nachar(s) if s<=1000nclob if s>1000character(s)character(s)string(s)nchar(s) if <= 4000nvarchar(max) > 4000ntextstringstringlongtextnclobtextvarcharstring(s)nchar(s) if <= 4000nvarchar(max) > 4000numericstringdecimal(38)decimal(18,0)number(18,0)numeric(18,0)numeric(18,0)float64numeric(18,0)numeric(p,s)numeric(p) if p<=38 and s<=8string if p>38 or s>9decimal(p,s) if p<=38, s<=37string if p>38 or s>37decimal(p,s) if p<=65 and s<=30text if p>65 or s>30number(p,s) if p<=38 and s<=127number if p>38 or s>127numeric(p,s)numeric(p) if p<=38 and s<=37varchar if p>38 or s>37float64 if p<=308 and s<=15string(max) if p>308 pr s>15numeric(p,s) if p<=38 and s<=38varchar(8000) if p>38 or s>38numeric(p)numeric(p) if p<=38string if p>38decimal(p,0) if p<=38string if p>38decimal(p) if p<=65decimal(65) if p>65number(p) if p<=38number(*,0) if p>38numeric(p)numeric(p) if p<=38varchar if p>38int64 if p<=20float64 if 20<p<=308string(max) if p>308numeric(p) if p<=38numeric(38) if p>38nvarchar(s)stringstringnvarchar(s) if s<65535longtext if s>65535nvarchar(s) if s<=4000nclob if s>4000character varyingvarcharstring(s)nvarchar(s) if <= 4000nvarchar(max) if > 4000realfloat64floatfloatfloatdouble precisionfloatnot supportedfloat(s)smalldatetimedatetimetimestampdatetimetimestamptimestamp without timezonetimestamptimestampdatetime2smallintint64bigintsmallintintsmallintintegerint64smallintsmallmoneynot supportednot supportednot supportednot supportednot supportednot supportednot supportedsmall moneytextstringstringlongtextclobtextvarcharstring(max)varchar(s) if <= 8000varchar(max) > 8000timetimetimestamptimevarchar2(100)timetimestring(max)timetinyintint64biginttinyint unsignedintsmallintintegerint64tinyintvarbinarybytesbinaryvarbinary(1)blobcharacter varying(1)varbinarybytes(1)varbinary(1)varbinary(s)bytesbinaryvarbinary(s)blobcharacter varying(s)varbinarybytes(s)varbinary(s)varchar(s)stringstringvarchar(s)varchar2(s) if s<=4000clob if s>4000character varying(s)varchar(s)string(s)varchar(s) if s<=8000varchar(max) if s>8000xmlstringstringlongtextxmltypexmlvarcharstring(max)xmlSQL Server source data types geography, geometry, rowversion, sql_variant, udt, and uniqueidentifier are not supported.In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2023-01-20


Handling schema evolutionSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideHandling schema evolutionPrevNextHandling schema evolutionFor the CDC sources listed below, Striim can capture certain changes to the DDL in the source tables. When CDDL Capture is enabled in a source, you must choose which of the following actions you want to happen when it encounters a DDL change:Halt in source: DDL changes are never expected in the source. Striim will halt the application so you can investigate the issue. Use this option when you want to prohibit changes to the source schema.Ignore in source: target is schemaless and does not have the CDDL Action property (for example, FileWriter with JSON Formatter, or MongoDB).Process in source, Halt in target: the application has one or more targets that support the Process action and one or more that have the CDDL Action property but do not support the Process action. In this case, set the CDDL Action property to Halt for the targets that do not support Process. Schema changes in the source will be replicated to the targets that support Process and the application will halt for you to deal with the others manually. If recovery is enabled for the application, after restart the DDL operation will be sent again.DDL changes to tables specified in Excluded Tables will not trigger Halt.Process in source, Ignore in target: the application has multiple targets that have the CDDL Action property and you do not want to replicate changes to this one.Process in source, Process in target: replicate changes to the target, keeping the target schema in sync with the source automatically without interrupting operation of the application. This is supported only for the targets listed below..When an unsupported DDL operation or unsupported data type is encountered in a DDL change, the application will halt for you to troubleshoot the problem.Quiesce in source: target does not support schema evolution, so when a DDL change is detected, Striim will write all the events received prior to the DDL operation to the target, then quiesce the application. Then you can update the target schema manually. If recovery is enabled for the application, after restart the DDL operation will not be sent again.Always select Process in the source when Using the Confluent or Hortonworks schema registry.Supported CDC sourcesMariaDBMySQLOracle Database 18c and earlierOracle GoldenGate 12.1 or later for Oracle Database onlyPostgreSQL (see PostgreSQL setup for schema evolution)SQL Server (with MSJet only)Targets with the CDDL Action propertyAzure SynapseDatabricks Writer: supports only CREATE TABLE, ADD COLUMN, DROP TABLE, and TRUNCATEGoogle BigQueryGoogle Cloud SpannerMariaDB (via Database Writer)MySQL (via Database Writer)Oracle Database (via Database Writer)PosgtgreSQL (via Database Writer)SAP Hana (via Database Writer - does not support Process)SnowflakeSQL Server (via Database Writer)Sybase (via Database Writer - does not support Process)Supported DDL operationsCREATE TABLE (default column values are not supported)ALTER TABLE ... ADD COLUMN (default column values are not supported)with MySQL, AFTER and ALGORITHM=INSTANT are not supported (known issues DEV-35539 and DEV-35681)with Oracle Database, adding NOT NULL constraints is not supported (known issues DEV-24666, DEV-25424)ALTER TABLE ... MODIFY COLUMN: The modification must be compatible with existing data, for example, you could change short to long, or varchar(20) to varchar(30). Default column values are not supported.not supported with BigQuery or Databricks targetsIf a ColumnMap is specified (see Mapping columns), the mapped target column will be modified.with Oracle Database, adding NOT NULL constraints is not supported (known issues DEV-24666, DEV-25424)Snowflake Writer: see limitations described in ALTER TABLE … ALTER COLUMNnot supported with SQL Server sources (known issue DEV-26386)ALTER TABLE ... ADD PRIMARY KEYnot supported with BigQuery or Databricks targetssupported with GoldenGate sources only when table has no primary key (known issues DEV-26575)not supported with MySQL or SQL Server sourcesALTER TABLE ... ADD CONSTRAINT ... PRIMARY KEYnot supported with BigQuery or Databricks targetsWith GoldenGate sources, use the syntax ALTER TABLE <name> ADD CONSTRAINT <name> PRIMARY KEY (id). (The syntax ALTER TABLE MODIFY ID NUMBER NOT NULL PRIMARY KEY will not work.)not supported with SQL Server sourcesALTER TABLE ... ADD CONSTRAINT ... UNIQUEnot supported with BigQuery or Databricks targetssupported with GoldenGate sources only when table has a primary key column and the constraint is not added to that column (known issues DEV-26575)not supported with SQL Server sources (known issue DEV-26386)ALTER TABLE ... ADD UNIQUEsupported with MariaDB and MySQL onlyALTER TABLE ... DROP COLUMN:not supported with BigQuery or Databricks targetsIf a ColumnMap is specified for the column (see Mapping columns), the application will halt. ALTER and RECOMPILE the application to remove the ColumnMap for the column, drop the column from the target table, and restart the application.with Oracle Database, adding NOT NULL constraints is not supported (known issues DEV-24666, DEV-25424)DROP TABLETRUNCATE TABLEsources that support TRUNCATE TABLEtargets that support TRUNCATE TABLEGG Trail ReaderMariaDB Xpand ReaderMySQL ReaderOJetOracle Reader (with Oracle version 18c or earlier only)PostgreSQL Reader (using wal2json version 2 only)Azure Synapse WriterBigQuery WriterDatabase Writer writing toMariaDBMariaDB XpandMySQLOraclePostgreSQLSQL ServerDatabricks WriterSnowflake WriterTRUNCATE TABLE ... CASCADE is not supportedTRUNCATE cannot be supported with Spanner or SQL Server sources as TRUNCATE operations is not included in their CDCData type support and mappingsSee Data type support & mapping for schema conversion & evolution.Process in the source is supported only when all source table data types are supported. Process in the target is supported only when all source types are mapped to target types.Monitoring schema evolutionThe MON command includes the following metrics for schema evolution:number of DDL operations, by tablelast captured / applied DDL statementtime of last captured / applied DDLignored DDL countNotes and limitationsSchema evolution is not supported when using Bidirectional replication.Striim can capture only those DDL changes made after schema evolution is enabled.The first time you start an application with CDDL Capture enabled, the CDC reader will take a snapshot of the source database's table metadata from the. It is essential that there are no DDL changes made to the database until startup completes. Otherwise, the schema captured in the snapshot will be out of date, which will eventually cause the application to terminate.When the Tables property in the reader uses a wildcard, the first time the application is started Striim must fetch the metadata for all tables in the schema. If there are many tables in the schema, this may take a significant amount of time.After an application with both recovery and schema evolution enabled is restarted, Striim will automatically use the correct schema for the restart positionIf the application halts due to an unsupported DDL change, an unsupported column data type, or a Parser Exception, you may add the table causing the halt to the Excluded Tables list and restart the application.LimitationsRenaming tables is not supported.TRUNCATE TABLE is not supported. Use DELETE FROM <table name>; or some other method for deleting all rows from a table.See also Schema evolution known issues and limitations.Sample WAEvents for DDL operations when schema evolution is enabledDDL commandexampleresulting WAEventCREATE TABLECREATE TABLE PRODUCT.CUSTOMER
(
    c_custkey     BIGINT not null,
    c_name        VARCHAR(25) not null,
    c_address     VARCHAR(40) not null,
    c_nationkey   INTEGER not null,
    c_phone       CHAR(15) not null,
    c_acctbal     DOUBLE PRECISION,,
    c_mktsegment  CHAR(10) not null
);WAEvent{
data: ["CREATE TABLE PRODUCT.CUSTOMER  …”]
metadata:{
"OperationName": "Create",
"TableName": "PRODUCT.CUSTOMER",
"SchemaName": "PRODUCT",
"OperationType": "DDL",
"CDDLMetadata": “<Info about DDL>”
}
};ALTER TABLE ADD COLUMNALTER TABLE PRODUCT.CUSTOMER
  ADD  c_comment VARCHAR(117) not null;WAEvent{
data: ["ALTER TABLE PRODUCT.CUSTOMER
  ADD  c_comment VARCHAR(117) not null;"]
metadata:{
"OperationName": "AlterColumns",
"OperationSubName": "AddColumn",
"TableName": "PRODUCT.CUSTOMER",
"SchemaName": "PRODUCT",
"OperationType": "DDL",
"CDDLMetadata": “<Info about DDL>”
}
};ALTER TABLE MODIFY COLUMNALTER TABLE PRODUCT.CUSTOMER
ALTER COLUMN c_address TYPE VARCHAR(200);WAEvent{
data: ["ALTER TABLE PRODUCT.CUSTOMER
ALTER COLUMN c_address TYPE VARCHAR(200);"]
metadata:{
"OperationName": "AlterColumns",
"OperationSubName": "AlterColumn",
"TableName": " PRODUCT.CUSTOMER",
"SchemaName": "PRODUCT",
"OperationType": "DDL",
"CDDLMetadata": “<Info about DDL>”
}
};
ALTER TABLE DROP COLUMNALTER TABLE PRODUCT.CUSTOMER
DROP COLUMN c_acctbal;WAEvent{
data: ["ALTER TABLE PRODUCT.CUSTOMER
DROP COLUMN c_acctbal;"]
metadata:{
"OperationName": "AlterColumns",
"OperationSubName": "DropColumn",
"TableName": "PRODUCT.CUSTOMER",
"SchemaName": "PRODUCT",
"OperationType": "DDL",
"CDDLMetadata": “<Info about DDL>”
}
};DROP TABLEDrop Table PRODUCT.CUSTOMER;WAEvent{
data: ["DROP TABLE PRODUCT.CUSTOMER"]
metadata:{
"OperationName": "Drop",
"TableName": "HR.EMP",
"SchemaName": "HR",
"OperationType": "DDL",
"CDDLMetadata": “<Info about DDL>”
}
};
Schema evolution known issues and limitationsThe following are known issues in this release related to schema evolution. Additional known issues are flagged by "DEV-#####"in Handling schema evolution.All sourcesColumns with a data type that has a scale set to a negative value (for example, number(1, -17) ) are not supported.Azure Synapse WriterIf using Optimized Merge mode, CREATE TABLE will cause the application to halt (DEV-29689).Adding a NOT NULL constraint on a column that already has a UNIQUE constraint is not supported. (DEV-26158)BigQuery WriterIf you are using the legacy streaming API to write to template tables, using the default setting of Process may cause the application to halt due to a limitation in BigQuery that does not allow writing for up to 90 minutes after a DDL change (see BigQuery > Documentation > Guides > Use the legacy streaming API > Creating tables automatically using template tables > Changing the template table schema). In this case, supporting schema evolution is impossible, so set CDDL Action to Ignore. This is not an issue if you are using partitioned tables.MSJetIf a table is dropped and a table of the same name is created, the application may terminate. (DEV-26417)The application will terminate if the database contains tables with names that vary only by case, for example, id and ID, even if those tables are not among those read by MSJet. (DEV-26872)MySQL ReaderAdding a UNQUE constraint syntax with a system generated constraint name (for example, ADD CONSTRAINT cs_01ec19f5f75caa91a1160eca1 UNIQUE (created_att) is not supported. (DEV-26678)OJetColumns of type ROWID are not supported.Invisible, virtual, and unused columns are not supported.Oracle ReaderColumns of type INTERVAL DAY(x) TO SECOND(y) are not supported. (DEV-24624).Invisible, virtual, and unused columns are not supported.PostgreSQL ReaderTo capture DDL changes when the command has more than 1024 characters (for example, a CREATE TABLE statement with many columns), you must raise PostgreSQL's track_activity_query_size parameter from its default value of 1024. (DEV-24650)Creating a table with a column of type serial or adding a column of type serial is not supported.In this section: Handling schema evolutionSample WAEvents for DDL operations when schema evolution is enabledSchema evolution known issues and limitationsSearch resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2023-06-02

Handling schema evolution


Creating and modifying apps using the Flow DesignerSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideCreating and modifying apps using the Flow DesignerPrevNextCreating and modifying apps using the Flow DesignerThe Flow Designer is a graphical programming tool that can do almost everything you can do with TQL.WarningMaking changes to a component that affect the number or type of fields in its output stream (for example, removing a field from a CQ's SELECT statement) can cause downstream objects to become invalid. Before attempting such changes in Flow Designer, make a TQL backup of the application by selecting Configuration > Export.Flow Designer may become unresponsive when a type has a very large number of fields, such as 1,000.If you encounter either of these problems, make your changes in TQL instead (see ALTER and RECOMPILE).See Modifying an application using Flow Designer for a walkthrough.The Base Components palette may be more efficient for experienced users. Any of the components in the Sources, Enrichers (caches), Processors (CQ and window), or Targets palettes may be created by setting the properties of a base component. See DDL and component reference for information on component properties. You may also search for a component by the label that is displayed in the palette (which is not always the same as the name in TQL, for example, Database Reader and Database Writer are both labeled Database).Some readers have a Click here to configure using wizard link that will launch the "to Striim" App Wizard for that source (see Configuring an app template source). Some of these readers also have a Test connection button to validate the properties.Configuring an app template sourceThe components on the Transformers palettes provide a graphical alternative to writing CQs. See Using database event transformers and Using database event transformers.When you view a running application in Flow Designer, input and output total event counts, current input and output rate, and a graph of input and output rates appear at the top of the page.Use Configuration > App Settings to configure recovery, data validation, encryption, exception handling, and auto-resume (see Recovering applications, Creating a data validation dashboard), CREATE APPLICATION ... END APPLICATION, and Automatically restarting an application).Recovering applicationsCREATE APPLICATION ... END APPLICATIONNotes:An application must be in the Created state to be editable. Applications in the Deployed or Running state are read-only.Click Metadata Browser at the top right or Copy from in source and target editors to view and copy settings from other flows or applications.Click and drag to select multiple components to cut and paste in another flow, or to copy and paste in another application.In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2022-12-14


Creating an app using the Flow DesignerSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideCreating and modifying apps using the Flow DesignerCreating an app using the Flow DesignerPrevNextCreating an app using the Flow DesignerThis tutorial uses the sample applications described in Running the CDC demo apps. The Docker PostgreSQL instance must be running. Kafka is not required.Running the CDC demo appsOn the Apps page, if ValidatePostgres is not running, deploy and start it.Click Create App > Start from scratch, name the app PG2File, select your personal namespace or enter a name for a new namespace, and click Save.Click the Metadata Browser icon, search for ReadCDCFromPostgresDB (located in Source), and click Copy to App.For Name , enter ReadPostgreSQLCDC; for Password, enter striim; for New Output, enter rawCDCstream; and click Save. Except for the name and output stream, this new source will have the same properties as SamplesDB.ReadCDCfromPostgresDB.Click the stream icon, click +, and select Connect next Target component.In the New Target dialog, set the properties as follows (leave other properties at their defaults), then click Save.Name: WriteRawDataAdapter: FileWriterFile Name: RawDataDirectory: MyDirectory (click Show Advanced Settings)Rollover Policy: leave as isFlush Policy: EventCount 1 (delete Interval)Formatter: JSONFormatterDeploy and start the application, then return to the View All Apps page and start SamplesDB.Execute250Inserts.Open Striim\MyDirectory\RawData.00. This contains the raw WAEvent output of PostgreSQLReader formatted as JSON. It should look similar to the following:[
 {
  "metadata":{"TableName":"public.customer","TxnID":"2175","OperationName":"INSERT","LSN":"0\/2F9803D8","NEXT_LSN":"0\/2F9942C8","Sequence":1,"Timestamp":"2022-02-25 14:51:31.919978-06"},
  "data":{
"c_custkey":150001,
"c_name":"Customer#150001",
"c_address":"IVhzIApeRb ot,c,E",
"c_nationkey":15,
"c_phone":"25-989-741-2988",
"c_acctbal":"711.56",
"c_mktsegment":"BUILDING  ",
"c_comment":"to the even, regular platelets. regular, ironic epitaphs nag e"
},
  "before":null,
  "userdata":null
 },
...This includes the column names and values for the row inserted into the public.customer table by transaction ID (TxnID).Modifying an app using the Flow DesignerTo parse this raw data, you must write a CQ as described in Parsing the data field of WAEvent. Reopen the PG2File app in Flow Designer, stop and undeploy it, click the stream icon, click +, and select Connect next CQ component.Name the CQ ParseData, name the new output stream ParsedDataStream, and copy and paste the following into the Query field:SELECT
  META(rawCDCstream,"OperationName").toString() AS OpType,
  TO_INT(data[0]) AS CustomerKey,
  TO_STRING(data[1]) AS CustomerName,
  TO_STRING(DATA[2]) AS CustomerAddress,
  TO_INT(DATA[3]) AS NationKey,
  TO_STRING(DATA[4]) AS CustomerPhone,
  TO_DOUBLE(data[5]) AS CustomerAccountBalance,
  TO_STRING(DATA[6]) AS MarketSegment,
  TO_STRING(data[7]) AS CustomerComment
FROM rawCDCstream;Make sure the properties look as shown below, then click Save.Click the ParseData stream icon, click +, and select Connect next Target component.Set the properties as shown below and click Save.Name: WriteParsedDataAdapter: FileWriterFile Name: ParsedDataDirectory: MyDirectory (click Show Advanced Settings)Rollover Policy: leave as isFlush Policy: EventCount 1 (delete Interval)Formatter: JSONFormatterDeploy and run the application, then return to the Apps page and start SamplesDB.Execute200Inserts.Open Striim\MyDirectory\ParsedData.00. This contains the parsed data in the custom format defined by the CQ as well as the operation type from the metadata."OpType":"INSERT",
"CustomerKey":151501,
"CustomerName":"Customer#151501",
"CustomerAddress":"IVhzIApeRb ot,c,E",
"NationKey":15,
"CustomerPhone":"25-989-741-2988",
"CustomerAccountBalance":711.56,
"MarketSegment":"BUILDING  ",
"CustomerComment":"to the even, regular platelets. regular, ironic epitaphs nag e"
In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2022-02-28

Creating an app using the Flow Designer


Using event transformersSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideCreating and modifying apps using the Flow DesignerUsing event transformersPrevNextUsing event transformersEvent transformers are a graphical alternative to CQs (see CREATE CQ (query)) for modifying data flows. For the transformers discussed in this section, the input stream must have a user-defined type.If you convert a transformer to a CQ, you cannot convert it back to a transformer.Field AdderAdds one or more fields and creates a corresponding type for the output stream. For each field, enter a name, select a data type, and enter an expression.Expressions may use the same Operators and Functions as CQ SELECT statements. Enclose literal string values in quotes.Field EnricherJoins fields from a stream with fields from a cache, event table, WActionStore, or window (see Joins). Select a stream from Input, another component from Enrichment Component, and the fields to join on from Input Field and Enrichment Component Field. All the fields from both components will appear in Output Fields (Input fields will start with i_ and Enrichment Component fields with e_.) Delete any fields you don't want by clicking x and reorder them as you like by dragging ↕ up or down. Optionally, click Convert to CQ to specify a particular type of join (see CREATE CQ (query)).CREATE CQ (query)Field MaskerSee Masking functions.Masking functionsField RemoverRemoves one or more fields and creates a corresponding type for the output stream. Click the X to remove a field. Click RESET to start over.Field RenamerRenames one or more fields and creates a corresponding type for the output stream.Field SplitterSplits a string field into multiple string fields using the specified delimiter. The delimiter may include any combination of alphanumeric characters, spaces, and punctuation.Field Type ModifierChanges the data type of one or more fields and creates a corresponding type for the output stream.The following conversions should work without problems. Others may cause the application to terminate.Input typeOutput typeByteDouble, Integer, Float, Long, Short, or StringDateTimeStringDoubleStringFloatDouble or StringIntegerDouble, Float, Long, or StringLongDouble, Float, or StringShortDouble, Integer, Float, Long, or StringStringByte, Double, Float, Integer, Long, or Short, provided the values are compatibleField Value FilterFilters the output stream based on one or more specified conditions. Each additional condition narrows the criteria, so for the output for the example above, the output will contain events with amounts from $5,000 to $10,000.To DB EventConverts events of a user-defined type to WAEvent format. The output stream's type is WAEvent and can be used as the input for a target using DatabaseWriter or another writer accept input from DatabaseReader, IncrementalBatchReader, or a SQL CDC source. In the output, the table name will be the value of the metadata TableName field. In this release, the value of the metadata OperationName field will always be INSERT.For example, using the settings shown above, the following input event:PosDataStream_Type_1_0{
  merchantId: "D6RJPwyuLXoLqQRQcOcouJ26KGxJSf6hgbu"
  dateTime: 1363134730000
  amount: 2.2
  zip: 41363
};
would be output as:WAEvent{
  data: ["D6RJPwyuLXoLqQRQcOcouJ26KGxJSf6hgbu",1363134730000,2.2,41363]
  metadata: {"TableName":"mydb.mytable","OperationName":"INSERT"}
  userdata: null
  before: null
  dataPresenceBitMap: "Dw=="
  beforePresenceBitMap: "AA=="
  typeUUID: {"uuidstring":"01e91cfe-b88a-8ef1-8e39-8cae4cf129d6"}
};In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2022-12-14

Using event transformers


Using database event transformersSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideCreating and modifying apps using the Flow DesignerUsing database event transformersPrevNextUsing database event transformersDatabase event transformers are a graphical alternative to CQs for modifying the output of SQL CDC sources (see CREATE CQ (query) and Working with SQL CDC readers).You can use DB event transformers with sources.If you convert a transformer to a CQ, you cannot convert it back to a transformer.Expressions may use the same Operators and Functions as CQ SELECT statements. Enclose literal string values in quotes.Custom Data AdderAdds one or more fields to the USERDATA map (see Adding user-defined data to WAEvent streams). For each field, enter a name, select a data type, and enter an expression.Data ModifierModifies one field in the DATA array. Select the table, select the column, and enter an expression.This transformer can be used only with MSSQLReader, MySQLReader, or OracleReader, and Striim must be able to connect to the database to download table descriptions.Metadata FilterFilters the input events based on one or more metadata field values.Operation FilterPasses only events with the selected operation type to the output stream. Select DELETE, INSERT, or UPDATE.Table FilterPasses only the selected table to the output stream.This transformer can be used only with MSSQLReader, MySQLReader, or OracleReader, and Striim must be able to connect to the database to download table descriptions.To EventCreates a Striim Type for the output stream based on the column names and data types of the selected table and converts WAEvent input to typed output (see Parsing the data field of WAEvent). Only events from the selected table are included in the output stream.This transformer can be used only with MSSQLReader, MySQLReader, or OracleReader, and Striim must be able to connect to the database to download table descriptions.To StagingCopies the values of the OperationName field in the METADATA map to the OrigOperationName field in the USERDATA map (see Adding user-defined data to WAEvent streams). Typically you would use this in an application that writes to a data warehouse or other target where UPDATE and DELETE operations are handled as INSERTs (see How update and delete operations are handled in writers).In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2022-02-24

Using database event transformers


Creating apps using templatesSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideCreating apps using templatesPrevNextCreating apps using templatesTemplates can make creating Striim applications faster and easier. Apps created with templates may be modified using Flow Designer or by exporting TQL, editing it, and importing the modified TQL.Striim has templates for many source-target combinations. Which are available in your installation will vary depending on which licenses you have (contact Striim support for more information).When you use a template, the app's exception store is enabled automatically. If you do not want an exception store, turn it off after you create the app (see CREATE EXCEPTIONSTORE).CREATE EXCEPTIONSTOREUse the Search Templates tool on the App Wizard page to see which templates are available for a particular source or target.Some initial load templates have Auto Schema Conversion, which allows you to automatically create schemas and tables in the target that correspond to those in the source. You must create the target database manually. In this release, tables can be created in the target only when Data type support & mapping for schema conversion & evolution includes mappings between all source and target column data types; other tables will be omitted.Sources includeTargets includeAmazon S3Azure Blob StorageCosmosDB ReaderGoogle Cloud StorageHDFSIncremental Batch ReaderMariaDB Initial Load (using Database Reader)MSJet (for initial load for MSJet, use SQL Server Initial Load)MariaDB Initial Load (using Database Reader)MongoDB CDC (using MongoDB Reader in Incremental mode)MongoDB Initial Load (using MongoDB Reader in Initial Load mode)MySQL CDCMySQL Initial Load (using Database Reader)Oracle CDCOracle Initial Load (using Database Reader)OJetPostgreSQL CDCPostgreSQL Initial Load (using Database Reader)SalesforceSalesforce PardotSQL Server CDCSQL Server Initial Load (using Database Reader)Amazon Kinesis Data StreamsAmazon RedshiftAmazon S3Apache CassandraApache Hive - ClouderaApache KuduAzure Blob StorageAzure Cosmos DB (after you configure the source and select schemas and tables, the wizard will prompt you to select the SQL, Cassandra, or MongoDB API)Azure Data Lake Store Gen1Azure Data Lake Store Gen2Azure Event HubsAzure PostgreSQLAzure SQL DatabaseAzure Synapse AnalyticsBigQueryCloud SQL for MySQLCloud SQL for PostgreSQLCloud SQL for SQL ServerDatabase (see Database Writer for supported databases)Databricks in AWS and AzureNote: Auto Schema Conversion is not supported when using Databricks' Unity Catalog.File (using File Writer)Google Cloud StorageHazelcast WriterHBaseHortonworks HiveJMSJPAWriterKafka 0.8.0, 0.9.0, 0.10.0, 0.11.0, 2.1.0MapR StreamsMariaDBMongoDBMQTTMySQL (using DatabaseWriter)Oracle Database (using DatabaseWriter)PostgreSQL (using DatabaseWriter)SAP HANA (using DatabaseWriter)ServiceNowSinglestore (MemSQL)SnowflakeSpannerSQL Server (using DatabaseWriter)Striim (app contains only the source)App template prerequisite checklistYou will need the assistance of a database administrator for some of these tasks.Configure the source database as detailed in the relevant topic in Sources. You will need to provide the user name and password created during database configuration when configuring the source. If using an initial load template with Auto Schema Conversion, also assign the privileges listed for the source in Database privileges required for Auto Schema Conversion.Configure the target as detailed in the relevant topic in Targets. If using an initial load template with Auto Schema Conversion, also assign the privileges listed for the target in Database privileges required for Auto Schema Conversion.If the source cannot be read directly by Striim (for example, if Striim is running in AWS, Azure, or Google Cloud, and the source is on your premises):Install the Forwarding Agent and appropriate JDBC driver on the source host as detailed in Striim Forwarding Agent installation and configuration.Before launching a template, make sure the Forwarding Agent is running.If not using an initial load template with Auto Schema Conversion, create tables in the target that match the tables in the source using any tool you wish.Database privileges required for Auto Schema ConversionTo use Auto Schema Conversion, a database administrator must assign privileges to the users or service accounts to be specified for the source and target connections using the following commands or procedures:Azure Synapse - target onlyUSE <DATABASENAME>; EXEC sp_addrolemember 'db_owner', '<USERNAME>';BigQuery - target onlyFollow the instructions in BigQuery setup to assign roles or permissions to the service account to be specified in the target properties.Cloud Spanner - target onlyGrant the Cloud Spanner Database User or higher role (see Cloud Spanner Roles) for the database to the service account to be specified in the target properties.Databricks - target onlyGRANT CREATE ON SCHEMA <SCHEMANAME> TO <USERNAME>;MariaDB / MySQL - sourceCreate a user as described in MariaDB setup or MySQL setup.MySQL / MariaDB setupAlternatively, assign the SELECT privilege:GRANT SELECT ON *.* TO <USERNAME>@<HOSTNAME>;MariaDB / MySQL - targetGRANT CREATE ON *.* TO <USERNAME>@<HOSTNAME>;Oracle - sourceCreate a user as described in Create an Oracle user with LogMiner privileges. or Running the OJet setup script on Oracle.Alternatively, assign the following privileges:grant CREATE SESSION to <username>;
grant SELECT ANY TABLE to <username>;
grant SELECT ANY DICTIONARY to <username>;Oracle - targetGRANT CREATE SESSION TO <USERNAME>;
GRANT CREATE USER TO <USERNAME>;
GRANT CREATE ANY TABLE TO <USERNAME>;
GRANT CREATE ANY INDEX TO <USERNAME>;
GRANT UNLIMITED TABLESPACE TO <USERNAME>;PostgreSQL - sourceCreate a user as described in PostgreSQL setup.Alternatively, assign the following privileges:GRANT CONNECT ON DATABASE <DATABASENAME> TO <USERNAME>;
GRANT SELECT ON <TABLENAMES> to <USERNAME>;PostgreSQL - targetGRANT CONNECT ON DATABASE <DATABASENAME> TO <USERNAME>;
GRANT CREATE ON <TABLENAMES> to <USERNAME>;Snowflake - target onlyGRANT USAGE ON <DATABASE | WAREHOUSE | SCHEMA | TABLE> TO ROLE <USERNAME>;SQL Server - sourceCreate a user as described in SQL Server setup.Alternatively, give the user the db_owner role:USE <DATABASENAME>;
EXEC sp_addrolemember 'db_owner', '<USERNAME>';SQL Server - targetUSE <DATABASENAME>; EXEC sp_addrolemember 'db_owner', '<USERNAME>';Creating an application using a templateComplete the App template prerequisite checklist.In the Striim web UI, select Apps > Create New.Find and click the template you want to use. If you don't see it, enter part of the source name in Search for templates, select the one you want, then do the same for the target.Enter a name for the app, select or enter the namespace in which you want to create it, and click Save.Continue with Configuring an app template source.Configuring an app template sourceComplete the steps in Creating an application using a template.Enter the host name, port, the user name and password for Striim, and any other properties required by the wizard. See the relevant reader properties reference in Sources or Change Data Capture (CDC) for more information.For an Incremental Batch Reader source, also specify the check column and start position as detailed in Incremental Batch Reader.If using a Forwarding Agent, set Deploy source on Agent on. (This option appears only when an agent is connected to Striim. (For file sources, this option does not appear, but the source components are put in their own flow which can be deployed on the agent.)Click Next. The wizard will connect to the database to validate the settings. If all the checks pass, click Next.If using a template with Auto Schema Conversion, continue with Using Auto Schema Conversion. Otherwise, select the tables to be read and click Next.If your template's target is not to Striim, continue with Configuring an app template target.If your template's target is to Striim, your new application will open in the Flow Designer.Using Auto Schema ConversionNoteKnown issue (DEV-31163): when the target is Snowflake, you must create the schemas manually.Select the schemas you want to copy to the target, then click Next.If validation is successful, click Next. Otherwise, click Back and resolve any issues.For each schema, select the tables to copy. Any tables with data types that can not be mapped to the target (see Data type support & mapping for schema conversion & evolution) will be greyed out and can not be copied. When done, click Next.Continue with Configuring an app template target.Configuring an app template targetComplete the steps in Configuring an app template source.Configuring an app template sourceEnter the appropriate properties. For more information, see the Targets entry for the selected target type.By default, the case of source table schema and table names will be preserved in the target. To avoid that, edit the Tables property and remove all double quotes.With a CDC source and a to Azure SQL Database, to Azure SQL Data Warehouse, or to Database target, for the Tables property, specify the source and target table names, separated by commas, separating each source-target pair with a semicolon: for example, dbo.Jobs,mydb.Jobs;dbo.Region,mydb.Region. Alternatively, use wildcards (see the discussion of the Tables property in Database Writer).If using a template with Auto Schema Conversion and you want to start initial load, click Migrate my schemas and move data. Otherwise, click Take me to Flow Designer.If using a template without Auto Schema Conversion, click Next. Your new application will open in the Flow Designer.Before enabling recovery for a template-created app with a to Azure SQL DB or to Database target, follow the instructions for the target DBMS in Creating the checkpoint table.In this section: Creating apps using templatesApp template prerequisite checklistDatabase privileges required for Auto Schema ConversionCreating an application using a templateConfiguring an app template sourceUsing Auto Schema ConversionConfiguring an app template targetSearch resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2023-05-24

Creating apps using templates


Creating apps by importing TQLSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideCreating apps by importing TQLPrevNextCreating apps by importing TQLYou can import TQL using the console (see @), the web UI (see Apps page), or the REST API (see POST / tungsten).Apps pageIn this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2021-09-23

Creating apps by importing TQL


Creating or modifying apps using Source PreviewSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideCreating or modifying apps using Source PreviewPrevNextCreating or modifying apps using Source PreviewSource Preview can create File Reader (click Browse) or HDFS Reader (click Connect) sources and caches in a new or existing application. FileReader source files can be selected on the Striim server or uploaded from a directory readable from your local system.Source Preview requires you to select a single file, but after creating a source, you can modify its wildcard property to select multiple files.See Creating sources and caches using Source Preview for a hands-on walkthrough.In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2023-01-17


Creating a data validation dashboardSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideCreating a data validation dashboardPrevNextCreating a data validation dashboardA data validation dashboard gives you a visual representation of data read and written by a Striim application. Supported sources and targets are:SourcesTargetsDatabase ReaderFile ReaderHP NonStop SQL/MX ReaderIncremental Batch ReaderJMS ReaderKafka Reader versions 0.8, 0.9, 0.10, and 0.11MariaDB ReaderMongoDB ReaderMS SQL Reader (SQL Server)MySQL ReaderOracle ReaderPostgreSQL ReaderSalesforce Platform Event ReaderSalesforce ReaderSalesforce Pardot ReaderSpanner Batch ReaderADLS Gen1 WriterADLS Gen2 WriterAzure Blob WriterAzure Event Hub WriterAzure SQL Data Warehouse WriterBigQuery WriterCassandra Cosmos DB WriterCassandra WriterCloudera Hive WriterCosmos DB WriterDatabase WriterDatabricks WriterFile WriterGCS WriterGoogle PubSub WriterHazelcast WriterHBase WriterHDFS WriterHortonworks Hive WriterKafka Writer versions 0.8, 0.9, 0.10, and 0.11Kinesis WriterKudu WriterMapR DB WriterMongoDB WriterS3 WriterServiceNow WriterSnowflake WriterSpanner WriterThe application may have multiple sources and/or targets so long as they are all of the same type. For example, you could have multiple sources all using DatabaseReader and multiple targets all using KafkaWriter.To create and view a data validation dashboard:Undeploy the application.Open the application in the Flow Designer, select Configuration > App Settings, enable data validation, and click Save.Deploy and start the application, then select Configuration > View Validation Dashboard.Hover the mouse pointer over a bar to see additional details for that source or target for the indicated time period. Click Pause to stop updating the dashboard temporarily so you can compare event counts and verify that all source data was written by the target. Validation dashboards do not appear on the Dashboard page.In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2023-05-31

Creating a data validation dashboard


Switching from initial load to continuous replicationSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideSwitching from initial load to continuous replicationPrevNextSwitching from initial load to continuous replicationIf there may be open transactions in the source database when the initial load completes, take the following steps to ensure that they are replicated to the target.Before running the initial load application, run the following command on the source database using an appropriate database client and record the value returned:MariaDB:select @@gtid_current_pos;MySQL:select current_timestamp;Oracle:select min(start_scn) from gv$transaction;PostgreSQL:For PostgreSQL version 10 or higher:SELECT pg_current_wal_lsn();For earlier versions:SELECT pg_current_xlog_location();SQL Server:SELECT sys.fn_cdc_get_max_lsn()AS max_lsn; When creating the continuous replication application:Enable recovery.Use the following setting for the CDC reader:MariaDB Reader: set Start Position to the GTID you recorded before performing the initial load.MySQL Reader, set Start Time to the timestamp you recorded before performing the initial load.Oracle Reader, set Start SCN to the SCN you recorded before performing the initial load.PostgreSQL Reader, set Start LSN to the LSN you recorded before performing the initial load.SQL Server (MSSQL Reader): set Start Position to the LSN you recorded before performing the initial load.Use the following setting for Database Writer's Ignorable Exception Code property:DUPLICATE_ROW_EXISTS, NO_OP_UPDATE, NO_OP_DELETEWhen you know all open transactions have completed and been written to the target, undeploy and stop the application, edit the target, clear the Ignorable Exception Code value, save, and deploy and start the application. Since recovery is enabled, writing will resume where it left off, and there should be no missing or duplicate rows.In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2023-04-18


Handing apps off to QA or productionSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideHanding apps off to QA or productionPrevNextHanding apps off to QA or productionWhen handing applications off to QA, production, or anyone else not using the same instance of Striim that you use, give them the following: an exported TQL file for the application (see Exporting applications and dashboards)exported JSON file(s) for any dashboard(s)the SCM file(s) for any Open Processor components (see Creating an open processor component)When writing applications that will be handed off, consider creating vaults with the same name in the various namespaces used by the teams. If the vaults' entries have the same names but different values, the applications can use different connection URLs, user names, passwords, keys, and so on with no need to revise the TQL. See Using vaults for more details.See also Managing the application lifecycle.In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2023-05-08

Handing apps off to QA or production


Intermediate TQL programming: common patternsSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideIntermediate TQL programming: common patternsPrevNextIntermediate TQL programming: common patternsThe topics in this section assume that you have read Fundamentals of TQL programming.In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2018-09-06


Getting data from sourcesSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideIntermediate TQL programming: common patternsGetting data from sourcesPrevNextGetting data from sourcesThe appropriate programming pattern for getting data from a source depends on the output type of the source's adapter.In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2017-01-10

Getting data from sources


Sources with WAEvent outputSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideIntermediate TQL programming: common patternsGetting data from sourcesSources with WAEvent outputPrevNextSources with WAEvent outputThe basic programming pattern for a source using an adapter with the WAEvent output type is:source > stream > CQ > streamYou do not need to define the streams explicitly: instead, define them implicitly as part of the source and CQ, and they will be created automatically. For example, from PosApp:CREATE SOURCE CsvDataSource USING FileReader (
  directory:'Samples/PosApp/appData',
  wildcard:'posdata.csv',
  blocksize: 10240,
  positionByEOF:false
)
PARSE USING DSVParser (
  header:Yes,
  trimquote:false
)
OUTPUT TO CsvStream;

CREATE CQ CsvToPosData
INSERT INTO PosDataStream
SELECT TO_STRING(data[1]) as merchantId,
  TO_DATEF(data[4],'yyyyMMddHHmmss') as dateTime,
  DHOURS(TO_DATEF(data[4],'yyyyMMddHHmmss')) as hourValue,
  TO_DOUBLE(data[7]) as amount,
  TO_STRING(data[9]) as zip
FROM CsvStream;In addition to the explicitly defined CsvDataSource and CsvToPosData, this will create the "implicit" stream CsvStream (of type WAEvent), PosDataStream_Type, and PosDataStream:You can inspect these using the DESCRIBE command, for example:W (admin) > describe type Samples.PosDataStream_Type;
Processing - describe type Samples.PosDataStream_Type
TYPE Samples.PosDataStream_Type CREATED 2016-01-22 12:35:34
ATTRIBUTES (
  merchantId java.lang.String
  dateTime org.joda.time.DateTime
  hourValue java.lang.Integer
  amount java.lang.Double
  zip java.lang.String
)For more information, see Parsing the data field of WAEvent.The code for XML sources is slightly different. For example, from MultiLogApp:CREATE SOURCE Log4JSource USING FileReader (
  directory:'Samples/MultiLogApp/appData',
  wildcard:'log4jLog.xml',
  positionByEOF:false
)
PARSE USING XMLParser(
  rootnode:'/log4j:event',
  columnlist:'log4j:event/@timestamp,
    log4j:event/@level,
    log4j:event/log4j:message,
    log4j:event/log4j:throwable,
    log4j:event/log4j:locationInfo/@class,
    log4j:event/log4j:locationInfo/@method,
    log4j:event/log4j:locationInfo/@file,
    log4j:event/log4j:locationInfo/@line'
)
OUTPUT TO RawXMLStream;Here, fields 0-7 in the WAEvent data array are defined by columnlist. These are then parsed by the CQ ParseLog4J:CREATE TYPE Log4JEntry (
  logTime DateTime,
  level String,
  message String,
  api String,
  sessionId String,
  userId String,
  sobject String,
  xception String,
  className String,
  method String,
  fileName String,
  lineNum String
);
CREATE STREAM Log4JStream OF Log4JEntry;

CREATE CQ ParseLog4J
INSERT INTO Log4JStream
SELECT TO_DATE(TO_LONG(data[0])),
  data[1],
  data[2],
  MATCH(data[2], '\\\\[api=([a-zA-Z0-9]*)\\\\]'),
  MATCH(data[2], '\\\\[session=([a-zA-Z0-9\\-]*)\\\\]'),
  MATCH(data[2], '\\\\[user=([a-zA-Z0-9\\-]*)\\\\]'),
  MATCH(data[2], '\\\\[sobject=([a-zA-Z0-9]*)\\\\]'),
  data[3],
  data[4],
  data[5],
  data[6],
  data[7]
FROM RawXMLStream;See MultiLogApp for a more detailed discussion including explanation of the MATCH function.If you preferred, instead of separately defining Log4JStream as above, you could define it within the ParseLog4J CQ, as follows:CREATE CQ ParseLog4J
INSERT INTO Log4JStream
SELECT TO_DATE(TO_LONG(data[0])) as logTime,
  TO_STRING(data[1]) as level,
  TO_STRING(data[2]) as message,
  TO_STRING(MATCH(data[2], '\\\\[api=([a-zA-Z0-9]*)\\\\]')) as api,
  TO_STRING(MATCH(data[2], '\\\\[session=([a-zA-Z0-9\\-]*)\\\\]')) as sessionId,
  TO_STRING(MATCH(data[2], '\\\\[user=([a-zA-Z0-9\\-]*)\\\\]')) as userId,
  TO_STRING(MATCH(data[2], '\\\\[sobject=([a-zA-Z0-9]*)\\\\]')) as sobject,
  TO_STRING(data[3]) as xception,
  TO_STRING(data[4]) as className,
  TO_STRING(data[5]) as method,
  TO_STRING(data[6]) as fileName,
  TO_STRING(data[7]) as lineNum
FROM RawXMLStream;With this approach, Log4JStream's automatically generated type would be Log4JStream_Type, so you would have to replace the four other references to Log4JEntry in the application with Log4JStream_Type.In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2021-09-01

Sources with WAEvent output


JSON sources with custom output typesSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideIntermediate TQL programming: common patternsGetting data from sourcesJSON sources with custom output typesPrevNextJSON sources with custom output typesThe basic programming pattern for sources using adapters with the JSONParser is:
type, stream, source
First define the type for the output stream, then the stream, then the source. For example:CREATE TYPE ScanResultType (
  timestamp1 String,
  rssi String
);
CREATE STREAM ScanResultStream OF ScanResultType;

CREATE SOURCE JSONSource USING FileReader (
  directory: 'Samples',
  WildCard: 'sample.json',
  positionByEOF: false
)
PARSE USING JSONParser (
  eventType: 'ScanResultType',
  fieldName: 'scanresult'
)
OUTPUT TO ScanResultStream;See JSONParser for a more detailed discussion.In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2017-01-10

JSON sources with custom output types


Filtering data in a sourceSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideIntermediate TQL programming: common patternsFiltering data in a sourcePrevNextFiltering data in a sourceThe following examples give an idea of various possibilities for filtering source data using OUTPUT TO and SELECT in sources. This syntax is supported both by the server and the Forwarding Agent (see Using the Striim Forwarding Agent).In sources, SELECT statements must use DATA[#] function functions (see also Parsing the data field of WAEvent. To select using DATA(x) and DATAORDERED(x) functions or META() function functions, you must create a CQ on the output of the source (see also Using the DATA(), DATAORDERED(), BEFORE(), and BEFOREORDERED() functions).Select only events where the fifth column value is greater than 10,000:CREATE SOURCE ...
OUTPUT TO CSVStream (a String, b String, c String, d String,e String)
  SELECT * WHERE TO_INT(data[4]) > 10000;Filter out second and third columns:... OUTPUT TO CSVStream (a String, b String, c String)
  SELECT data[0], data[3],data[4] ;Cast the first and third columns as integers:... OUTPUT TO CSVStream (a Integer, b String, c Integer, d String, e String)
  SELECT TO_INT(data[0]), data[1], TO_INT(data[2]), data[3], data[4];Cast the first and third columns as integers and select only events where the fifth column value is greater than 10,000:... OUTPUT TO CSVStream (a Integer, b String, c Integer, d String, e String)
SELECT TO_INT(data[0]), data[1], TO_INT(data[2]), data[3], data[4]
  where TO_INT(data[4]) > 10000)Add the first and third columns and output as a single field a:... OUTPUT TO CSVStream (a Integer, b String, c String, d String)
  SELECT TO_INT(data[0])+TO_INT(data[2]), data[1], data[3],data[4] ;You can also use OUTPUT TO to split events among multiple streams based on their field values.When the fifth column value is over 10,000, output the event to to HighOrderStream, when the value is 10,000 or less, output it to LowOrderStream:... OUTPUT to HighOrderStream (a Integer, b String, c Integer, d String, e String)
  SELECT TO_INT(data[0]), data[1], TO_INT(data[2]), data[3], data[4]
  WHERE TO_INT(data[4]) > 10000),
OUTPUT to LowOrderStream (a Integer, b String, c Integer, d String, e String)
  SELECT TO_INT(data[0]), data[1], TO_INT(data[2]), data[3], data[4]
  WHERE TO_INT(data[4]) <= 10000);Output all events to FullStream and only events where the fifth column value is 10,000 or less to LowOrderStream:... OUTPUT to FullStream,
OUTPUT to LowOrderStream (a Integer, b String, c Integer, d String, e String)
SELECT TO_INT(data[0]), data[1], TO_INT(data[2]), data[3], data[4]
  WHERE TO_INT(data[4]) <= 10000);In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2018-10-04

Filtering data in a source


Parsing sources with regular expressionsSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideIntermediate TQL programming: common patternsParsing sources with regular expressionsPrevNextParsing sources with regular expressionsRegular expressions are frequently used to parse source data. See Using regular expressions (regex) for an introduction.In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2017-01-10


Parsing HTTP log entriesSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideIntermediate TQL programming: common patternsParsing sources with regular expressionsParsing HTTP log entriesPrevNextParsing HTTP log entriesThe following example uses the FreeFormTextParser regex property to match several patterns in the following log entry.log {start 1234567890.123456} {addr 123.456.789.012} {port 12345} {method POST} {url /abc/def.ghi} {agent {Mozilla/4.0 (compatible; MSIE 6.0; MS Web Services Client Protocol 1.1.12345.1234)}} {bytes 1234} {status 200} {end 1234567890.123456} {host 123.456.789.012}
...In this case we use a  positive lookbehind construct to match the start, addr, port, method, url, bytes, status, and end patterns, while excluding the log and agent patterns:regex:'((?<=start ).[^}]+)|((?<=addr ).[^}]+)|((?<=port ).[^}]+)|((?<=method ).[^}]+)|((?<=url ).[^}]+)|((?<=bytes ).[^}]+)|(?<=\\(\\#)[^\\)]+|((?<=status ).[^}]+)|((?<=end ).[^}]+)|((?<=host ).[^}]+)'Note that each capture group uses two sets of parentheses. For example,((?<=start ).[^}]+)The inner parentheses are used for the positive lookbehind syntax:(?<=start )The outer parentheses are used for the capture group. In this example, group[1]=1234567890.123456, which is used by the parser in its data array.Here is the TQL of the PARSE statement using the regex expression within a FreeFormTextParser:PARSE USING FreeFormTextParser (
  RecordBegin:'^log ',
  RecordEnd:'\n',
  regex:'((?<=start ).[^}]+)|((?<=addr ).[^}]+)|((?<=port ).[^}]+)|((?<=method ).[^}]+)|((?<=url ).[^}]+)|((?<=bytes ).[^}]+)|(?<=\\(\\#)[^\\)]+|((?<=status ).[^}]+)|((?<=end ).[^}]+)|((?<=host ).[^}]+)',
  separator:'~'
)In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2017-01-10

Parsing HTTP log entries


Extracting substrings from log entriesSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideIntermediate TQL programming: common patternsParsing sources with regular expressionsExtracting substrings from log entriesPrevNextExtracting substrings from log entriesThe MATCH function allows you to match a string using a regex expression (see Functions for details about this function). The 3rd parameter indicates which capture group is used, which may be useful when the input string to a regex can result in multiple capture groups of differing values. In the TQL example, log data in which session information is captured is extracted via the use of basic regex expressions:MATCH(data[5], "(?<=process: )([a-zA-Z0-9//$]*)",1), /* process */
MATCH(data[5], "(?<=pathway: )([a-zA-Z0-9//$]*)",1), /* pathway */
MATCH(data[5], "(?<=service code: )([a-zA-Z0-9//_]*)",1), /* service code */
MATCH(data[5], "(?<=model: )([a-zA-Z0-9]*)",1), /* model */
MATCH(data[5], "(?<=user id: )([0-9]*)",1), /* userId */
MATCH(data[5], "(?<=session IP: )([a-zA-Z0-9//.]*)",1), /* session IP */
MATCH(data[5], "(?<=source: )([a-zA-Z0-9//.//_///]*)",1), /* source */
MATCH(data[5], "(?<=detail: )(.*$)",1) /* detail message */Here is an example of a typical log entry that may contain SEVERE or WARNING messages:Aug 22, 2014 11:17:19 AM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: org.apache.solr.search.SyntaxError: Cannot parse '((suggest_title:(04) AND suggest_title:(gmc) AND suggest_title: AND suggest_title:(6.6l) AND suggest_title:(lb7) AND suggest_title:(p1094,p0234)) AND NOT (deleted:(true)))': Encountered " <AND> "AND "" at line 1, column 64.
Was expecting one of:
    <BAREOPER> ...
    "(" ...
    "*" ...
    <QUOTED> ...
    <TERM> ...
    <PREFIXTERM> ...
    <WILDTERM> ...
    <REGEXPTERM> ...
    "[" ...
    "{" ...
    <LPARAMS> ...
    <NUMBER> ...

    at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:147)
    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:187)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) ...We would like to reduce this information to the following:fftpInfo: fftpOutType_1_0{
  msg: "SEVERE: org.apache.solr.common.SolrException: org.apache.solr.search.SyntaxError: Cannot parse '((suggest_title:(04) AND suggest_title:(gmc) AND suggest_title: AND suggest_title:(6.6l) AND suggest_title:(lb7) AND suggest_title:(p1094,p0234)) AND NOT (deleted:(true)))': Encountered \" <AND> \"AND \"\" at line 1, column 64."
  origTs: 1408731439000
};To do this, we will include the following regex in the FreeFormTextParser properties:regex:'(SEVERE:.*|WARNING:.*)'Here is the complete TQL:create source fftpSource using FileReader (
  directory:'Samples/',
  WildCard:'catalina*.log',
  charset:'UTF-8',
  positionByEOF:false
)
parse using FreeFormTextParser (
-- Timestamp format in log is  "Aug 21, 2014 8:33:56 AM"
  TimeStamp:'%mon %d, %yyyy %H:%M:%S %p',
  RecordBegin:'%mon %d, %yyyy %H:%M:%S %p',
  regex:'(SEVERE:.*|WARNING:.*)'
)
OUTPUT TO fftpInStream;

CREATE TYPE fftpOutType (
  msg String,
  origTs long
);

create stream fftpOutStream of fftpOutType;

create cq fftpOutCQ
insert into fftpOutStream
select data[0],
  TO_LONG(META(x,'OriginTimestamp'))
from fftpInStream x;

create Target fftpTarget using SysOut(name:fftpInfo) input from fftpOutStream;
See FreeFormTextParser for more information.In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2017-01-10

Extracting substrings from log entries


Matching IPv4 subnet octetsSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideIntermediate TQL programming: common patternsParsing sources with regular expressionsMatching IPv4 subnet octetsPrevNextMatching IPv4 subnet octetsAs noted previously, multiple escapes for [ and ] in regular expressions are required because they are reserved characters in both the Striim TQL compiler and Java.match(s.srcIp,'(^\\\\d{1,3}\\\\.)')  - first octet
match(s.srcIp,'(^\\\\d{1,3}\\\\.\\\\d{1,3})\\\\.'), - first and second octet
match(s.srcIp,'(^\\\\d{1,3}\\\\.\\\\d{1,3}\\\\.\\\\d{1,3}\\\\.)') - first, second, and third octetIn this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2017-01-10

Matching IPv4 subnet octets


Parsing SOAP entriesSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideIntermediate TQL programming: common patternsParsing sources with regular expressionsParsing SOAP entriesPrevNextParsing SOAP entriesWe will use the FreeFormTextParser regex property to match several patterns in the following log entry:>> 2015/1/14 16:20:10: :<< Received request from remote address: 123.45.6.789
>> 2015/1/14 16:20:10: :<< Path Name: $name1, Class Name: CLASS-1
>> 2015/1/14 16:20:11: :<< Service Name: Service_1, Response Time: 123.456789 milliseconds
<model>E</model>
<userid>0000000103</userid>
...In this case we also use a  positive lookbehind construct to match the remote address, path name, service name, model, and user ID:regex:'((?<=remote address: )[\\d\\.]+)|((?<=Path Name: )[^ ]+)|((?<=\\<\\< Service Name: )[^,]+)|
  ((?<=Response Time: )[^ ]+)|((?<=\\<model\\>)([a-zA-Z0-9]+))|((?<=\\<userid\\>)([0-9]+))',Here is the TQL of the PARSE statement using the regex expression within a FreeFormTextParser:  PARSE USING FreeFormTextParser (
    RecordBegin:'Start>>> POST INPUT',
    TimeStamp:'>> %yyyy/%m/%d %H:%M:%S: :<<',
    linecontains:'>> %yyyy/%m/%d %H:%M:%S: :<<',
    RecordEnd:' milliseconds',
    regex:'((?<=remote address: )[\\d\\.]+)|((?<=Path Name: )[^ ]+)|
      ((?<=\\<\\< Service Name: )[^,]+)|
      ((?<=Response Time: )[^ ]+)|((?<=\\<model\\>)([a-zA-Z0-9]+))|
      ((?<=\\<userid\\>)([0-9]+))',
    separator:'~'
  )In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2018-06-26

Parsing SOAP entries


Bounding data with windowsSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideIntermediate TQL programming: common patternsBounding data with windowsPrevNextBounding data with windowsWhen a CQ's source is a stream, it is limited to acting on single events, as in this example from MultiLogApp:CREATE CQ GetErrors
INSERT INTO ErrorStream
SELECT log4j
FROM Log4ErrorWarningStream log4j WHERE log4j.level = 'ERROR';This triggers an alert whenever an error appears in the log.To aggregate, join, or perform calculations on the data, you must create a bounded data set. The usual way to do this is with a window, which bounds the stream by a specified number of events or period of time. As discussed in the Concepts Guide (see Window), this may be a sliding window, which always contains the most recent set of events, or a jumping window, which breaks the stream up into successive chunks.The basic programming pattern for a window is:stream > window > CQWhat properties a window should have depend on the nature of the data in the input stream and what you want the CQ to do with it. There are six basic variations depending on whether you are bounding the data batches (jumping) or continuously (sliding) and whether you are bounding by by time, event count, or both.In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2019-11-19

Bounding data with windows


Bound data in batches by timeSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideIntermediate TQL programming: common patternsBounding data with windowsBound data in batches by timePrevNextBound data in batches by timeThe following uses aggregate functions to summarize events in a stream every fifteen minutes:CREATE TYPE OrderType(
  storeId      String,
  orderId      String,
  sku          String,
  orderAmount  double,
  dateTime     DateTime
);
CREATE STREAM RetailOrders Of OrderType;
...
CREATE JUMPING WINDOW ProductData_15MIN
OVER RetailOrders
KEEP WITHIN 15 MINUTE ON dateTime;

CREATE CQ GetProductActivity
INSERT INTO ProductTrackingStream
SELECT pd.sku, COUNT(*), SUM(pd.orderAmount), FIRST(pd.dateTime)
FROM ProductData_15MIN pd;Every 15 minutes, the output stream receives one event per SKU for which there was at least one order. Each SKU's event includes the number of orders (COUNT(*)), the total amount of those orders (SUM(pd.orderAmount)), and the timestamp of the first order in the batch (FIRST(pd.dateTime)). You could use this data to graph changes over time or trigger alerts when the count or total amount vary significantly from what you expect.Say that orders are not received 24 hours a day, seven days a week, but instead drop to zero after closing hours. In that case, the code above could leave events in the window overnight, where they would be mistakenly reported as occurring in the morning. To avoid that, use RANGE to add a timeout:CREATE JUMPING WINDOW ProductData_15MIN
OVER RetailOrders
KEEP RANGE 15 MINUTE ON dateTime WITHIN 16 MINUTE;Now, so long as orders are being placed, the window will jump every 15 minutes, but if orders stop, the window will jump after 16 minutes.In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2019-11-19

Bound data in batches by time


Bound data in batches by event countSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideIntermediate TQL programming: common patternsBounding data with windowsBound data in batches by event countPrevNextBound data in batches by event countAlternatively, you can aggregate data in batches of n number of events. For example:CREATE JUMPING WINDOW ProductData_100
OVER RetailOrders
KEEP 100 ROWS
PARTITION BY storeId;

CREATE CQ GetProductActivity
INSERT INTO ProductTrackingStream
SELECT pd.storeId, COUNT(*), SUM(pd.orderAmount), FIRST(pd.dateTime)
FROM ProductData_100 pd;The output stream receives an event for each batch of 100 events from each store. You might use code like this to trigger a potential fraud alert when a store's order count or amount is anomalously high. (PARTITION BY storeId means the window will contain the data for the most recent 100 orders for each store. Without this clause, the window would contain 100 events total for all stores.)In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2017-01-10

Bound data in batches by event count


Bound data continuously by timeSkip to main contentToggle navigationSelect versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveToggle navigationStriim Cloud 4.2.0 Select versionStriim Cloud 4.2.0 (current release)Striim Platform 4.2.0 (current release)Striim Cloud 4.1.2Striim Platform 4.1.2Striim Cloud 4.1.0Striim Platform 4.1.0Striim documentation archiveprintToggle navigationStriim Cloud 4.2.0 Programmer's GuideIntermediate TQL programming: common patternsBounding data with windowsBound data continuously by timePrevNextBound data continuously by timeThe following variation on ProductData_15MIN summarizes events continuously:CREATE WINDOW ProductData_15MIN
OVER RetailOrders
KEEP WITHIN 15 MINUTE ON dateTime;

CREATE CQ GetProductActivity
INSERT INTO ProductTrackingStream
SELECT pd.sku, COUNT(*), SUM(pd.orderAmount)
FROM ProductData_15MIN pd;Omitting JUMPING from the window definition creates a sliding window (see Window in the Concepts Guide). Every time a new order is received, or one is dropped because it has been in the window for 15 minutes, the output stream receives a new event updating the number of orders and total amount of those orders for the past 15 minutes. FIRST(pd.dateTime) is unnecessary since you know the window always contains the most recent orders.In this section: Search resultsNo results foundWould you like to provide feedback? Just click here to suggest edits.PrevNextSee how streaming data integration can work for
						you.Schedule a
						DemoDownload © 2023 Striim, Inc. All rights reserved. Last modified: 2017-01-10

Bound data continuously by time