Striim

Welcome to the Striim Help Center and Community Site

3.10.1 Creating apps using templates

Follow

Templates can make creating Striim applications faster and easier. Apps created with templates may be modified using Flow Designer or by exporting TQL, editing it, and importing the modified TQL.

Striim has templates for many source-target combinations. Which are available in your installation will vary depending on which licenses you have (contact Striim support for more information).

When you use a template, the app's exception store is enabled automatically. If you do not want an exception store, turn it off after you create the app (see CREATE EXCEPTIONSTORE).

Use the Search Templates tool on the App Wizard page to see which templates are available for a particular source or target.

Sources include:

  • file (FileReader)

  • Incremental Batch Reader (MySQL, Oracle, PostgreSQL, SQL Server using JDBC)

  • MongoDB (MongoReader)

  • Oracle CDC (OracleReader

  • PostgreSQL CDC (PostgreSQL Reader)

  • SQL Server CDC (MSSQLReader)

Targets include:

  • Azure Blob storage

  • Azure Cosmos DB

  • Azure Event Hub

  • Azure PostgreSQL

  • Azure SQL DB

  • Azure SQL Data Warehouse

  • Cassandra (using DatabaseWriter)

  • Cloudera Hive

  • database (see Database Writer for supported DBMSes)

  • file (using FileWriter)

  • Google BigQuery

  • Google Cloud SQL MySQL

  • Google Cloud SQL PostgreSQL

  • Google Cloud Storage

  • Hadoop (using HDFSWriter)

  • Hazelcast

  • HBase

  • Hive

  • Hortonworks Hive

  • JMS

  • JPA

  • Kafka 0.8, 0.9, 0.10, or 0.11

  • Kinesis

  • Kudu

  • MapR Streams

  • MemSQL

  • MQTT

  • MySQL (using DatabaseWriter)

  • Oracle (using DatabaseWriter)

  • PostgreSQL (using DatabaseWriter)

  • Amazon Redshift

  • Snowflake

  • Google Spanner

  • SQL Server (using DatabaseWriter)

  • Striim (app contains only the source)

App template prerequisite checklist

You will need the assistance of a database administrator for some of these tasks.

  1. Configure the source database as detailed in Incremental Batch Reader, MySQL / MariaDB setupOracle configurationSQL Server setup. You will need to provide the user name and password created during database configuration when configuring the source.

  2. Configure the target as detailed in the relevant topic in the Writers reference.

  3. Install the MySQL, Oracle, or SQL Server JDBC driver in Striim as detailed in Installing third-party drivers.

  4. If the source cannot be read directly by Striim (for example, if Striim is running in Azure or Amazon EC2 and the source is on your premises):

    1. Install the Forwarding Agent and appropriate JDBC driver on the source host as detailed in Striim Forwarding Agent installation and configuration.

    2. Before launching a template, make sure Striim and the Forwarding Agent are both running.

Creating a new application using a template

  1. Complete the App template prerequisite checklist.

  2. In the Striim web UI, go to the Apps page and select Add App > Start with Template.

  3. Click the template you want to use. If you don't see it, enter the source or target in Search Templates.

  4. Enter a name for the app, select or enter the namespace in which you want to create it, and click Save.

Continue with Configuring an app template source.

Configuring an app template source

Note

You must have completed the steps in App template prerequisite checklist or the wizard will not be able to go past this step.

  1. Enter whatever name you wish for this source.

  2. Enter the connection URL, Striim's database user name and password, and the name of the database containing the tables. For Oracle, also select LogMiner or XStreamOut as the log reader. See Incremental Batch Reader, MySQLReader and MariaDBReader propertiesOracle Reader properties, or MS SQL Reader properties for more information.

  3. For an Incremental Batch Reader source, also specify the check column and start position as detailed in Incremental Batch Reader.

  4. If using a Forwarding Agent, set Deploy source on Agent on. (This option appears only when an agent is connected to Striim. For file sources, this option does not appear, but the source components are put in their own flow which can be deployed on the agent.)

  5. Click Next. The wizard will connect to the database to validate the settings. If all the checks pass, click Next.

  6. Select the tables to be read and click Next.

  7. Click Advanced Setup if you wish to map columns to fields or specify multiple output streams based on various criteria. See Using Advanced Setup to filter data based on table valuesUsing Advanced Setup to filter data based on WAEvent values, or Using Advanced Setup to map tables to streams. Otherwise, click Next.

If your template's target is not to Striim, continue with Configuring an app template target.

If your template's target is to Striim, your new application will open in the Flow Designer.

Using Advanced Setup to map tables to streams

The following is a general overview of the process. The details will vary depending on which source and target you use and the columns in the selected table(s).

The following example assumes a MySQL table created as follows:

CREATE DATABASE mydb;
USE mydb;
CREATE TABLE mydb.testtable (merchantId char(36), dateTime datetime, amount decimal(10,2), zip char(5));
GRANT INSERT ON `mydb`.`testtable` TO 'striim'@'%'

1. In the App Wizard, click the application type you want to create. For the following, we will use MySQL to Striim.

2. Enter the properties for the database and click Next.

my01.png

3. After all checks pass, click Next. If the Striim user has access to many tables, the Read Table Metadata step may take a minute or more.

my02.png

4. Select one or more tables to map.

my03.png

5. Click Advanced Setup, then click Add Stream.

my05.png

6. Click Auto Column Map to have Striim automatically pick up field names from the source table(s). Alternatively, click Manual Column Map to specify your own names.

my06.png

7. Select the table.

my07.png

8. Repeat steps 5-7 for each additional table.

9. If you choose Manual Column Map, rename the fields of each stream as you wish.

10. Click Save > Next.

If your template's target is not to Striim, continue with Configuring an app template target.

If your template's target is to Striim, your new application will open in the Flow Designer:

my09.png

In the Flow Designer, you can see that the tables have been mapped. The TQL for this is:

CREATE SOURCE ...
OUTPUT TO MySQL2FileWithMaps_ChangeDataStream,
OUTPUT TO MySQL2FileWithMaps_testtable_Stream1 MAP ( table:'mydb.testtable'),
OUTPUT TO MySQL2FileWithMaps_testtable2_Stream2 MAP ( table:'mydb.testtable2');

See Using the MAP function for more details about this syntax.

Using Advanced Setup to filter data based on table values

The following is a general overview of the process. The details will vary depending on which source and target you use and the columns in the selected table(s).

1. In the App Wizard, click the application type you want to create. For the following, we will use Oracle to File.

2. Enter the properties for the database and click Next.

Screen_Shot_2016-06-24_at_8.39.23_AM.png

3. After all checks pass, click Next. If the Striim user has access to many tables, the Read Table Metadata step may take a minute or more.

Screen_Shot_2016-06-24_at_8.40.30_AM.png

4. Select one or more tables, then click Next..

Screen_Shot_2016-06-24_at_8.41.08_AM.png

5. Click Advanced Setup, then click Add Stream.

2016-06-24_08-42-24.png

6. Click Auto Column Map.

my06.png

7. Select the table.

2016-06-24_08-43-16.png

8. Click Add Filter, then select the field you want to filter by from the drop-down menu.

Screen_Shot_2016-06-24_at_8.46.43_AM.png

9. Set the Operator and Value, then click Save.

filters.png

10. Add additional filters if desired, then click Next.

If your template's target is not to Striim, continue with Configuring an app template target.

If your template's target is to Striim, your new application will open in the Flow Designer. See Filtering events using OUTPUT TO and WHERE for a discussion of the underlying TQL.

Using Advanced Setup to filter data based on WAEvent values

The following is a general overview of the process. The details will vary depending on which source and target you use and the columns in the selected table(s).

1. In the App Wizard, click the application type you want to create. For the following, we will use Oracle to File.

2. Enter the properties for the database and click Next.

Screen_Shot_2016-06-24_at_8.39.23_AM.png

3. After all checks pass, click Next. If the Striim user has access to many tables, the Read Table Metadata step may take a minute or more.

Screen_Shot_2016-06-24_at_8.40.30_AM.png

4. Select one or more tables, then click Next..

Screen_Shot_2016-06-24_at_8.41.08_AM.png

5. Click Advanced Setup, then click Add Stream.

2016-06-24_08-42-24.png

6. Click DB Change Data Stream.

7. Click Add Filter, then select the field you want to filter by from the drop-down menu.

8. Set the Operator and Value, then click Save.

10. Add additional filters if desired, then click Next.

If your template's target is not to Striim, continue with Configuring an app template target.

If your template's target is to Striim, your new application will open in the Flow Designer.

Configuring an app template target

  1. Enter whatever name you wish for this target.

  2. Enter the appropriate properties. For more information, see the Writers entry for the selected target type.

    With a CDC source and a to Azure SQL Database, to Azure SQL Data Warehouse, or to Database target, for the Tables property, specify the source and target table names, separated by commas, separating each source-target pair with a semicolon: for example, dbo.Jobs,mydb.Jobs;dbo.Region,mydb.Region. Alternatively, use wildcards (see the discussion of the Tables property in Database Writer).

  3. Click Next. Your new application will open in the Flow Designer.

Before enabling recovery for a template-created app with a to Azure SQL DB or to Database target, follow the instructions for the target DBMS in Creating the checkpoint table.

Known issue (DEV-14440): If the target is KafkaWriter 0.10 or 0.11 with AvroFormatter, after the application opens in the Flow Designer, edit the target and specify either Schema File Name (see Kafka Writer) or Schema Registry URL (see Using the Confluent or Hortonworks schema registry).

3.10.1
Was this article helpful?
0 out of 0 found this helpful

Comments