Striim

Welcome to the Striim Help Center and Community Site

3.10.3 Creating or modifying apps using the Flow Designer

Follow

The Flow Designer is a graphical programming tool that can do almost everything you can do with TQL.

Warning

Making changes to a component that affect the number or type of fields in its output stream (for example, removing a field from a CQ's SELECT statement) can cause downstream objects to become invalid. Before attempting such changes in Flow Designer, make a TQL backup of the application by selecting Configuration > Export.

The Flow Designer may become unresponsive when a type has a very large number of fields, such as 1,000.

If you encounter either of these problems, make your changes in TQL instead (see ALTER and RECOMPILE).

See Modifying an application using the Flow Designer for a walkthrough.

component_palette.png

The Base Components palette may be more efficient for experienced users. Any of the components in the Sources, Enrichers (caches), Processors (CQ and window), or Targets palettes may be created by setting the properties of a base component. See DDL and component reference for information on component properties.

ConfigureSourceUsingWizard.png

Some readers have a Configure using wizard button that will launch the App Wizard (see Configuring an app template source).

The components on the Transformers palettes provide a graphical alternative to writing CQs. See Using database event transformers and Using database event transformers.

When you view a running application in Flow Designer, input and output total event counts, current input and output rate, and a graph of input and output rates appear at the top of the page.

Use Configuration > App Settings to configure recovery, data validation, encryption, exception handling, and auto-restart (see Recovering applications, Creating a data validation dashboard), CREATE APPLICATION ... END APPLICATION, and Automatically restarting an application after a crash).

Notes:

  • An application must be in the Created state to be editable. Applications in the Deployed or Running state are read-only.

  • Click Metadata Browser at the top right or copy from in source and target editorssto view and copy settings from other flows or applications.

  • Click and drag to select multiple components to cut and paste in another flow, or to copy and paste in another application.

Creating an app using the Flow Designer

This tutorial uses the sample applications described in Running the CDC demo apps. The Docker PostgreSQL instance must be running. (Kafka is not required.)

  1. On the Apps page, if ValidatePostgres is not running, deploy and start it.

  2. Select Add App > Start from Scratch, name the app PG2File, select your personal namespace or enter a name for a new namespace, and click Save.

  3. Open the Metadata Browser, open the Source folder, hover the mouse over ReadCDCfromPostgreDB, and click Copy to App.

    MetadataBrowser.png
  4. Name the source ReadPostgreSQLCDC, name the new output stream rawCDCstream, and click Save. Except for the name and output stream, this new source will have the same properties as SamplesDB.ReadCDCfromPostgreDB.

    ReadPostgreSQLCDC.png
  5. Hover the mouse over the stream icon, click +, and select Connect next Target component.

    ConnectNextComponent1.png
  6. Set the properties as shown below and click Save.

    WriteRawData2.png
  7. Deploy and run the application, then return to the Apps page and start SamplesDB.Execute200Inserts.

  8. Open Striim\MyDirectory\RawData.00. This contains the raw WAEvent output of PostgreSQLReader formatted as JSON:

     {
      "metadata":{"TableName":"public.customer","TxnID":584,"OperationName":"INSERT" ...
      "data":{
    "C_CUSTKEY":151251,
    "C_NAME":"Customer#151251",
    "C_ADDRESS":"IVhzIApeRb ot,c,E",
    "C_NATIONKEY":15,
    "C_PHONE":"25-989-741-2988",
    "C_ACCTBAL":"711.56",
    "C_MKTSEGMENT":"BUILDING  ",
    "C_COMMENT":"to the even, regular platelets. regular, ironic epitaphs nag e"
    },
      "before":null,
      "userdata":null
     }

    This includes the column names and values for the row inserted into the public.customer table by transaction ID 584.

  9. To parse this raw data, you must write a CQ as described in Parsing the data field of WAEvent. Reopen the PG2File app in Flow Designer, hover the mouse over the stream icon, click +, and select Connect next CQ component.

    ConnectNextComponent2.png
  10. Name the CQ ParseData, name the new output stream ParsedDataStream, and copy and paste the following into the Query field:

    SELECT 
      META(rawCDCstream,"OperationName").toString() AS OpType,
      TO_INT(data[0]) AS CustomerKey,
      TO_STRING(data[1]) AS CustomerName,
      TO_STRING(DATA[2]) AS CustomerAddress,
      TO_INT(DATA[3]) AS NationKey,
      TO_STRING(DATA[4]) AS CustomerPhone,
      TO_DOUBLE(data[5]) AS CustomerAccountBalance,
      TO_STRING(DATA[6]) AS MarketSegment,
      TO_STRING(data[7]) AS CustomerComment
    FROM rawCDCstream;

    Make sure the properties look as shown below, then click Save.

    ParseDataCQ2.png
  11. Hover the mouse over the lower stream icon, click +, and select Connect next Target component.

    ConnectNextComponent3.png
  12. Set the properties as shown below and click Save.

    WriteParsedData2.png
  13. Deploy and run the application, then return to the Apps page and start SamplesDB.Execute200Inserts.

  14. Open Striim\MyDirectory\ParsedData.00. This contains the parsed data in the custom format defined by the CQ as well as the operation type from the metadata.

    "OpType":"INSERT",
    "CustomerKey":151501,
    CustomerName":"Customer#151501",
    "CustomerAddress":"IVhzIApeRb ot,c,E",
    "NationKey":15,
    "CustomerPhone":"25-989-741-2988",
    "CustomerAccountBalance":711.56,
    "MarketSegment":"BUILDING  ",
    "CustomerComment":"to the even, regular platelets. regular, ironic epitaphs nag e"
    

Using event transformers

Event transformers are a graphical alternative to CQs (see CREATE CQ (query)) for modifying data flows. For the transformers discussed in this section, the input stream must have a user-defined type.

If you convert a transformer to a CQ, you can not convert it back to a transformer.

Field Adder
FieldAdder.png

Adds one or more fields and creates a corresponding type for the output stream. For each field, enter a name, select a data type, and enter an expression.

Expressions may use the same Operators and Functions as CQ SELECT statements. Enclose literal string values in quotes.

Field Enricher
FieldEnricher.png

Joins fields from a stream with fields from a cache, event table, WActionStore, or window (see Joins). Select a stream from Input, another component from Enrichment Component, and the fields to join on from Input Field and Enrichment Component Field. All the fields from both components will appear in Output Fields (Input fields will start with i_ and Enrichment Component fields with e_.) Delete any fields you don't want by clicking x and reorder them as you like by dragging up or down. Optionally, click Convert to CQ to specify a particular type of join (see CREATE CQ (query)).

Field Masker

See Masking functions.

Field Remover
FieldRemover.png

Removes one or more fields and creates a corresponding type for the output stream. Click the X to remove a field. Click RESET to start over.

Field Renamer
FieldRenamer.png

Renames one or more fields and creates a corresponding type for the output stream.

Field Splitter
FieldSplitter.png

Splits a string field into multiple string fields using the specified delimiter. The delimiter may include any combination of alphanumeric characters, spaces, and punctuation.

Field Type Modifier
TypeModifier.png

Changes the data type of one or more fields and creates a corresponding type for the output stream.

The following conversions should work without problems. Others may cause the application to crash.

input type

output type

Byte

Double, Integer, Float, Long, Short, or String

DateTime

String

Double

String

Float

Double or String

Integer

Double, Float, Long, or String

Long

Double, Float, or String

Short

Double, Integer, Float, Long, or String

String

Byte, Double, Float, Integer, Long, or Short, provided the values are compatible

Field Value Filter
FieldValueFilter.png

Filters the output stream based on one or more specified conditions. Each additional condition narrows the criteria, so for the output for the example above the output will contain events with amounts from $5,000 to $10,000.

To DB Event
ToDBEvent.png

Converts events of a user-defined type to WAEvent format. The output stream's type is WAEvent and can be used as the input for a target using DatabaseWriter or another writer accept input from DatabaseReader, IncrementalBatchReader, or a SQL CDC source. In the output, the table name will be the value of the metadata TableName field. In this release, the value of the metadata OperationName field will always be INSERT.

For example, using the settings shown above, the following input event:

PosDataStream_Type_1_0{
  merchantId: "D6RJPwyuLXoLqQRQcOcouJ26KGxJSf6hgbu"
  dateTime: 1363134730000
  amount: 2.2
  zip: 41363
};

would be output as:

WAEvent{
  data: ["D6RJPwyuLXoLqQRQcOcouJ26KGxJSf6hgbu",1363134730000,2.2,41363]
  metadata: {"TableName":"mydb.mytable","OperationName":"INSERT"}
  userdata: null
  before: null
  dataPresenceBitMap: "Dw=="
  beforePresenceBitMap: "AA=="
  typeUUID: {"uuidstring":"01e91cfe-b88a-8ef1-8e39-8cae4cf129d6"}
};

Using database event transformers

Database event transformers are a graphical alternative to CQs for modifying the output of JDBC CDC sources (see CREATE CQ (query) and Working with SQL CDC readers).

In this release, you may use DB event transformers with sources.

If you convert a transformer to a CQ, you can not convert it back to a transformer.

Expressions may use the same Operators and Functions as CQ SELECT statements. Enclose literal string values in quotes.

Custom Data Adder
CustomDataAdder.png

Adds one or more fields to the USERDATA map (see Adding user-defined data to WAEvent streams). For each field, enter a name, select a data type, and enter an expression.

Data Modifier
DataModifier.png

Modifies one field in the DATA array. Select the table, select the column, and enter an expression.

In this release, this transformer can be used only with MSSQLReader, MySQLReader, or OracleReader, and Striim must be able to connect to the database to download table descriptions.

Metadata Filter
MetadataFilter.png

Filters the input events based on one or more metadata field values.

Operation Filter
OperationFilter.png

Passes only events with the selected operation type to the output stream. Select DELETE, INSERT, or UPDATE.

Table Filter
TableFilter.png

Passes only the selected table to the output stream.

In this release, this transformer can be used only with MSSQLReader, MySQLReader, or OracleReader, and Striim must be able to connect to the database must be online so Striim can connect to it to download table descriptions.

To Event
ToEvent.png

Creates a Striim Type for the output stream based on the column names and data types of the selected table and converts WAEvent input to typed output (see Parsing the data field of WAEvent). Only events from the selected table are included in the output stream.

In this release, this transformer can be used only with MSSQLReader, MySQLReader, or OracleReader, and Striim must be able to connect to the database must be online so Striim can connect to it to download table descriptions.

To Staging
ToStaging.png

Copies the values of the OperationName field in the METADATA map to the OrigOperationName field in the USERDATA map (see Adding user-defined data to WAEvent streams). Typically you would use this in an application that writes to a data warehouse or other target where UPDATE and DELETE operations are handled as INSERTs (see How update and delete operations are handled in writers).

3.10.3
Was this article helpful?
0 out of 0 found this helpful

Comments