3.10.3 Configuring remote hosts

This section describes various ways to configure remote hosts to send data to Striim.

Apache Flume integration

On the system running Flume:

1. Extract striim_FlumeWebActionSink_....tgz on the system running Flume.

2. Save the following as flume/conf/flume-env.sh (replace ... with the correct path):

JAVA_OPTS="-Xms512m -Xmx1024m"
FLUME_CLASSPATH="/.../WebActionSink/lib/*"

If you are setting up the sample application, return to Using Apache Flume and continue with step 2.

3. Create a configuration such as flume/conf/waflume.conf, as described in Using Apache Flume. Alternatively, add the properties to an existing .conf file.

4. Start Flume, specifying the configuration file:

bin/flume-ng agent --conf conf --conf-file conf/waflume.conf --name agent -Dflume.root.logger=INFO,console

collectd configuration

Before an application can use the CollectdParser, the following properties must be set in collectd.conf:

LoadPlugin syslog
LoadPlugin aggregation
LoadPlugin cpu
LoadPlugin df
LoadPlugin disk
LoadPlugin interface
LoadPlugin load
LoadPlugin memory
LoadPlugin network
LoadPlugin rrdtool
LoadPlugin swap
LoadPlugin uptime
<Plugin "aggregation">
  <Aggregation>
    Plugin "cpu"
    Type "cpu"
    GroupBy "Host"
    GroupBy "TypeInstance"
    CalculateNum false
    CalculateSum false
    CalculateAverage true
    CalculateMinimum false
    CalculateMaximum false
    CalculateStddev false
  </Aggregation>
  <Aggregation>
    Plugin "memory"
    Type "memory"
    GroupBy "Host"
    GroupBy "TypeInstance"
    CalculateNum false
    CalculateSum false
    CalculateAverage true
    CalculateMinimum false
    CalculateMaximum false
    CalculateStddev false
  </Aggregation>
  <Aggregation>
    Plugin "swap"
    Type "swap"
    GroupBy "Host"
    GroupBy "TypeInstance"
    CalculateNum false
    CalculateSum false
    CalculateAverage true
    CalculateMinimum false
    CalculateMaximum false
    CalculateStddev false
  </Aggregation>
</Plugin>
<Plugin df>
	MountPoint "/"
	IgnoreSelected false
	ReportReserved false
	ReportInodes false
</Plugin>
<Plugin disk>
	Disk "/^[hs]d[a-f][0-9]?$/"
	IgnoreSelected false
</Plugin>
<Plugin network>
	<Server "127.0.0.1">
	</Server>
</Plugin>
<Plugin "swap">
	ReportByDevice false
	ReportBytes true
</Plugin>

Replace 127.0.0.1 with the UDPReader's IP address. If the UDPReader's port setting is not 25826, specify it as well:

<Plugin network>
  Server "192.168.0.42" "25827"
</Plugin>

See collectd.org for more information.

Multiple remote hosts may send data to the same UDPReader source.

SNMP configuration

This sample snmpd.conf will cause SNMP to collect system information and send it to the receiver of the sample application discussed in SNMPParser.

rocommunity  public
# Default SNMPV3 user name
agentSecName disman
# Creating user, this user will be used by 'monitor' process to fetch system metrics.
createUser disman MD5 sercrt@1
# Setting user to have read only access to system metrics.
rouser disman auth
# SNMPV1 trap destination.
trapsink 10.1.10.114:15021 public
# Instruct agent to monitor disk space
disk / 50% 
# Instruct agent to monitor CPU load.
load 1 1 1
# Setting up monitoring job to send alert/trap message.
monitor -S -D -r 5 -i sysName.0 -i hrSystemDate.0 -i sysUpTime.0 -o ifIndex 
  -o ifSpeed -o ifHighSpeed  -o ifPhysAddress -o ifInOctets -o ifInUcastPkts -o ifInDiscards 
  -o ifInErrors -o ifOutOctets -o ifOutUcastPkts -o ifOutDiscards -o ifOutErrors 
  -o ifOperStatus "Interface Details" ifOutOctets 0 0

Multiple remote hosts may be configured to send to the same trap destination.

Striim Forwarding Agent installation and configuration

The Striim Forwarding Agent is a stripped-down version of the Striim server that can be used to run sources locally on a remote host. For more information, see Using the Striim Forwarding Agent.

To install the Forwarding Agent when you deploy Striim from the AWS Marketplace, contact Striim support.

Striim Forwarding Agent system requirements

memory: 256MB to 1GB depending on adapter used
free disk space: 500MB (free disk space must never drop below 10%)
supported operating systems: any 64-bit version of Linux, Mac OS X, or Microsoft Windows
supported Java environments:
- recommended: 64-bit Oracle Java SE 8 JRE (JDK required to use HTTPReader or SNMPParser)
- also supported: 64-bit OpenJDK 8 JRE
firewall: the following ports must be open:
- port 5701 inbound for TCP (Hazelcast): if 5701 is in use, Hazelcast will use 5702; if both 5701 and 5702 are busy, it will use 5703; and so on
- ports 49152-49162 outbound for TCP (Java Message Queue) for an application sending a single stream to the server (increase the high end of the range by one for each additional stream)
Any driver required by a source that will run on the Agent must be installed in Agent/lib. See Installing third-party drivers.

The Forwarding Agent gets its license from the Striim cluster. It does not need to be specified on the remote host where the agent is installed.

Running the Forwarding Agent as a process

Using a supported web browser, log into Striim at <DNS name>:9080 with username admin and the admin password you specified when creating the Striim server.
Select Help > Download Agent and save Striim_Agent_3.10.3.zip to an appropriate location. If you can not download it directly to the host, download it to your local system and use scp to copy it to the host.
Unzip the file. If you used scp to copy the file to the host, use ssh to log into the host and unzip it.
Follow the instructions in Configuring the Forwarding Agent.

To start the agent, make sure the Striim cluster the Agent will connect to is running, open a shell terminal, command prompt, or ssh session, change to the Agent directory, and enter bin/agent.sh or, in Windows, bin\agent.bat.

Once you have successfully started the agent and connected to Striim, you may delete the .zip file.

To stop the agent, switch to the command prompt in which it is running and press Ctrl-C.

Running the Forwarding Agent as a service in CentOS

Log in to Linux.
Download striim-agent-3.10.3-Linux.rpm.
Install the package: sudo rpm -ivh striim-agent-3.10.3-Linux.rpm. By default, the agent is installed in /opt/striim/agent/.
Follow the instructions in Configuring the Forwarding Agent.
Make sure the Striim server is running, then:
- For CentOS 6, enter sudo start striim-agent.
- For CentOS 7, enter:
```
sudo systemctl enable striim-agent
sudo systemctl start striim-agent
```

To verify that the agent connected to the server, look on the Monitoring page (see Monitoring using the web UI).

Running the Forwarding Agent as a service in Ubuntu

Log in to Linux and run sudo su to switch to root.
Download striim-agent-3.10.3-Linux.deb.
Install the package: dpkg -i striim-agent-3.10.3-Linux.deb. The agent is installed at /opt/striim/agent.
Follow the instructions in Configuring the Forwarding Agent.
Make sure the Striim server is running, then:
- for Ubuntu 14.04, enter start striim-agent
- for Ubuntu 16.04 or later, enter systemctl enable striim-agent && systemctl start striim-agent

To verify that the agent connected to the server, look on the Monitoring page (see Monitoring using the web UI).

Running the Forwarding Agent as a service in Windows

This requires Windows PowerShell 5.0 or later.

Follow the instructions in Running the Forwarding Agent as a process, but do not start the agent.
Start Windows PowerShell as administrator and run the script Agent\conf\windowsAgent\setupWindowsAgent.ps1. Note that if you are in the windowsAgent directory, to run the script you must include the path: .\setupWindowsAgent.ps1.

The message Error in OpenSCManager is of no concern if the script completes successfully.
Start the Striim Agent service manually, or reboot to verify that it starts automatically.

To uninstall the service, stop it, then run this batch file:

Agent/conf/windowsAgent/yajsw_agent/bat/uninstallService.bat

Configuring the Forwarding Agent

Before starting the agent, you must:

Run agent/bin/aksConfig.sh or aksConfig.bat. When prompted, enter a password for the agent's local keystore and the sys password (used to authenticate the agent when it connects to the Striim cluster).
Install any third-party drivers required by the readers that will run in the Forwarding agent (see Installing third-party drivers). Note that the PostgreSQL and SQL Server drivers bundled with the Striim server must be installed manually in the Forwarding Agent.
Edit agent/conf/agent.conf to specify the agent's settings, as follows:

Always required: for the striim.cluster.clusterName property value, specify the cluster to connect to. Cluster names are case-sensitive.
Always required: for the striim.node.servernode.address property value, specify the server's IP address or fully-qualified domain name. If the agent may connect to multiple servers, specify them all, separated by commas.
If the system on which the agent is running has more than one IP address, specify the one you want the agent to use as the value for the striim.node.interface property.
By default, the agent will join the Agent deployment group. If you wish to change that, specify another deployment group as the value for the striim.cluster.deploymentGroups property. If the specified deployment group does not exist, it will be created automatically. When multiple agents on different remote hosts will be used by the same source, they must belong to the same deployment group.
To change the agent's log level, see Setting the log levels.
If the HTTPS port is not 9081, see Configuring HTTP and HTTPS.
When Striim is running in Amazon EC2, specify the server's IPv4 Public IP address (displayed on the EC2 instance's Description tab) as the value for the property.
When Striim is running in Azure, specify the server's DNS name (displayed on the Essentials tab for the Striim VM) as the value for the striim.node.servernode.address property.
When Striim is running in Google Cloud, specify the server's IP address as the value for the striim.node.servernode.address property. (Note that you should change the public IP address for the virtual machine running Striim from Ephemeral to Static or it will change every time the VM is restarted. See Promoting an ephemeral external IP address for instructions.)
When the Striim cluster is not discoverable using multicast UDP (for example, if it is running in Amazon EC2), specify the server's IP address or fully-qualified domain name as the value for the striim.node.servernode.address property. If the server's Hazelcast port is not the default 5701, include that as well (for example, 192.0.2.21:5702).
When the Striim cluster is Using TCP/IP instead of multicast UDP, specify a comma-delimited list of server IP addresses or ranges, for example, ‑S "192.0.2.0-7,192.0.2.21", as the value for the striim.node.servernode.address property.
When the Striim cluster and Forwarding Agent are on different private networks linked by VPN with a NAT gateway, you cannot specify the server's IP address as the value for striim.node.servernode.address since it is not reachable by the agent. To work around this, create a DNS name for the Striim server that resolves to a different IP address in each environment.

For example, say that in the Striim cluster's network the DNS name striim.mycompany.com resolves to 192.0.2.101 and on the Forwarding Agent's network it resolves to 192.51.100.25. On each server in the Striim cluster, edit /etc/hosts and add 192.0.2.101 striim.mycompany.com and in startUp.properties set ServerFQDN=striim.mycompany.com. On the agent system, edit /etc/hosts and add 192.51.100.25 striim.mycompany.com. Restart the server(s), then restart the agent.

Testing the Forwarding Agent

You may use this application (the sample data is installed with the agent) to verify that the agent is working:

CREATE APPLICATION agentTest;

CREATE FLOW AgentFlow;
CREATE SOURCE CsvDataSource USING FileReader (
  directory:'SampleData',
  wildcard:'PosDataAgentSample.csv',
  positionByEOF:false)
PARSE USING DSVParser (
  header:Yes,
  trimquote:false)
OUTPUT TO CsvStream;
END FLOW AgentFlow;

CREATE FLOW ServerFlow;
CREATE TARGET t USING FileWriter( filename:'AgentOut')
FORMAT USING DSVFormatter ()
INPUT FROM CsvStream;
END FLOW ServerFlow;

END APPLICATION agentTest;

Deploy AgentFlow in the Agent group and ServerFlow and agentTest in the Default group. Then run the application and verify that AgentOut.00 has been written to the Striim program directory.

Updating the sys user password on the Forwarding Agent

If the sys user's password changes, use the following command to update the password (required to connect to a Striim cluster) on the agent:

agent/bin/aksConfig.sh  -p <new password>

Enabling agent failover

When a flow in an application with recovery enabled is deployed ON ONE (see DEPLOY APPLICATION) in a deployment group with more than one agent, and that agent goes down, Striim will automatically deploy the flow on another agent in the deployment group. The flow will continue running on the other agent even after the original agent comes back up.

3.10.3

Striim