Do you, your teams, and your organization need to automatically export your Industrial IoT sensor measurements to any cloud platform?
With your Tools.Valarm.net Data Exports features, you can set up Export Jobs for automated delivery of your asset and environmental monitoring information, whether you’re monitoring water levels or anything else. Your Export Jobs can be configured for various online, cloud-based platforms like:
- Google Drive (and Google Sheets)
- Microsoft OneDrive
- Amazon AWS S3 (covered here in the rest of this blog post below)
- FTP Servers
- Dropbox
In this article, we’ll go over step by step details on how to set up IoT data export jobs for Amazon AWS S3, a.k.a., Simple Storage Service, which is a simple cloud storage service.
Let’s get going. First off you’ll log in to your Tools.Valarm.net account and go to your Device Manager.
Find which of your Industrial IoT devices you’d like to use for data export to your Amazon S3 account folders.
Then click the Export Jobs button on your Device Manager page so we can set up your new S3 export job.
Once you’ve arrived at your Export Jobs page on Tools.Valarm.net, you’ll see a green button for creating a new export job.
Click the button to create a new Export Job for your Industrial IoT sensor measurements.
You’ll see a window pop up where we’ll configure your Amazon AWS S3 IoT Export Job.
First up in your configuration is the Basics section.
Give your Export Job a Name and some Notes in the text fields. Then when you, your teams, or anyone at your organization comes back to this you’ll know the details of what this export job does and why you created it.
Now we’ll configure the Output / Destination settings for your Export Job.
Under Storage Service choose Amazon S3 in the dropdown menu.
For your AWS Region enter the name of the AWS Region where your S3 bucket is located. Examples are ‘us-west-2’ or ‘eu-east-3’. For a complete list see AWS Regions and Availability Zones.
Under S3 Bucket Name, type in the name of your S3 bucket.
Enter your IAM User Access Key and your IAM User Secret in the indicated fields.
Next up is your S3 Storage Class configuration. Select the Storage Class you’d like to use for the exported files. Storage Classes allow you to scale your need for durability and availability against cost. You can read more at Amazon’s S3 Storage Classes page.
If you’d like to save space on your servers, choose a Compression type in the dropdown menu. Compression will greatly reduce the size of the exported file, saving significant space on your storage medium while speeding transit time and reducing consumed bandwidth. In most cases, GZ offers the best compression, and Zip is also excellent. The only reason to select ‘None’ is if you are running an automated process against the exported file which absolutely cannot handle a compressed file.
Type in the Folder Directory where you’d like to save your exported Industrial IoT sensor measurements.
Now choose a File Name Format for your IoT Sensor Export Job:
- Device Name and Date Range: A filename will be generated using the name of the Device or Device Group that is the source of the data, and the date/time range of the query.
- Job Name and Time Stamp: The filename will be generated from the Job Name field of this Export Job, appended with a timestamp for the moment it runs.
- Prefix and Time Stamp‘: A prefix you supply in the Filename Prefix field will be combined with the timestamp of the moment the job runs.
- Static / Fixed: Input the explicit, complete, and exact filename to use for your export. If you leave this field blank, a default filename will be used. Note: Unlike the other filenaming options, this option does NOT append the file ‘format’ extension and/or compression type extensions to the filename. Whatever you enter in the ‘Static/Fixed Filename’ field will be exactly the name of the exported file. If you include characters which are illegal in the output destination, your export may fail.
Based on your choice for your File Name Format, you may enter additional file name info in the text field.
Data Generation is the last configuration section we’ll fill out.
Choose the Schedule of how often you’d like to automatically export your IoT sensor measurements to Amazon S3. These schedule times are ‘on the hour’. Thus ‘Daily’ means at midnight. Twice-daily is at noon and midnight. Weekly is Saturday night at midnight.
Select ‘Manual’ if you wish to trigger the job manually whenever you want it to run.
Next up you’ll choose a Time Zone like UTC. This timezone will be used to render timestamps in the exported data, as well as to name the output file. It will also be used when specifying a query time-span for Manual triggered jobs.
If you’re configuring a Manually Scheduled Export Job, then you can choose a Date Range from and to dates.
Select a Single Device Source Type and choose the Source for which of your Industrial IoT Devices you’d like to export data.
For your Output Format your file can be made in your choice of formats like:
- JSON
- SHEF
- CSV
Choose your Output Columns custom column names group and aliases you’d like to use for column headers on your exported IoT sensor measurements. For more info, have a gander at this tutorial for configuring aliases and custom column names.
If you’d like to filter out any of your data like only exporting IoT sensor measurements less than or greater than your custom value, then you can Enable Threshold Filtering. Export Filtering allows you to set logical-comparator threshold values that each Event record must satisfy in order to be INCLUDED in the exported data. The configurable options are identical to Outbound Alerts and other functions which use Thresholds to make decisions about incoming data.
Last up you’ll choose your Sort order. This is the order that your exported IoT data file will list the measurements based on timestamps. You can choose to sort in descending or ascending order, based on whether you want your latest sensor measurements to be at the top or bottom of your exported IoT data file.
Now remember to Save your IoT Sensor Data Export for Amazon AWS S3.
Then you can click the button Run Now for your new IoT Export Job.
That’s it! After you’ve run your Export Job, you’ll see your IoT sensor measurements in your desired S3 folder.
You can download your sensor information and do what you need to do. In the example screenshot you can see we’ve downloaded the IoT device data from Amazon S3 and opened it in Microsoft Excel to verify the contents.
That’s your overview of using Tools.Valarm.net IoT Export Jobs with Amazon S3 AWS.
Please don’t hesitate to get in touch with me at Info@Valarm.net if you’ve got any questions.