In my last blog, I was mentioning how a user can trigger a batch ETL job dynamically using the power of ISD. And we saw how the user can dynamically choose the input and output file. But there was a limitation, these files need to be available on the server where we have InfoSphere Information Server installed. Can we modify the behavior in a way that the input and output file exist on Cloud? In that case we have made the ETL batch job a step closer to be Cloud ready. Lets explore this.
In last blog we saw the following job taking the input and output from a file.
In my earlier blog I mentioned how in Information Server 11.3 release there were some Cloud Additions (to S3). Now we can replace the input and output of the above DataStage job with S3 connector (connector that can directly connect to the storage available on Amazon). This is as simple as deleting the Sequential File input and output and replace them with S3 connector. The job now looks like the following.
To access a File on S3, we require the following parameters. Access Key, Secret Key, Bucket, File name. Now we double click the Amazon_S3 stage to edit the properties and parametrize these values for both the input and output stage.
In another minute we can compile and deploy this DataStage job as SOAP or REST Service. Now dynamically we can pick a file from Cloud, process it and then put the result back on Cloud! We do not have to write an app to test it also as there are many free plugins that can be used to verify this new Rest Service that was created. I typically use the Poster Plugin of Firefox Web Browser to compose the Rest request and view the response.