Big Data and Infosphere Streams – Are you future ready?

Big data is more than a challenge; it is an opportunity to find insight in new and emerging types of data, to make your business more agile, and to answer questions that, in the past, were beyond reach. IBM InfoSphere Streams allows you to capture and act on all of your business data… all of the time… just in time. InfoSphere Streams radically extends the state-of-the-art in big data processing; it’s a high-performance computing platform that allows user-developed applications to rapidly ingest, analyze, & correlate information as it arrives from thousands of real-time sources.

Stream Computing:
Stream computing is a new paradigm. In “traditional” processing, one can think of running analytic queries against historic data: for instance – calculate the distance walked last month from a data set of subscribers who transmit Global Positioning System (GPS) location data while walking. With stream computing, one can execute a process similar to a “continuous query” that keeps running totals, as location information from GPS data is refreshed moment by moment. In the first case, questions are asked of historic data, in the second case, data is continuously evaluated by static questions. InfoSphere Streams goes further by allowing the continuous analysis to be modified over time.

Stream computing is a new paradigm. In “traditional” processing, one can think of running analytic queries against historic data: for instance – calculate the distance walked last month from a data set of subscribers who transmit Global Positioning System (GPS) location data while walking. With stream computing, one can execute a process similar to a “continuous query” that keeps running totals, as location information from GPS data is refreshed moment by moment. In the first case, questions are asked of historic data, in the second case, data is continuously evaluated by static questions. InfoSphere Streams goes further by allowing the continuous analysis to be modified over time.

Historic Data Vs Streaming Data
Historic Data Vs Streaming Data


Architecture:
A college professor once filled a large glass jar with rocks, and then asked students if the jar was full. Most replied yes, it was full. The professor proceeded to add small pebbles, and after shaking the jar was able to add a significant number of pebbles. The professor again asked if the jar was full. Having learned, most students replied no. The professor next added sand, and after again shaking the jar had sand filling all the small openings up to the top of the jar. Again, the professor asked, “Is the jar full?” Several students answered that yes, the jar was full; thinking no more rocks, pebbles or sand could be added. The professor next pulled out a pitcher of water and poured it into the jar. The professor explained that the jar was analogous to life and that if people didn’t fill their life with important things first, just like the big rocks, they would never find time to add them later.

But the story illustrates another principle. Using the smallest of building blocks, there is less wasted space. Water molecules easily fill nearly invisible spaces in the sand filled jar. Streams uses this principle to optimize performance and latency. Very fine grained operators are used in streaming applications, which are then fused together into Processing Elements for deployment onto Streams for execution. An advanced compiler converts the high level declarative Streams Processing Language (SPL) into machine language, ready for execution. The compiler also detects stateless and stateful operators, enabling the use of multiple threads on multicore computers. This advanced capability not only facilitates developer agility to solve the many core programming challenge, but enables outstanding performance and very low latency processing.

For more information
InfoSphere Streams and associated products to build them
Bringing Big Data to enterprise
IBM InfoSphere Streams White Paper
IBM Software Universe 2011

One thought on “Big Data and Infosphere Streams – Are you future ready?

  1. ”the jar was analogous to life and that if people didn’t fill their life with important things first, just like the big rocks, they would never find time to add them later” … This is interesting😉

    But I am not able to relate this story with Streams Technology. While you were penning this blog entry, were you immersed in thinking about some big rocks in life that you need to fill with ?😉

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s