IBM Bluemix Data Connect

I have been tracking the development on IBM Bluemix Data Connect quite closely. One of the reason is that I was a key developer in the one of the first few services that it launched almost two years back under the name of DataWorks. Two weeks back I attended a session on Data Connect by the architect and saw a demo. I am impressed at the way it has evolved since then. Therefore I am planning to re-visit DataWorks again, now as IBM Bluemix Data Connect. In this blog I will reconcile the role that IBM Bluemix Data Connect play in the era of cloud computing, big data and the Internet of Things.

Research from Forrester found that 68 percent of simple BI requests take weeks, months or longer for IT to fulfill due to lack of technical resources. So this entails that the enterprises must find ways to transform line of business professionals into skilled data workers, taking some of the burden off of IT. It means business users should be empowered work with data from many sources—both on premises and in the cloud—without requiring the deep technical expertise of a database administrator or data scientist.

This is where cloud services like IBM Bluemix Data Connect comes into picture. It enables both technical and non-technical business users to derive useful insights from data, with point and click access—whether it’s a few Excel sheets stored locally, or a massive database hosted in the cloud.

Data Connect is a fully managed data preparation and movement service that enables users to put data to work through a simple yet powerful cloud-based interface. The design team has taken lot of pain to design the solution in most simplistic way, so that a basic user can quickly get started with it. Data Connect empowers the business analyst to discover, cleanse, standardize, transform and move data in support of application development and analytics use cases.

Through its integration with cloud data services like IBM Watson Analytics, Data Connect is a seamless tool for preparing and moving data from on premises and off premises to an analytics cloud ecosystem where it can be quickly analyzed and visualized. Furthermore, Data Connect is backed by continuous delivery, which adds robust new features and functionality on a regular basis. Its processing engine is built on Apache Spark, the leading open source analytics project, with a large and continuously growing development community. The result is a best-of-breed solution that can keep up with the rapid pace of innovation in big data and cloud computing.

So here are highlights of IBM Bluemix Data Connect:

  • Allow technical and non-technical users to draw value from data quickly and easily.
  • Ensure data quality with simple data preparation and movement services in the cloud.
  • Integrate with leading cloud data services to create a seamless data management platform.
  • Continuous inflow of new and robust features
  • Best-of-breed ETL solution available on Bluemix  – IBMs Next-Generation Cloud App Development Platform

Cognitive 7 – Challenging Cyber-Criminals

It’s estimated that cyber-crime will cost the global economy more than $2 trillion by 2019—and could be the greatest threat to every company in the world! Since I have your attention now, lets spend some more time detailing this statement that I made.

cybercrimeOrganized criminals, hacktivists, governments and adversaries are compelled by financial gain, politics and notoriety to attack organizations’ most valuable assets. Their operations are well funded and business-like. However, cybersecurity solutions of the past often fail to protect against increasingly sophisticated attack methods. The result is more severe security breaches happening more frequently (50% increase in past year alone) which are geared towards stealing higher-value records such as health-related personally identifiable information (PII) and other highly sensitive data.

3 Top Security concerns of companies:

  • Optimizing their security program to unite security silos, reduce complexity and lower costs as opposed to buying dozens of products from dozens of vendors to nail down all kind of security risks.
  • Stopping advanced threats to prevent, detect and respond to known and unknown threats throughout the entire attack lifecycle. Currently on average a company takes 201 days to identify a data breach!
  • Safeguarding critical assets across users, applications, sensitive data and infrastructure, in the cloud, on mobile or on-premise. Note more than 55 percent of all attacks are carried out by malicious insiders or inadvertent actors

Protecting against security threats:

Cybersecurity  can be compared to an immune system. When you get a cold or virus, the immune system understands the virus and sends data through your central nervous system (security intelligence) to create white blood cells and antibodies to gather info, prioritize and take action. This is what’s called the “immune response.” Faced with an attack—whether to a server, mobile device, application, etc.—an effective security solution should detect that breach and work to stop it, similar to how an immune system would function.

Continuing with our comparison to fighting infection, think about how global health organizations share data to stop a pandemic. For cybersecurity, a partner ecosystem serves that same purpose. When organizations collaborate and share their data on new or changing threats, the other organizations can more quickly assess their own vulnerability based on that data and put appropriate protections in place.

Why Cognitive security?

Because even with major advances in technology, the tools available to today’s security analysts aren’t enough to keep up with the number of security events they see every day. There’s a tremendous amount of security knowledge created for human consumption, yet most of it is untapped. 80% of all data on the internet, including blogs, articles, videos, reports, alerts and other information cannot be processed by traditional security tools, and it’s this unstructured data which often proves most valuable in detecting and stopping threats before they cause harm.

Cognitive computing has the ability to tap into and make sense of all of the security data that has previously been dark. A cognitive systems continues to observe events and behaviors—distinguishing the good from the bad—the ability to use integrated defenses to block new threats gets stronger and stronger. By helping to make security analysts more effective and accelerating the response to emerging threats, cognitive security will help to address the current security skills gap, bringing heightened levels of confidence and risk control.

Cognitive 6 – Meet The Watson Family

Watson is a technology that can learn, compare, analyze, reason, justify and process human language, so there must be countless ways to create benefit from it. Watson doesn’t replace people, but acts as a trusted advisor that can quickly read and understand massive amounts of information and provide suggestions or warnings to help people make the best decisions. In fact, Watson is already interacting with 1 billion people in 8 languages. Let us look at some specific Watson offerings that are helping clients.

Some specific Watson offerings helping clients

Watson Knowledge Studio
Enables subject matter experts and developers to teach Watson the linguistic nuances of industries and knowledge domains—without writing a single line of code. Click here to get a series of videos on Watson Knowledge Studio. It can help an auto manufacturer to proactively identify safety defects using traffic incident reports as shown below.
Watson Explorer
Accesses and analyzes structured and unstructured content—presenting data, analytics and cognitive insights in a single view, giving users the information they’re looking for while uncovering trends, patterns and relationships.

Watson Company Analyzer
Watson Company Analyzer applies cognitive technology to enhance understanding of business accounts – creating holistic company profiles. It does this by collecting, filtering and analyzing structured and unstructured information. By building these comprehensive company profiles enables companies to better understand customers and accelerate opportunities.

Watson Engagement Advisor
Interacts with customers, listens to questions and offers solutions, learning with every human interaction and growing its collection of knowledge—quickly adapting to the way people think.

 Watson Health Solutions
From advancing personal training to cancer treatment, from clinical trial matching to insulin pumps that think, Watson Health is transforming the way organizations deliver health and wellness. Imagine you’re a doctor treating cancer patients. It’s impossible for you to keep up with the vast amount of new data from medical journals, clinical trials, academic research and other sources that would help you provide the best treatment recommendations. That’s where Watson comes in. It can ingest all of this data and find potential connections or correlations that you wouldn’t find on your own—providing you the insight to make the best decisions about treatment options for the specific type of cancer your patient has.

To discover more about Watson through education modules and courseware, visit IBM Watson Academy.

Cognitive 5 – IBM Watson as a cloud service

After its success on Jeopardy!, IBM research worked to make this technology open for public use. Later, IBM established a separate business unit for Watson called The Watson Group and a dedicated workforce to continuously improve Watson’s capabilities. The aim is to bring the power of Watson and cognitive computing to market using cloud delivery models. With those efforts, some of the Watson capabilities are now available on IBM Bluemix. These are available “as a service,” meaning you can use them in your own applications and services, embed them anywhere using Watson APIs and enhance your application capabilities dramatically. This also means that soon you might be able to do all the magic behind the Jeopardy! challenge within your application, with just a couple of clicks!

We can open the Bluemix dashboard and start using these services. Here they are:


Using AlchemyAPI, developers can perform tasks such as extracting the people, places, companies, and other entities mentioned in any publicly-accessible webpage, posted HTML/text document, or a predefined corpus of news articles.

Using Conversation APIs, you can add a natural language interface to your application to automate interactions with your end users. Common applications include virtual agents and chat bots that can integrate and communicate on any channel or device.

Document conversion
The IBM Watson Document conversion service converts a single HTML, PDF, or Microsoft Word™ document into a normalized HTML, plain text, or a set of JSON-formatted Answer units that can be used with other Watson services.

Language Translation:
It dynamically translate news, patents, or conversational documents and can instantly publish content in multiple languages. As a result French-speaking staff can be empowered to instantly send emails in English.

Watson Personality Insights:
Personality Insights derives insights from transactional and social media data to identify psychological traits which determine purchase decisions, intent and behavioral traits; This can be utilized to improve conversion rates.

Retrieve and Rank service
Watson Retrieve and Rank service helps users find the most relevant information for their query by using a combination of search and machine learning algorithms to detect “signals” in the data. The Retrieve and Rank Service can be applied to a number of information retrieval scenarios. For example, an experienced technician who is going onsite and requires help troubleshooting a problem can use this.

The Text to Speech and Speech to Text
Text to Speech service processes text and natural language to generate synthesized audio output complete with appropriate cadence and intonation (multi language support exists). The Speech to Text service converts the human voice into the written word.

Tone Analyzer
This API leverages cognitive linguistic analysis to identify a variety of tones at both the sentence and document level. It detects three types of tones, including emotion (anger, disgust, fear, joy and sadness), social propensities (openness, conscientiousness, extroversion, agreeableness, and emotional range), and language styles (analytical, confident and tentative) from text. This insight can then used to refine and improve communications.

Tradeoff Analytics
This API helps people make better choices while taking into account multiple, often conflicting, goals that matter when making that choice. The service can be used to help make complex decisions like what mortgage to take, and also for helping with more everyday ones like which laptop to purchase.

Using these Watson APIs now you can build cognitive apps that help enhance, scale, and accelerate human expertise. In our next blog, we will explore some of these cognitive apps. Stay tuned.

Cognitve 4 -Cognitive Computing in Action

There is growing interest and imagination about how Cognitive computing can help organizations to think in whole new ways. To see cognitive in action, here are some Industry specific examples of organizations benefiting from power of cognitive computing.


Cognitive systems are driving more personalized shopping experiences and helping reveal customer trends. Macy’s, a U.S. department store chain, is using Watson cognitive technology to create an in-store mobile companion that assists in servicing customer needs. It allows customers to input natural language questions regarding each participating store’s unique product assortment, services and facilities and receive a customized response to the inquiry. Click here for a video.


Cognitive systems are improving the client experience and driving more efficient operations. Bradesco, the second largest bank in Brazil, used Watson as a virtual agent assistant to answer banking questions, reducing the number of calls; improved its client experience and delivery using Watson Self Service Assistant for a new virtual channel; and kept its innovation leadership position by implementing cognitive computing and being the first Watson Portuguese implementation.


Cognitive systems are transforming global health. Medtronic, a U.S. and Ireland-based medical device company, has insulin pumps which, combined with Watson, will help predict dangerous spikes or drops in blood sugar hours in advance, then notify people living with diabetes so they can take action before it becomes a problem.

Energy and Utilities

Cognitive systems are acting as a trusted advisor. Woodside, an Australian oil and gas producer, needed to access knowledge from decades worth of projects to save the company millions of dollars. With Watson Engagement Advisor, they created a service that culls through 30 years of documented expertise, recognizing patterns and continually learning from them. Watson acts as a trusted advisor answering questions from engineers, while learning through newly adapted knowledge. Here is a video on it:

In next blog we will explore some of the Watson offerings (aka APIs) available for consumption. So stay tuned.

Cognitive 3 – What is Watson ?

In my previous blog, we discussed how cognitive business understands, reasons, learns and interacts. Watson is IBM’s brand for cognitive capabilities.

Watson came in lime light when it appeared as a contestant on the US game show Jeopardy! where it handsomely beat two of the show’s best ever contestants (it’s winning total was more than three times that of second placed Ken Jennings). The show poses answers, and contestants must correctly identify the question being asked. For example, one puzzle Watson faced was “Jodie Foster took this home for her role in Silence of the Lambs”. Watson correctly inferred that in this content “took this home” meant “winning an Oscar”. Sometimes “took this home” infers a cold, groceries, or any number of things. Watson’s cognitive system enabled it to behave with human-like characteristics and correctly understand the context.


How does Watson provided answers to those questions?

Watson did the following to provide the correct answer:
1. Question Analysis – In this step, Watson tries to figure out what the question is asking for, and what the answer type (should) be.
2. Hypothesis Generation – Here, Watson creates hundreds of different possible candidate answers. Later Watson will prioritize one of the answer as correct.
3. Hypothesis and Evidence Scoring – Now, Watson weighs each answer. It downgrades or upgrades answers, by looking at the evidence that does not or does support the hypothesis.
4. Final Merging and Ranking – Finally, Watson ranks all the candidate answers, and displays the top 3 answers. It gives confidence scores for each candidate answer and says out the final, first ranked answer.

In 2011 comprised what is now a single API—Q&A—built on five underlying technologies (Natural Language Processing, Machine Learning, Question Analysis, Feature Engineering, and Ontology Analysis). Since then, Watson has grown to a family of 28 APIs. By the end of 2016, there will be nearly 50 Watson APIs— with more added every year! Each API is capable of performing a different task, from recognizing bias in language to gathering information in news reports. In combination, these APIs can be adapted to solve any number of business problems or create deeply engaging experiences. And soon Watson will have ability to interpret data that human senses cannot, such as infrared and sonar.

How Watson is different from Traditional computer Systems?

Traditional computer systems depend on a knowledge base of structured information. They are limited in the kinds of information they can use, and that information must be analyzed and structured for them before they can use it. In contrast, Watson can read unstructured information and figure out its contents, giving it access to a much larger body of information and allowing it to digest that information with much less pre-processing. Because Watson is trained on a corpus of knowledge rather than being programmed, it has more flexibly and so it understands what we are looking for. And it’s ranking of answers helps humans make better decision.

I will explore some of Watson’s APIs / customer use cases in my future blogs. Stay tuned…

Cognitive 2 – The 3 eras of computing

In my last blog I mentioned that today we are at the beginning or the dawn of the cognitive computing era.  And so, what does that mean?  There are three eras of computing and each different from the one preceding it.

Screen Shot 2016-08-03 at 11.28.34 AM

Let us start with the first era, which actually we consider as the tabulating era.  This started maybe in the late 1800s, turn of the century, and was really about machines that were doing counting and tabulation, punch card readers or maybe special purpose machines. These helped with tasks such as controlling industrial looms or measuring voting and census data. They did it really well, but were ultimately limited in that single task.

Then came the programmable era. This starts with the dawn of the digital computer around in the 1950s.  And the big change here is that you have general purpose computing systems that are programmable — they can be reprogrammed to perform different tasks and do a variety of things.  People created software that used programming languages to give computers more complex tasks. Their performance, however, was limited by their adherence to established processes and decision-trees, by their inability to find relationships within unstructured information and they were also somewhat constrained in the way they interact with humans.

But what we see today and emerging over the last few years is something we call cognitive computing era. The major driver for this era is this sudden exponential increase in the amount of unstructured data that’s out there.  And so the challenge is, what are the computing technologies that can really leverage all of this unstructured information in a much more natural way?  And we believe that the only way we’re really going to survive with this onslaught of data is to create whats being called cognitive computing systems.

Today’s cognitive systems understand, learn, and communicate with people in natural language rather than software code. They can extract meaning from large amounts of visual, verbal, and numerical unstructured information; and they can learn as they do so, helping people make complex decisions based on Big Data. This era is leading to creation of automated IT systems that are capable of solving problems without requiring human assistance.

Lets get a taste of the cognitive systems through the means of an example. Lets consider the following question -“Did Maya shoot an elephant in her pajamas?
There are some ambiguities in the statement –  Did Maya shoot a gun or a photo? Was she or the elephant wearing pajamas? Although a human mind can resolves the question’s ambiguities based on context, but a conventional analytic software program cannot. A cognitive system is capable of resolving these ambiguities. First it creates multiple hypotheses about the meaning of question elements such as what object was shooting and who was wearing pajamas. Then it examines those elements against the context of its corpus–the set of documents that constitutes its essential knowledge. And thus it understands the most likely meaning of the question. Then it can answer the question providing a measure of its confidence in that answer.