From RDBMS to NoSQL to DBaaS

RDBMS
Relational Database Management Systems had been a subject I paid special attention to, when I was at college.  Having used DB2 and Oracle, the attention that the NoSQL databases are getting over the past two years or so, made me think why we need it, and whether RDBMS stay for the coming decades.

The ACID (Atomicity, Consistency, Isolation and Durability) properties and referential integrity that RDBMS provides us cannot be compromised in many systems.  That means, that for some systems, there were some needs, that led to the emergence and high usage of NoSQL (Not Only SQL ) systems.

NoSQL
An obvious reason that convinced me for the emergence of NoSQL is the increasing volume of un-structured and semi-structured data, and other documents that are used in the social networks which most of us use.  Then the other reasons slowly came by.  The speed at which we want a post to be published,  the number of reads and likes to displayed is one.  And the concurrency at which the social networks and other systems are being used, by many of us around the world, is also on the rise.  Above all this, we want a 24 X 7 availability of most of the sites, without which our satisfaction rate will drop down

It is to satisfy the above requirements that the use of NoSQL is on the rise.  Let us see how this is made possible.

1. Use of distributed databases
Distributed databases can be located at servers at any geographic location.  This means that they could be available on servers across the internet.  Also they could be located on the cloud infrastructure.  Distributed databases support replication and duplication, thereby enabling continuous availability.  Since data is available across many locations, concurrent usage is also made possible.

2. Horizontal scaling (sharding)
All users of Facebook want a quick login and quicker updates.  With this, a wiser way to store the database of Indian users in servers in India and Canadian users in servers in Canada, than storing data at any place in the globe.  This is an example of sharding.

3. Scalability
Many of the NoSQL databases are capable of storing large quantities of data.  With BigData the volume of data that is generated every second is on the rise.  Hence the ability to store data becomes important.

4. Schema-less databases
To enable storage of semi-structured and unstructured data, the databases do not store data in tables.  Data is stored as Documents, Columns, Key Value Store or Graph Databases.

Let us have a look at a couple of ways in which data is stored.

a) Document
Documents that contain semi-structured data are stored.  The MongoDB database stores documents.  This database is platform independent and holds JSON like documents.

b) Column
A column in a tuple of three arguments (name, column, timestamp)
student_name: {name: "student_name", value: "vishnu", timestamp: 123456789}

A Column Family is a set of Columns.  This is in some ways similar to a table, but the main difference is that, the same set of Columns need not be provided for all Column Family objects.  Please notice the difference between the column families given below.

{
    student_name: {name: "student_name", value: "aditya", timestamp: 123456789}
    school_name: {name: "school_name", value: "sun shine", timestamp: 123456789}
    city: {name: "city", value: "bangalore", timestamp: 123456789},
}
{
    student_name: {name: "student_name", value: "lily", timestamp: 123456789}
    school_name: {name: "school_name", value: "sun shine", timestamp: 123456789}
    standard: {name: "standard", value: ""IV", timestamp: 123456789},
}

hbase which is like a  BigTable for Hadoop uses column type storage.  This is an open source database from the Apache Foundation.  Hadoop uses hbase to store critical data, the size of which is much smaller when compared to the Big Data that Hadoop can store.

Database As A Service
The name indicates here that the database is provided as a service by a cloud provider.  The cloud provider will do the installation, upgrades and maintenance activities on the database and the customers can invoke services on it.  DBaaS reduces the time taken for installation and maintenance and manpower required for database management for the customer.  With the emergence of Cloud, DBaaS is not a surprise.  IBM Cloudant is a DBaaS, that stores JSON documents.

IBM MDM in the Big Data - Interoperability between products in the Big Data platform

The goal of Big Data is to obtain valuable insights through analysis.  With the IBM InfoSphere Master Data Management system serving as the single repository to obtain trusted data, let us discuss some of the key Big Data products with which IBM InfoSphere MDM could be integrated with.


Customer data from a single source or from multiple sources are loaded into MDM.  Also, there are downstream systems that receive data from MDM.  Some of the MDM APIs are integrated with InfoSphere DataStage for the Extract Transform and Load (ETL) operations, to form the MDM Connector.  The MDM Connector can be used for ETL operations using MDM.

It is important to determine the quality of data from a data source before it is data is loaded to the MDM server.  The IBM InfoSphere Information Analyzer can be used for accessing the data quality and its structure before loading data into MDM.  In addition, MDM can be configured to leverage the standardization and matching features of the IBM InfoSphere QualityStage.

The term Big Data encompasses structured data and unstructured data.  The IBM MDM provides a trusted single view of structured data.  The IBM InfoSphere Data Explorer, the tool used to derive Insights from Big Data, uses the MDM Connectors to access data from the MDM database to obtain a holistic view of the entities.

InfoSphere MDM has a Probabilistic Matching Engine, that can be used for matching parties to identify suspected duplicates.  This Probabilistic Matching Engine can be configured for use by InfoSphere BigInsights.  InfoSphere BigInsights is a product that supports storage of large volumes of un-structured, semi-structured and structured data and provides data analysis capabilities on such data.  The InfoSphere Data Click can also be used with MDM, to load master data into BigInsights system and other analysis sytems.

While MDM provides a single trusted view of data, business processes are required to ensure that the master data is accurate from the point of creation.  IBM Business Process Management Process Center and Process Designer components can be used to create workflows that govern data steward oriented tasks.  Master Data Management along with Business Process Management enable organizations to immediately take critical business decisions.

Salesforce.com, a Customer Relationship Management (CRM) solution available in the the Cloud (SaaS) is integrated with IBM MDM, which enables it to obtain a 360 degree view of its customers.  

MDM data can be exported and predictive analysis can be performed using the Cognos Business Intelligence reports.

Details on the given integrations and integrations with other products could be obtained in the below links.
IBM InfoSphere Master Data Management v 11.3.0
Master Data Management, Business Process Management and Services Oriented Architecture 

Master Data Management (MDM) in Big Data

"Maintaining a golden record of every entity" - this is precisely what a Master Data Management (MDM) system does.

MDM stores a cleansed, de-duplicated trusted view of structured data and plays a major role amidst big data flowing in from the social networks and streaming data.

I do see many organizations use master data to improve their performance.

A Diabetes clinic calls a patient's mobile number when his/her consultation is due.  So they do maintain the master data of their patients.  When the system is good enough to store all medical data about the patient, based on the tests he/she undergoes, the Doctor's analysis reports and the medicines prescribed, each time the patient consults a Doctor, then the system becomes capable of providing a complete view of the patient's health.

A retailer sends a customer an SMS, a month before the customer's Birthday, with a greeting and offering a 5% discount for what the customer shops during that one month.  So, here the master data of a customer is stored along with the mobile and date of birth and there is a system to send a message a month before his/her birthday.  By doing so, the retailer ensures that they maintain a good relationship with their customers.  When this retailer stores the list of items that the customer purchases, the total cost he/she pays, the mode of payment (cash or card) along with the Date of purchase, the retailer will be able to predict when the customer may visit again.

An insurance firm informs its customer through an SMS or email that insurance payment is due in a month.  This is again a system in which the customer's details are stored along with the payment date of insurance.  Hence the company makes sure that they do not lose their customer.  The company has to ensure that such a message is being sent each time an installment has to be paid. 

Some banks are able to classify their customers as Classic, Premium etc., based on the balance they maintain in their account, over a period of time.  This easily indicates that extent to which they maintain big data.

One of the example we consider for Big Data is Facebook.  This social website also holds master data of its users.  It asks each user for name, city, employment and date of birth.  The family relationships, close friends list, friends and Likes of user, along with the other primary details contribute to the master data.  Alerts on friend's birthdays, the list of probable friends of a user, the groups in which a user may like to join and the personalities a user likes may be derived based on the master data.


A good Master Data Management system will ensure that the data is cleansed, duplicate data is not present and the data is trusted.

This master data plays a major role in analysis.  With a complete view of  patient's current health and history, a Doctor who consults, will be able to easily make out the drugs to which this patient is allergic, what medicines would not suit a patient due to the medicines being (or been) taken and prescribe treatment accordingly.  A Banker would be able to suggest a Recurring Deposit (or some other plan) to an account holder, based on the balance in his/her account or based on the monthly salary being deposited to that account.

With these being the uses of analysis to a user, the benefits that a customer of MDM would be much more.  A retailer can find out the lean seasons and try to give discounts during those periods of a year.  During the peak seasons, they can increase the stock.  They can also find out which products sell well in a particular geography and increase the stock of that product.  By having fast networks, stocks can be replenished as required.

When hospital chains start having Master Data Management System, it would make the life of a patient much easier.  This becomes all the more important for patients with critical illnesses. Having such a system could also help in medical research.


With these just being examples I have noticed, further details could be obtained from the below links.
IBM Master Data Management for Big Data
IBM Think Big - Big Data & MDM
IBM Master Data Management: The key to leveraging Big Data
How MDM Fits with Big Data, Mobile and Cloud
IBM Master Data Management - Solutions for Healthcare

Analyzing Big Data

Let us take a quick look at some of the ways in which big data is processed, maintained and analyzed to provide valuable insights.

Quite a good volume of data had been available since a decade, but the extent to which it was used was less than today.  We have started exploring data - that was already available, and new data that is coming in every other milli second (streaming data) - to make valuable decisions.

Types of Data
Data can be categorized as structured, unstructured and semi structured.

1. Structured data is data that is in a pre-defined format.  Data stored in databases, and spread sheets are examples of structured data.  Structured data can be analyzed easily.  
2. Unstructured data refers to data that does not have a pre-defined format. Sentences, texts, stories, pictures are all examples of unstructured data.  Text mining tools have to be used to uncover data in unstructured format.
3. Data in xml files (and other markup languages) are semi-structured - some parts of the data is structured.

Data Cleansing
Analysis performed from inaccurate, erroneous, or duplicate data will reduce the value.  Hence it is required that the data available is cleansed before analysis.

For example, a person in India gives a 5 digit pin code in his address.  The system has to immediately highlight that the data is inaccurate.

Maintaining a Golden Record for every entity
For every entity 'a single version of true data' has to be stored.  Details of operations that are performed on data has to be stored and a copy of the data before modification also needs to be maintained.

For example, a customer of a Bank has a Savings account and a Fixed Deposit.  It is good to maintain one record of the customer, containing all his details (like name, date of birth, gender, address), rather than having two copies of this data.

Analyzing Streaming Data
Data that is transferred at a high speed rate is known as streaming data. An example that we might have noticed is heart beat monitors attached to patients.  Other examples include network signals and transactions over the internet.  In some cases, monitoring streaming data becomes very important.  Product that can easily ingest and analyze data can help when critical decisions have to be made using streaming data.

Data Integration and Governance
A software that can integrated data from multiple systems and provide a complete 360 degree view of each entity involved is a required.  Also at each stage of the processing, managing data quality is important.

For example, let us again consider a bank customer, having Rs. 50,00,000/- Home Loan, Rs.60,000/- balance is credit card and a Savings account with balance less than Rs.1,000/-  If the bank can get a complete view of this customer and drill in to his history records, it is easy for the bank manager to decide, if the customer approaches for a car loan for Rs, 10,00,000/-

Data Exploration
Sometimes, data outside the organization (eg., number of Likes in Facebook, Analysis done by government or third party agencies) also become crucial during analysis.  A software that helps to uncover value from data in internal and external sources and is a key component of big data analysis.

For example, a popular brand wants to compare its performance in various cities in a county.  It also wants to compare itself with its competitors based on the Like votes in Facebook, in those cities.

Predictive Analysis
Predictive Analysis makes use of historical data and current data to make predictions about the future.  This is one of the most frequently used analysis technique.

For example, based on the number of enterprises that have started using big data and analysis, it is possible to predict the number of data scientists required after five years.
 
Some of the products that provide these capabilities are given below.
IBM InfoSphere Information Server
IBM InfoSphere Master Data Management
IBM Information Integration and Governance
IBM InfoSphere BigInsights
IBM Stream Computing
IBM InfoSphere Data Explorer
IBM SPSS Software

Big Data, Analytics and Insights

What is Big Data
Big data is one of the most important topic today.  So what is "big data"?  Does it mean "data" has become big?

For my point of view, I feel that the data that can be stored in digital form has increased.  Information has always been there, from stone inscriptions to those written on books, but the readers were limited, owing to knowledge, access, language, distance and the time period up to which it could be preserved.

Today data that can be stored and accessed has increased in quantity.  All of us here have email ids, send emails, have mobile devices, send SMSes, many of us have installed Whats App and use it, tweet and use Facebook.  We use these to exchange information - data.  All this data is stored somewhere, hence we are able to search for a mail which we received four years ago.  We are able to see a photograph of our friend's child.  We get an SMS from Railways giving the PNR number and requesting not to waste paper, an instant message from a Bank intimating that the EMI for Housing Loan has been deducted, a message in Facebook that a movie that you wanted to see is not so good and that a shop your regularly visit has opened a store in your neighbourhood.  You wish your friend on her Birthday, and Thank Facebook for the timely reminder that it gave you.  You too, tweet, write a blog and post your favourite photo in Facebook.  The newspapers and magazines are available over the Internet, so it is not necessary for you to spend time with them in 'paper' form early in the  morning.  With all this, we get to know that the 'Volume' of data has increased. 

By the time I complete this post, so many others would be also be adding other data by posting photographs, marking 'Like' for a garment brand, tweeting comments on a product, greeting a friend for passing an examination, booking railway tickets, transferring money from a bank account using NEFT etc.  The data that already exists keeps changing very fast (Velocity)

To store all these Varieties of data - written text, photographs, number of Likes for a garment brand, railway ticket booking, managing bank balance and transfer of money securely, the systems should be appropriately equipped.

In some of the above examples, the Veracity (how true the data is) plays a very important role. Incorrectly tagging a person in Facebook is a common example false data.

Hence Big Data is a huge Volume of data, that keeps on increasing, of different Varieties and the Velocity at which this data changes is significant.  Veracity of this data has to be ensured.  We realize its importance based on the Value it provides.

Why Big Data?
Consciously or sub consciously we have been using analyzed Big Data.  Google displays the search results using maximum access as the criteria.  Facebook and LinkedIn do show us your "Could be" Friends.

Big Data is not there just because the extent of storage capability has increased.  It is there because it can be analyzed to provide us valuable Insights. 

For example, after I Publish a Post, Blogger allows me to view valuable Statistics.  It provides details on number of times each post has been read, read count based on countries, browsers, operating systems and the link using which the reader read to the Post, which provides me some insights.  As soon as I publish, the read count starts increasing slowly, it increases sharply once I give the link in Facebook, and it decreases after the post is a week old or so.

When an individual is so much interested in knowing the number of readers, it is obvious that Governments, Banks, Insurance, and Retailers would want to know Insights which would help them improve their services, and / or maximize their profits.

Actionable Insights
1. Some of the actions that Banks take based on the Insights might have already been experienced or observed.  A Bank in which you have your salary account (or a good amount as balance) is  ready to give you a Personal Loan without provision of documents.  The organization is also treats you as a Privileged customer, thereby ensuring that you are satisfied with their service and will remain their customer.

2. When you are regularly consulting a Diabetes hospital, the hospital staff call you up when your next Consultation is due.

3. A reputed food chain has started offering Vegetarian food after getting to know closely that most of the potential customers are Vegetarians.

Given below are other examples.
1. On knowing that a store is running short of a particular drug, the Pharma company can immediately replenish the stock there.  If there are many such stores run short of the same drug, the company may need to decide whether the production of the drug has to be increased.

2. A particular garment brand compared itself with its competitors using details from a social website and found that its performance has dramatically reduced.  It then started replenishing stores with stocks based on the geography and reducing prices to make the clothes suit the style of the locals and affordable to them.  Also it started advertising through the social website and adding new designs.

3. Currently, organizations have started recognizing the importance of big data and analysis and have predicted that the industry would require a number of Data Scientists over the years to come. (Predictive analysis).

With Big Data and Analysis playing a key role in almost all domains, we need to understand the components that are used to obtain the actionable insights and the opportunities that it provides us.

The below links will enable you get a good understanding of Big Data and Insights.
What is Big Data Analysis
Big Data - What it is and how it matters
IBM Big Data Use Cases
IBM Big Data in Action

IBM InfoSphere Master Data Management on Cloud

Cloud computing refers to delivery of computing resources over the Internet on a pay for use basis.

Cloud computing services are available in three ways.
1. Software As A Service (SaaS) where the software is installed on servers (in the cloud) and the user can connect through it through the Internet. (Example: IBM Sterling Supply Chain Visibility, Google, Facebook)

2. Platform As A Service (PaaS) where a cloud based environment is provided using which the application can be built and deployed. (Example: IBM BlueMix, IBM SmartCloud Application Services)

3. Infrastructure As A Service (IaaS) where servers, storage space and networking is provided on a pay per use basis. (Example IBM SoftLayer)

Managing Big Data and analysis is becoming critical to businesses and they slowly start realizing the importance of having a Master Data Management solution to meet their objectives.

IBM InfoSphere Master Data Management (from version 11.0.0) can now be deployed on the cloud, thereby providing customers the below advantages.
1. Accelerated deployment
2. Pay per use model
3. Maintenance of the MDM solution on cloud is addressed by IBM

These facilities enable the businesses to obtain value quickly, without initial capital expenditure.  Hence small and medium sized enterprises can also experience the benefits of using IBM Master Data Management.

Please refer the below link for further details.
IBM Master Data Management

IBM InfoSphere Master Data Management

With Big Data and Analysis forming a basis for successful enterprises, we still find many organizations where data is in silos.

For example, there is an organization, which runs Banking, Insurance and other businesses.  A customer have a Savings Account and three insurance policies with this organization.  For each insurance policy, the organization has given a Customer Id.  So this customer, has three Customer Ids and one Savings Bank Account Number.  To update any single detail, the update has to be done four times, one for each Customer Id.  If a customer updates only two of them, the organization is left with inconsistent data.

From the organization's perspective. If they keep holding a separate Customer Id for each insurance policy, the extent of data they have to maintain keeps increasing.  The extent of inconsistent data is unknown.  It is difficult to come up with a single view of a customer.

Having a single view of every customer would help organizations improve their revenue.  Suppose this organization is able to make out that customer A who has an Insurance policy also has a Savings Account, an option of direct debit from the account could be suggested to the Customer.  It would also be easy for the organization to get a complete view of the Customer.

IBM InfoSphere Master Data Management (MDM) is a solution that helps organizations to obtain a single view of each customer.  Master Data refers to data that is shared across the organization.  Hence in our example, master data refers to data about the customer. MDM provides capabilities to match and merge records from multiple sources. This master data can then be used across the organization.

For further details on the features that this product provides, please refer IBM InfoSphere Master Data Management