The Continuing Role of the Database in the New Era of Big Data

Big data is all about scaling the use of data beyond the norms of the current era of information technology.

You could reasonably argue that the first big data era began more than a half-century ago. On May 25, 1961, President John F. Kennedy gave a speech to the U.S. Congress in which he declared the goal of landing a man on the moon, and returning him safely to Earth.  The amount of data generated and managed throughout the program quickly outgrew data systems of the time.  A brand new “Information Management System” (IMS) was created by IBM and other members of the Apollo team to tackle this new big data challenge.

Now, fast forward more than 50 years and we have ushered in a new era of big data, ignited by the global “Internet of things,” mobile, social and cloud computing, and instrumented systems of all kinds.  Now every transaction, tweet or meter reading has potential value to enhance or destroy a customer relationship; to drive a new business opportunity; or to catch a bad guy.   New types of data systems are needed to handle more data and more types of data, faster and more cost effectively than systems that were state of the art just a few years ago.

The key to making big data work for business is using systems that are designed for workload optimized performance and simplicity.  In some cases that means completely new systems to handle challenges like analyzing data in motion, or spreading complex work among a large number of distributed systems.  In other cases, new capabilities are added to proven systems such as IBM DB2 and Informix, to provide a new mix of production grade capabilities – e.g., for both SQL and NoSQL databases.

Solving today’s big data challenges often requires combining the structured, optimized approach of traditional database systems with the less structured, exploratory approach of new systems.   In fact, modern versions of technology created decades ago may be the best choice for new enterprise challenges; ones that also benefit from their time-proven stability, maturity, and manageability.

So what’s the role of a relational data system in this big data era? 

Some IT professionals may take relational and pre-relational database technologies for granted, but they remain the trusty workhorse in most data centers.  These proven platforms continue to handle the growing volume of data and faster transactions from applications that conduct business every second of every day.   They also enable deep analysis of that data to help organizations make better decisions with the speed needed to affect business operations as they execute.

Organizations leading the pack in big data ingenuity are the ones using the best combination of systems – traditional or new – for each need.  For many organizations building complex systems, running global banking networks, or delivering millions of packages around the world everyday, that includes using the modern descendent of the data system that played a small role in a giant leap for mankind.

Look for more thoughts about Big Data at the speed of business from me and other followers of database technology in the coming weeks.

And if you’re interested in IBM’s next Big Data event, go to this link for details. http://ibm.co/BigDataEvent