We are entering into the age of IOT, lot of connected devices will talk to each other using sensors/signals and it is expected that those devices will generate an enormous amount of data.
Handling that much of data [Big data] and generating actionable signals from that would be a challenge.
Companies are investing a lot of resources to build the platforms that can process the big data and retrieve some intelligence out of it. I am wondering that IOT will generate limitless data. Would these platforms will still be able to handle that situation and would it still remain cost effective to process the data of thousands of devices and then retrieving the signals that could be understand able to other devices.
Today SkyServer had data on 200 million galaxies in the database. The SDSS data exceeds 150 terabytes, covering more than 220 million galaxies and 260 million stars. The images alone include 2.5 trillion pixels of original raw data.
As the accuracy of devices will improve with the passage of time, their discovery capabilities will also improve resulting into more petabytes of data. Would we continue adding Servers, RAM, Hard drives and processors to manage all this?
Why we cannot invest all of these resources in making the devices intelligent? Every device do necessary intelligence at its own end its output interface expose only the actionable data to the outside world?
For example the sky telescope can compare all the images itself and only send the images to sky server which has some new insights. Some kind of machine learning built-in to sky telescope which could help it to learn it from the past and generate actionable insights based on that…in simple words, fit a human brain inside telescope that will transform big data to smart data. Storing, managing and processing garbage [Big data] at SkyServer do not make any sense to me. Why we should bear the data transfer cost of garbage? May be hardware device keep it in its unconsciousness and can use it if needed to generate the smart data.
In the world of IOT if output interface of all devices could expose smart data, it will tremendously decrease the data size that needs to be traveled on the IOT wire and software world would be more comfortable with handling those megabytes of data.
We also need to rethink the way we use the internet today, the data being generated from the internet is tremendously increasing with the passage of time. Software and hardware technology cannot handle the situation at all in future.
May be we think of a new language for the internet which can help to convey the message in a few signals instead of writing a story of 1000 words. If you see conversations between deaf people, they use few signals and tell a full a story in few actions. Consider internet users as deaf people and think of a language which could tell a story in few signals.
We think of a platform that could help to reduce the size of the data that we are generating today on the internet.
Transform the world towards the smart data instead of wasting money to handle the garbage [Big Data]
Originally posted on Data Science Central