Defining Big Data) but if we adopt the simplest definition we see that it. It seems our abilities to generate data always are one step ahead of our abilities to process all of it.
It may seem far-fetched but think about it.The problems with computing back in 80s and 90s are remarkably similar to the ones related to Big Data these days.If we peel off the layers of salesman talking the problems with large data sets boil down to:
- Long times in manipulating stored data (save, load, update);
- Long times for extracting and processing the data;
("Long" is defined in specific ways according to the application but in general, time is of less and less supply and in more and more demand these days.)
Overcoming these problems required careful work with memory, calculating time for execution and selecting good calculation methods. All these problems were easily solved back then by a more powerful processor and larger and faster storage. Fortunately, new and better models were coming out every six months! The scale and speed of data now are so big that it requires applying radically new technologies. It does not come as a surprise that the core of the modern Big Data technologies were invented long time ago and now are simply revived and updated.
History repeats itself and big data will be just data sooner than we think. We are at the peak of the hype curve and shortly we will stop hearing much for Big Data and probably stop being very careful about memory usage and processing time again. At least for a while. Until we hit again the ceiling of our abilities to deal with the data we generate. Then will come the time of what? Mega Data?