Big Data vs Traditional Data Analysis

Traditional Data Analysis focuses on using spreadsheets and databases to analyse small, easy to manage structured data sets. Programs like SQL and spreadsheet and commonly used in traditional data analysis to provide statistics and identify trends in the data.

Big Data on the other hand analyses vast amounts of unstructured and semi-structured data by utilising powerful AI and machine learning tools to process far more data than traditional data analysis methods.

The advantage of traditional data analysis over other analysis methods is its simplicity. You know what your getting, it is reliable, easy to use and integrate and provides accurate, useful information and statistics that help businesses operate daily.

However compared to big data analysis it has a few limitations. It's only able to sort through limited small structured data sets whereas big data analysis can handle large amounts of unstructured data. It is only able to process data at a relatively slow speed so it struggles with larger volumes of data. And it's unable to reveal the complex patterns that Big data analysis can.

Despite its limitations traditional data analysis is still the most common type of data analysis used by businesses to find simple patterns, it is also very reliable and doesn't require the same level of AI hardware that big data analysis does.

https://infomineo.com/services/data-analytics/big-data-analytics-versus-traditional-data-analytics 

Descriptive and Inferential Statistics

Traditional Statistics

Besides using Big Data and AI to analyse data and conduct research, descriptive and inferential statistics are also used to analyse data sets. These are commonly used when researching groups of people and provide different outcomes. 

Descriptive Statistics are used to provide information based on raw data collected. This usually includes summaries of the data that make it understandable often using different data point like mean, mode, median. This can be used to identify trends and distributions. It is important to note that descriptive statistics don't make any assumptions based on the data and merely provide information based on concrete data. 

Inferential Statistics however use data from a small sample and use that to make assumptions about a broader population. This type of statistic is useful for making connections between different data points, for example linking poverty with higher crime rates. These statistics are only estimates and differentiate from descriptive statistics which use more concrete data points.

https://statistics.laerd.com/statistical-guides/descriptive-inferential-statistics.php

Value of Big Data

 The Value of Big Data

Data is one of the most valuable assets for companies, but it can sometimes be hard to understand this value, as data does not have a traditional market value the way that stocks, property and other commodities do. 

The value of big data can be seen in how it is used by businesses and corporations to better understand their users and customers, and to optimise their business strategy and decisions based on data evidence. This data driven approach is a very effective strategy for companies as it relies on hard evidence rather than opinions. Some of the areas in which data is used are marketing, operation efficiency, and security strategy. 

These days its is also common for companies to buy and sell consumer data to each other, often without permission and usually to target people with personalised advertisements. This is one of the main ways that big corporations make a profit, they collect your data with or without your permission then use it themselves or sell it to another company. If something is free, most of the time it means you are the product. 

The global big data market is currently worth around $300 billion and is expected to grow significantly over the next decade and beyond. 

https://www.talend.com/resources/data-value

History of Big Data

History of Big Data

Early History

When we think of Big data, we think about the overwhelming amounts of data that are collected and stored today. But the idea of big data has actually been around a lot longer that that. Most people believe the term first became popular in the 1990s with some crediting American computer scientist John Mashey with coining the term. Before this however the first data centre had already been built a couple of decades prior in 1945 at the University of Pennsylvania ENIAC the world’s first general purpose digital computer. 

World Wide Web and Internet

With the development of the world wide web and internet in early 90s, data became much more accessible, and the opportunity to use data as a tool started to present itself. Companies realised they could collect and analyse commercial customer data, and use it to build customer profiles based on their behaviour.   

2000s and Web 2.0

Into the early 2000s the term big data became more common as well as the first ideas of Volume Velocity and Variety. But it wasn’t until the introduction of Web 2.0 and user created applications that generated a wealth of web content big data really started to take off. Apache Hadoop, a type of open-source software was created in 2005 by Doug Cutting and Mike Cafarella for data processing.

Recent History

Recently we have seen more and more data being collected on everything in our daily lives. Today powerful AI machine learning tools are used to analyse this data providing companies with invaluable information.

Big Data 1 - What is Big Data

What is Big Data?

Big Data is a term that is used to describe the extremely vast amounts of complex data that is unable to be processed by the traditional data management systems. This data can be categorised as either structured, unstructured or semi-structured data:

Structured data is data that can be organised neatly into tables and spreadsheets as part of  databases. It includes things like dates, emails, names, phone numbers, prices and much more. This type of data is easily processed by machine learning and other data management tools. 

Unstructured Data can be anything that cannot be easily categorised and put into a table or spreadsheet. This type of data is complex and common in big data comprising of 80 - 90% of all collected data. Examples of unstructured data can be anything from emails and social media posts to photos and smart phone activity. 

Semi-structured data cannot be as easily categorised as structured data however it is more flexible that structured data and uses metadata like tags and markers to help it to be part of structured data sets. 

The 5 Vs of Big Data

The 5 Vs of Big data refers to the five main characteristics that define big data.

Volume - The word 'big' in big data isn't there for no reason. These days data is generated from everything we do in our daily lives from browsing social media to going to the shops, this all contributes to unfathomably huge amounts of data that is unable that cannot be processed easily. 

Velocity - The speed at which data is generated and moves around is so fast that data often needs to be analysed in real time for organisations to make the most of it. 

Variety - Data can be almost anything and everything, it is important that data is sorted accordingly to the type of data like structured or unstructured. 

Veracity - How reliable and accurate the data is. Because of the volume and variety in big data, quality data that businesses need should be be filtered from any poor quality inaccurate  data. 

Value - The reason some much data collected is its value. Data should be valuable to the company or organisation that collects it. This allows them to analyse the data and use that information to improve their business platform or product. 

https://www.mongodb.com/resources/basics/big-data-explained

https://www.ibm.com/think/topics/structured-vs-unstructured-data

https://cloud.google.com/learn/what-is-big-data

https://www.ibm.com/think/topics/big-data