How big MNCs stores, manages and manipulates thousands of terabytes of data with high speed and efficiency?


Big data is a field that treats ways to analyze, systematically extract information from, or otherwise deal with data sets that are too large or complex to be dealt with by traditional data-processing application software. Data with many cases (rows) offer greater statistical power, while data with higher complexity (more attributes or columns) may lead to a higher false discovery rate. Big data challenges include capturing data, data storage, data analysis, search, sharing, transfer, visualization, querying, updating, information privacy and data source. Big data was originally associated with three key concepts: volume, variety, and velocity. When we handle big data, we may not sample but simply observe and track what happens. Therefore, big data often includes data with sizes that exceed the capacity of traditional software to process within an acceptable time and value.

Key Concepts of Big Data:

Volume: Organizations collect data from a variety of sources. In the past, storing it would have been a problem — but cheaper storage on platforms like data lakes and Hadoop have eased the burden.

Variety: Data comes in all types of formats — from structured, numeric data in traditional databases to unstructured text documents, emails, videos, audios, stock ticker data and financial transactions.

Velocity: With the growth in the Internet of Things, data streams in to businesses at an unprecedented speed and must be handled in a timely manner. RFID tags, sensors and smart meters are driving the need to deal with these torrents of data in near-real time.

Companies using Big Data:

1. Amazon:

In today’s consumer landscape, Amazon is an e-commerce giant. Amazon’s success does not come by an accident. Amazon becomes successful because it utilizes the advantage of big data to make decisions, please customers, and stimulate purchase.

Amazon has access to the vast amount of data of its customers like customers’ names, addresses, payments made by customers and search histories.

2. Facebook:

Face Book processes 2.5 billion pieces of content and 500+ terabytes of data each day. It’s pulling in 2.7 billion Like actions and 300 million photos per day, and it scans roughly 105 terabytes of data each half hour.

Facebook generates 4 petabytes of data per day — that’s a million gigabytes. All that data is stored in what is known as the Hive, which contains about 300 petabytes of data.

Every 60 seconds, 510,000 comments are posted, 293,000 statuses are updated, 4 million posts are liked, and 136,000 photos are uploaded.

3. Netflix

Netflix has surpassed Disney with a company valuation of over $164 billion. Netflix’s success is attributable more to user experience and content rather than marketing. Netflix content is influenced by big data. It uses Big Data to find out what users want to see and give it to them.

4. American Express

American Express handles about more than 25% of the U.S. credit card activity. Big Data is the heart of decision-making at American Express. American Express interacts with people on both sides that are millions of buyers and millions of businesses. The Company uses big data for fraud detection and bringing customers and merchants closer.

5. Starbucks

Starbucks is one of the best-known companies in the world. It has over 27,000+ stores. The secret ingredient for Starbucks’ success is its use of data analytics.

6. LinkedIn

LinkedIn is the first largest network that connects professionals and employers all over the world. It is the biggest social network platform for professionals, with 660 million users spread over 200+ countries. As per the November 2019 report, over 30 million companies have profiles on LinkedIn. Every second, more than two new members join LinkedIn.

Importance of Big Data:

Big data analytics helps organizations harness their data and use it to identify new opportunities. That, in turn, leads to smarter business moves, more efficient operations, higher profits and happier customers. In his report Big Data in Big Companies, IIA Director of Research Tom Davenport interviewed more than 50 businesses to understand how they used big data. He found they got value in the following ways:

  1. Cost reduction. Big data technologies such as Hadoop and cloud-based analytics bring significant cost advantages when it comes to storing large amounts of data — plus they can identify more efficient ways of doing business.
  2. Faster, better decision making. With the speed of Hadoop and in-memory analytics, combined with the ability to analyze new sources of data, businesses are able to analyze information immediately — and make decisions based on what they’ve learned.
  3. New products and services. With the ability to gauge customer needs and satisfaction through analytics comes the power to give customers what they want. Davenport points out that with big data analytics, more companies are creating new products to meet customers’ needs.

Challenges in Big Data:

  • Lack of Understanding of Big Data
  • Quality of Data
  • Spending a Huge Amount of Money
  • Integration of Platform
  • Security Issues

Hadoop is designed to handle the three V’s of Big Data: volume, variety, velocity. First lets look at volume, Hadoop is a distributed architecture that scales cost effectively. In other words, Hadoop was designed to scale out, and it is much more cost effective to grow the system. As you need more storage or computing capacity, all you need to do is add more nodes to the cluster. Second is variety, Hadoop allows you to store data in any format, be that structured or unstructured data. This means that you will not need to alter your data to fit any single schema before putting it into Hadoop. Next is velocity, with Hadoop you can load raw data into the system and then later define how you want to view it. Because of the flexibility of the system, you are able to avoid many network and processing bottlenecks associated with loading raw data. Since data is always changing, the flexibility of the system makes it much easier to integrate any changes.

Hadoop will allow you to process massive amounts of data very quickly. Hadoop is known as a distributing processing engine which leverages data locality. That means it was designed to execute transformations and processes where the data actually exists. Another benefit of value is from an analytics perspective, Hadoop allows you load raw data and then define the structure of the data at the time of query. This means that Hadoop is quick, flexible, and able to handle any type of analysis you want to conduct.

Organizations begin to utilize Hadoop when they need faster processing on large data sets, and often find they save the organization some money too. Large users of Hadoop include: Facebook, Amazon, Adobe, EBay, and LinkedIn. It is also in use throughout the financial sector and the US government. These organizations are a testament to what can be done at internet speed by utilizing big data to its fullest extent.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store