mirror of
https://github.com/linkedin/school-of-sre
synced 2026-01-20 15:38:03 +00:00
62 lines
2.6 KiB
Markdown
62 lines
2.6 KiB
Markdown
# School of SRE: Big Data
|
|
|
|
## Pre - Reads
|
|
|
|
- Basics of Linux File systems.
|
|
- Basic understanding of System Design.
|
|
|
|
## Target Audience
|
|
|
|
The concept of Big Data has been around for years; most organizations now understand that if they capture all the data that streams into their businesses, they can apply analytics and get significant value from it.
|
|
This training material covers the basics of Big Data(using Hadoop) for beginners, who would like to quickly get started and get their hands dirty in this domain.
|
|
|
|
## What to expect from this training
|
|
|
|
This course covers the basics of Big Data and how it has evolved to become what it is today. We will take a look at a few realistic scenarios where Big Data would be a perfect fit. An interesting assignment on designing a Big Data system is followed by understanding the architecture of Hadoop and the tooling around it.
|
|
|
|
## What is not covered under this training
|
|
|
|
Writing programs to draw analytics from data.
|
|
|
|
## TOC:
|
|
|
|
1. Overview of Big Data
|
|
2. Usage of Big Data techniques
|
|
3. Evolution of Hadoop
|
|
4. Architecture of hadoop
|
|
1. HDFS
|
|
2. Yarn
|
|
5. MapReduce framework
|
|
6. Other tooling around hadoop
|
|
1. Hive
|
|
2. Pig
|
|
3. Spark
|
|
4. Presto
|
|
7. Data Serialisation and storage
|
|
|
|
|
|
# Overview of Big Data
|
|
|
|
1. Big Data is a collection of large datasets that cannot be processed using traditional computing techniques. It is not a single technique or a tool, rather it has become a complete subject, which involves various tools, techniques and frameworks.
|
|
2. Big Data could consist of
|
|
1. Structured data
|
|
2. Unstructured data
|
|
3. Semi-structured data
|
|
3. Characteristics of Big Data:
|
|
1. Volume
|
|
2. Variety
|
|
3. Velocity
|
|
4. Variability
|
|
4. Examples of Big Data generation include stock exchanges, social media sites, jet engines, etc.
|
|
|
|
|
|
# Usage of Big Data techniques
|
|
|
|
1. Take the example of the traffic lights problem.
|
|
1. There are more than 300,000 traffic lights in the US as of 2018.
|
|
2. Let us assume that we placed a device on each of them to collect metrics and send it to a central metrics collection system.
|
|
3. If each of the IOT devices sends 10 events per minute, we have 300000x10x60x24 = 432x10^7 events per day.
|
|
4. How would you go about processing that and telling me how many of the signals were “green” at 10:45 am on a particular day?
|
|
2. Consider the next example on Unified Payments Interface (UPI) transactions:
|
|
1. We had about 1.15 billion UPI transactions in the month of October, 2019 in India.
|
|
12. If we try to extrapolate this data to about a year and try to find out some common payments that were happening through a particular UPI ID, how do you suggest we go about that? |