diff --git a/courses/big_data/architecture.md b/courses/big_data/architecture.md deleted file mode 100644 index 4563dc9..0000000 --- a/courses/big_data/architecture.md +++ /dev/null @@ -1,78 +0,0 @@ -# Architecture of Hadoop - -1. **HDFS** - 1. The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. - 2. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large data sets. - 3. HDFS is part of the Apache Hadoop Core project. - - ![HDFS Architecture](images/hdfs_architecture.png) - - 1. NameNode: is the arbitrator and central repository of file namespace in the cluster. The NameNode executes the operations such as opening, closing, and renaming files and directories. - 2. DataNode: manages the storage attached to the node on which it runs. It is responsible for serving all the read and write requests. It performs operations on instructions on NameNode such as creation, deletion, and replications of blocks. - 3. Client: Responsible for getting the required metadata from the namenode and then communicating with the datanodes for reads and writes. - -2. **YARN** - 1. YARN stands for “Yet Another Resource Negotiator“. It was introduced in Hadoop 2.0 to remove the bottleneck on Job Tracker which was present in Hadoop 1.0. YARN was described as a “Redesigned Resource Manager” at the time of its launching, but it has now evolved to be known as a large-scale distributed operating system used for Big Data processing. - 2. The main components of YARN architecture include: - - ![YARN Architecture](images/yarn_architecture.gif) - - 1. Client: It submits map-reduce jobs to the resource manager. - 2. Resource Manager: It is the master daemon of YARN and is responsible for resource assignment and management among all the applications. Whenever it receives a processing request, it forwards it to the corresponding node manager and allocates resources for the completion of the request accordingly. It has two major components: - 3. Scheduler: It performs scheduling based on the allocated application and available resources. It is a pure scheduler, which means that it does not perform other tasks such as monitoring or tracking and does not guarantee a restart if a task fails. The YARN scheduler supports plugins such as Capacity Scheduler and Fair Scheduler to partition the cluster resources. - 4. Application manager: It is responsible for accepting the application and negotiating the first container from the resource manager. It also restarts the Application Manager container if a task fails. - 5. Node Manager: It takes care of individual nodes on the Hadoop cluster and manages application and workflow and that particular node. Its primary job is to keep-up with the Node Manager. It monitors resource usage, performs log management and also kills a container based on directions from the resource manager. It is also responsible for creating the container process and starting it on the request of the Application master. - 6. Application Master: An application is a single job submitted to a framework. The application manager is responsible for negotiating resources with the resource manager, tracking the status and monitoring progress of a single application. The application master requests the container from the node manager by sending a Container Launch Context(CLC) which includes everything an application needs to run. Once the application is started, it sends the health report to the resource manager from time-to-time. - 7. Container: It is a collection of physical resources such as RAM, CPU cores and disk on a single node. The containers are invoked by Container Launch Context(CLC) which is a record that contains information such as environment variables, security tokens, dependencies etc. - - -# MapReduce framework - -![MapReduce Framework](images/map_reduce.jpg) - - 1. The term MapReduce represents two separate and distinct tasks Hadoop programs perform-Map Job and Reduce Job. Map jobs take data sets as input and process them to produce key value pairs. Reduce job takes the output of the Map job i.e. the key value pairs and aggregates them to produce desired results. - 2. Hadoop MapReduce (Hadoop Map/Reduce) is a software framework for distributed processing of large data sets on computing clusters. Mapreduce helps to split the input data set into a number of parts and run a program on all data parts parallel at once. - 3. Please find the below Word count example demonstrating the usage of MapReduce framework: - -![Word Count Example](images/mapreduce_example.jpg) - -# Other tooling around hadoop - -1. **Hive** - 1. Uses a language called HQL which is very SQL like. Gives non-programmers the ability to query and analyze data in Hadoop. Is basically an abstraction layer on top of map-reduce. - 2. Ex. HQL query: - 1. _SELECT pet.name, comment FROM pet JOIN event ON (pet.name = event.name);_ - 3. In mysql: - 1. _SELECT pet.name, comment FROM pet, event WHERE pet.name = event.name;_ -2. **Pig** - 1. Uses a scripting language called Pig Latin, which is more workflow driven. Don't need to be an expert Java programmer but need a few coding skills. Is also an abstraction layer on top of map-reduce. - 2. Here is a quick question for you: - What is the output of running the pig queries in the right column against the data present in the left column in the below image? - - ![Pig Example](images/pig_example.png) - - Output: - ```mysql - 7,Komal,Nayak,24,9848022334,trivendram - 8,Bharathi,Nambiayar,24,9848022333,Chennai - 5,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar - 6,Archana,Mishra,23,9848022335,Chennai - ``` -3. **Spark** - 1. Spark provides primitives for in-memory cluster computing that allows user programs to load data into a cluster’s memory and query it repeatedly, making it well suited to machine learning algorithms. -4. **Presto** - 1. Presto is a high performance, distributed SQL query engine for Big Data. - 2. Its architecture allows users to query a variety of data sources such as Hadoop, AWS S3, Alluxio, MySQL, Cassandra, Kafka, and MongoDB. - 3. Example presto query: - ```mysql - use studentDB; - show tables; - SELECT roll_no, name FROM studentDB.studentDetails where section=’A’ limit 5; - ``` - - -# Data Serialisation and storage - -1. In order to transport the data over the network or to store on some persistent storage, we use the process of translating data structures or objects state into binary or textual form. We call this process serialization.. -2. Avro data is stored in a container file (a .avro file) and its schema (the .avsc file) is stored with the data file. -3. Apache Hive provides support to store a table as Avro and can also query data in this serialisation format. \ No newline at end of file diff --git a/courses/big_data/evolution.md b/courses/big_data/evolution.md index 058c5c7..7f6922d 100644 --- a/courses/big_data/evolution.md +++ b/courses/big_data/evolution.md @@ -1,3 +1,83 @@ # Evolution of Hadoop -![Evolution of hadoop](images/hadoop_evolution.png) \ No newline at end of file +![Evolution of hadoop](images/hadoop_evolution.png) + +# Architecture of Hadoop + +1. **HDFS** + 1. The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. + 2. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large data sets. + 3. HDFS is part of the [Apache Hadoop Core project](https://github.com/apache/hadoop). + + ![HDFS Architecture](images/hdfs_architecture.png) + + 1. NameNode: is the arbitrator and central repository of file namespace in the cluster. The NameNode executes the operations such as opening, closing, and renaming files and directories. + 2. DataNode: manages the storage attached to the node on which it runs. It is responsible for serving all the read and write requests. It performs operations on instructions on NameNode such as creation, deletion, and replications of blocks. + 3. Client: Responsible for getting the required metadata from the namenode and then communicating with the datanodes for reads and writes.


+ +2. **YARN** + 1. YARN stands for “Yet Another Resource Negotiator“. It was introduced in Hadoop 2.0 to remove the bottleneck on Job Tracker which was present in Hadoop 1.0. YARN was described as a “Redesigned Resource Manager” at the time of its launching, but it has now evolved to be known as a large-scale distributed operating system used for Big Data processing. + 2. The main components of YARN architecture include: + + ![YARN Architecture](images/yarn_architecture.gif) + + 1. Client: It submits map-reduce(MR) jobs to the resource manager. + 2. Resource Manager: It is the master daemon of YARN and is responsible for resource assignment and management among all the applications. Whenever it receives a processing request, it forwards it to the corresponding node manager and allocates resources for the completion of the request accordingly. It has two major components: + 3. Scheduler: It performs scheduling based on the allocated application and available resources. It is a pure scheduler, which means that it does not perform other tasks such as monitoring or tracking and does not guarantee a restart if a task fails. The YARN scheduler supports plugins such as Capacity Scheduler and Fair Scheduler to partition the cluster resources. + 4. Application manager: It is responsible for accepting the application and negotiating the first container from the resource manager. It also restarts the Application Manager container if a task fails. + 5. Node Manager: It takes care of individual nodes on the Hadoop cluster and manages application and workflow and that particular node. Its primary job is to keep-up with the Node Manager. It monitors resource usage, performs log management and also kills a container based on directions from the resource manager. It is also responsible for creating the container process and starting it on the request of the Application master. + 6. Application Master: An application is a single job submitted to a framework. The application manager is responsible for negotiating resources with the resource manager, tracking the status and monitoring progress of a single application. The application master requests the container from the node manager by sending a Container Launch Context(CLC) which includes everything an application needs to run. Once the application is started, it sends the health report to the resource manager from time-to-time. + 7. Container: It is a collection of physical resources such as RAM, CPU cores and disk on a single node. The containers are invoked by Container Launch Context(CLC) which is a record that contains information such as environment variables, security tokens, dependencies etc.

+ + +# MapReduce framework + +![MapReduce Framework](images/map_reduce.jpg) + +1. The term MapReduce represents two separate and distinct tasks Hadoop programs perform-Map Job and Reduce Job. Map jobs take data sets as input and process them to produce key value pairs. Reduce job takes the output of the Map job i.e. the key value pairs and aggregates them to produce desired results. +2. Hadoop MapReduce (Hadoop Map/Reduce) is a software framework for distributed processing of large data sets on computing clusters. Mapreduce helps to split the input data set into a number of parts and run a program on all data parts parallel at once. +3. Please find the below Word count example demonstrating the usage of MapReduce framework: + +![Word Count Example](images/mapreduce_example.jpg) +

+ +# Other tooling around hadoop + +1. [**Hive**](https://hive.apache.org/) + 1. Uses a language called HQL which is very SQL like. Gives non-programmers the ability to query and analyze data in Hadoop. Is basically an abstraction layer on top of map-reduce. + 2. Ex. HQL query: + 1. _SELECT pet.name, comment FROM pet JOIN event ON (pet.name = event.name);_ + 3. In mysql: + 1. _SELECT pet.name, comment FROM pet, event WHERE pet.name = event.name;_ +2. [**Pig**](https://pig.apache.org/) + 1. Uses a scripting language called Pig Latin, which is more workflow driven. Don't need to be an expert Java programmer but need a few coding skills. Is also an abstraction layer on top of map-reduce. + 2. Here is a quick question for you: + What is the output of running the pig queries in the right column against the data present in the left column in the below image? + + ![Pig Example](images/pig_example.png) + + Output: + ```mysql + 7,Komal,Nayak,24,9848022334,trivendram + 8,Bharathi,Nambiayar,24,9848022333,Chennai + 5,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar + 6,Archana,Mishra,23,9848022335,Chennai + ``` +3. [**Spark**](https://spark.apache.org/) + 1. Spark provides primitives for in-memory cluster computing that allows user programs to load data into a cluster’s memory and query it repeatedly, making it well suited to machine learning algorithms. +4. [**Presto**](https://prestodb.io/) + 1. Presto is a high performance, distributed SQL query engine for Big Data. + 2. Its architecture allows users to query a variety of data sources such as Hadoop, AWS S3, Alluxio, MySQL, Cassandra, Kafka, and MongoDB. + 3. Example presto query: + ```mysql + use studentDB; + show tables; + SELECT roll_no, name FROM studentDB.studentDetails where section=’A’ limit 5; + ``` +
+ +# Data Serialisation and storage + +1. In order to transport the data over the network or to store on some persistent storage, we use the process of translating data structures or objects state into binary or textual form. We call this process serialization.. +2. Avro data is stored in a container file (a .avro file) and its schema (the .avsc file) is stored with the data file. +3. Apache Hive provides support to store a table as Avro and can also query data in this serialisation format. \ No newline at end of file diff --git a/courses/big_data/intro.md b/courses/big_data/intro.md index 143cc8d..5f67995 100644 --- a/courses/big_data/intro.md +++ b/courses/big_data/intro.md @@ -13,20 +13,46 @@ This course covers the basics of Big Data and how it has evolved to become what Writing programs to draw analytics from data. -## Course Content +## Course Contents -### Table of Contents - -1. [Overview of Big Data](https://linkedin.github.io/school-of-sre/big_data/overview/) -2. [Usage of Big Data techniques](https://linkedin.github.io/school-of-sre/big_data/overview/) +1. [Overview of Big Data](https://linkedin.github.io/school-of-sre/big_data/intro/#overview-of-big-data) +2. [Usage of Big Data techniques](https://linkedin.github.io/school-of-sre/big_data/intro/#usage-of-big-data-techniques) 3. [Evolution of Hadoop](https://linkedin.github.io/school-of-sre/big_data/evolution/) -4. [Architecture of hadoop](https://linkedin.github.io/school-of-sre/big_data/architecture/) +4. [Architecture of hadoop](https://linkedin.github.io/school-of-sre/big_data/evolution/#architecture-of-hadoop) 1. HDFS 2. Yarn -5. [MapReduce framework](https://linkedin.github.io/school-of-sre/big_data/architecture/#mapreduce-framework) -6. [Other tooling around hadoop](https://linkedin.github.io/school-of-sre/big_data/architecture/#other-tooling-around-hadoop) +5. [MapReduce framework](https://linkedin.github.io/school-of-sre/big_data/evolution/#mapreduce-framework) +6. [Other tooling around hadoop](https://linkedin.github.io/school-of-sre/big_data/evolution/#other-tooling-around-hadoop) 1. Hive 2. Pig 3. Spark 4. Presto -7. [Data Serialisation and storage](https://linkedin.github.io/school-of-sre/big_data/architecture/#data-serialisation-and-storage) +7. [Data Serialisation and storage](https://linkedin.github.io/school-of-sre/big_data/evolution/#data-serialisation-and-storage) + + +# Overview of Big Data + +1. Big Data is a collection of large datasets that cannot be processed using traditional computing techniques. It is not a single technique or a tool, rather it has become a complete subject, which involves various tools, techniques and frameworks. +2. Big Data could consist of + 1. Structured data + 2. Unstructured data + 3. Semi-structured data +3. Characteristics of Big Data: + 1. Volume + 2. Variety + 3. Velocity + 4. Variability +4. Examples of Big Data generation include stock exchanges, social media sites, jet engines, etc. + + +# Usage of Big Data Techniques + +1. Take the example of the traffic lights problem. + 1. There are more than 300,000 traffic lights in the US as of 2018. + 2. Let us assume that we placed a device on each of them to collect metrics and send it to a central metrics collection system. + 3. If each of the IOT devices sends 10 events per minute, we have 300000x10x60x24 = 432x10^7 events per day. + 4. How would you go about processing that and telling me how many of the signals were “green” at 10:45 am on a particular day? +2. Consider the next example on Unified Payments Interface (UPI) transactions: + 1. We had about 1.15 billion UPI transactions in the month of October, 2019 in India. + 12. If we try to extrapolate this data to about a year and try to find out some common payments that were happening through a particular UPI ID, how do you suggest we go about that? + diff --git a/courses/big_data/overview.md b/courses/big_data/overview.md deleted file mode 100644 index 78b618f..0000000 --- a/courses/big_data/overview.md +++ /dev/null @@ -1,13 +0,0 @@ -# Overview of Big Data - -1. Big Data is a collection of large datasets that cannot be processed using traditional computing techniques. It is not a single technique or a tool, rather it has become a complete subject, which involves various tools, techniques and frameworks. -2. Big Data could consist of - 1. Structured data - 2. Unstructured data - 3. Semi-structured data -3. Characteristics of Big Data: - 1. Volume - 2. Variety - 3. Velocity - 4. Variability -4. Examples of Big Data generation include stock exchanges, social media sites, jet engines, etc. \ No newline at end of file diff --git a/courses/big_data/usage.md b/courses/big_data/usage.md deleted file mode 100644 index 620b07f..0000000 --- a/courses/big_data/usage.md +++ /dev/null @@ -1,10 +0,0 @@ -# Usage of Big Data techniques - -1. Take the example of the traffic lights problem. - 1. There are more than 300,000 traffic lights in the US as of 2018. - 2. Let us assume that we placed a device on each of them to collect metrics and send it to a central metrics collection system. - 3. If each of the IOT devices sends 10 events per minute, we have 300000x10x60x24 = 432x10^7 events per day. - 4. How would you go about processing that and telling me how many of the signals were “green” at 10:45 am on a particular day? -2. Consider the next example on Unified Payments Interface (UPI) transactions: - 1. We had about 1.15 billion UPI transactions in the month of October, 2019 in India. - 12. If we try to extrapolate this data to about a year and try to find out some common payments that were happening through a particular UPI ID, how do you suggest we go about that? \ No newline at end of file diff --git a/courses/databases_nosql/further_reading.md b/courses/databases_nosql/further_reading.md new file mode 100644 index 0000000..f7024c3 --- /dev/null +++ b/courses/databases_nosql/further_reading.md @@ -0,0 +1,25 @@ +# Further reading: + +NoSQL: + +[https://hostingdata.co.uk/nosql-database/](https://hostingdata.co.uk/nosql-database/) + +[https://www.mongodb.com/nosql-explained](https://www.mongodb.com/nosql-explained) + +[https://www.mongodb.com/nosql-explained/nosql-vs-sql](https://www.mongodb.com/nosql-explained/nosql-vs-sql) + +Cap Theorem + +[http://www.julianbrowne.com/article/brewers-cap-theorem](http://www.julianbrowne.com/article/brewers-cap-theorem) + +Scalability + +[http://www.slideshare.net/jboner/scalability-availability-stability-patterns](http://www.slideshare.net/jboner/scalability-availability-stability-patterns) + +Eventual Consistency + +[https://www.allthingsdistributed.com/2008/12/eventually_consistent.html](https://www.allthingsdistributed.com/2008/12/eventually_consistent.html) + +[https://www.toptal.com/big-data/consistent-hashing](https://www.toptal.com/big-data/consistent-hashing) + +[https://web.stanford.edu/class/cs244/papers/chord_TON_2003.pdf](https://web.stanford.edu/class/cs244/papers/chord_TON_2003.pdf) diff --git a/courses/databases_nosql/images/Quorum.png b/courses/databases_nosql/images/Quorum.png new file mode 100644 index 0000000..8e79ec5 Binary files /dev/null and b/courses/databases_nosql/images/Quorum.png differ diff --git a/courses/databases_nosql/images/cluster_quorum.png b/courses/databases_nosql/images/cluster_quorum.png new file mode 100644 index 0000000..1091fb4 Binary files /dev/null and b/courses/databases_nosql/images/cluster_quorum.png differ diff --git a/courses/databases_nosql/images/consistent_hashing.png b/courses/databases_nosql/images/consistent_hashing.png new file mode 100644 index 0000000..31564bc Binary files /dev/null and b/courses/databases_nosql/images/consistent_hashing.png differ diff --git a/courses/databases_nosql/images/database_sharding.png b/courses/databases_nosql/images/database_sharding.png new file mode 100644 index 0000000..b3f83db Binary files /dev/null and b/courses/databases_nosql/images/database_sharding.png differ diff --git a/courses/databases_nosql/images/vector_clocks.png b/courses/databases_nosql/images/vector_clocks.png new file mode 100644 index 0000000..c4e9361 Binary files /dev/null and b/courses/databases_nosql/images/vector_clocks.png differ diff --git a/courses/databases_nosql/intro.md b/courses/databases_nosql/intro.md new file mode 100644 index 0000000..caf1d7f --- /dev/null +++ b/courses/databases_nosql/intro.md @@ -0,0 +1,217 @@ +# NoSQL Concepts + + +## What to expect from this course + +At the end of training, you will have an understanding of what a NoSQL database is, what kind of advantages or disadvantages it has over traditional RDBMS, learn about different types of NoSQL databases and understand some of the underlying concepts & trade offs w.r.t to NoSQL. + + +## What is not covered under this course + +We will not be deep diving into any specific NoSQL Database. + + +## Course Contents + + + +* [Introduction to NoSQL](https://linkedin.github.io/school-of-sre/databases_nosql/intro/#introduction) +* [CAP Theorem](https://linkedin.github.io/school-of-sre/databases_nosql/key_concepts/#cap-theorem) +* [Data versioning](https://linkedin.github.io/school-of-sre/databases_nosql/key_concepts/#versioning-of-data-in-distributed-systems) +* [Partitioning](https://linkedin.github.io/school-of-sre/databases_nosql/key_concepts/#partitioning) +* [Hashing](https://linkedin.github.io/school-of-sre/databases_nosql/key_concepts/#hashing) +* [Quorum](https://linkedin.github.io/school-of-sre/databases_nosql/key_concepts/#quorum) + + +## Introduction + +When people use the term “NoSQL database”, they typically use it to refer to any non-relational database. Some say the term “NoSQL” stands for “non SQL” while others say it stands for “not only SQL.” Either way, most agree that NoSQL databases are databases that store data in a format other than relational tables. + +A common misconception is that NoSQL databases or non-relational databases don’t store relationship data well. NoSQL databases can store relationship data—they just store it differently than relational databases do. In fact, when compared with SQL databases, many find modeling relationship data in NoSQL databases to be _easier_, because related data doesn’t have to be split between tables. + +Such databases have existed since the late 1960s, but the name "NoSQL" was only coined in the early 21st century. NASA used a NoSQL database to track inventory for the Apollo mission. NoSQL databases emerged in the late 2000s as the cost of storage dramatically decreased. Gone were the days of needing to create a complex, difficult-to-manage data model simply for the purposes of reducing data duplication. Developers (rather than storage) were becoming the primary cost of software development, so NoSQL databases optimized for developer productivity. With the rise of Agile development methodology, NoSQL databases were developed with a focus on scaling, fast performance and at the same time allowed for frequent application changes and made programming easier. + + +### Types of NoSQL databases: + +Over time due to the way these NoSQL databases were developed to suit requirements at different companies, we ended up with quite a few types of them. However, they can be broadly classified into 4 types. Some of the databases can overlap between different types. They are + + + +1. **Document databases: **They store data in documents similar to [JSON](https://www.json.org/json-en.html) (JavaScript Object Notation) objects. Each document contains pairs of fields and values. The values can typically be a variety of types including things like strings, numbers, booleans, arrays, or objects, and their structures typically align with objects developers are working with in code. The advantages include intuitive data model & flexible schemas. Because of their variety of field value types and powerful query languages, document databases are great for a wide variety of use cases and can be used as a general purpose database. They can horizontally scale-out to accomodate large data volumes. Ex: MongoDB, Couchbase +2. **Key-Value databases:** These are a simpler type of databases where each item contains keys and values. A value can typically only be retrieved by referencing its value, so learning how to query for a specific key-value pair is typically simple. Key-value databases are great for use cases where you need to store large amounts of data but you don’t need to perform complex queries to retrieve it. Common use cases include storing user preferences or caching. Ex: [Redis](https://redis.io/), [DynamoDB](https://aws.amazon.com/dynamodb/), [Voldemort](https://www.project-voldemort.com/voldemort/)/[Venice](https://engineering.linkedin.com/blog/2017/04/building-venice--a-production-software-case-study) (Linkedin), +3. **Wide-Column stores:** They store data in tables, rows, and dynamic columns. Wide-column stores provide a lot of flexibility over relational databases because each row is not required to have the same columns. Many consider wide-column stores to be two-dimensional key-value databases. Wide-column stores are great for when you need to store large amounts of data and you can predict what your query patterns will be. Wide-column stores are commonly used for storing Internet of Things data and user profile data. [Cassandra](https://cassandra.apache.org/) and [HBase](https://hbase.apache.org/) are two of the most popular wide-column stores. +4. Graph Databases: These databases store data in nodes and edges. Nodes typically store information about people, places, and things while edges store information about the relationships between the nodes. The underlying storage mechanism of graph databases can vary. Some depend on a relational engine and “store” the graph data in a table (although a table is a logical element, therefore this approach imposes another level of abstraction between the graph database, the graph database management system and the physical devices where the data is actually stored). Others use a key-value store or document-oriented database for storage, making them inherently NoSQL structures. Graph databases excel in use cases where you need to traverse relationships to look for patterns such as social networks, fraud detection, and recommendation engines. Ex: [Neo4j](https://neo4j.com/) + + +### **Comparison** + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ Performance + Scalability + Flexibility + Complexity + Functionality +
Key Value + high + high + high + none + Variable +
Document stores + high + Variable (high) + high + low + Variable (low) +
Column DB + high + high + moderate + low + minimal +
Graph + Variable + Variable + high + high + Graph theory +
+ + + +### Differences between SQL and NoSQL + +The table below summarizes the main differences between SQL and NoSQL databases. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ SQL Databases + NoSQL Databases +
Data Storage Model + Tables with fixed rows and columns + Document: JSON documents, Key-value: key-value pairs, Wide-column: tables with rows and dynamic columns, Graph: nodes and edges +
Primary Purpose + General purpose + Document: general purpose, Key-value: large amounts of data with simple lookup queries, Wide-column: large amounts of data with predictable query patterns, Graph: analyzing and traversing relationships between connected data +
Schemas + Rigid + Flexible +
Scaling + Vertical (scale-up with a larger server) + Horizontal (scale-out across commodity servers) +
Multi-Record ACID Transactions + Supported + Most do not support multi-record ACID transactions. However, some—like MongoDB—do. +
Joins + Typically required + Typically not required +
Data to Object Mapping + Requires ORM (object-relational mapping) + Many do not require ORMs. Document DB documents map directly to data structures in most popular programming languages. +
+ + + +### Advantages + + + +* Flexible Data Models + + Most NoSQL systems feature flexible schemas. A flexible schema means you can easily modify your database schema to add or remove fields to support for evolving application requirements. This facilitates with continuous application development of new features without database operation overhead. + +* Horizontal Scaling + + Most NoSQL systems allow you to scale horizontally, which means you can add in cheaper & commodity hardware, whenever you want to scale a system. On the other hand SQL systems generally scale Vertically (a more powerful server). NoSQL systems can also host huge data sets when compared to traditional SQL systems. + +* Fast Queries + + NoSQL can generally be a lot faster than traditional SQL systems due to data denormalization and horizontal scaling. Most NoSQL systems also tend to store similar data together facilitating faster query responses. + +* Developer productivity + + NoSQL systems tend to map data based on the programming data structures. As a result developers need to perform fewer data transformations leading to increased productivity & fewer bugs. diff --git a/courses/databases_nosql/key_concepts.md b/courses/databases_nosql/key_concepts.md new file mode 100644 index 0000000..a150d40 --- /dev/null +++ b/courses/databases_nosql/key_concepts.md @@ -0,0 +1,274 @@ +# Key Concepts + +Lets looks at some of the key concepts when we talk about NoSQL or distributed systems + + +### CAP Theorem + + + +In a keynote titled “[Towards Robust Distributed Systems](https://sites.cs.ucsb.edu/~rich/class/cs293b-cloud/papers/Brewer_podc_keynote_2000.pdf)” at ACM’s PODC symposium in 2000 Eric Brewer came up with the so-called CAP-theorem which is widely adopted today by large web companies as well as in the NoSQL community. The CAP acronym stands for **C**onsistency, **A**vailability & **P**artition Tolerance. + + + +* **Consistency** + + It refers to how consistent a system is after an execution. A distributed system is called consistent when a write made by a source is available for all readers of that shared data. Different NoSQL systems support different levels of consistency. + +* **Availability** + + It refers to how a system responds to loss of functionality of different systems due to hardware and software failures. A high availability implies that a system is still available to handle operations (reads and writes) when a certain part of the system is down due to a failure or upgrade. + +* **Partition Tolerance** + + It is the ability of the system to continue operations in the event of a network partition. A network partition occurs when a failure causes two or more islands of networks where the systems can’t talk to each other across the islands temporarily or permanently. + + +Brewer alleges that one can at most choose two of these three characteristics in a shared-data system. The CAP-theorem states that a choice can only be made for two options out of consistency, availability and partition tolerance. A growing number of use cases in large scale applications tend to value reliability implying that availability & redundancy are more valuable than consistency. As a result these systems struggle to meet ACID properties. They attain this by loosening on the consistency requirement i.e Eventual Consistency. + +**Eventual Consistency **means that all readers will see writes, as time goes on: “In a steady state, the system will eventually return the last written value”. Clients therefore may face an inconsistent state of data as updates are in progress. For instance, in a replicated database updates may go to one node which replicates the latest version to all other nodes that contain a replica of the modified dataset so that the replica nodes eventually will have the latest version. + +NoSQL systems support different levels of eventual consistency models. For example: + + + +* Read Your Own Writes Consistency + + A client will see his updates immediately after they are written. The reads can hit nodes other than the one where it was written. However he might not see updates by other clients immediately. + +* Session Consistency: + + A client will see the updates to his data within a session scope. This generally indicates that reads & writes occur on the same server. Other clients using the same nodes will receive the same updates. + +* Casual Consistency + + A system provides causal consistency if the following condition holds: write operations that are related by potential causality are seen by each process of the system in order. Different processes may observe concurrent writes in different orders + + + + +Eventual consistency is useful if concurrent updates of the same partitions of data are unlikely and if clients do not immediately depend on reading updates issued by themselves or by other clients. + +Depending on what consistency model was chosen for the system (or parts of it), determines where the requests are routed, ex: replicas. + +CAP alternatives illustration + + + + + + + + + + + + + + + + + + + + + + + +
Choice + Traits + Examples +
Consistency + Availability +

+(Forfeit Partitions) +

2-phase commits +

+Cache invalidation protocols +

Single-site databases Cluster databases +

+LDAP +

+xFS file system +

Consistency + Partition tolerance +

+ (Forfeit Availability) +

Pessimistic locking +

+Make minority partitions unavailable +

Distributed databases Distributed locking Majority protocols +
Availability + Partition tolerance (Forfeit Consistency) + expirations/leases +

+conflict resolution optimistic +

DNS +

+Web caching +

+ + +### Versioning of Data in distributed systems + +When data is distributed across nodes, it can be modified on different nodes at the same time (assuming strict consistency is enforced). Questions arise on conflict resolution for concurrent updates. Some of the popular conflict resolution mechanism are + + + +* **Timestamps** + + This is the most obvious solution. You sort updates based on chronological order and choose the latest update. However this relies on clock synchronization across different parts of the infrastructure. This gets even more complicated when parts of systems are spread across different geographic locations. + +* **Optimistic Locking** + + You associate a unique value like a clock or counter with every data update. When a client wants to update data, it has to specify which version of data needs to be updated. This would mean you need to keep track of history of the data versions. + +* **Vector Clocks** + + A vector clock is defined as a tuple of clock values from each node. In a distributed environment, each node maintains a tuple of such clock values which represent the state of the nodes itself and its peers/replicas. A clock value may be real timestamps derived from local clock or version no. + +

+ + +![alt_text](images/vector_clocks.png "Vector Clocks") + + + + +

Vector clocks illustration

+ +Vector clocks have the following advantages over other conflict resolution mechanism + + + +1. No dependency on synchronized clocks +2. No total ordering of revision nos required for casual reasoning + +No need to store and maintain multiple versions of the data on different nodes.** ** + + +### Partitioning + +When the amount of data crosses the capacity of a single node, we need to think of splitting data, creating replicas for load balancing & disaster recovery. Depending on how dynamic the infrastructure is, we have a few approaches that we can take. + + + +1. **Memory cached** + + These are partitioned in-memory databases that are primarily used for transient data. These databases are generally used as a front for traditional RDBMS. Most frequently used data is replicated from a rdbms into a memory database to facilitate fast queries and to take the load off from backend DB’s. A very common example is memcached or couchbase. + +2. **Clustering** + + Traditional cluster mechanisms abstract away the cluster topology from clients. A client need not know where the actual data is residing and which node it is talking to. Clustering is very commonly used in traditional RDBMS where it can help scaling the persistent layer to a certain extent. + +3. **Separating reads from writes** + + In this method, you will have multiple replicas hosting the same data. The incoming writes are typically sent to a single node (Leader) or multiple nodes (multi-Leader), while the rest of the replicas (Follower) handle reads requests. The leader replicates writes asynchronously to all followers. However the write lag can’t be completely avoided. Sometimes a leader can crash before it replicates all the data to a follower. When this happens, a follower with the most consistent data can be turned into a leader. As you can realize now, it is hard to enforce full consistency in this model. You also need to consider the ratio of read vs write traffic. This model won’t make sense when writes are higher than reads. The replication methods can also vary widely. Some systems do a complete transfer of state periodically, while others use a delta state transfer approach. You could also transfer the state by transferring the operations in order. The followers can then apply the same operations as the leader to catch up. + +4. **Sharding** + + Sharing refers to dividing data in such a way that data is distributed evenly (both in terms of storage & processing power) across a cluster of nodes. It can also imply data locality, which means similar & related data is stored together to facilitate faster access. A shard in turn can be further replicated to meet load balancing or disaster recovery requirements. A single shard replica might take in all writes (single leader) or multiple replicas can take writes (multi-leader). Reads can be distributed across multiple replicas. Since data is now distributed across multiple nodes, clients should be able to consistently figure out where data is hosted. We will look at some of the common techniques below. The downside of sharding is that joins between shards is not possible. So an upstream/downstream application has to aggregate the results from multiple shards. + +

+ + +![alt_text]( images/database_sharding.png "Sharding") + + +

Sharding example

+ + +### Hashing + +A hash function is a function that maps one piece of data—typically describing some kind of object, often of arbitrary size—to another piece of data, typically an integer, known as _hash code_, or simply _hash_. In a partitioned database, it is important to consistently map a key to a server/replica. + +For ex: you can use a very simple hash as a modulo function. + + + _p = k mod n_ + +Where + + + p -> partition, + + + k -> primary key + + + n -> no of nodes + +The downside of this simple hash is that, whenever the cluster topology changes, the data distribution also changes. When you are dealing with memory caches, it will be easy to distribute partitions around. Whenever a node joins/leaves a topology, partitions can reorder themselves, a cache miss can be re-populated from backend DB. However when you look at persistent data, it is not possible as the new node doesn’t have the data needed to serve it. This brings us to consistent hashing. + + +#### Consistent Hashing + +Consistent hashing is a distributed hashing scheme that operates independently of the number of servers or objects in a distributed _hash table_ by assigning them a position on an abstract circle, or _hash ring_. This allows servers and objects to scale without affecting the overall system. + +Say that our hash function h() generates a 32-bit integer. Then, to determine to which server we will send a key k, we find the server s whose hash h(s) is the smallest integer that is larger than h(k). To make the process simpler, we assume the table is circular, which means that if we cannot find a server with a hash larger than h(k), we wrap around and start looking from the beginning of the array. + +

+ + +![alt_text]( images/consistent_hashing.png "Consistent Hashing") + + +

Consistent hashing illustration

+ +In consistent hashing when a server is removed or added then only the keys from that server are relocated. For example, if server S3 is removed then, all keys from server S3 will be moved to server S4 but keys stored on server S4 and S2 are not relocated. But there is one problem, when server S3 is removed then keys from S3 are not equally distributed among remaining servers S4 and S2. They are only assigned to server S4 which increases the load on server S4. + +To evenly distribute the load among servers when a server is added or removed, it creates a fixed number of replicas ( known as virtual nodes) of each server and distributes it along the circle. So instead of server labels S1, S2 and S3, we will have S10 S11…S19, S20 S21…S29 and S30 S31…S39. The factor for a number of replicas is also known as _weight_, depending on the situation. + + + + +All keys which are mapped to replicas Sij are stored on server Si. To find a key we do the same thing, find the position of the key on the circle and then move forward until you find a server replica. If the server replica is Sij then the key is stored in server Si. + +Suppose server S3 is removed, then all S3 replicas with labels S30 S31 … S39 must be removed. Now the objects keys adjacent to S3X labels will be automatically re-assigned to S1X, S2X and S4X. All keys originally assigned to S1, S2 & S4 will not be moved. + +Similar things happen if we add a server. Suppose we want to add a server S5 as a replacement of S3 then we need to add labels S50 S51 … S59. In the ideal case, one-fourth of keys from S1, S2 and S4 will be reassigned to S5. + +When applied to persistent storages, further issues arise: if a node has left the scene, data stored on this node becomes unavailable, unless it has been replicated to other nodes before; in the opposite case of a new node joining the others, adjacent nodes are no longer responsible for some pieces of data which they still store but not get asked for anymore as the corresponding objects are no longer hashed to them by requesting clients. In order to address this issue, a replication factor (r) can be introduced. + +Introducing replicas in a partitioning scheme—besides reliability benefits—also makes it possible to spread workload for read requests that can go to any physical node responsible for a requested piece of data. Scalability doesn’t work if the clients have to decide between multiple versions of the dataset, because they need to read from a quorum of servers which in turn reduces the efficiency of load balancing. + + + + +### Quorum + +Quorum is the minimum number of nodes in a cluster that must be online and be able to communicate with each other. If any additional node failure occurs beyond this threshold, the cluster will stop running. + + + + + +To attain a quorum, you need a majority of the nodes. Commonly it is (N/2 + 1), where N is the total no of nodes in the system. For ex, + +In a 3 node cluster, you need 2 nodes for a majority, + +In a 5 node cluster, you need 3 nodes for a majority, + +In a 6 node cluster, you need 4 nodes for a majority. + +

+ + +![alt_text](images/Quorum.png "image_tooltip") + +

Quorum example

+ + + +Network problems can cause communication failures among cluster nodes. One set of nodes might be able to communicate together across a functioning part of a network but not be able to communicate with a different set of nodes in another part of the network. This is known as split brain in cluster or cluster partitioning. + +Now the partition which has quorum is allowed to continue running the application. The other partitions are removed from the cluster. + +Eg: In a 5 node cluster, consider what happens if nodes 1, 2, and 3 can communicate with each other but not with nodes 4 and 5. Nodes 1, 2, and 3 constitute a majority, and they continue running as a cluster. Nodes 4 and 5, being a minority, stop running as a cluster. If node 3 loses communication with other nodes, all nodes stop running as a cluster. However, all functioning nodes will continue to listen for communication, so that when the network begins working again, the cluster can form and begin to run. + +Below diagram demonstrates Quorum selection on a cluster partitioned into two sets. + +

+ + +![alt_text](images/cluster_quorum.png "image_tooltip") + +**

Cluster Quorum example

** + diff --git a/courses/git/branches.md b/courses/git/branches.md index 92e07e8..856db58 100644 --- a/courses/git/branches.md +++ b/courses/git/branches.md @@ -83,7 +83,7 @@ Above tree structure should make things clear. Notice a clear branch/fork on com ## Merges -Now say the feature you were working on branch `b1` is complete. And you need to merge it on master branch, where all the final version of code goes. So first you will checkout to branch master and then you will pull the latest code from upstream (eg: GitHub). Then you need to merge your code from `b1` into master. And there could be two ways this can be done. +Now say the feature you were working on branch `b1` is complete and you need to merge it on master branch, where all the final version of code goes. So first you will checkout to branch master and then you pull the latest code from upstream (eg: GitHub). Then you need to merge your code from `b1` into master. There could be two ways this can be done. Here is the current history: @@ -96,7 +96,7 @@ spatel1-mn1:school-of-sre spatel1$ git log --oneline --graph --all * df2fb7a adding file 1 ``` -**Option 1: Directly merge the branch.** Merging the branch b1 into master will result in a new merge commit which will merge changes from two different lines of history and create a new commit of the result. +**Option 1: Directly merge the branch.** Merging the branch b1 into master will result in a new merge commit. This will merge changes from two different lines of history and create a new commit of the result. ```bash spatel1-mn1:school-of-sre spatel1$ git merge b1 diff --git a/courses/git/conclusion.md b/courses/git/conclusion.md new file mode 100644 index 0000000..32f45b6 --- /dev/null +++ b/courses/git/conclusion.md @@ -0,0 +1,10 @@ +## What next from here? + +There are a lot of git commands and features which we have not explored here. But with the base built-up, be sure to explore concepts like + +- Cherrypick +- Squash +- Amend +- Stash +- Reset + diff --git a/courses/git/git-basics.md b/courses/git/git-basics.md index d5a84c3..1fd3774 100644 --- a/courses/git/git-basics.md +++ b/courses/git/git-basics.md @@ -10,15 +10,13 @@ ## What to expect from this course -As an engineer in the field of computer science, having knowledge of version control tools becomes almost a requirement. While there are a lot of version control tools that exist today, Git perhaps is the most used one and this course we will be working with Git. While this course does not start with Git 101 and expects basic knowledge of git as a prerequisite, it will reintroduce the git concepts known by you with details covering what is happening under the hood as you execute various git commands. So that next time you run a git command, you will be able to press enter more confidently! +As an engineer in the field of computer science, having knowledge of version control tools becomes almost a requirement. While there are a lot of version control tools that exist today like SVN, Mercurial, etc, Git perhaps is the most used one and this course we will be working with Git. While this course does not start with Git 101 and expects basic knowledge of git as a prerequisite, it will reintroduce the git concepts known by you with details covering what is happening under the hood as you execute various git commands. So that next time you run a git command, you will be able to press enter more confidently! ## What is not covered under this course Advanced usage and specifics of internal implementation details of Git. -## Course Content - -### Table of Contents +## Course Contents 1. [Git Basics](https://linkedin.github.io/school-of-sre/git/git-basics/#git-basics) 2. [Working with Branches](https://linkedin.github.io/school-of-sre/git/branches/) @@ -27,7 +25,7 @@ Advanced usage and specifics of internal implementation details of Git. ## Git Basics -Though you might be aware already, let's revisit why we need a version control system. As the project grows and multiple developers start working on it, an efficient method for collaboration is warranted. Git helps the team collaborate easily and also maintains history of the changes happened with the codebase. +Though you might be aware already, let's revisit why we need a version control system. As the project grows and multiple developers start working on it, an efficient method for collaboration is warranted. Git helps the team collaborate easily and also maintains the history of the changes happening with the codebase. ### Creating a Git Repo @@ -92,7 +90,7 @@ spatel1-mn1:school-of-sre spatel1$ git commit -m "adding file 1" create mode 100644 file1.txt ``` -Notice how after adding the file, git status says `Changes to be commited:`. What it means is whatever is listed there, will be included in the next commit. Then we go ahead and create a commit, with an attached messaged via `-m`. +Notice how after adding the file, git status says `Changes to be committed:`. What it means is whatever is listed there, will be included in the next commit. Then we go ahead and create a commit, with an attached messaged via `-m`. ### More About a Commit diff --git a/courses/git/github-hooks.md b/courses/git/github-hooks.md index db78e76..74ef622 100644 --- a/courses/git/github-hooks.md +++ b/courses/git/github-hooks.md @@ -40,13 +40,3 @@ this is from pre commit hook # <===== THE MESSAGE FROM HOOK EXECUTION 1 file changed, 1 insertion(+) create mode 100644 sample.txt ``` - -## What next from here? - -There are a lot of git commands and features which we have not explored here. But with the base built-up, be sure to explore concepts like - -- Cherrypick -- Squash -- Amend -- Stash -- Reset diff --git a/courses/index.md b/courses/index.md index 28d24aa..3f7c49a 100644 --- a/courses/index.md +++ b/courses/index.md @@ -1,5 +1,7 @@ # School of SRE -![School of SRE](img/sos.png) + + + Early 2019, we started visiting campuses to recruit the brightest minds to ensure LinkedIn and all the services that it is composed of is always available for everyone. This function at Linkedin falls in the purview of the Site Reliability Engineering team and Site Reliability Engineers ( SRE ) who are Software Engineers who specialize in reliability. SREs apply the principles of computer science and engineering to the design and development of computer systems: generally, large distributed ones. As we continued on this journey we started getting a lot of questions from these campuses on what exactly site engineering roll entails? and, how could someone learn the skills and the disciplines involved to become a successful site engineer? Fast forward a few months, and a few of these campus students had joined LinkedIn either as Interns or as full time engineers to become a part of the Site Engineering team, we also had a few lateral hires who joined our organization who were not from a traditional SRE background. That's when a few of us got together and started to think about how we can on board new new graduate engineers to the site engineering team. @@ -15,7 +17,7 @@ In this course we are focusing on building strong foundational skills. The cours - [Python and Web](https://linkedin.github.io/school-of-sre/python_web/intro/) - Data - Relational databases (MySQL) - - NoSQL concepts + - [NoSQL concepts](https://linkedin.github.io/school-of-sre/databases_nosql/intro/) - [Big Data](https://linkedin.github.io/school-of-sre/big_data/intro/) - [Systems Design](https://linkedin.github.io/school-of-sre/systems_design/intro/) - [Security](https://linkedin.github.io/school-of-sre/security/intro/) diff --git a/courses/linux_basics/command_line_basics.md b/courses/linux_basics/command_line_basics.md index 54845ff..33294b0 100644 --- a/courses/linux_basics/command_line_basics.md +++ b/courses/linux_basics/command_line_basics.md @@ -1,7 +1,12 @@ # Command Line Basics -## What is a command ? +## Lab Environment Setup +One can use an online bash interpreter to run all the commands that are provided as examples in this course. This will also help you in getting a hands-on experience of various linux commands. + +[REPL](https://repl.it/languages/bash) is one of the popular online bash interpreters for running linux commands. We will be using it for running all the commands mentioned in this course. + +## What is a Command A command is a program that tells the operating system to perform specific work. Programs are stored as files in linux. Therefore, a @@ -438,22 +443,3 @@ prints the unique numbers from the input. I/O redirection - [https://tldp.org/LDP/abs/html/io-redirection.html](https://tldp.org/LDP/abs/html/io-redirection.html) - -## Applications in SRE Role - -- As a SRE, you will be required to perform some general tasks on these linux servers. You will also be using the command line when you are troubleshooting issues. - -- Moving from one location to another in the filesystem will require the help of ls, pwd and cd commands - -- You may need to search some specific information in the log files. Grep command would be very useful here. I/O redirection will become handy if you want to store the output in a file or pass it as an input to another command. - -- Tail command is very useful to view the latest data in the log file. - -## Useful courses and tutorials - - -- [Edx linuxcourse](https://courses.edx.org/courses/course-v1:LinuxFoundationX+LFS101x+1T2020/course/) - - This video course can be very helpful in developing the basics of linux command line. This course is provided - in both free and paidmodes by edX. If you take the free course, you will not be able to access the assignments. - -- [https://linuxcommand.org/lc3_learning_the_shell.php](https://linuxcommand.org/lc3_learning_the_shell.php) diff --git a/courses/linux_basics/conclusion.md b/courses/linux_basics/conclusion.md new file mode 100644 index 0000000..783340f --- /dev/null +++ b/courses/linux_basics/conclusion.md @@ -0,0 +1,25 @@ +# Conclusion + +With this we have covered the basics of linux operating systems along with basic commands +which are used in linux. We have also covered the linux server administration commands. + +We hope that this course will make it easier for you to operate on the command line. + +## Applications in SRE Role + +1. As a SRE, you will be required to perform some general tasks on these linux servers. You will also be using the command line when you are troubleshooting issues. +2. Moving from one location to another in the filesystem will require the help of ls, pwd and cd commands +3. You may need to search some specific information in the log files. Grep command would be very useful here. I/O redirection will become handy if you want to store the output in a file or pass it as an input to another command. +4. Tail command is very useful to view the latest data in the log file. +5. Different users will have different permissions depending on their roles. We will also not want everyone in the company to access our servers for security reasons. Users permissions can be restricted with chown, chmod and chgrp commands. +6. SSH is one of the most frequently used commands for a SRE. Logging into servers and troubleshooting along with performing basic administration tasks will only be possible if we are able to login into the server. +7. What if we want to run an apache server or nginx on a server ? We will first install it using the package manager. Package management commands become important here. +8. Managing services on servers is another critical responsibility of a SRE. Systemd related commands can help in troubleshooting issues. If a service goes down, we can start it using systemctl start command. We can also stop a service in case it is not needed. +9. Monitoring is another core responsibility of a SRE. Memory and CPU are two important system level metrics which should be monitored. Commands like top and free are quite helpful here. +10. If a service is throwing an error, how do we find out the root cause of the error ? We will certainly need to check logs to find out the whole stack trace of the error. The log file will also tell us the number of times the error has occurred along with time when it started. + +## Useful Courses and tutorials + +* [Edx basic linux commands course](https://courses.edx.org/courses/course-v1:LinuxFoundationX+LFS101x+1T2020/course/) +* [Edx Red Hat Enterprise Linux Course](https://courses.edx.org/courses/course-v1:RedHat+RH066x+2T2017/course/) +* [https://linuxcommand.org/lc3_learning_the_shell.php](https://linuxcommand.org/lc3_learning_the_shell.php) diff --git a/courses/linux_basics/images/linux/admin/image1.png b/courses/linux_basics/images/linux/admin/image1.png index a0847e0..365ad09 100644 Binary files a/courses/linux_basics/images/linux/admin/image1.png and b/courses/linux_basics/images/linux/admin/image1.png differ diff --git a/courses/linux_basics/images/linux/admin/image10.png b/courses/linux_basics/images/linux/admin/image10.png index 2022fab..73d1a2a 100644 Binary files a/courses/linux_basics/images/linux/admin/image10.png and b/courses/linux_basics/images/linux/admin/image10.png differ diff --git a/courses/linux_basics/images/linux/admin/image11.png b/courses/linux_basics/images/linux/admin/image11.png index c45dffb..7710bdc 100644 Binary files a/courses/linux_basics/images/linux/admin/image11.png and b/courses/linux_basics/images/linux/admin/image11.png differ diff --git a/courses/linux_basics/images/linux/admin/image12.png b/courses/linux_basics/images/linux/admin/image12.png index 328b0a6..74199df 100644 Binary files a/courses/linux_basics/images/linux/admin/image12.png and b/courses/linux_basics/images/linux/admin/image12.png differ diff --git a/courses/linux_basics/images/linux/admin/image13.png b/courses/linux_basics/images/linux/admin/image13.png index 1d701a7..5044f3d 100644 Binary files a/courses/linux_basics/images/linux/admin/image13.png and b/courses/linux_basics/images/linux/admin/image13.png differ diff --git a/courses/linux_basics/images/linux/admin/image14.png b/courses/linux_basics/images/linux/admin/image14.png index 42d25e5..5a0f468 100644 Binary files a/courses/linux_basics/images/linux/admin/image14.png and b/courses/linux_basics/images/linux/admin/image14.png differ diff --git a/courses/linux_basics/images/linux/admin/image15.png b/courses/linux_basics/images/linux/admin/image15.png index c0d9979..e0aa749 100644 Binary files a/courses/linux_basics/images/linux/admin/image15.png and b/courses/linux_basics/images/linux/admin/image15.png differ diff --git a/courses/linux_basics/images/linux/admin/image16.png b/courses/linux_basics/images/linux/admin/image16.png index e5fc1a0..947658d 100644 Binary files a/courses/linux_basics/images/linux/admin/image16.png and b/courses/linux_basics/images/linux/admin/image16.png differ diff --git a/courses/linux_basics/images/linux/admin/image17.png b/courses/linux_basics/images/linux/admin/image17.png index 9416248..26d777a 100644 Binary files a/courses/linux_basics/images/linux/admin/image17.png and b/courses/linux_basics/images/linux/admin/image17.png differ diff --git a/courses/linux_basics/images/linux/admin/image18.png b/courses/linux_basics/images/linux/admin/image18.png index 8e21fa8..ac72d6d 100644 Binary files a/courses/linux_basics/images/linux/admin/image18.png and b/courses/linux_basics/images/linux/admin/image18.png differ diff --git a/courses/linux_basics/images/linux/admin/image19.png b/courses/linux_basics/images/linux/admin/image19.png index fc61c5f..f782f43 100644 Binary files a/courses/linux_basics/images/linux/admin/image19.png and b/courses/linux_basics/images/linux/admin/image19.png differ diff --git a/courses/linux_basics/images/linux/admin/image2.png b/courses/linux_basics/images/linux/admin/image2.png index 0b84a99..bf056a0 100644 Binary files a/courses/linux_basics/images/linux/admin/image2.png and b/courses/linux_basics/images/linux/admin/image2.png differ diff --git a/courses/linux_basics/images/linux/admin/image20.png b/courses/linux_basics/images/linux/admin/image20.png index 1e49ce0..eaf3dd4 100644 Binary files a/courses/linux_basics/images/linux/admin/image20.png and b/courses/linux_basics/images/linux/admin/image20.png differ diff --git a/courses/linux_basics/images/linux/admin/image21.png b/courses/linux_basics/images/linux/admin/image21.png index b8034fa..1eb0234 100644 Binary files a/courses/linux_basics/images/linux/admin/image21.png and b/courses/linux_basics/images/linux/admin/image21.png differ diff --git a/courses/linux_basics/images/linux/admin/image22.png b/courses/linux_basics/images/linux/admin/image22.png index 8a339a5..bc77e51 100644 Binary files a/courses/linux_basics/images/linux/admin/image22.png and b/courses/linux_basics/images/linux/admin/image22.png differ diff --git a/courses/linux_basics/images/linux/admin/image23.png b/courses/linux_basics/images/linux/admin/image23.png index 2d43eac..56345f0 100644 Binary files a/courses/linux_basics/images/linux/admin/image23.png and b/courses/linux_basics/images/linux/admin/image23.png differ diff --git a/courses/linux_basics/images/linux/admin/image24.png b/courses/linux_basics/images/linux/admin/image24.png index 8aed08a..9b5d955 100644 Binary files a/courses/linux_basics/images/linux/admin/image24.png and b/courses/linux_basics/images/linux/admin/image24.png differ diff --git a/courses/linux_basics/images/linux/admin/image25.png b/courses/linux_basics/images/linux/admin/image25.png index 0606aab..50b894f 100644 Binary files a/courses/linux_basics/images/linux/admin/image25.png and b/courses/linux_basics/images/linux/admin/image25.png differ diff --git a/courses/linux_basics/images/linux/admin/image26.png b/courses/linux_basics/images/linux/admin/image26.png index a35f697..6b561a8 100644 Binary files a/courses/linux_basics/images/linux/admin/image26.png and b/courses/linux_basics/images/linux/admin/image26.png differ diff --git a/courses/linux_basics/images/linux/admin/image27.png b/courses/linux_basics/images/linux/admin/image27.png index 394b79f..7eee5e2 100644 Binary files a/courses/linux_basics/images/linux/admin/image27.png and b/courses/linux_basics/images/linux/admin/image27.png differ diff --git a/courses/linux_basics/images/linux/admin/image28.png b/courses/linux_basics/images/linux/admin/image28.png index b4bd8ba..1e7a557 100644 Binary files a/courses/linux_basics/images/linux/admin/image28.png and b/courses/linux_basics/images/linux/admin/image28.png differ diff --git a/courses/linux_basics/images/linux/admin/image29.png b/courses/linux_basics/images/linux/admin/image29.png index 6ccaf73..2f8cdfe 100644 Binary files a/courses/linux_basics/images/linux/admin/image29.png and b/courses/linux_basics/images/linux/admin/image29.png differ diff --git a/courses/linux_basics/images/linux/admin/image3.png b/courses/linux_basics/images/linux/admin/image3.png index 6d9ebc4..a4d5c3c 100644 Binary files a/courses/linux_basics/images/linux/admin/image3.png and b/courses/linux_basics/images/linux/admin/image3.png differ diff --git a/courses/linux_basics/images/linux/admin/image30.png b/courses/linux_basics/images/linux/admin/image30.png index d9dccae..9e9caa6 100644 Binary files a/courses/linux_basics/images/linux/admin/image30.png and b/courses/linux_basics/images/linux/admin/image30.png differ diff --git a/courses/linux_basics/images/linux/admin/image31.jpg b/courses/linux_basics/images/linux/admin/image31.jpg index d781a27..b7d99c1 100644 Binary files a/courses/linux_basics/images/linux/admin/image31.jpg and b/courses/linux_basics/images/linux/admin/image31.jpg differ diff --git a/courses/linux_basics/images/linux/admin/image32.png b/courses/linux_basics/images/linux/admin/image32.png index e11722a..99be765 100644 Binary files a/courses/linux_basics/images/linux/admin/image32.png and b/courses/linux_basics/images/linux/admin/image32.png differ diff --git a/courses/linux_basics/images/linux/admin/image33.png b/courses/linux_basics/images/linux/admin/image33.png index 17a3bad..f56c4ed 100644 Binary files a/courses/linux_basics/images/linux/admin/image33.png and b/courses/linux_basics/images/linux/admin/image33.png differ diff --git a/courses/linux_basics/images/linux/admin/image34.png b/courses/linux_basics/images/linux/admin/image34.png index fba8d86..0357536 100644 Binary files a/courses/linux_basics/images/linux/admin/image34.png and b/courses/linux_basics/images/linux/admin/image34.png differ diff --git a/courses/linux_basics/images/linux/admin/image35.png b/courses/linux_basics/images/linux/admin/image35.png index ae39c08..f1000ea 100644 Binary files a/courses/linux_basics/images/linux/admin/image35.png and b/courses/linux_basics/images/linux/admin/image35.png differ diff --git a/courses/linux_basics/images/linux/admin/image36.png b/courses/linux_basics/images/linux/admin/image36.png index f38a037..85b9a9d 100644 Binary files a/courses/linux_basics/images/linux/admin/image36.png and b/courses/linux_basics/images/linux/admin/image36.png differ diff --git a/courses/linux_basics/images/linux/admin/image37.png b/courses/linux_basics/images/linux/admin/image37.png index 021d4a9..f153b21 100644 Binary files a/courses/linux_basics/images/linux/admin/image37.png and b/courses/linux_basics/images/linux/admin/image37.png differ diff --git a/courses/linux_basics/images/linux/admin/image38.png b/courses/linux_basics/images/linux/admin/image38.png index e90f505..e59c478 100644 Binary files a/courses/linux_basics/images/linux/admin/image38.png and b/courses/linux_basics/images/linux/admin/image38.png differ diff --git a/courses/linux_basics/images/linux/admin/image39.png b/courses/linux_basics/images/linux/admin/image39.png index 286308d..de54428 100644 Binary files a/courses/linux_basics/images/linux/admin/image39.png and b/courses/linux_basics/images/linux/admin/image39.png differ diff --git a/courses/linux_basics/images/linux/admin/image4.png b/courses/linux_basics/images/linux/admin/image4.png index 6c34447..6b58219 100644 Binary files a/courses/linux_basics/images/linux/admin/image4.png and b/courses/linux_basics/images/linux/admin/image4.png differ diff --git a/courses/linux_basics/images/linux/admin/image40.png b/courses/linux_basics/images/linux/admin/image40.png index 6f699ae..de24aa5 100644 Binary files a/courses/linux_basics/images/linux/admin/image40.png and b/courses/linux_basics/images/linux/admin/image40.png differ diff --git a/courses/linux_basics/images/linux/admin/image41.png b/courses/linux_basics/images/linux/admin/image41.png index e445a74..e94f7d3 100644 Binary files a/courses/linux_basics/images/linux/admin/image41.png and b/courses/linux_basics/images/linux/admin/image41.png differ diff --git a/courses/linux_basics/images/linux/admin/image42.png b/courses/linux_basics/images/linux/admin/image42.png index 2fb65b1..df8889d 100644 Binary files a/courses/linux_basics/images/linux/admin/image42.png and b/courses/linux_basics/images/linux/admin/image42.png differ diff --git a/courses/linux_basics/images/linux/admin/image43.png b/courses/linux_basics/images/linux/admin/image43.png index b2effc4..ac08e10 100644 Binary files a/courses/linux_basics/images/linux/admin/image43.png and b/courses/linux_basics/images/linux/admin/image43.png differ diff --git a/courses/linux_basics/images/linux/admin/image44.png b/courses/linux_basics/images/linux/admin/image44.png index 00661ff..aa9cd1f 100644 Binary files a/courses/linux_basics/images/linux/admin/image44.png and b/courses/linux_basics/images/linux/admin/image44.png differ diff --git a/courses/linux_basics/images/linux/admin/image45.png b/courses/linux_basics/images/linux/admin/image45.png index 8c85219..2ca25a2 100644 Binary files a/courses/linux_basics/images/linux/admin/image45.png and b/courses/linux_basics/images/linux/admin/image45.png differ diff --git a/courses/linux_basics/images/linux/admin/image46.png b/courses/linux_basics/images/linux/admin/image46.png index ff22c92..ec95a7b 100644 Binary files a/courses/linux_basics/images/linux/admin/image46.png and b/courses/linux_basics/images/linux/admin/image46.png differ diff --git a/courses/linux_basics/images/linux/admin/image47.png b/courses/linux_basics/images/linux/admin/image47.png index 2258509..b032aa7 100644 Binary files a/courses/linux_basics/images/linux/admin/image47.png and b/courses/linux_basics/images/linux/admin/image47.png differ diff --git a/courses/linux_basics/images/linux/admin/image48.png b/courses/linux_basics/images/linux/admin/image48.png index 00ec3be..b3b8e40 100644 Binary files a/courses/linux_basics/images/linux/admin/image48.png and b/courses/linux_basics/images/linux/admin/image48.png differ diff --git a/courses/linux_basics/images/linux/admin/image49.png b/courses/linux_basics/images/linux/admin/image49.png index 1197ea4..526187f 100644 Binary files a/courses/linux_basics/images/linux/admin/image49.png and b/courses/linux_basics/images/linux/admin/image49.png differ diff --git a/courses/linux_basics/images/linux/admin/image5.png b/courses/linux_basics/images/linux/admin/image5.png index 749506c..0bfcda2 100644 Binary files a/courses/linux_basics/images/linux/admin/image5.png and b/courses/linux_basics/images/linux/admin/image5.png differ diff --git a/courses/linux_basics/images/linux/admin/image50.png b/courses/linux_basics/images/linux/admin/image50.png index 25da00e..fc42ad9 100644 Binary files a/courses/linux_basics/images/linux/admin/image50.png and b/courses/linux_basics/images/linux/admin/image50.png differ diff --git a/courses/linux_basics/images/linux/admin/image51.png b/courses/linux_basics/images/linux/admin/image51.png index 90eb629..e30c4d2 100644 Binary files a/courses/linux_basics/images/linux/admin/image51.png and b/courses/linux_basics/images/linux/admin/image51.png differ diff --git a/courses/linux_basics/images/linux/admin/image52.png b/courses/linux_basics/images/linux/admin/image52.png index 62f8843..2a0d0b6 100644 Binary files a/courses/linux_basics/images/linux/admin/image52.png and b/courses/linux_basics/images/linux/admin/image52.png differ diff --git a/courses/linux_basics/images/linux/admin/image53.png b/courses/linux_basics/images/linux/admin/image53.png index 2866610..ac3d18e 100644 Binary files a/courses/linux_basics/images/linux/admin/image53.png and b/courses/linux_basics/images/linux/admin/image53.png differ diff --git a/courses/linux_basics/images/linux/admin/image54.png b/courses/linux_basics/images/linux/admin/image54.png index 599a905..c4c64b1 100644 Binary files a/courses/linux_basics/images/linux/admin/image54.png and b/courses/linux_basics/images/linux/admin/image54.png differ diff --git a/courses/linux_basics/images/linux/admin/image55.png b/courses/linux_basics/images/linux/admin/image55.png index 64a41df..ef88e35 100644 Binary files a/courses/linux_basics/images/linux/admin/image55.png and b/courses/linux_basics/images/linux/admin/image55.png differ diff --git a/courses/linux_basics/images/linux/admin/image56.png b/courses/linux_basics/images/linux/admin/image56.png index 8717179..6413b24 100644 Binary files a/courses/linux_basics/images/linux/admin/image56.png and b/courses/linux_basics/images/linux/admin/image56.png differ diff --git a/courses/linux_basics/images/linux/admin/image57.png b/courses/linux_basics/images/linux/admin/image57.png index 382f95d..c8325d7 100644 Binary files a/courses/linux_basics/images/linux/admin/image57.png and b/courses/linux_basics/images/linux/admin/image57.png differ diff --git a/courses/linux_basics/images/linux/admin/image58.png b/courses/linux_basics/images/linux/admin/image58.png index 2d8a003..d068a10 100644 Binary files a/courses/linux_basics/images/linux/admin/image58.png and b/courses/linux_basics/images/linux/admin/image58.png differ diff --git a/courses/linux_basics/images/linux/admin/image6.png b/courses/linux_basics/images/linux/admin/image6.png index f1f270d..a6b2851 100644 Binary files a/courses/linux_basics/images/linux/admin/image6.png and b/courses/linux_basics/images/linux/admin/image6.png differ diff --git a/courses/linux_basics/images/linux/admin/image7.png b/courses/linux_basics/images/linux/admin/image7.png index 4153cbe..e43a493 100644 Binary files a/courses/linux_basics/images/linux/admin/image7.png and b/courses/linux_basics/images/linux/admin/image7.png differ diff --git a/courses/linux_basics/images/linux/admin/image8.png b/courses/linux_basics/images/linux/admin/image8.png index 003f462..5aa9637 100644 Binary files a/courses/linux_basics/images/linux/admin/image8.png and b/courses/linux_basics/images/linux/admin/image8.png differ diff --git a/courses/linux_basics/images/linux/admin/image9.png b/courses/linux_basics/images/linux/admin/image9.png index 8ab7b9e..ce8b0c2 100644 Binary files a/courses/linux_basics/images/linux/admin/image9.png and b/courses/linux_basics/images/linux/admin/image9.png differ diff --git a/courses/linux_basics/images/linux/commands/image1.png b/courses/linux_basics/images/linux/commands/image1.png index c6eb3d1..a88d782 100644 Binary files a/courses/linux_basics/images/linux/commands/image1.png and b/courses/linux_basics/images/linux/commands/image1.png differ diff --git a/courses/linux_basics/images/linux/commands/image10.png b/courses/linux_basics/images/linux/commands/image10.png index 36dec6b..a62ea9c 100644 Binary files a/courses/linux_basics/images/linux/commands/image10.png and b/courses/linux_basics/images/linux/commands/image10.png differ diff --git a/courses/linux_basics/images/linux/commands/image11.png b/courses/linux_basics/images/linux/commands/image11.png index 1f80575..109ff4e 100644 Binary files a/courses/linux_basics/images/linux/commands/image11.png and b/courses/linux_basics/images/linux/commands/image11.png differ diff --git a/courses/linux_basics/images/linux/commands/image12.png b/courses/linux_basics/images/linux/commands/image12.png index 7ca5dd7..59e2981 100644 Binary files a/courses/linux_basics/images/linux/commands/image12.png and b/courses/linux_basics/images/linux/commands/image12.png differ diff --git a/courses/linux_basics/images/linux/commands/image13.png b/courses/linux_basics/images/linux/commands/image13.png index fe2ed7f..1a73b0b 100644 Binary files a/courses/linux_basics/images/linux/commands/image13.png and b/courses/linux_basics/images/linux/commands/image13.png differ diff --git a/courses/linux_basics/images/linux/commands/image14.png b/courses/linux_basics/images/linux/commands/image14.png index 474377a..120f049 100644 Binary files a/courses/linux_basics/images/linux/commands/image14.png and b/courses/linux_basics/images/linux/commands/image14.png differ diff --git a/courses/linux_basics/images/linux/commands/image15.png b/courses/linux_basics/images/linux/commands/image15.png index 31b39c8..95a1ddd 100644 Binary files a/courses/linux_basics/images/linux/commands/image15.png and b/courses/linux_basics/images/linux/commands/image15.png differ diff --git a/courses/linux_basics/images/linux/commands/image16.png b/courses/linux_basics/images/linux/commands/image16.png index a85334f..9ab4190 100644 Binary files a/courses/linux_basics/images/linux/commands/image16.png and b/courses/linux_basics/images/linux/commands/image16.png differ diff --git a/courses/linux_basics/images/linux/commands/image17.png b/courses/linux_basics/images/linux/commands/image17.png index 04ce543..987a586 100644 Binary files a/courses/linux_basics/images/linux/commands/image17.png and b/courses/linux_basics/images/linux/commands/image17.png differ diff --git a/courses/linux_basics/images/linux/commands/image18.png b/courses/linux_basics/images/linux/commands/image18.png index 592c5dd..f06e5be 100644 Binary files a/courses/linux_basics/images/linux/commands/image18.png and b/courses/linux_basics/images/linux/commands/image18.png differ diff --git a/courses/linux_basics/images/linux/commands/image19.png b/courses/linux_basics/images/linux/commands/image19.png index 63eaf66..e0761e3 100644 Binary files a/courses/linux_basics/images/linux/commands/image19.png and b/courses/linux_basics/images/linux/commands/image19.png differ diff --git a/courses/linux_basics/images/linux/commands/image2.png b/courses/linux_basics/images/linux/commands/image2.png index 49c10e8..b199ffb 100644 Binary files a/courses/linux_basics/images/linux/commands/image2.png and b/courses/linux_basics/images/linux/commands/image2.png differ diff --git a/courses/linux_basics/images/linux/commands/image20.png b/courses/linux_basics/images/linux/commands/image20.png index 1f389b6..9c04837 100644 Binary files a/courses/linux_basics/images/linux/commands/image20.png and b/courses/linux_basics/images/linux/commands/image20.png differ diff --git a/courses/linux_basics/images/linux/commands/image21.png b/courses/linux_basics/images/linux/commands/image21.png index c605d6f..56db7cd 100644 Binary files a/courses/linux_basics/images/linux/commands/image21.png and b/courses/linux_basics/images/linux/commands/image21.png differ diff --git a/courses/linux_basics/images/linux/commands/image22.png b/courses/linux_basics/images/linux/commands/image22.png index 8a53a36..e2e96c9 100644 Binary files a/courses/linux_basics/images/linux/commands/image22.png and b/courses/linux_basics/images/linux/commands/image22.png differ diff --git a/courses/linux_basics/images/linux/commands/image23.png b/courses/linux_basics/images/linux/commands/image23.png index b5e7c2d..2fbbaad 100644 Binary files a/courses/linux_basics/images/linux/commands/image23.png and b/courses/linux_basics/images/linux/commands/image23.png differ diff --git a/courses/linux_basics/images/linux/commands/image24.png b/courses/linux_basics/images/linux/commands/image24.png index f80d6c7..1b15077 100644 Binary files a/courses/linux_basics/images/linux/commands/image24.png and b/courses/linux_basics/images/linux/commands/image24.png differ diff --git a/courses/linux_basics/images/linux/commands/image25.png b/courses/linux_basics/images/linux/commands/image25.png index 8dfaad9..39ce92c 100644 Binary files a/courses/linux_basics/images/linux/commands/image25.png and b/courses/linux_basics/images/linux/commands/image25.png differ diff --git a/courses/linux_basics/images/linux/commands/image26.png b/courses/linux_basics/images/linux/commands/image26.png index c171945..559c13b 100644 Binary files a/courses/linux_basics/images/linux/commands/image26.png and b/courses/linux_basics/images/linux/commands/image26.png differ diff --git a/courses/linux_basics/images/linux/commands/image27.png b/courses/linux_basics/images/linux/commands/image27.png index f0dcc70..9004989 100644 Binary files a/courses/linux_basics/images/linux/commands/image27.png and b/courses/linux_basics/images/linux/commands/image27.png differ diff --git a/courses/linux_basics/images/linux/commands/image28.png b/courses/linux_basics/images/linux/commands/image28.png index 9467313..e11d3fd 100644 Binary files a/courses/linux_basics/images/linux/commands/image28.png and b/courses/linux_basics/images/linux/commands/image28.png differ diff --git a/courses/linux_basics/images/linux/commands/image29.png b/courses/linux_basics/images/linux/commands/image29.png index 4d9dd96..a628cad 100644 Binary files a/courses/linux_basics/images/linux/commands/image29.png and b/courses/linux_basics/images/linux/commands/image29.png differ diff --git a/courses/linux_basics/images/linux/commands/image3.png b/courses/linux_basics/images/linux/commands/image3.png index 0693a5a..7a36c72 100644 Binary files a/courses/linux_basics/images/linux/commands/image3.png and b/courses/linux_basics/images/linux/commands/image3.png differ diff --git a/courses/linux_basics/images/linux/commands/image30.png b/courses/linux_basics/images/linux/commands/image30.png index 7d9088c..b3ebf55 100644 Binary files a/courses/linux_basics/images/linux/commands/image30.png and b/courses/linux_basics/images/linux/commands/image30.png differ diff --git a/courses/linux_basics/images/linux/commands/image31.png b/courses/linux_basics/images/linux/commands/image31.png index 1912203..daf7f54 100644 Binary files a/courses/linux_basics/images/linux/commands/image31.png and b/courses/linux_basics/images/linux/commands/image31.png differ diff --git a/courses/linux_basics/images/linux/commands/image32.png b/courses/linux_basics/images/linux/commands/image32.png index fb2bf71..b6d6c30 100644 Binary files a/courses/linux_basics/images/linux/commands/image32.png and b/courses/linux_basics/images/linux/commands/image32.png differ diff --git a/courses/linux_basics/images/linux/commands/image4.png b/courses/linux_basics/images/linux/commands/image4.png index c589d5f..1cd900c 100644 Binary files a/courses/linux_basics/images/linux/commands/image4.png and b/courses/linux_basics/images/linux/commands/image4.png differ diff --git a/courses/linux_basics/images/linux/commands/image5.png b/courses/linux_basics/images/linux/commands/image5.png index d993a83..1060bce 100644 Binary files a/courses/linux_basics/images/linux/commands/image5.png and b/courses/linux_basics/images/linux/commands/image5.png differ diff --git a/courses/linux_basics/images/linux/commands/image6.png b/courses/linux_basics/images/linux/commands/image6.png index 0996234..8f19b15 100644 Binary files a/courses/linux_basics/images/linux/commands/image6.png and b/courses/linux_basics/images/linux/commands/image6.png differ diff --git a/courses/linux_basics/images/linux/commands/image7.png b/courses/linux_basics/images/linux/commands/image7.png index 40625ed..bcbac6a 100644 Binary files a/courses/linux_basics/images/linux/commands/image7.png and b/courses/linux_basics/images/linux/commands/image7.png differ diff --git a/courses/linux_basics/images/linux/commands/image8.png b/courses/linux_basics/images/linux/commands/image8.png index a96fad7..c8b8813 100644 Binary files a/courses/linux_basics/images/linux/commands/image8.png and b/courses/linux_basics/images/linux/commands/image8.png differ diff --git a/courses/linux_basics/images/linux/commands/image9.png b/courses/linux_basics/images/linux/commands/image9.png index 1de865d..501229a 100644 Binary files a/courses/linux_basics/images/linux/commands/image9.png and b/courses/linux_basics/images/linux/commands/image9.png differ diff --git a/courses/linux_basics/intro.md b/courses/linux_basics/intro.md index 94e01d7..d4fb1bf 100644 --- a/courses/linux_basics/intro.md +++ b/courses/linux_basics/intro.md @@ -27,35 +27,40 @@ In the second and third part, we will be taking examples to understand the conce We are not covering advanced linux commands and bash scripting in this course. We will also not be covering linux internals. -## Course Content - -### Table of Contents +## Course Contents The following topics has been covered in this course: -- Introduction to Linux +- [Introduction to Linux](https://linkedin.github.io/school-of-sre/linux_basics/intro/) - [What are Linux Operating Systems](https://linkedin.github.io/school-of-sre/linux_basics/intro/#what-are-linux-operating-systems) - - [Linux Distributions](https://linkedin.github.io/school-of-sre/linux_basics/intro/#what-are-popular-linux-distributions) + - [What are popular Linux distributions](https://linkedin.github.io/school-of-sre/linux_basics/intro/#what-are-popular-linux-distributions) - [Uses of Linux Operating Systems](https://linkedin.github.io/school-of-sre/linux_basics/intro/#uses-of-linux-operating-systems) - [Linux Architecture](https://linkedin.github.io/school-of-sre/linux_basics/intro/#linux-architecture) - - [GUI vs CLI](https://linkedin.github.io/school-of-sre/linux_basics/intro/#graphical-user-interface-gui-vs-command-line-interface-cli) + - [Graphical user interface (GUI) vs Command line interface (CLI)](https://linkedin.github.io/school-of-sre/linux_basics/intro/#graphical-user-interface-gui-vs-command-line-interface-cli) - [Command Line Basics](https://linkedin.github.io/school-of-sre/linux_basics/command_line_basics/) + - [Lab Environment Setup](https://linkedin.github.io/school-of-sre/linux_basics/command_line_basics/lab-environment-setup) + - [What is a Command](https://linkedin.github.io/school-of-sre/linux_basics/command_line_basics/#what-is-a-command) + - [File System Organization](https://linkedin.github.io/school-of-sre/linux_basics/command_line_basics/#file-system-organization) - [Navigating File System](https://linkedin.github.io/school-of-sre/linux_basics/command_line_basics/#commands-for-navigating-the-file-system) - [Manipulating Files](https://linkedin.github.io/school-of-sre/linux_basics/command_line_basics/#commands-for-manipulating-files) - [Viewing Files](https://linkedin.github.io/school-of-sre/linux_basics/command_line_basics/#commands-for-viewing-files) + - [Echo Command](https://linkedin.github.io/school-of-sre/linux_basics/command_line_basics/#echo-command) - [Text Processing Commands](https://linkedin.github.io/school-of-sre/linux_basics/command_line_basics/#text-processing-commands) - [I/O Redirection](https://linkedin.github.io/school-of-sre/linux_basics/command_line_basics/#io-redirection) - [Linux system administration](https://linkedin.github.io/school-of-sre/linux_basics/linux_server_administration/) - - [User/Groups management](https://linkedin.github.io/school-of-sre/linux_basics/linux_server_administration/#usergroup-management-in-linux) - - [Superuser in Linux](https://linkedin.github.io/school-of-sre/linux_basics/linux_server_administration/#becoming-a-superuser-in-linux) - - [File Permissions](https://linkedin.github.io/school-of-sre/linux_basics/linux_server_administration/#file-permissions-in-linux) + - [Lab Environment Setup](https://linkedin.github.io/school-of-sre/linux_basics/linux_server_administration/lab-environment-setup) + - [User/Groups management](https://linkedin.github.io/school-of-sre/linux_basics/linux_server_administration/#usergroup-management) + - [Becoming a Superuser](https://linkedin.github.io/school-of-sre/linux_basics/linux_server_administration/#becoming-a-superuser) + - [File Permissions](https://linkedin.github.io/school-of-sre/linux_basics/linux_server_administration/#file-permissions) - [SSH Command](https://linkedin.github.io/school-of-sre/linux_basics/linux_server_administration/#ssh-command) - [Package Management](https://linkedin.github.io/school-of-sre/linux_basics/linux_server_administration/#package-management) - [Process Management](https://linkedin.github.io/school-of-sre/linux_basics/linux_server_administration/#process-management) - [Memory Management](https://linkedin.github.io/school-of-sre/linux_basics/linux_server_administration/#memory-management) - [Daemons and Systemd](https://linkedin.github.io/school-of-sre/linux_basics/linux_server_administration/#daemons) - [Logs](https://linkedin.github.io/school-of-sre/linux_basics/linux_server_administration/#logs) - +- [Conclusion](https://linkedin.github.io/school-of-sre/linux_basics/conclusion) + - [Applications in SRE Role](https://linkedin.github.io/school-of-sre/linux_basics/conclusion/#applications-in-sre-role) + - [Useful Courses and tutorials](https://linkedin.github.io/school-of-sre/linux_basics/conclusion/#useful-courses-and-tutorials) ## What are Linux operating systems diff --git a/courses/linux_basics/linux_server_administration.md b/courses/linux_basics/linux_server_administration.md index 9da1e44..f02e8be 100644 --- a/courses/linux_basics/linux_server_administration.md +++ b/courses/linux_basics/linux_server_administration.md @@ -25,7 +25,7 @@ As a server administrator, we are mostly concerned with the linux servers which Since linux supports multiple users, we need to have a method which can protect the users from each other. One user should not be able to access and modify files of other users -## User/Group Management in Linux +## User/Group Management - Each user in linux has an associated user ID called UID attached to him @@ -35,7 +35,7 @@ Since linux supports multiple users, we need to have a method which can protect - Each group has a group ID called GID associated with it. -### id command in linux +### id command id command can be used to find the uid and gid associated with an user. It also lists down the groups to which the user belongs to. @@ -166,7 +166,7 @@ We will now try to add user "shivam" to the group we have created above. ![](images/linux/admin/image33.png) -## Becoming a Superuser in Linux +## Becoming a Superuser **Before running the below commands, do make sure that you have set up a password for user "shivam" and user "root" using the passwd command @@ -243,7 +243,7 @@ accessed with the sudo privileges. We have already given sudo privileges to user “shivam” by adding him to the group “wheel”. -## File Permissions in Linux +## File Permissions On a linux operating system, each file and directory is assigned access permissions for the owner of the file, the members of a group of related @@ -263,7 +263,7 @@ related to file permissions. ![](images/linux/admin/image57.png) -### Chmod command in linux +### Chmod command The chmod command is used to modify files and directories permissions in linux. @@ -296,7 +296,7 @@ command. Chmod command can be also used to change the permissions of a directory in the similar way. -### Chown command in linux +### Chown command The chown command is used to change the owner of files or directories in linux. @@ -315,7 +315,7 @@ command. Chown command can also be used to change the owner of a directory in the similar way. -### Chgrp command in linux +### Chgrp command The chgrp command can be used to change the group ownership of files or directories in linux. The syntax is very similar to that of chown @@ -518,7 +518,7 @@ additional information about io and cpu usage. ![](images/linux/admin/image38.png) -## Checking Disk Space in Linux +## Checking Disk Space In this section, we will study about some useful commands that can be used to view disk space on linux. @@ -585,38 +585,3 @@ in linux. These logs can be very useful when you are troubleshooting on the system. ![](images/linux/admin/image58.png) - -## Applications in SRE Role - -- Different users will have different permissions depending on their - roles. We will also not want everyone in the company to access our - servers for security reasons. Users permissions can be restricted - with chown, chmod and chgrp commands. - -- SSH is one of the most frequently used commands for a SRE. Logging - into servers and troubleshooting along with performing basic - administration tasks will only be possible if we are able to login - into the server. - -- What if we want to run an apache server or nginx on a server ? We - will first install it using the package manager. Package - management commands become important here. - -- Managing services on servers is another critical responsibility of a - SRE. Systemd related commands can help in troubleshooting issues. - If a service goes down, we can start it using systemctl start - command. We can also stop a service in case it is not needed. - -- Monitoring is another core responsibility of a SRE. Memory and CPU - are two important system level metrics which should be monitored. - Commands like top and free are quite helpful here. - -- If a service is throwing an error, how do we find out the root cause - of the error ? We will certainly need to check logs to find out - the whole stack trace of the error. The log file will also tell us - the number of times the error has occurred along with time when it - started. - -## Useful courses and tutorials - -- Edx Red Hat Enterprise Linux Course - [https://courses.edx.org/courses/course-v1:RedHat+RH066x+2T2017/course/](https://courses.edx.org/courses/course-v1:RedHat+RH066x+2T2017/course/) diff --git a/courses/linux_networking/intro.md b/courses/linux_networking/intro.md index 34f7484..bf57073 100644 --- a/courses/linux_networking/intro.md +++ b/courses/linux_networking/intro.md @@ -12,13 +12,11 @@ Throughout the course, we cover how an SRE can optimize the system to improve th This course spends time on the fundamentals. We are not covering concepts like HTTP/2.0, QUIC, TCP congestion control protocols, Anycast, BGP, CDN, Tunnels and Multicast. We expect that this course will provide the relevant basics to understand such concepts -## Course Content - -### Birds eye view of the course +## Birds eye view of the course The course covers the question “What happens when you open linkedin.com in your browser?” The course follows the flow of TCP/IP stack.More specifically, the course covers topics of Application layer protocols DNS and HTTP, transport layer protocols UDP and TCP, networking layer protocol IP and Data Link Layer protocol -## Table of Contents +## Course Contents 1. [DNS](https://linkedin.github.io/school-of-sre/linux_networking/dns/) 2. [UDP](https://linkedin.github.io/school-of-sre/linux_networking/udp/) 3. [HTTP](https://linkedin.github.io/school-of-sre/linux_networking/http/) diff --git a/courses/python_web/intro.md b/courses/python_web/intro.md index 52df081..089010c 100644 --- a/courses/python_web/intro.md +++ b/courses/python_web/intro.md @@ -17,13 +17,11 @@ And to introduce SRE flavour to the course, we will design, develop and deploy ( Extensive knowledge of python internals and advanced python. -## Course Content - -### Lab Environment Setup +## Lab Environment Setup Have latest version of python installed -### Table of Contents +## Course Contents 1. [The Python Language](https://linkedin.github.io/school-of-sre/python_web/intro/#the-python-language) 1. Some Python Concepts diff --git a/courses/python_web/sre-conclusion.md b/courses/python_web/sre-conclusion.md index bd76f27..721d609 100644 --- a/courses/python_web/sre-conclusion.md +++ b/courses/python_web/sre-conclusion.md @@ -25,7 +25,7 @@ Now let's think in terms of the given url shortening app. We need to monitor it. 3. We also want to keep an eye on the database so depending on the database solution chosen. Query times, volumes, disk usage etc. 4. Finally, there also needs to be some external monitoring which runs periodic tests from devices outside of your data centers. This emulates customers and ensures that from customer point of view, the system is working as expected. -## SRE Use-cases +## Applications in SRE role In the world of SRE, python is a widely used language. For small scripts and tooling developed for various purposes. Since tooling developed by SRE works with critical pieces of infrastructure and has great power (to bring things down), it is important to know what you are doing while using a programming language and its features. Also it is equally important to know the language and its characteristics while debugging the issues. As an SRE having a deeper understanding of python language, it has helped me a lot to debug very sneaky bugs and be generally more aware and informed while making certain design decisions. diff --git a/courses/security/conclusion.md b/courses/security/conclusion.md new file mode 100644 index 0000000..edfe1ce --- /dev/null +++ b/courses/security/conclusion.md @@ -0,0 +1,25 @@ +# Conclusion + +Now that you have completed this course on Security you are now aware of the possible security threats to computer systems & networks. Not only that, but you are now better able to protect your systems as well as recommend security measures to others. + +This course provides fundamental everyday knowledge on security domain which will also help you keep security at the top of your priority. + +## Other Resources + +Some books that would be a great resource + +- Holistic Info-Sec for Web Developers - Free and downloadable book series with very broad and deep coverage of what Web Developers and DevOps Engineers need to know in order to create robust, reliable, maintainable and secure software, networks and other, that are delivered continuously, on time, with no nasty surprises +- Docker Security - Quick Reference: For DevOps Engineers - A book on understanding the Docker security defaults, how to improve them (theory and practical), along with many tools and techniques. +- How to Hack Like a Legend - A hacker’s tale breaking into a secretive offshore company, Sparc Flow, 2018 +- How to Investigate Like a Rockstar - Live a real crisis to master the secrets of forensic analysis, Sparc Flow, 2017 +- Real World Cryptography - This early-access book teaches you applied cryptographic techniques to understand and apply security at every level of your systems and applications. +- AWS Security - This early-access book covers commong AWS security issues and best practices for access policies, data protection, auditing, continuous monitoring, and incident response. + +## Post Training asks/ Further Reading + +- CTF Events like : +- Penetration Testing : +- Threat Intelligence : +- Threat Detection & Hunting : +- Web Security: +- Building Secure and Reliable Systems : diff --git a/courses/security/intro.md b/courses/security/intro.md index e05622e..4c31ab9 100644 --- a/courses/security/intro.md +++ b/courses/security/intro.md @@ -2,9 +2,9 @@ ## Prerequisites -1. Basics of Linux fundamentals & command line usage +1. [Linux Basics](https://linkedin.github.io/school-of-sre/linux_basics/intro/) -2. Networking Module +2. [Linux Networking](https://linkedin.github.io/school-of-sre/linux_networking/intro/) ## What to expect from this course @@ -17,21 +17,10 @@ The course covers fundamentals of information security along with touching on su The courseware is not an ethical hacking workshop or a very deep dive into the fundamentals of the problems. The course does not deal with hacking or breaking into systems but rather an approach on how to ensure you don’t get into those situations and also to make you aware of different ways a system can be compromised. -## Course Content - -### Table of Contents +## Course Contents 1. [Fundamentals](https://linkedin.github.io/school-of-sre/security/fundamentals/) 2. [Network Security](https://linkedin.github.io/school-of-sre/security/network_security/) 3. [Threats, Attacks & Defence](https://linkedin.github.io/school-of-sre/security/threats_attacks_defences/) 4. [Writing Secure Code & More](https://linkedin.github.io/school-of-sre/security/writing_secure_code/) - - -## Post Training asks/ Further Reading - -- CTF Events like : -- Penetration Testing : -- Threat Intelligence : -- Threat Detection & Hunting : -- Web Security: -- Building Secure and Reliable Systems : +5. [Conclusion](https://linkedin.github.io/school-of-sre/security/conclusion/) diff --git a/courses/systems_design/availability.md b/courses/systems_design/availability.md index a4f9c65..20d9a0e 100644 --- a/courses/systems_design/availability.md +++ b/courses/systems_design/availability.md @@ -76,7 +76,7 @@ The combined availability of the system is **KEY TAKEAWAYS:** Strive for active/active rather than active/passive solutions, they have a lesser risk of cross over being unreliable. Use LB and right load balancing methods to ensure reliable failover. Model and build your data systems to ensure data is correctly handled when crossover happens. Generally DB systems follow active/passive semantics for writes. Masters accept writes and when master goes down, follower is promoted to master(active from being passive) to accept writes. We have to be careful here that the cutover never introduces more than one masters. This problem is called a split brain. -## SRE Use cases +## Applications in SRE role 1. SRE works on deciding an acceptable SLA and make sure system is available to achieve the SLA 2. SRE is involved in architecture design right from building the data center to make sure site is not affected by network switch, hardware, power or software failures 3. SRE also run mock drills of failures to see how the system behaves in uncharted territory and comes up with a plan to improve availability if there are misses. diff --git a/courses/systems_design/fault-tolerance.md b/courses/systems_design/fault-tolerance.md index bc97d45..39e4ea7 100644 --- a/courses/systems_design/fault-tolerance.md +++ b/courses/systems_design/fault-tolerance.md @@ -60,7 +60,7 @@ Let's take another example, we come up with a new model for our Content sharing - https://learning.oreilly.com/library/view/the-art-of/9780134031408/ch21.html#ch21 -### SRE Use cases: +### Applications in SRE role 1. Work with the DC tech or cloud team to distribute infrastructure such that its immune to switch or power failures by creating fault zones within a Data Center https://docs.microsoft.com/en-us/azure/virtual-machines/manage-availability#use-availability-zones-to-protect-from-datacenter-level-failures 2. Work with the partners and design interaction between services such that one service breakdown is not amplified in a cascading fashion to all upstreams diff --git a/courses/systems_design/intro.md b/courses/systems_design/intro.md index be222b2..a346b29 100644 --- a/courses/systems_design/intro.md +++ b/courses/systems_design/intro.md @@ -3,9 +3,11 @@ ## Prerequisites Fundamentals of common software system components: -- Operating Systems -- Networking -- Databases RDBMS/NoSQL + +- [Linux Basics](https://linkedin.github.io/school-of-sre/linux_basics/intro/) +- [Linux Networking](https://linkedin.github.io/school-of-sre/linux_networking/intro/) +- Databases RDBMS +- [NoSQL Concepts](https://linkedin.github.io/school-of-sre/databases_nosql/intro/) ## What to expect from this course @@ -17,9 +19,7 @@ Individual software components’ scalability and reliability concerns like e.g. More light will be shed on concepts rather than on setting up and configuring components like Loadbalancers to achieve scalability, availability and reliability of systems -## Course Content - -### Table of Contents +## Course Contents - [Introduction](https://linkedin.github.io/school-of-sre/systems_design/intro/#backstory) - [Scalability](https://linkedin.github.io/school-of-sre/systems_design/scalability/) diff --git a/courses/systems_design/scalability.md b/courses/systems_design/scalability.md index 87fa6a7..265fc4a 100644 --- a/courses/systems_design/scalability.md +++ b/courses/systems_design/scalability.md @@ -171,7 +171,7 @@ Here the whole data centre is split and replicated and clients are directed to a -## SRE Use cases +## Applications in SRE role 1. SREs in coordination with the network team work on how to map users traffic to a particular site. https://engineering.linkedin.com/blog/2017/05/trafficshift--load-testing-at-scale 2. SREs work closely with the Dev team to split monoliths to multiple microservices that are easy to run and manage diff --git a/mkdocs.yml b/mkdocs.yml index 39f0927..d25cf93 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -4,7 +4,7 @@ theme: name: material logo: img/sos.png favicon: img/favicon.ico -plugins: [] + custom_dir: overrides nav: - Home: index.md - Fundamentals Series: @@ -12,10 +12,12 @@ nav: - Introduction: linux_basics/intro.md - Command Line Basics: linux_basics/command_line_basics.md - Server Administration: linux_basics/linux_server_administration.md + - Conclusion: linux_basics/conclusion.md - Git: - Git Basics: git/git-basics.md - Working With Branches: git/branches.md - Github and Hooks: git/github-hooks.md + - Conclusion: git/conclusion.md - Linux Networking: - Introduction: linux_networking/intro.md - DNS: linux_networking/dns.md @@ -31,13 +33,14 @@ nav: - The URL Shortening App: python_web/url-shorten-app.md - Conclusion: python_web/sre-conclusion.md - Data: + - NoSQL Concepts: + - Introduction: databases_nosql/intro.md + - Key Concepts: databases_nosql/key_concepts.md + - Conclusion: databases_nosql/further_reading.md - Big Data: - Introduction: big_data/intro.md - - Overview of Big Data: big_data/overview.md - - Usage of Big Data techniques: big_data/usage.md - - Evolution of Hadoop: big_data/evolution.md - - Architecture of Hadoop: big_data/architecture.md - - Tasks and conclusion: big_data/tasks.md + - Evolution and Architecure of Hadoop: big_data/evolution.md + - Conclusion: big_data/tasks.md - Systems Design: - Introduction: systems_design/intro.md - Scalability: systems_design/scalability.md @@ -50,4 +53,5 @@ nav: - Network Security: security/network_security.md - Threat, Attacks & Defences: security/threats_attacks_defences.md - Writing Secure code: security/writing_secure_code.md + - Conclusion: security/conclusion.md - Contribute: CONTRIBUTING.md diff --git a/overrides/partials/header.html b/overrides/partials/header.html new file mode 100644 index 0000000..0f88369 --- /dev/null +++ b/overrides/partials/header.html @@ -0,0 +1,47 @@ +{% set site_url = config.site_url | default(nav.homepage.url, true) | url %} +{% if not config.use_directory_urls and site_url[0] == site_url[-1] == "." %} + {% set site_url = site_url ~ "/index.html" %} +{% endif %} +
+ +
diff --git a/overrides/partials/nav.html b/overrides/partials/nav.html new file mode 100644 index 0000000..564981b --- /dev/null +++ b/overrides/partials/nav.html @@ -0,0 +1,18 @@ +{% set site_url = config.site_url | default(nav.homepage.url, true) | url %} +{% if not config.use_directory_urls and site_url[0] == site_url[-1] == "." %} + {% set site_url = site_url ~ "/index.html" %} +{% endif %} +