Initially we can start with deploying this app on one virtual machine on any cloud provider. But this is a Single point of failure which is something we never allow as an SRE (or even as an engineer). So an improvement here can be having multiple instances of applications deployed behind a load balancer. This certainly prevents problems of one machine going down.
Scaling here would mean adding more instances behind the load balancer. But this is scalable upto only a certain point. After that, other bottlenecks in the system will start appearing. ie: DB will become the bottleneck, or perhaps the load balancer itself. How do you know what is the bottleneck? You need to have observability into each aspects of the application architecture.
Only after you have metrics, you will be able to know what is going wrong where. What gets measured, gets fixed!
-Get deeper insights into scaling from School Of SRE's Scalability module and post going through it, apply your learnings and takeaways to this app. Think how will we make this app geographically distributed and highly available and scalable.
+Get deeper insights into scaling from School Of SRE's Scalability module and post going through it, apply your learnings and takeaways to this app. Think how will we make this app geographically distributed and highly available and scalable.
Once we have our application deployed. It will be working ok. But not forever. Reliability is in the title of our job and we make systems reliable by making the design in a certain way. But things still will go down. Machines will fail. Disks will behave weirdly. Buggy code will get pushed to production. And all these possible scenarios will make the system less reliable. So what do we do? We monitor!
We keep an eye on the system's health and if anything is not going as expected, we want ourselves to get alerted.
diff --git a/python_web/url-shorten-app/index.html b/python_web/url-shorten-app/index.html index dddcc4d..e1068fc 100644 --- a/python_web/url-shorten-app/index.html +++ b/python_web/url-shorten-app/index.html @@ -1156,7 +1156,7 @@Finally, storage. Where will we store the data that we will generate over time? There are multiple database solutions available and we will need to choose the one that suits this app the most. Relational database like MySQL would be a fair choice but be sure to checkout School of SRE's database section for deeper insights into making a more informed decision.
+Finally, storage. Where will we store the data that we will generate over time? There are multiple database solutions available and we will need to choose the one that suits this app the most. Relational database like MySQL would be a fair choice but be sure to checkout School of SRE's SQL database section and NoSQL databases section for deeper insights into making a more informed decision.
We are not accounting for users into our app and other possible features like rate limiting, customized links etc but it will eventually come up with time. Depending on the requirements, they too might need to get incorporated.
The minimal working code is given below for reference but I'd encourage you to come up with your own.
diff --git a/search/search_index.json b/search/search_index.json index fecd7a8..9623a94 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"School of SRE In early 2019, we started visiting campuses across India to recruit the best and brightest minds to ensure LinkedIn, and all the services that make up its complex technology stack, is always available for everyone. This critical function at Linkedin falls under the purview of the Site Engineering team and Site Reliability Engineers (SREs) who are Software Engineers specializing in reliability. SREs apply the principles of computer science and engineering to the design, development and operation of computer systems: generally, large scale, distributed ones As we continued on this journey we started getting a lot of questions from these campuses on what exactly the site reliability engineering role entails? And, how could someone learn the skills and the disciplines involved to become a successful site reliability engineer? Fast forward a few months, and a few of these campus students had joined LinkedIn either as interns or as full-time engineers to become a part of the Site Engineering team; we also had a few lateral hires who joined our organization who were not from a traditional SRE background. That's when a few of us got together and started to think about how we can onboard new graduate engineers to the Site Engineering team. There is a vast amount of resources scattered throughout the web on what the roles and responsibilities of SREs are, how to monitor site health, production incidents, define SLO/SLI etc. But there are very few resources out there guiding someone on the basic skill sets one has to acquire as a beginner. Because of the lack of these resources, we felt that individuals have a tough time getting into open positions in the industry. We created the School Of SRE as a starting point for anyone wanting to build their career as an SRE. In this course, we are focusing on building strong foundational skills. The course is structured in a way to provide more real life examples and how learning each of these topics can play an important role in day to day SRE life. Currently we are covering the following topics under the School Of SRE: Fundamentals Series Linux Basics Git Linux Networking Python and Web Data Relational databases(MySQL) NoSQL concepts Big Data Systems Design Security We believe continuous learning will help in acquiring deeper knowledge and competencies in order to expand your skill sets, every module has added references which could be a guide for further learning. Our hope is that by going through these modules we should be able to build the essential skills required for a Site Reliability Engineer. At Linkedin, we are using this curriculum for onboarding our non-traditional hires and new college grads into the SRE role. We had multiple rounds of successful onboarding experience with new employees and the course helped them be productive in a very short period of time. This motivated us to open source the content for helping other organizations in onboarding new engineers into the role and provide guidance for aspiring individuals to get into the role. We realize that the initial content we created is just a starting point and we hope that the community can help in the journey of refining and expanding the content.","title":"Home"},{"location":"#school-of-sre","text":"In early 2019, we started visiting campuses across India to recruit the best and brightest minds to ensure LinkedIn, and all the services that make up its complex technology stack, is always available for everyone. This critical function at Linkedin falls under the purview of the Site Engineering team and Site Reliability Engineers (SREs) who are Software Engineers specializing in reliability. SREs apply the principles of computer science and engineering to the design, development and operation of computer systems: generally, large scale, distributed ones As we continued on this journey we started getting a lot of questions from these campuses on what exactly the site reliability engineering role entails? And, how could someone learn the skills and the disciplines involved to become a successful site reliability engineer? Fast forward a few months, and a few of these campus students had joined LinkedIn either as interns or as full-time engineers to become a part of the Site Engineering team; we also had a few lateral hires who joined our organization who were not from a traditional SRE background. That's when a few of us got together and started to think about how we can onboard new graduate engineers to the Site Engineering team. There is a vast amount of resources scattered throughout the web on what the roles and responsibilities of SREs are, how to monitor site health, production incidents, define SLO/SLI etc. But there are very few resources out there guiding someone on the basic skill sets one has to acquire as a beginner. Because of the lack of these resources, we felt that individuals have a tough time getting into open positions in the industry. We created the School Of SRE as a starting point for anyone wanting to build their career as an SRE. In this course, we are focusing on building strong foundational skills. The course is structured in a way to provide more real life examples and how learning each of these topics can play an important role in day to day SRE life. Currently we are covering the following topics under the School Of SRE: Fundamentals Series Linux Basics Git Linux Networking Python and Web Data Relational databases(MySQL) NoSQL concepts Big Data Systems Design Security We believe continuous learning will help in acquiring deeper knowledge and competencies in order to expand your skill sets, every module has added references which could be a guide for further learning. Our hope is that by going through these modules we should be able to build the essential skills required for a Site Reliability Engineer. At Linkedin, we are using this curriculum for onboarding our non-traditional hires and new college grads into the SRE role. We had multiple rounds of successful onboarding experience with new employees and the course helped them be productive in a very short period of time. This motivated us to open source the content for helping other organizations in onboarding new engineers into the role and provide guidance for aspiring individuals to get into the role. We realize that the initial content we created is just a starting point and we hope that the community can help in the journey of refining and expanding the content.","title":"School of SRE"},{"location":"CONTRIBUTING/","text":"We realise that the initial content we created is just a starting point and our hope is that the community can help in the journey refining and extending the contents. As a contributor, you represent that the content you submit is not plagiarised. By submitting the content, you (and, if applicable, your employer) are licensing the submitted content to LinkedIn and the open source community subject to the Creative Commons Attribution 4.0 International Public License. We suggest to open an issue first and seek advice for your changes before submitting a pull request. Repository URL : https://github.com/linkedin/school-of-sre Building and testing locally Run the following commands to build and view the site locally before opening a PR. pip install -r requirements.txt mkdocs build mkdocs serve","title":"Contribute"},{"location":"CONTRIBUTING/#building-and-testing-locally","text":"Run the following commands to build and view the site locally before opening a PR. pip install -r requirements.txt mkdocs build mkdocs serve","title":"Building and testing locally"},{"location":"big_data/evolution/","text":"Evolution of Hadoop Architecture of Hadoop HDFS The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large data sets. HDFS is part of the Apache Hadoop Core project . NameNode: is the arbitrator and central repository of file namespace in the cluster. The NameNode executes the operations such as opening, closing, and renaming files and directories. DataNode: manages the storage attached to the node on which it runs. It is responsible for serving all the read and writes requests. It performs operations on instructions on NameNode such as creation, deletion, and replications of blocks. Client: Responsible for getting the required metadata from the namenode and then communicating with the datanodes for reads and writes. YARN YARN stands for \u201cYet Another Resource Negotiator\u201c. It was introduced in Hadoop 2.0 to remove the bottleneck on Job Tracker which was present in Hadoop 1.0. YARN was described as a \u201cRedesigned Resource Manager\u201d at the time of its launching, but it has now evolved to be known as a large-scale distributed operating system used for Big Data processing. The main components of YARN architecture include: Client: It submits map-reduce(MR) jobs to the resource manager. Resource Manager: It is the master daemon of YARN and is responsible for resource assignment and management among all the applications. Whenever it receives a processing request, it forwards it to the corresponding node manager and allocates resources for the completion of the request accordingly. It has two major components: Scheduler: It performs scheduling based on the allocated application and available resources. It is a pure scheduler, which means that it does not perform other tasks such as monitoring or tracking and does not guarantee a restart if a task fails. The YARN scheduler supports plugins such as Capacity Scheduler and Fair Scheduler to partition the cluster resources. Application manager: It is responsible for accepting the application and negotiating the first container from the resource manager. It also restarts the Application Manager container if a task fails. Node Manager: It takes care of individual nodes on the Hadoop cluster and manages application and workflow and that particular node. Its primary job is to keep up with the Node Manager. It monitors resource usage, performs log management, and also kills a container based on directions from the resource manager. It is also responsible for creating the container process and starting it at the request of the Application master. Application Master: An application is a single job submitted to a framework. The application manager is responsible for negotiating resources with the resource manager, tracking the status, and monitoring the progress of a single application. The application master requests the container from the node manager by sending a Container Launch Context(CLC) which includes everything an application needs to run. Once the application is started, it sends the health report to the resource manager from time-to-time. Container: It is a collection of physical resources such as RAM, CPU cores, and disk on a single node. The containers are invoked by Container Launch Context(CLC) which is a record that contains information such as environment variables, security tokens, dependencies, etc. MapReduce framework The term MapReduce represents two separate and distinct tasks Hadoop programs perform-Map Job and Reduce Job. Map jobs take data sets as input and process them to produce key-value pairs. Reduce job takes the output of the Map job i.e. the key-value pairs and aggregates them to produce desired results. Hadoop MapReduce (Hadoop Map/Reduce) is a software framework for distributed processing of large data sets on computing clusters. Mapreduce helps to split the input data set into a number of parts and run a program on all data parts parallel at once. Please find the below Word count example demonstrating the usage of the MapReduce framework: Other tooling around Hadoop Hive Uses a language called HQL which is very SQL like. Gives non-programmers the ability to query and analyze data in Hadoop. Is basically an abstraction layer on top of map-reduce. Ex. HQL query: SELECT pet.name, comment FROM pet JOIN event ON (pet.name = event.name); In mysql: SELECT pet.name, comment FROM pet, event WHERE pet.name = event.name; Pig Uses a scripting language called Pig Latin, which is more workflow driven. Don't need to be an expert Java programmer but need a few coding skills. Is also an abstraction layer on top of map-reduce. Here is a quick question for you: What is the output of running the pig queries in the right column against the data present in the left column in the below image? Output: mysql 7,Komal,Nayak,24,9848022334,trivendram 8,Bharathi,Nambiayar,24,9848022333,Chennai 5,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar 6,Archana,Mishra,23,9848022335,Chennai 3. Spark 1. Spark provides primitives for in-memory cluster computing that allows user programs to load data into a cluster\u2019s memory and query it repeatedly, making it well suited to machine learning algorithms. 4. Presto 1. Presto is a high performance, distributed SQL query engine for Big Data. 2. Its architecture allows users to query a variety of data sources such as Hadoop, AWS S3, Alluxio, MySQL, Cassandra, Kafka, and MongoDB. 3. Example presto query: mysql use studentDB; show tables; SELECT roll_no, name FROM studentDB.studentDetails where section=\u2019A\u2019 limit 5; Data Serialisation and storage In order to transport the data over the network or to store on some persistent storage, we use the process of translating data structures or objects state into binary or textual form. We call this process serialization.. Avro data is stored in a container file (a .avro file) and its schema (the .avsc file) is stored with the data file. Apache Hive provides support to store a table as Avro and can also query data in this serialisation format.","title":"Evolution and Architecture of Hadoop"},{"location":"big_data/evolution/#evolution-of-hadoop","text":"","title":"Evolution of Hadoop"},{"location":"big_data/evolution/#architecture-of-hadoop","text":"HDFS The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large data sets. HDFS is part of the Apache Hadoop Core project . NameNode: is the arbitrator and central repository of file namespace in the cluster. The NameNode executes the operations such as opening, closing, and renaming files and directories. DataNode: manages the storage attached to the node on which it runs. It is responsible for serving all the read and writes requests. It performs operations on instructions on NameNode such as creation, deletion, and replications of blocks. Client: Responsible for getting the required metadata from the namenode and then communicating with the datanodes for reads and writes. YARN YARN stands for \u201cYet Another Resource Negotiator\u201c. It was introduced in Hadoop 2.0 to remove the bottleneck on Job Tracker which was present in Hadoop 1.0. YARN was described as a \u201cRedesigned Resource Manager\u201d at the time of its launching, but it has now evolved to be known as a large-scale distributed operating system used for Big Data processing. The main components of YARN architecture include: Client: It submits map-reduce(MR) jobs to the resource manager. Resource Manager: It is the master daemon of YARN and is responsible for resource assignment and management among all the applications. Whenever it receives a processing request, it forwards it to the corresponding node manager and allocates resources for the completion of the request accordingly. It has two major components: Scheduler: It performs scheduling based on the allocated application and available resources. It is a pure scheduler, which means that it does not perform other tasks such as monitoring or tracking and does not guarantee a restart if a task fails. The YARN scheduler supports plugins such as Capacity Scheduler and Fair Scheduler to partition the cluster resources. Application manager: It is responsible for accepting the application and negotiating the first container from the resource manager. It also restarts the Application Manager container if a task fails. Node Manager: It takes care of individual nodes on the Hadoop cluster and manages application and workflow and that particular node. Its primary job is to keep up with the Node Manager. It monitors resource usage, performs log management, and also kills a container based on directions from the resource manager. It is also responsible for creating the container process and starting it at the request of the Application master. Application Master: An application is a single job submitted to a framework. The application manager is responsible for negotiating resources with the resource manager, tracking the status, and monitoring the progress of a single application. The application master requests the container from the node manager by sending a Container Launch Context(CLC) which includes everything an application needs to run. Once the application is started, it sends the health report to the resource manager from time-to-time. Container: It is a collection of physical resources such as RAM, CPU cores, and disk on a single node. The containers are invoked by Container Launch Context(CLC) which is a record that contains information such as environment variables, security tokens, dependencies, etc.","title":"Architecture of Hadoop"},{"location":"big_data/evolution/#mapreduce-framework","text":"The term MapReduce represents two separate and distinct tasks Hadoop programs perform-Map Job and Reduce Job. Map jobs take data sets as input and process them to produce key-value pairs. Reduce job takes the output of the Map job i.e. the key-value pairs and aggregates them to produce desired results. Hadoop MapReduce (Hadoop Map/Reduce) is a software framework for distributed processing of large data sets on computing clusters. Mapreduce helps to split the input data set into a number of parts and run a program on all data parts parallel at once. Please find the below Word count example demonstrating the usage of the MapReduce framework:","title":"MapReduce framework"},{"location":"big_data/evolution/#other-tooling-around-hadoop","text":"Hive Uses a language called HQL which is very SQL like. Gives non-programmers the ability to query and analyze data in Hadoop. Is basically an abstraction layer on top of map-reduce. Ex. HQL query: SELECT pet.name, comment FROM pet JOIN event ON (pet.name = event.name); In mysql: SELECT pet.name, comment FROM pet, event WHERE pet.name = event.name; Pig Uses a scripting language called Pig Latin, which is more workflow driven. Don't need to be an expert Java programmer but need a few coding skills. Is also an abstraction layer on top of map-reduce. Here is a quick question for you: What is the output of running the pig queries in the right column against the data present in the left column in the below image? Output: mysql 7,Komal,Nayak,24,9848022334,trivendram 8,Bharathi,Nambiayar,24,9848022333,Chennai 5,Trupthi,Mohanthy,23,9848022336,Bhuwaneshwar 6,Archana,Mishra,23,9848022335,Chennai 3. Spark 1. Spark provides primitives for in-memory cluster computing that allows user programs to load data into a cluster\u2019s memory and query it repeatedly, making it well suited to machine learning algorithms. 4. Presto 1. Presto is a high performance, distributed SQL query engine for Big Data. 2. Its architecture allows users to query a variety of data sources such as Hadoop, AWS S3, Alluxio, MySQL, Cassandra, Kafka, and MongoDB. 3. Example presto query: mysql use studentDB; show tables; SELECT roll_no, name FROM studentDB.studentDetails where section=\u2019A\u2019 limit 5;","title":"Other tooling around Hadoop"},{"location":"big_data/evolution/#data-serialisation-and-storage","text":"In order to transport the data over the network or to store on some persistent storage, we use the process of translating data structures or objects state into binary or textual form. We call this process serialization.. Avro data is stored in a container file (a .avro file) and its schema (the .avsc file) is stored with the data file. Apache Hive provides support to store a table as Avro and can also query data in this serialisation format.","title":"Data Serialisation and storage"},{"location":"big_data/intro/","text":"Big Data Prerequisites Basics of Linux File systems. Basic understanding of System Design. What to expect from this course This course covers the basics of Big Data and how it has evolved to become what it is today. We will take a look at a few realistic scenarios where Big Data would be a perfect fit. An interesting assignment on designing a Big Data system is followed by understanding the architecture of Hadoop and the tooling around it. What is not covered under this course Writing programs to draw analytics from data. Course Contents Overview of Big Data Usage of Big Data techniques Evolution of Hadoop Architecture of hadoop HDFS Yarn MapReduce framework Other tooling around hadoop Hive Pig Spark Presto Data Serialisation and storage Overview of Big Data Big Data is a collection of large datasets that cannot be processed using traditional computing techniques. It is not a single technique or a tool, rather it has become a complete subject, which involves various tools, techniques, and frameworks. Big Data could consist of Structured data Unstructured data Semi-structured data Characteristics of Big Data: Volume Variety Velocity Variability Examples of Big Data generation include stock exchanges, social media sites, jet engines, etc. Usage of Big Data Techniques Take the example of the traffic lights problem. There are more than 300,000 traffic lights in the US as of 2018. Let us assume that we placed a device on each of them to collect metrics and send it to a central metrics collection system. If each of the IoT devices sends 10 events per minute, we have 300000x10x60x24 = 432x10^7 events per day. How would you go about processing that and telling me how many of the signals were \u201cgreen\u201d at 10:45 am on a particular day? Consider the next example on Unified Payments Interface (UPI) transactions: We had about 1.15 billion UPI transactions in the month of October 2019 in India. If we try to extrapolate this data to about a year and try to find out some common payments that were happening through a particular UPI ID, how do you suggest we go about that?","title":"Introduction"},{"location":"big_data/intro/#big-data","text":"","title":"Big Data"},{"location":"big_data/intro/#prerequisites","text":"Basics of Linux File systems. Basic understanding of System Design.","title":"Prerequisites"},{"location":"big_data/intro/#what-to-expect-from-this-course","text":"This course covers the basics of Big Data and how it has evolved to become what it is today. We will take a look at a few realistic scenarios where Big Data would be a perfect fit. An interesting assignment on designing a Big Data system is followed by understanding the architecture of Hadoop and the tooling around it.","title":"What to expect from this course"},{"location":"big_data/intro/#what-is-not-covered-under-this-course","text":"Writing programs to draw analytics from data.","title":"What is not covered under this course"},{"location":"big_data/intro/#course-contents","text":"Overview of Big Data Usage of Big Data techniques Evolution of Hadoop Architecture of hadoop HDFS Yarn MapReduce framework Other tooling around hadoop Hive Pig Spark Presto Data Serialisation and storage","title":"Course Contents"},{"location":"big_data/intro/#overview-of-big-data","text":"Big Data is a collection of large datasets that cannot be processed using traditional computing techniques. It is not a single technique or a tool, rather it has become a complete subject, which involves various tools, techniques, and frameworks. Big Data could consist of Structured data Unstructured data Semi-structured data Characteristics of Big Data: Volume Variety Velocity Variability Examples of Big Data generation include stock exchanges, social media sites, jet engines, etc.","title":"Overview of Big Data"},{"location":"big_data/intro/#usage-of-big-data-techniques","text":"Take the example of the traffic lights problem. There are more than 300,000 traffic lights in the US as of 2018. Let us assume that we placed a device on each of them to collect metrics and send it to a central metrics collection system. If each of the IoT devices sends 10 events per minute, we have 300000x10x60x24 = 432x10^7 events per day. How would you go about processing that and telling me how many of the signals were \u201cgreen\u201d at 10:45 am on a particular day? Consider the next example on Unified Payments Interface (UPI) transactions: We had about 1.15 billion UPI transactions in the month of October 2019 in India. If we try to extrapolate this data to about a year and try to find out some common payments that were happening through a particular UPI ID, how do you suggest we go about that?","title":"Usage of Big Data Techniques"},{"location":"big_data/tasks/","text":"Tasks and conclusion Post-training tasks: Try setting up your own 3 node Hadoop cluster. A VM based solution can be found here Write a simple spark/MR job of your choice and understand how to generate analytics from data. Sample dataset can be found here References: Hadoop documentation HDFS Architecture YARN Architecture Google GFS paper","title":"Conclusion"},{"location":"big_data/tasks/#tasks-and-conclusion","text":"","title":"Tasks and conclusion"},{"location":"big_data/tasks/#post-training-tasks","text":"Try setting up your own 3 node Hadoop cluster. A VM based solution can be found here Write a simple spark/MR job of your choice and understand how to generate analytics from data. Sample dataset can be found here","title":"Post-training tasks:"},{"location":"big_data/tasks/#references","text":"Hadoop documentation HDFS Architecture YARN Architecture Google GFS paper","title":"References:"},{"location":"databases_nosql/further_reading/","text":"Conclusion We have covered basic concepts of NoSQL databases. There is much more to learn and do. We hope this course gives you a good start and inspires you to explore further. Further reading NoSQL: https://hostingdata.co.uk/nosql-database/ https://www.mongodb.com/nosql-explained https://www.mongodb.com/nosql-explained/nosql-vs-sql Cap Theorem http://www.julianbrowne.com/article/brewers-cap-theorem Scalability http://www.slideshare.net/jboner/scalability-availability-stability-patterns Eventual Consistency https://www.allthingsdistributed.com/2008/12/eventually_consistent.html https://www.toptal.com/big-data/consistent-hashing https://web.stanford.edu/class/cs244/papers/chord_TON_2003.pdf","title":"Conclusion"},{"location":"databases_nosql/further_reading/#conclusion","text":"We have covered basic concepts of NoSQL databases. There is much more to learn and do. We hope this course gives you a good start and inspires you to explore further.","title":"Conclusion"},{"location":"databases_nosql/further_reading/#further-reading","text":"NoSQL: https://hostingdata.co.uk/nosql-database/ https://www.mongodb.com/nosql-explained https://www.mongodb.com/nosql-explained/nosql-vs-sql Cap Theorem http://www.julianbrowne.com/article/brewers-cap-theorem Scalability http://www.slideshare.net/jboner/scalability-availability-stability-patterns Eventual Consistency https://www.allthingsdistributed.com/2008/12/eventually_consistent.html https://www.toptal.com/big-data/consistent-hashing https://web.stanford.edu/class/cs244/papers/chord_TON_2003.pdf","title":"Further reading"},{"location":"databases_nosql/intro/","text":"NoSQL Concepts Prerequisites Relational Databases What to expect from this course At the end of training, you will have an understanding of what a NoSQL database is, what kind of advantages or disadvantages it has over traditional RDBMS, learn about different types of NoSQL databases and understand some of the underlying concepts & trade offs w.r.t to NoSQL. What is not covered under this course We will not be deep diving into any specific NoSQL Database. Course Contents Introduction to NoSQL CAP Theorem Data versioning Partitioning Hashing Quorum Introduction When people use the term \u201cNoSQL database\u201d, they typically use it to refer to any non-relational database. Some say the term \u201cNoSQL\u201d stands for \u201cnon SQL\u201d while others say it stands for \u201cnot only SQL.\u201d Either way, most agree that NoSQL databases are databases that store data in a format other than relational tables. A common misconception is that NoSQL databases or non-relational databases don\u2019t store relationship data well. NoSQL databases can store relationship data\u2014they just store it differently than relational databases do. In fact, when compared with SQL databases, many find modeling relationship data in NoSQL databases to be easier , because related data doesn\u2019t have to be split between tables. Such databases have existed since the late 1960s, but the name \"NoSQL\" was only coined in the early 21st century. NASA used a NoSQL database to track inventory for the Apollo mission. NoSQL databases emerged in the late 2000s as the cost of storage dramatically decreased. Gone were the days of needing to create a complex, difficult-to-manage data model simply for the purposes of reducing data duplication. Developers (rather than storage) were becoming the primary cost of software development, so NoSQL databases optimized for developer productivity. With the rise of Agile development methodology, NoSQL databases were developed with a focus on scaling, fast performance and at the same time allowed for frequent application changes and made programming easier. Types of NoSQL databases: Over time due to the way these NoSQL databases were developed to suit requirements at different companies, we ended up with quite a few types of them. However, they can be broadly classified into 4 types. Some of the databases can overlap between different types. They are Document databases: They store data in documents similar to JSON (JavaScript Object Notation) objects. Each document contains pairs of fields and values. The values can typically be a variety of types including things like strings, numbers, booleans, arrays, or objects, and their structures typically align with objects developers are working with in code. The advantages include intuitive data model & flexible schemas. Because of their variety of field value types and powerful query languages, document databases are great for a wide variety of use cases and can be used as a general purpose database. They can horizontally scale-out to accomodate large data volumes. Ex: MongoDB, Couchbase Key-Value databases: These are a simpler type of databases where each item contains keys and values. A value can typically only be retrieved by referencing its value, so learning how to query for a specific key-value pair is typically simple. Key-value databases are great for use cases where you need to store large amounts of data but you don\u2019t need to perform complex queries to retrieve it. Common use cases include storing user preferences or caching. Ex: Redis , DynamoDB , Voldemort / Venice (Linkedin), Wide-Column stores: They store data in tables, rows, and dynamic columns. Wide-column stores provide a lot of flexibility over relational databases because each row is not required to have the same columns. Many consider wide-column stores to be two-dimensional key-value databases. Wide-column stores are great for when you need to store large amounts of data and you can predict what your query patterns will be. Wide-column stores are commonly used for storing Internet of Things data and user profile data. Cassandra and HBase are two of the most popular wide-column stores. Graph Databases: These databases store data in nodes and edges. Nodes typically store information about people, places, and things while edges store information about the relationships between the nodes. The underlying storage mechanism of graph databases can vary. Some depend on a relational engine and \u201cstore\u201d the graph data in a table (although a table is a logical element, therefore this approach imposes another level of abstraction between the graph database, the graph database management system and the physical devices where the data is actually stored). Others use a key-value store or document-oriented database for storage, making them inherently NoSQL structures. Graph databases excel in use cases where you need to traverse relationships to look for patterns such as social networks, fraud detection, and recommendation engines. Ex: Neo4j Comparison Performance Scalability Flexibility Complexity Functionality Key Value high high high none Variable Document stores high Variable (high) high low Variable (low) Column DB high high moderate low minimal Graph Variable Variable high high Graph theory Differences between SQL and NoSQL The table below summarizes the main differences between SQL and NoSQL databases. SQL Databases NoSQL Databases Data Storage Model Tables with fixed rows and columns Document: JSON documents, Key-value: key-value pairs, Wide-column: tables with rows and dynamic columns, Graph: nodes and edges Primary Purpose General purpose Document: general purpose, Key-value: large amounts of data with simple lookup queries, Wide-column: large amounts of data with predictable query patterns, Graph: analyzing and traversing relationships between connected data Schemas Rigid Flexible Scaling Vertical (scale-up with a larger server) Horizontal (scale-out across commodity servers) Multi-Record ACID Transactions Supported Most do not support multi-record ACID transactions. However, some\u2014like MongoDB\u2014do. Joins Typically required Typically not required Data to Object Mapping Requires ORM (object-relational mapping) Many do not require ORMs. Document DB documents map directly to data structures in most popular programming languages. Advantages Flexible Data Models Most NoSQL systems feature flexible schemas. A flexible schema means you can easily modify your database schema to add or remove fields to support for evolving application requirements. This facilitates with continuous application development of new features without database operation overhead. Horizontal Scaling Most NoSQL systems allow you to scale horizontally, which means you can add in cheaper & commodity hardware, whenever you want to scale a system. On the other hand SQL systems generally scale Vertically (a more powerful server). NoSQL systems can also host huge data sets when compared to traditional SQL systems. Fast Queries NoSQL can generally be a lot faster than traditional SQL systems due to data denormalization and horizontal scaling. Most NoSQL systems also tend to store similar data together facilitating faster query responses. Developer productivity NoSQL systems tend to map data based on the programming data structures. As a result developers need to perform fewer data transformations leading to increased productivity & fewer bugs.","title":"Introduction"},{"location":"databases_nosql/intro/#nosql-concepts","text":"","title":"NoSQL Concepts"},{"location":"databases_nosql/intro/#prerequisites","text":"Relational Databases","title":"Prerequisites"},{"location":"databases_nosql/intro/#what-to-expect-from-this-course","text":"At the end of training, you will have an understanding of what a NoSQL database is, what kind of advantages or disadvantages it has over traditional RDBMS, learn about different types of NoSQL databases and understand some of the underlying concepts & trade offs w.r.t to NoSQL.","title":"What to expect from this course"},{"location":"databases_nosql/intro/#what-is-not-covered-under-this-course","text":"We will not be deep diving into any specific NoSQL Database.","title":"What is not covered under this course"},{"location":"databases_nosql/intro/#course-contents","text":"Introduction to NoSQL CAP Theorem Data versioning Partitioning Hashing Quorum","title":"Course Contents"},{"location":"databases_nosql/intro/#introduction","text":"When people use the term \u201cNoSQL database\u201d, they typically use it to refer to any non-relational database. Some say the term \u201cNoSQL\u201d stands for \u201cnon SQL\u201d while others say it stands for \u201cnot only SQL.\u201d Either way, most agree that NoSQL databases are databases that store data in a format other than relational tables. A common misconception is that NoSQL databases or non-relational databases don\u2019t store relationship data well. NoSQL databases can store relationship data\u2014they just store it differently than relational databases do. In fact, when compared with SQL databases, many find modeling relationship data in NoSQL databases to be easier , because related data doesn\u2019t have to be split between tables. Such databases have existed since the late 1960s, but the name \"NoSQL\" was only coined in the early 21st century. NASA used a NoSQL database to track inventory for the Apollo mission. NoSQL databases emerged in the late 2000s as the cost of storage dramatically decreased. Gone were the days of needing to create a complex, difficult-to-manage data model simply for the purposes of reducing data duplication. Developers (rather than storage) were becoming the primary cost of software development, so NoSQL databases optimized for developer productivity. With the rise of Agile development methodology, NoSQL databases were developed with a focus on scaling, fast performance and at the same time allowed for frequent application changes and made programming easier.","title":"Introduction"},{"location":"databases_nosql/intro/#types-of-nosql-databases","text":"Over time due to the way these NoSQL databases were developed to suit requirements at different companies, we ended up with quite a few types of them. However, they can be broadly classified into 4 types. Some of the databases can overlap between different types. They are Document databases: They store data in documents similar to JSON (JavaScript Object Notation) objects. Each document contains pairs of fields and values. The values can typically be a variety of types including things like strings, numbers, booleans, arrays, or objects, and their structures typically align with objects developers are working with in code. The advantages include intuitive data model & flexible schemas. Because of their variety of field value types and powerful query languages, document databases are great for a wide variety of use cases and can be used as a general purpose database. They can horizontally scale-out to accomodate large data volumes. Ex: MongoDB, Couchbase Key-Value databases: These are a simpler type of databases where each item contains keys and values. A value can typically only be retrieved by referencing its value, so learning how to query for a specific key-value pair is typically simple. Key-value databases are great for use cases where you need to store large amounts of data but you don\u2019t need to perform complex queries to retrieve it. Common use cases include storing user preferences or caching. Ex: Redis , DynamoDB , Voldemort / Venice (Linkedin), Wide-Column stores: They store data in tables, rows, and dynamic columns. Wide-column stores provide a lot of flexibility over relational databases because each row is not required to have the same columns. Many consider wide-column stores to be two-dimensional key-value databases. Wide-column stores are great for when you need to store large amounts of data and you can predict what your query patterns will be. Wide-column stores are commonly used for storing Internet of Things data and user profile data. Cassandra and HBase are two of the most popular wide-column stores. Graph Databases: These databases store data in nodes and edges. Nodes typically store information about people, places, and things while edges store information about the relationships between the nodes. The underlying storage mechanism of graph databases can vary. Some depend on a relational engine and \u201cstore\u201d the graph data in a table (although a table is a logical element, therefore this approach imposes another level of abstraction between the graph database, the graph database management system and the physical devices where the data is actually stored). Others use a key-value store or document-oriented database for storage, making them inherently NoSQL structures. Graph databases excel in use cases where you need to traverse relationships to look for patterns such as social networks, fraud detection, and recommendation engines. Ex: Neo4j","title":"Types of NoSQL databases:"},{"location":"databases_nosql/intro/#comparison","text":"Performance Scalability Flexibility Complexity Functionality Key Value high high high none Variable Document stores high Variable (high) high low Variable (low) Column DB high high moderate low minimal Graph Variable Variable high high Graph theory","title":"Comparison"},{"location":"databases_nosql/intro/#differences-between-sql-and-nosql","text":"The table below summarizes the main differences between SQL and NoSQL databases. SQL Databases NoSQL Databases Data Storage Model Tables with fixed rows and columns Document: JSON documents, Key-value: key-value pairs, Wide-column: tables with rows and dynamic columns, Graph: nodes and edges Primary Purpose General purpose Document: general purpose, Key-value: large amounts of data with simple lookup queries, Wide-column: large amounts of data with predictable query patterns, Graph: analyzing and traversing relationships between connected data Schemas Rigid Flexible Scaling Vertical (scale-up with a larger server) Horizontal (scale-out across commodity servers) Multi-Record ACID Transactions Supported Most do not support multi-record ACID transactions. However, some\u2014like MongoDB\u2014do. Joins Typically required Typically not required Data to Object Mapping Requires ORM (object-relational mapping) Many do not require ORMs. Document DB documents map directly to data structures in most popular programming languages.","title":"Differences between SQL and NoSQL"},{"location":"databases_nosql/intro/#advantages","text":"Flexible Data Models Most NoSQL systems feature flexible schemas. A flexible schema means you can easily modify your database schema to add or remove fields to support for evolving application requirements. This facilitates with continuous application development of new features without database operation overhead. Horizontal Scaling Most NoSQL systems allow you to scale horizontally, which means you can add in cheaper & commodity hardware, whenever you want to scale a system. On the other hand SQL systems generally scale Vertically (a more powerful server). NoSQL systems can also host huge data sets when compared to traditional SQL systems. Fast Queries NoSQL can generally be a lot faster than traditional SQL systems due to data denormalization and horizontal scaling. Most NoSQL systems also tend to store similar data together facilitating faster query responses. Developer productivity NoSQL systems tend to map data based on the programming data structures. As a result developers need to perform fewer data transformations leading to increased productivity & fewer bugs.","title":"Advantages"},{"location":"databases_nosql/key_concepts/","text":"Key Concepts Lets looks at some of the key concepts when we talk about NoSQL or distributed systems CAP Theorem In a keynote titled \u201c Towards Robust Distributed Systems \u201d at ACM\u2019s PODC symposium in 2000 Eric Brewer came up with the so-called CAP-theorem which is widely adopted today by large web companies as well as in the NoSQL community. The CAP acronym stands for C onsistency, A vailability & P artition Tolerance. Consistency It refers to how consistent a system is after an execution. A distributed system is called consistent when a write made by a source is available for all readers of that shared data. Different NoSQL systems support different levels of consistency. Availability It refers to how a system responds to loss of functionality of different systems due to hardware and software failures. A high availability implies that a system is still available to handle operations (reads and writes) when a certain part of the system is down due to a failure or upgrade. Partition Tolerance It is the ability of the system to continue operations in the event of a network partition. A network partition occurs when a failure causes two or more islands of networks where the systems can\u2019t talk to each other across the islands temporarily or permanently. Brewer alleges that one can at most choose two of these three characteristics in a shared-data system. The CAP-theorem states that a choice can only be made for two options out of consistency, availability and partition tolerance. A growing number of use cases in large scale applications tend to value reliability implying that availability & redundancy are more valuable than consistency. As a result these systems struggle to meet ACID properties. They attain this by loosening on the consistency requirement i.e Eventual Consistency. Eventual Consistency means that all readers will see writes, as time goes on: \u201cIn a steady state, the system will eventually return the last written value\u201d. Clients therefore may face an inconsistent state of data as updates are in progress. For instance, in a replicated database updates may go to one node which replicates the latest version to all other nodes that contain a replica of the modified dataset so that the replica nodes eventually will have the latest version. NoSQL systems support different levels of eventual consistency models. For example: Read Your Own Writes Consistency Clients will see their updates immediately after they are written. The reads can hit nodes other than the one where it was written. However they might not see updates by other clients immediately. Session Consistency Clients will see the updates to their data within a session scope. This generally indicates that reads & writes occur on the same server. Other clients using the same nodes will receive the same updates. Casual Consistency A system provides causal consistency if the following condition holds: write operations that are related by potential causality are seen by each process of the system in order. Different processes may observe concurrent writes in different orders Eventual consistency is useful if concurrent updates of the same partitions of data are unlikely and if clients do not immediately depend on reading updates issued by themselves or by other clients. Depending on what consistency model was chosen for the system (or parts of it), determines where the requests are routed, ex: replicas. CAP alternatives illustration Choice Traits Examples Consistency + Availability (Forfeit Partitions) 2-phase commits Cache invalidation protocols Single-site databases Cluster databases LDAP xFS file system Consistency + Partition tolerance (Forfeit Availability) Pessimistic locking Make minority partitions unavailable Distributed databases Distributed locking Majority protocols Availability + Partition tolerance (Forfeit Consistency) expirations/leases conflict resolution optimistic DNS Web caching Versioning of Data in distributed systems When data is distributed across nodes, it can be modified on different nodes at the same time (assuming strict consistency is enforced). Questions arise on conflict resolution for concurrent updates. Some of the popular conflict resolution mechanism are Timestamps This is the most obvious solution. You sort updates based on chronological order and choose the latest update. However this relies on clock synchronization across different parts of the infrastructure. This gets even more complicated when parts of systems are spread across different geographic locations. Optimistic Locking You associate a unique value like a clock or counter with every data update. When a client wants to update data, it has to specify which version of data needs to be updated. This would mean you need to keep track of history of the data versions. Vector Clocks A vector clock is defined as a tuple of clock values from each node. In a distributed environment, each node maintains a tuple of such clock values which represent the state of the nodes itself and its peers/replicas. A clock value may be real timestamps derived from local clock or version no. Vector clocks illustration Vector clocks have the following advantages over other conflict resolution mechanism No dependency on synchronized clocks No total ordering of revision nos required for casual reasoning No need to store and maintain multiple versions of the data on different nodes. Partitioning When the amount of data crosses the capacity of a single node, we need to think of splitting data, creating replicas for load balancing & disaster recovery. Depending on how dynamic the infrastructure is, we have a few approaches that we can take. Memory cached These are partitioned in-memory databases that are primarily used for transient data. These databases are generally used as a front for traditional RDBMS. Most frequently used data is replicated from a rdbms into a memory database to facilitate fast queries and to take the load off from backend DB\u2019s. A very common example is memcached or couchbase. Clustering Traditional cluster mechanisms abstract away the cluster topology from clients. A client need not know where the actual data is residing and which node it is talking to. Clustering is very commonly used in traditional RDBMS where it can help scaling the persistent layer to a certain extent. Separating reads from writes In this method, you will have multiple replicas hosting the same data. The incoming writes are typically sent to a single node (Leader) or multiple nodes (multi-Leader), while the rest of the replicas (Follower) handle reads requests. The leader replicates writes asynchronously to all followers. However the write lag can\u2019t be completely avoided. Sometimes a leader can crash before it replicates all the data to a follower. When this happens, a follower with the most consistent data can be turned into a leader. As you can realize now, it is hard to enforce full consistency in this model. You also need to consider the ratio of read vs write traffic. This model won\u2019t make sense when writes are higher than reads. The replication methods can also vary widely. Some systems do a complete transfer of state periodically, while others use a delta state transfer approach. You could also transfer the state by transferring the operations in order. The followers can then apply the same operations as the leader to catch up. Sharding Sharing refers to dividing data in such a way that data is distributed evenly (both in terms of storage & processing power) across a cluster of nodes. It can also imply data locality, which means similar & related data is stored together to facilitate faster access. A shard in turn can be further replicated to meet load balancing or disaster recovery requirements. A single shard replica might take in all writes (single leader) or multiple replicas can take writes (multi-leader). Reads can be distributed across multiple replicas. Since data is now distributed across multiple nodes, clients should be able to consistently figure out where data is hosted. We will look at some of the common techniques below. The downside of sharding is that joins between shards is not possible. So an upstream/downstream application has to aggregate the results from multiple shards. Sharding example Hashing A hash function is a function that maps one piece of data\u2014typically describing some kind of object, often of arbitrary size\u2014to another piece of data, typically an integer, known as hash code , or simply hash . In a partitioned database, it is important to consistently map a key to a server/replica. For ex: you can use a very simple hash as a modulo function. _p = k mod n_ Where p -> partition, k -> primary key n -> no of nodes The downside of this simple hash is that, whenever the cluster topology changes, the data distribution also changes. When you are dealing with memory caches, it will be easy to distribute partitions around. Whenever a node joins/leaves a topology, partitions can reorder themselves, a cache miss can be re-populated from backend DB. However when you look at persistent data, it is not possible as the new node doesn\u2019t have the data needed to serve it. This brings us to consistent hashing. Consistent Hashing Consistent hashing is a distributed hashing scheme that operates independently of the number of servers or objects in a distributed hash table by assigning them a position on an abstract circle, or hash ring . This allows servers and objects to scale without affecting the overall system. Say that our hash function h() generates a 32-bit integer. Then, to determine to which server we will send a key k, we find the server s whose hash h(s) is the smallest integer that is larger than h(k). To make the process simpler, we assume the table is circular, which means that if we cannot find a server with a hash larger than h(k), we wrap around and start looking from the beginning of the array. Consistent hashing illustration In consistent hashing when a server is removed or added then only the keys from that server are relocated. For example, if server S3 is removed then, all keys from server S3 will be moved to server S4 but keys stored on server S4 and S2 are not relocated. But there is one problem, when server S3 is removed then keys from S3 are not equally distributed among remaining servers S4 and S2. They are only assigned to server S4 which increases the load on server S4. To evenly distribute the load among servers when a server is added or removed, it creates a fixed number of replicas ( known as virtual nodes) of each server and distributes it along the circle. So instead of server labels S1, S2 and S3, we will have S10 S11\u2026S19, S20 S21\u2026S29 and S30 S31\u2026S39. The factor for a number of replicas is also known as weight , depending on the situation. All keys which are mapped to replicas Sij are stored on server Si. To find a key we do the same thing, find the position of the key on the circle and then move forward until you find a server replica. If the server replica is Sij then the key is stored in server Si. Suppose server S3 is removed, then all S3 replicas with labels S30 S31 \u2026 S39 must be removed. Now the objects keys adjacent to S3X labels will be automatically re-assigned to S1X, S2X and S4X. All keys originally assigned to S1, S2 & S4 will not be moved. Similar things happen if we add a server. Suppose we want to add a server S5 as a replacement of S3 then we need to add labels S50 S51 \u2026 S59. In the ideal case, one-fourth of keys from S1, S2 and S4 will be reassigned to S5. When applied to persistent storages, further issues arise: if a node has left the scene, data stored on this node becomes unavailable, unless it has been replicated to other nodes before; in the opposite case of a new node joining the others, adjacent nodes are no longer responsible for some pieces of data which they still store but not get asked for anymore as the corresponding objects are no longer hashed to them by requesting clients. In order to address this issue, a replication factor (r) can be introduced. Introducing replicas in a partitioning scheme\u2014besides reliability benefits\u2014also makes it possible to spread workload for read requests that can go to any physical node responsible for a requested piece of data. Scalability doesn\u2019t work if the clients have to decide between multiple versions of the dataset, because they need to read from a quorum of servers which in turn reduces the efficiency of load balancing. Quorum Quorum is the minimum number of nodes in a cluster that must be online and be able to communicate with each other. If any additional node failure occurs beyond this threshold, the cluster will stop running. To attain a quorum, you need a majority of the nodes. Commonly it is (N/2 + 1), where N is the total no of nodes in the system. For ex, In a 3 node cluster, you need 2 nodes for a majority, In a 5 node cluster, you need 3 nodes for a majority, In a 6 node cluster, you need 4 nodes for a majority. Quorum example Network problems can cause communication failures among cluster nodes. One set of nodes might be able to communicate together across a functioning part of a network but not be able to communicate with a different set of nodes in another part of the network. This is known as split brain in cluster or cluster partitioning. Now the partition which has quorum is allowed to continue running the application. The other partitions are removed from the cluster. Eg: In a 5 node cluster, consider what happens if nodes 1, 2, and 3 can communicate with each other but not with nodes 4 and 5. Nodes 1, 2, and 3 constitute a majority, and they continue running as a cluster. Nodes 4 and 5, being a minority, stop running as a cluster. If node 3 loses communication with other nodes, all nodes stop running as a cluster. However, all functioning nodes will continue to listen for communication, so that when the network begins working again, the cluster can form and begin to run. Below diagram demonstrates Quorum selection on a cluster partitioned into two sets. Cluster Quorum example","title":"Key Concepts"},{"location":"databases_nosql/key_concepts/#key-concepts","text":"Lets looks at some of the key concepts when we talk about NoSQL or distributed systems","title":"Key Concepts"},{"location":"databases_nosql/key_concepts/#cap-theorem","text":"In a keynote titled \u201c Towards Robust Distributed Systems \u201d at ACM\u2019s PODC symposium in 2000 Eric Brewer came up with the so-called CAP-theorem which is widely adopted today by large web companies as well as in the NoSQL community. The CAP acronym stands for C onsistency, A vailability & P artition Tolerance. Consistency It refers to how consistent a system is after an execution. A distributed system is called consistent when a write made by a source is available for all readers of that shared data. Different NoSQL systems support different levels of consistency. Availability It refers to how a system responds to loss of functionality of different systems due to hardware and software failures. A high availability implies that a system is still available to handle operations (reads and writes) when a certain part of the system is down due to a failure or upgrade. Partition Tolerance It is the ability of the system to continue operations in the event of a network partition. A network partition occurs when a failure causes two or more islands of networks where the systems can\u2019t talk to each other across the islands temporarily or permanently. Brewer alleges that one can at most choose two of these three characteristics in a shared-data system. The CAP-theorem states that a choice can only be made for two options out of consistency, availability and partition tolerance. A growing number of use cases in large scale applications tend to value reliability implying that availability & redundancy are more valuable than consistency. As a result these systems struggle to meet ACID properties. They attain this by loosening on the consistency requirement i.e Eventual Consistency. Eventual Consistency means that all readers will see writes, as time goes on: \u201cIn a steady state, the system will eventually return the last written value\u201d. Clients therefore may face an inconsistent state of data as updates are in progress. For instance, in a replicated database updates may go to one node which replicates the latest version to all other nodes that contain a replica of the modified dataset so that the replica nodes eventually will have the latest version. NoSQL systems support different levels of eventual consistency models. For example: Read Your Own Writes Consistency Clients will see their updates immediately after they are written. The reads can hit nodes other than the one where it was written. However they might not see updates by other clients immediately. Session Consistency Clients will see the updates to their data within a session scope. This generally indicates that reads & writes occur on the same server. Other clients using the same nodes will receive the same updates. Casual Consistency A system provides causal consistency if the following condition holds: write operations that are related by potential causality are seen by each process of the system in order. Different processes may observe concurrent writes in different orders Eventual consistency is useful if concurrent updates of the same partitions of data are unlikely and if clients do not immediately depend on reading updates issued by themselves or by other clients. Depending on what consistency model was chosen for the system (or parts of it), determines where the requests are routed, ex: replicas. CAP alternatives illustration Choice Traits Examples Consistency + Availability (Forfeit Partitions) 2-phase commits Cache invalidation protocols Single-site databases Cluster databases LDAP xFS file system Consistency + Partition tolerance (Forfeit Availability) Pessimistic locking Make minority partitions unavailable Distributed databases Distributed locking Majority protocols Availability + Partition tolerance (Forfeit Consistency) expirations/leases conflict resolution optimistic DNS Web caching","title":"CAP Theorem"},{"location":"databases_nosql/key_concepts/#versioning-of-data-in-distributed-systems","text":"When data is distributed across nodes, it can be modified on different nodes at the same time (assuming strict consistency is enforced). Questions arise on conflict resolution for concurrent updates. Some of the popular conflict resolution mechanism are Timestamps This is the most obvious solution. You sort updates based on chronological order and choose the latest update. However this relies on clock synchronization across different parts of the infrastructure. This gets even more complicated when parts of systems are spread across different geographic locations. Optimistic Locking You associate a unique value like a clock or counter with every data update. When a client wants to update data, it has to specify which version of data needs to be updated. This would mean you need to keep track of history of the data versions. Vector Clocks A vector clock is defined as a tuple of clock values from each node. In a distributed environment, each node maintains a tuple of such clock values which represent the state of the nodes itself and its peers/replicas. A clock value may be real timestamps derived from local clock or version no. Vector clocks illustration Vector clocks have the following advantages over other conflict resolution mechanism No dependency on synchronized clocks No total ordering of revision nos required for casual reasoning No need to store and maintain multiple versions of the data on different nodes.","title":"Versioning of Data in distributed systems"},{"location":"databases_nosql/key_concepts/#partitioning","text":"When the amount of data crosses the capacity of a single node, we need to think of splitting data, creating replicas for load balancing & disaster recovery. Depending on how dynamic the infrastructure is, we have a few approaches that we can take. Memory cached These are partitioned in-memory databases that are primarily used for transient data. These databases are generally used as a front for traditional RDBMS. Most frequently used data is replicated from a rdbms into a memory database to facilitate fast queries and to take the load off from backend DB\u2019s. A very common example is memcached or couchbase. Clustering Traditional cluster mechanisms abstract away the cluster topology from clients. A client need not know where the actual data is residing and which node it is talking to. Clustering is very commonly used in traditional RDBMS where it can help scaling the persistent layer to a certain extent. Separating reads from writes In this method, you will have multiple replicas hosting the same data. The incoming writes are typically sent to a single node (Leader) or multiple nodes (multi-Leader), while the rest of the replicas (Follower) handle reads requests. The leader replicates writes asynchronously to all followers. However the write lag can\u2019t be completely avoided. Sometimes a leader can crash before it replicates all the data to a follower. When this happens, a follower with the most consistent data can be turned into a leader. As you can realize now, it is hard to enforce full consistency in this model. You also need to consider the ratio of read vs write traffic. This model won\u2019t make sense when writes are higher than reads. The replication methods can also vary widely. Some systems do a complete transfer of state periodically, while others use a delta state transfer approach. You could also transfer the state by transferring the operations in order. The followers can then apply the same operations as the leader to catch up. Sharding Sharing refers to dividing data in such a way that data is distributed evenly (both in terms of storage & processing power) across a cluster of nodes. It can also imply data locality, which means similar & related data is stored together to facilitate faster access. A shard in turn can be further replicated to meet load balancing or disaster recovery requirements. A single shard replica might take in all writes (single leader) or multiple replicas can take writes (multi-leader). Reads can be distributed across multiple replicas. Since data is now distributed across multiple nodes, clients should be able to consistently figure out where data is hosted. We will look at some of the common techniques below. The downside of sharding is that joins between shards is not possible. So an upstream/downstream application has to aggregate the results from multiple shards. Sharding example","title":"Partitioning"},{"location":"databases_nosql/key_concepts/#hashing","text":"A hash function is a function that maps one piece of data\u2014typically describing some kind of object, often of arbitrary size\u2014to another piece of data, typically an integer, known as hash code , or simply hash . In a partitioned database, it is important to consistently map a key to a server/replica. For ex: you can use a very simple hash as a modulo function. _p = k mod n_ Where p -> partition, k -> primary key n -> no of nodes The downside of this simple hash is that, whenever the cluster topology changes, the data distribution also changes. When you are dealing with memory caches, it will be easy to distribute partitions around. Whenever a node joins/leaves a topology, partitions can reorder themselves, a cache miss can be re-populated from backend DB. However when you look at persistent data, it is not possible as the new node doesn\u2019t have the data needed to serve it. This brings us to consistent hashing.","title":"Hashing"},{"location":"databases_nosql/key_concepts/#consistent-hashing","text":"Consistent hashing is a distributed hashing scheme that operates independently of the number of servers or objects in a distributed hash table by assigning them a position on an abstract circle, or hash ring . This allows servers and objects to scale without affecting the overall system. Say that our hash function h() generates a 32-bit integer. Then, to determine to which server we will send a key k, we find the server s whose hash h(s) is the smallest integer that is larger than h(k). To make the process simpler, we assume the table is circular, which means that if we cannot find a server with a hash larger than h(k), we wrap around and start looking from the beginning of the array. Consistent hashing illustration In consistent hashing when a server is removed or added then only the keys from that server are relocated. For example, if server S3 is removed then, all keys from server S3 will be moved to server S4 but keys stored on server S4 and S2 are not relocated. But there is one problem, when server S3 is removed then keys from S3 are not equally distributed among remaining servers S4 and S2. They are only assigned to server S4 which increases the load on server S4. To evenly distribute the load among servers when a server is added or removed, it creates a fixed number of replicas ( known as virtual nodes) of each server and distributes it along the circle. So instead of server labels S1, S2 and S3, we will have S10 S11\u2026S19, S20 S21\u2026S29 and S30 S31\u2026S39. The factor for a number of replicas is also known as weight , depending on the situation. All keys which are mapped to replicas Sij are stored on server Si. To find a key we do the same thing, find the position of the key on the circle and then move forward until you find a server replica. If the server replica is Sij then the key is stored in server Si. Suppose server S3 is removed, then all S3 replicas with labels S30 S31 \u2026 S39 must be removed. Now the objects keys adjacent to S3X labels will be automatically re-assigned to S1X, S2X and S4X. All keys originally assigned to S1, S2 & S4 will not be moved. Similar things happen if we add a server. Suppose we want to add a server S5 as a replacement of S3 then we need to add labels S50 S51 \u2026 S59. In the ideal case, one-fourth of keys from S1, S2 and S4 will be reassigned to S5. When applied to persistent storages, further issues arise: if a node has left the scene, data stored on this node becomes unavailable, unless it has been replicated to other nodes before; in the opposite case of a new node joining the others, adjacent nodes are no longer responsible for some pieces of data which they still store but not get asked for anymore as the corresponding objects are no longer hashed to them by requesting clients. In order to address this issue, a replication factor (r) can be introduced. Introducing replicas in a partitioning scheme\u2014besides reliability benefits\u2014also makes it possible to spread workload for read requests that can go to any physical node responsible for a requested piece of data. Scalability doesn\u2019t work if the clients have to decide between multiple versions of the dataset, because they need to read from a quorum of servers which in turn reduces the efficiency of load balancing.","title":"Consistent Hashing"},{"location":"databases_nosql/key_concepts/#quorum","text":"Quorum is the minimum number of nodes in a cluster that must be online and be able to communicate with each other. If any additional node failure occurs beyond this threshold, the cluster will stop running. To attain a quorum, you need a majority of the nodes. Commonly it is (N/2 + 1), where N is the total no of nodes in the system. For ex, In a 3 node cluster, you need 2 nodes for a majority, In a 5 node cluster, you need 3 nodes for a majority, In a 6 node cluster, you need 4 nodes for a majority. Quorum example Network problems can cause communication failures among cluster nodes. One set of nodes might be able to communicate together across a functioning part of a network but not be able to communicate with a different set of nodes in another part of the network. This is known as split brain in cluster or cluster partitioning. Now the partition which has quorum is allowed to continue running the application. The other partitions are removed from the cluster. Eg: In a 5 node cluster, consider what happens if nodes 1, 2, and 3 can communicate with each other but not with nodes 4 and 5. Nodes 1, 2, and 3 constitute a majority, and they continue running as a cluster. Nodes 4 and 5, being a minority, stop running as a cluster. If node 3 loses communication with other nodes, all nodes stop running as a cluster. However, all functioning nodes will continue to listen for communication, so that when the network begins working again, the cluster can form and begin to run. Below diagram demonstrates Quorum selection on a cluster partitioned into two sets. Cluster Quorum example","title":"Quorum"},{"location":"databases_sql/concepts/","text":"Relational DBs are used for data storage. Even a file can be used to store data, but relational DBs are designed with specific goals: Efficiency Ease of access and management Organized Handle relations between data (represented as tables) Transaction: a unit of work that can comprise multiple statements, executed together ACID properties Set of properties that guarantee data integrity of DB transactions Atomicity: Each transaction is atomic (succeeds or fails completely) Consistency: Transactions only result in valid state (which includes rules, constraints, triggers etc.) Isolation: Each transaction is executed independently of others safely within a concurrent system Durability: Completed transactions will not be lost due to any later failures Let\u2019s take some examples to illustrate the above properties. Account A has a balance of \u20b9200 & B has \u20b9400. Account A is transferring \u20b9100 to Account B. This transaction has a deduction from sender and an addition into the recipient\u2019s balance. If the first operation passes successfully while the second fails, A\u2019s balance would be \u20b9100 while B would be having \u20b9400 instead of \u20b9500. Atomicity in a DB ensures this partially failed transaction is rolled back. If the second operation above fails, it leaves the DB inconsistent (sum of balance of accounts before and after the operation is not the same). Consistency ensures that this does not happen. There are three operations, one to calculate interest for A\u2019s account, another to add that to A\u2019s account, then transfer \u20b9100 from B to A. Without isolation guarantees, concurrent execution of these 3 operations may lead to a different outcome every time. What happens if the system crashes before the transactions are written to disk? Durability ensures that the changes are applied correctly during recovery. Relational data Tables represent relations Columns (fields) represent attributes Rows are individual records Schema describes the structure of DB SQL A query language to interact with and manage data. CRUD operations - create, read, update, delete queries Management operations - create DBs/tables/indexes etc, backup, import/export, users, access controls Exercise: Classify the below queries into the four types - DDL (definition), DML(manipulation), DCL(control) and TCL(transactions) and explain in detail. insert, create, drop, delete, update, commit, rollback, truncate, alter, grant, revoke You can practise these in the lab section . Constraints Rules for data that can be stored. Query fails if you violate any of these defined on a table. Primary key: one or more columns that contain UNIQUE values, and cannot contain NULL values. A table can have only ONE primary key. An index on it is created by default. Foreign key: links two tables together. Its value(s) match a primary key in a different table \\ Not null: Does not allow null values \\ Unique: Value of column must be unique across all rows \\ Default: Provides a default value for a column if none is specified during insert Check: Allows only particular values (like Balance >= 0) Indexes Most indexes use B+ tree structure. Why use them: Speeds up queries (in large tables that fetch only a few rows, min/max queries, by eliminating rows from consideration etc) Types of indexes: unique, primary key, fulltext, secondary Write-heavy loads, mostly full table scans or accessing large number of rows etc. do not benefit from indexes Joins Allows you to fetch related data from multiple tables, linking them together with some common field. Powerful but also resource-intensive and makes scaling databases difficult. This is the cause of many slow performing queries when run at scale, and the solution is almost always to find ways to reduce the joins. Access control DBs have privileged accounts for admin tasks, and regular accounts for clients. There are finegrained controls on what actions(DDL, DML etc. discussed earlier )are allowed for these accounts. DB first verifies the user credentials (authentication), and then examines whether this user is permitted to perform the request (authorization) by looking up these information in some internal tables. Other controls include activity auditing that allows examining the history of actions done by a user, and resource limits which define the number of queries, connections etc. allowed. Popular databases Commercial, closed source - Oracle, Microsoft SQL Server, IBM DB2 Open source with optional paid support - MySQL, MariaDB, PostgreSQL Individuals and small companies have always preferred open source DBs because of the huge cost associated with commercial software. In recent times, even large organizations have moved away from commercial software to open source alternatives because of the flexibility and cost savings associated with it. Lack of support is no longer a concern because of the paid support available from the developer and third parties. MySQL is the most widely used open source DB, and it is widely supported by hosting providers, making it easy for anyone to use. It is part of the popular Linux-Apache-MySQL-PHP ( LAMP ) stack that became popular in the 2000s. We have many more choices for a programming language, but the rest of that stack is still widely used.","title":"Key Concepts"},{"location":"databases_sql/concepts/#popular-databases","text":"Commercial, closed source - Oracle, Microsoft SQL Server, IBM DB2 Open source with optional paid support - MySQL, MariaDB, PostgreSQL Individuals and small companies have always preferred open source DBs because of the huge cost associated with commercial software. In recent times, even large organizations have moved away from commercial software to open source alternatives because of the flexibility and cost savings associated with it. Lack of support is no longer a concern because of the paid support available from the developer and third parties. MySQL is the most widely used open source DB, and it is widely supported by hosting providers, making it easy for anyone to use. It is part of the popular Linux-Apache-MySQL-PHP ( LAMP ) stack that became popular in the 2000s. We have many more choices for a programming language, but the rest of that stack is still widely used.","title":"Popular databases"},{"location":"databases_sql/conclusion/","text":"Conclusion We have covered basic concepts of SQL databases. We have also covered some of the tasks that an SRE may be responsible for - there is so much more to learn and do. We hope this course gives you a good start and inspires you to explore further. Further reading More practice with online resources like this one Normalization Routines , triggers Views Transaction isolation levels Sharding Setting up HA , monitoring , backups","title":"Conclusion"},{"location":"databases_sql/conclusion/#conclusion","text":"We have covered basic concepts of SQL databases. We have also covered some of the tasks that an SRE may be responsible for - there is so much more to learn and do. We hope this course gives you a good start and inspires you to explore further.","title":"Conclusion"},{"location":"databases_sql/conclusion/#further-reading","text":"More practice with online resources like this one Normalization Routines , triggers Views Transaction isolation levels Sharding Setting up HA , monitoring , backups","title":"Further reading"},{"location":"databases_sql/innodb/","text":"Why should you use this? General purpose, row level locking, ACID support, transactions, crash recovery and multi-version concurrency control etc. Architecture Key components: Memory: Buffer pool: LRU cache of frequently used data(table and index) to be processed directly from memory, which speeds up processing. Important for tuning performance. Change buffer: Caches changes to secondary index pages when those pages are not in the buffer pool and merges it when they are fetched. Merging may take a long time and impact live queries. It also takes up part of the buffer pool. Avoids the extra I/O to read secondary indexes in. Adaptive hash index: Supplements InnoDB\u2019s B-Tree indexes with fast hash lookup tables like a cache. Slight performance penalty for misses, also adds maintenance overhead of updating it. Hash collisions cause AHI rebuilding for large DBs. Log buffer: Holds log data before flush to disk. Size of each above memory is configurable, and impacts performance a lot. Requires careful analysis of workload, available resources, benchmarking and tuning for optimal performance. Disk: Tables: Stores data within rows and columns. Indexes: Helps find rows with specific column values quickly, avoids full table scans. Redo Logs: all transactions are written to them, and after a crash, the recovery process corrects data written by incomplete transactions and replays any pending ones. Undo Logs: Records associated with a single transaction that contains information about how to undo the latest change by a transaction.","title":"InnoDB"},{"location":"databases_sql/innodb/#why-should-you-use-this","text":"General purpose, row level locking, ACID support, transactions, crash recovery and multi-version concurrency control etc.","title":"Why should you use this?"},{"location":"databases_sql/innodb/#architecture","text":"","title":"Architecture"},{"location":"databases_sql/innodb/#key-components","text":"Memory: Buffer pool: LRU cache of frequently used data(table and index) to be processed directly from memory, which speeds up processing. Important for tuning performance. Change buffer: Caches changes to secondary index pages when those pages are not in the buffer pool and merges it when they are fetched. Merging may take a long time and impact live queries. It also takes up part of the buffer pool. Avoids the extra I/O to read secondary indexes in. Adaptive hash index: Supplements InnoDB\u2019s B-Tree indexes with fast hash lookup tables like a cache. Slight performance penalty for misses, also adds maintenance overhead of updating it. Hash collisions cause AHI rebuilding for large DBs. Log buffer: Holds log data before flush to disk. Size of each above memory is configurable, and impacts performance a lot. Requires careful analysis of workload, available resources, benchmarking and tuning for optimal performance. Disk: Tables: Stores data within rows and columns. Indexes: Helps find rows with specific column values quickly, avoids full table scans. Redo Logs: all transactions are written to them, and after a crash, the recovery process corrects data written by incomplete transactions and replays any pending ones. Undo Logs: Records associated with a single transaction that contains information about how to undo the latest change by a transaction.","title":"Key components:"},{"location":"databases_sql/intro/","text":"Relational Databases Prerequisites Complete Linux course Install Docker (for lab section) What to expect from this course You will have an understanding of what relational databases are, their advantages, and some MySQL specific concepts. What is not covered under this course In depth implementation details Advanced topics like normalization, sharding Specific tools for administration Introduction The main purpose of database systems is to manage data. This includes storage, adding new data, deleting unused data, updating existing data, retrieving data within a reasonable response time, other maintenance tasks to keep the system running etc. Pre-reads RDBMS Concepts Course Contents Key Concepts MySQL Architecture InnoDB Operational Concepts Lab Further Reading","title":"Introduction"},{"location":"databases_sql/intro/#relational-databases","text":"","title":"Relational Databases"},{"location":"databases_sql/intro/#prerequisites","text":"Complete Linux course Install Docker (for lab section)","title":"Prerequisites"},{"location":"databases_sql/intro/#what-to-expect-from-this-course","text":"You will have an understanding of what relational databases are, their advantages, and some MySQL specific concepts.","title":"What to expect from this course"},{"location":"databases_sql/intro/#what-is-not-covered-under-this-course","text":"In depth implementation details Advanced topics like normalization, sharding Specific tools for administration","title":"What is not covered under this course"},{"location":"databases_sql/intro/#introduction","text":"The main purpose of database systems is to manage data. This includes storage, adding new data, deleting unused data, updating existing data, retrieving data within a reasonable response time, other maintenance tasks to keep the system running etc.","title":"Introduction"},{"location":"databases_sql/intro/#pre-reads","text":"RDBMS Concepts","title":"Pre-reads"},{"location":"databases_sql/intro/#course-contents","text":"Key Concepts MySQL Architecture InnoDB Operational Concepts Lab Further Reading","title":"Course Contents"},{"location":"databases_sql/lab/","text":"Prerequisites Install Docker Setup Create a working directory named sos or something similar, and cd into it. Enter the following into a file named my.cnf under a directory named custom. sos $ cat custom/my.cnf [mysqld] # These settings apply to MySQL server # You can set port, socket path, buffer size etc. # Below, we are configuring slow query settings slow_query_log=1 slow_query_log_file=/var/log/mysqlslow.log long_query_time=0.1 Start a container and enable slow query log with the following: sos $ docker run --name db -v custom:/etc/mysql/conf.d -e MYSQL_ROOT_PASSWORD=realsecret -d mysql:8 sos $ docker cp custom/mysqld.cnf $(docker ps -qf \"name=db\"):/etc/mysql/conf.d/custom.cnf sos $ docker restart $(docker ps -qf \"name=db\") Import a sample database sos $ git clone git@github.com:datacharmer/test_db.git sos $ docker cp test_db $(docker ps -qf \"name=db\"):/home/test_db/ sos $ docker exec -it $(docker ps -qf \"name=db\") bash root@3ab5b18b0c7d:/# cd /home/test_db/ root@3ab5b18b0c7d:/# mysql -uroot -prealsecret mysql < employees.sql root@3ab5b18b0c7d:/etc# touch /var/log/mysqlslow.log root@3ab5b18b0c7d:/etc# chown mysql:mysql /var/log/mysqlslow.log Workshop 1: Run some sample queries Run the following $ mysql -uroot -prealsecret mysql mysql> # inspect DBs and tables # the last 4 are MySQL internal DBs mysql> show databases; +--------------------+ | Database | +--------------------+ | employees | | information_schema | | mysql | | performance_schema | | sys | +--------------------+ > use employees; mysql> show tables; +----------------------+ | Tables_in_employees | +----------------------+ | current_dept_emp | | departments | | dept_emp | | dept_emp_latest_date | | dept_manager | | employees | | salaries | | titles | +----------------------+ # read a few rows mysql> select * from employees limit 5; # filter data by conditions mysql> select count(*) from employees where gender = 'M' limit 5; # find count of particular data mysql> select count(*) from employees where first_name = 'Sachin'; Workshop 2: Use explain and explain analyze to profile a query, identify and add indexes required for improving performance # View all indexes on table #(\\G is to output horizontally, replace it with a ; to get table output) mysql> show index from employees from employees\\G *************************** 1. row *************************** Table: employees Non_unique: 0 Key_name: PRIMARY Seq_in_index: 1 Column_name: emp_no Collation: A Cardinality: 299113 Sub_part: NULL Packed: NULL Null: Index_type: BTREE Comment: Index_comment: Visible: YES Expression: NULL # This query uses an index, idenitfied by 'key' field # By prefixing explain keyword to the command, # we get query plan (including key used) mysql> explain select * from employees where emp_no < 10005\\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: employees partitions: NULL type: range possible_keys: PRIMARY key: PRIMARY key_len: 4 ref: NULL rows: 4 filtered: 100.00 Extra: Using where # Compare that to the next query which does not utilize any index mysql> explain select first_name, last_name from employees where first_name = 'Sachin'\\G *************************** 1. row *************************** id: 1 select_type: SIMPLE table: employees partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 299113 filtered: 10.00 Extra: Using where # Let's see how much time this query takes mysql> explain analyze select first_name, last_name from employees where first_name = 'Sachin'\\G *************************** 1. row *************************** EXPLAIN: -> Filter: (employees.first_name = 'Sachin') (cost=30143.55 rows=29911) (actual time=28.284..3952.428 rows=232 loops=1) -> Table scan on employees (cost=30143.55 rows=299113) (actual time=0.095..1996.092 rows=300024 loops=1) # Cost(estimated by query planner) is 30143.55 # actual time=28.284ms for first row, 3952.428 for all rows # Now lets try adding an index and running the query again mysql> create index idx_firstname on employees(first_name); Query OK, 0 rows affected (1.25 sec) Records: 0 Duplicates: 0 Warnings: 0 mysql> explain analyze select first_name, last_name from employees where first_name = 'Sachin'; +--------------------------------------------------------------------------------------------------------------------------------------------+ | EXPLAIN | +--------------------------------------------------------------------------------------------------------------------------------------------+ | -> Index lookup on employees using idx_firstname (first_name='Sachin') (cost=81.20 rows=232) (actual time=0.551..2.934 rows=232 loops=1) | +--------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.01 sec) # Actual time=0.551ms for first row # 2.934ms for all rows. A huge improvement! # Also notice that the query involves only an index lookup, # and no table scan (reading all rows of table) # ..which vastly reduces load on the DB. Workshop 3: Identify slow queries on a MySQL server # Run the command below in two terminal tabs to open two shells into the container. docker exec -it $(docker ps -qf \"name=db\") bash # Open a mysql prompt in one of them and execute this command # We have configured to log queries that take longer than 1s, # so this sleep(3) will be logged mysql -uroot -prealsecret mysql mysql> sleep(3); # Now, in the other terminal, tail the slow log to find details about the query root@62c92c89234d:/etc# tail -f /var/log/mysqlslow.log /usr/sbin/mysqld, Version: 8.0.21 (MySQL Community Server - GPL). started with: Tcp port: 3306 Unix socket: /var/run/mysqld/mysqld.sock Time Id Command Argument # Time: 2020-11-26T14:53:44.822348Z # User@Host: root[root] @ localhost [] Id: 9 # Query_time: 5.404938 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1 use employees; # Time: 2020-11-26T14:53:58.015736Z # User@Host: root[root] @ localhost [] Id: 9 # Query_time: 10.000225 Lock_time: 0.000000 Rows_sent: 1 Rows_examined: 1 SET timestamp=1606402428; select sleep(3); These were simulated examples with minimal complexity. In real life, the queries would be much more complex and the explain/analyze and slow query logs would have more details.","title":"Lab"},{"location":"databases_sql/mysql/","text":"MySQL architecture MySQL architecture enables you to select the right storage engine for your needs, and abstracts away all implementation details from the end users (application engineers and DBA ) who only need to know a consistent stable API. Application layer: Connection handling - each client gets its own connection which is cached for the duration of access) Authentication - server checks (username,password,host) info of client and allows/rejects connection Security: server determines whether the client has privileges to execute each query (check with show privileges command) Server layer: Services and utilities - backup/restore, replication, cluster etc SQL interface - clients run queries for data access and manipulation SQL parser - creates a parse tree from the query (lexical/syntactic/semantic analysis and code generation) Optimizer - optimizes queries using various algorithms and data available to it(table level stats), modifies queries, order of scanning, indexes to use etc. (check with explain command) Caches and buffers - cache stores query results, buffer pool(InnoDB) stores table and index data in LRU fashion Storage engine options: InnoDB: most widely used, transaction support, ACID compliant, supports row-level locking, crash recovery and multi-version concurrency control. Default since MySQL 5.5+. MyISAM: fast, does not support transactions, provides table-level locking, great for read-heavy workloads, mostly in web and data warehousing. Default upto MySQL 5.1. Archive: optimised for high speed inserts, compresses data as it is inserted, does not support transactions, ideal for storing and retrieving large amounts of seldom referenced historical, archived data Memory: tables in memory. Fastest engine, supports table-level locking, does not support transactions, ideal for creating temporary tables or quick lookups, data is lost after a shutdown CSV: stores data in CSV files, great for integrating into other applications that use this format \u2026 etc. It is possible to migrate from one storage engine to another. But this migration locks tables for all operations and is not online, as it changes the physical layout of the data. It takes a long time and is generally not recommended. Hence, choosing the right storage engine at the beginning is important. General guideline is to use InnoDB unless you have a specific need for one of the other storage engines. Running mysql> SHOW ENGINES; shows you the supported engines on your MySQL server.","title":"MySQL"},{"location":"databases_sql/mysql/#mysql-architecture","text":"MySQL architecture enables you to select the right storage engine for your needs, and abstracts away all implementation details from the end users (application engineers and DBA ) who only need to know a consistent stable API. Application layer: Connection handling - each client gets its own connection which is cached for the duration of access) Authentication - server checks (username,password,host) info of client and allows/rejects connection Security: server determines whether the client has privileges to execute each query (check with show privileges command) Server layer: Services and utilities - backup/restore, replication, cluster etc SQL interface - clients run queries for data access and manipulation SQL parser - creates a parse tree from the query (lexical/syntactic/semantic analysis and code generation) Optimizer - optimizes queries using various algorithms and data available to it(table level stats), modifies queries, order of scanning, indexes to use etc. (check with explain command) Caches and buffers - cache stores query results, buffer pool(InnoDB) stores table and index data in LRU fashion Storage engine options: InnoDB: most widely used, transaction support, ACID compliant, supports row-level locking, crash recovery and multi-version concurrency control. Default since MySQL 5.5+. MyISAM: fast, does not support transactions, provides table-level locking, great for read-heavy workloads, mostly in web and data warehousing. Default upto MySQL 5.1. Archive: optimised for high speed inserts, compresses data as it is inserted, does not support transactions, ideal for storing and retrieving large amounts of seldom referenced historical, archived data Memory: tables in memory. Fastest engine, supports table-level locking, does not support transactions, ideal for creating temporary tables or quick lookups, data is lost after a shutdown CSV: stores data in CSV files, great for integrating into other applications that use this format \u2026 etc. It is possible to migrate from one storage engine to another. But this migration locks tables for all operations and is not online, as it changes the physical layout of the data. It takes a long time and is generally not recommended. Hence, choosing the right storage engine at the beginning is important. General guideline is to use InnoDB unless you have a specific need for one of the other storage engines. Running mysql> SHOW ENGINES; shows you the supported engines on your MySQL server.","title":"MySQL architecture"},{"location":"databases_sql/operations/","text":"Explain and explain+analyze EXPLAINYou should be redirected automatically to target URL: https://linkedin.com. If not click the link. \"\"\"","title":"The URL Shortening App"},{"location":"python_web/url-shorten-app/#the-url-shortening-app","text":"Let's build a very simple URL shortening app using flask and try to incorporate all aspects of the development process including the reliability aspects. We will not be building the UI and we will come up with a minimal set of API that will be enough for the app to function well.","title":"The URL Shortening App"},{"location":"python_web/url-shorten-app/#design","text":"We don't jump directly to coding. First thing we do is gather requirements. Come up with an approach. Have the approach/design reviewed by peers. Evolve, iterate, document the decisions and tradeoffs. And then finally implement. While we will not do the full blown design document here, we will raise certain questions here that are important to the design.","title":"Design"},{"location":"python_web/url-shorten-app/#1-high-level-operations-and-api-endpoints","text":"Since it's a URL shortening app, we will need an API for generating the shorten link given an original link. And an API/Endpoint which will accept the shorten link and redirect to original URL. We are not including the user aspect of the app to keep things minimal. These two API should make app functional and usable by anyone.","title":"1. High Level Operations and API Endpoints"},{"location":"python_web/url-shorten-app/#2-how-to-shorten","text":"Given a url, we will need to generate a shortened version of it. One approach could be using random characters for each link. Another thing that can be done is to use some sort of hashing algorithm. The benefit here is we will reuse the same hash for the same link. ie: if lot of people are shortening https://www.linkedin.com they all will have the same value, compared to multiple entries in DB if chosen random characters. What about hash collisions? Even in random characters approach, though there is a less probability, hash collisions can happen. And we need to be mindful of them. In that case we might want to prepend/append the string with some random value to avoid conflict. Also, choice of hash algorithm matters. We will need to analyze algorithms. Their CPU requirements and their characteristics. Choose one that suits the most.","title":"2. How to shorten?"},{"location":"python_web/url-shorten-app/#3-is-url-valid","text":"Given a URL to shorten, how do we verify if the URL is valid? Do we even verify or validate? One basic check that can be done is see if the URL matches a regex of a URL. To go even further we can try opening/visiting the URL. But there are certain gotchas here. We need to define success criteria. ie: HTTP 200 means it is valid. What is the URL is in private network? What if URL is temporarily down?","title":"3. Is URL Valid?"},{"location":"python_web/url-shorten-app/#4-storage","text":"Finally, storage. Where will we store the data that we will generate over time? There are multiple database solutions available and we will need to choose the one that suits this app the most. Relational database like MySQL would be a fair choice but be sure to checkout School of SRE's database section for deeper insights into making a more informed decision.","title":"4. Storage"},{"location":"python_web/url-shorten-app/#5-other","text":"We are not accounting for users into our app and other possible features like rate limiting, customized links etc but it will eventually come up with time. Depending on the requirements, they too might need to get incorporated. The minimal working code is given below for reference but I'd encourage you to come up with your own. from flask import Flask, redirect, request from hashlib import md5 app = Flask(\"url_shortener\") mapping = {} @app.route(\"/shorten\", methods=[\"POST\"]) def shorten(): global mapping payload = request.json if \"url\" not in payload: return \"Missing URL Parameter\", 400 # TODO: check if URL is valid hash_ = md5() hash_.update(payload[\"url\"].encode()) digest = hash_.hexdigest()[:5] # limiting to 5 chars. Less the limit more the chances of collission if digest not in mapping: mapping[digest] = payload[\"url\"] return f\"Shortened: r/{digest}\\n\" else: # TODO: check for hash collission return f\"Already exists: r/{digest}\\n\" @app.route(\"/r/ You should be redirected automatically to target URL: https://linkedin.com. If not click the link. \"\"\"","title":"5. Other"},{"location":"security/conclusion/","text":"Conclusion Now that you have completed this course on Security you are now aware of the possible security threats to computer systems & networks. Not only that, but you are now better able to protect your systems as well as recommend security measures to others. This course provides fundamental everyday knowledge on security domain which will also help you keep security at the top of your priority. Other Resources Some books that would be a great resource Holistic Info-Sec for Web Developers https://holisticinfosecforwebdevelopers.com/ - Free and downloadable book series with very broad and deep coverage of what Web Developers and DevOps Engineers need to know in order to create robust, reliable, maintainable and secure software, networks and other, that are delivered continuously, on time, with no nasty surprises Docker Security - Quick Reference: For DevOps Engineers https://holisticinfosecforwebdevelopers.com/ - A book on understanding the Docker security defaults, how to improve them (theory and practical), along with many tools and techniques. How to Hack Like a Legend https://amzn.to/2uWh1Up - A hacker\u2019s tale breaking into a secretive offshore company, Sparc Flow, 2018 How to Investigate Like a Rockstar https://books2read.com/u/4jDWoZ - Live a real crisis to master the secrets of forensic analysis, Sparc Flow, 2017 Real World Cryptography https://www.manning.com/books/real-world-cryptography - This early-access book teaches you applied cryptographic techniques to understand and apply security at every level of your systems and applications. AWS Security https://www.manning.com/books/aws-security?utm_source=github&utm_medium=organic&utm_campaign=book_shields_aws_1_31_20 - This early-access book covers commong AWS security issues and best practices for access policies, data protection, auditing, continuous monitoring, and incident response. Post Training asks/ Further Reading CTF Events like : https://github.com/apsdehal/awesome-ctf Penetration Testing : https://github.com/enaqx/awesome-pentest Threat Intelligence : https://github.com/hslatman/awesome-threat-intelligence Threat Detection & Hunting : https://github.com/0x4D31/awesome-threat-detection Web Security: https://github.com/qazbnm456/awesome-web-security Building Secure and Reliable Systems : https://landing.google.com/sre/resources/foundationsandprinciples/srs-book/","title":"Conclusion"},{"location":"security/conclusion/#conclusion","text":"Now that you have completed this course on Security you are now aware of the possible security threats to computer systems & networks. Not only that, but you are now better able to protect your systems as well as recommend security measures to others. This course provides fundamental everyday knowledge on security domain which will also help you keep security at the top of your priority.","title":"Conclusion"},{"location":"security/conclusion/#other-resources","text":"Some books that would be a great resource Holistic Info-Sec for Web Developers https://holisticinfosecforwebdevelopers.com/ - Free and downloadable book series with very broad and deep coverage of what Web Developers and DevOps Engineers need to know in order to create robust, reliable, maintainable and secure software, networks and other, that are delivered continuously, on time, with no nasty surprises Docker Security - Quick Reference: For DevOps Engineers https://holisticinfosecforwebdevelopers.com/ - A book on understanding the Docker security defaults, how to improve them (theory and practical), along with many tools and techniques. How to Hack Like a Legend https://amzn.to/2uWh1Up - A hacker\u2019s tale breaking into a secretive offshore company, Sparc Flow, 2018 How to Investigate Like a Rockstar https://books2read.com/u/4jDWoZ - Live a real crisis to master the secrets of forensic analysis, Sparc Flow, 2017 Real World Cryptography https://www.manning.com/books/real-world-cryptography - This early-access book teaches you applied cryptographic techniques to understand and apply security at every level of your systems and applications. AWS Security https://www.manning.com/books/aws-security?utm_source=github&utm_medium=organic&utm_campaign=book_shields_aws_1_31_20 - This early-access book covers commong AWS security issues and best practices for access policies, data protection, auditing, continuous monitoring, and incident response.","title":"Other Resources"},{"location":"security/conclusion/#post-training-asks-further-reading","text":"CTF Events like : https://github.com/apsdehal/awesome-ctf Penetration Testing : https://github.com/enaqx/awesome-pentest Threat Intelligence : https://github.com/hslatman/awesome-threat-intelligence Threat Detection & Hunting : https://github.com/0x4D31/awesome-threat-detection Web Security: https://github.com/qazbnm456/awesome-web-security Building Secure and Reliable Systems : https://landing.google.com/sre/resources/foundationsandprinciples/srs-book/","title":"Post Training asks/ Further Reading"},{"location":"security/fundamentals/","text":"Part I: Fundamentals Introduction to Security Overview for SRE If you look closely, both Site Reliability Engineering and Security Engineering are concerned with keeping a system usable. Issues like broken releases, capacity shortages, and misconfigurations can make a system unusable (at least temporarily). Security or privacy incidents that break the trust of users also undermine the usefulness of a system. Consequently, system security should be top of mind for SREs. SREs should be involved in both significant design discussions and actual system changes. They have quite a big role in System design & hence are quite sometimes the first line of defence. SRE\u2019s help in preventing bad design & implementations which can affect the overall security of the infrastructure. Successfully designing, implementing, and maintaining systems requires a commitment to the full system lifecycle . This commitment is possible only when security and reliability are central elements in the architecture of systems. Core Pillars of Information Security : Confidentiality \u2013 only allow access to data for which the user is permitted Integrity \u2013 ensure data is not tampered or altered by unauthorized users Availability \u2013 ensure systems and data are available to authorized users when they need it Thinking like a Security Engineer When starting a new application or re-factoring an existing application, you should consider each functional feature, and consider: Is the process surrounding this feature as safe as possible? In other words, is this a flawed process? If I were evil, how would I abuse this feature? Or more specifically failing to address how a feature can be abused can cause design flaws. Is the feature required to be on by default? If so, are there limits or options that could help reduce the risk from this feature? Security Principles By OWASP (Open Web Application Security Project) Minimize attack surface area : Every feature that is added to an application adds a certain amount of risk to the overall application. The aim of secure development is to reduce the overall risk by reducing the attack surface area. For example, a web application implements online help with a search function. The search function may be vulnerable to SQL injection attacks. If the help feature was limited to authorized users, the attack likelihood is reduced. If the help feature\u2019s search function was gated through centralized data validation routines, the ability to perform SQL injection is dramatically reduced. However, if the help feature was re-written to eliminate the search function (through a better user interface, for example), this almost eliminates the attack surface area, even if the help feature was available to the Internet at large. Establish secure defaults: There are many ways to deliver an \u201cout of the box\u201d experience for users. However, by default, the experience should be secure, and it should be up to the user to reduce their security \u2013 if they are allowed. For example, by default, password ageing and complexity should be enabled. Users might be allowed to turn these two features off to simplify their use of the application and increase their risk. Default Passwords of routers, IoT devices should be changed Principle of Least privilege The principle of least privilege recommends that accounts have the least amount of privilege required to perform their business processes. This encompasses user rights, resource permissions such as CPU limits, memory, network, and file system permissions. For example, if a middleware server only requires access to the network, read access to a database table, and the ability to write to a log, this describes all the permissions that should be granted. Under no circumstances should the middleware be granted administrative privileges. Principle of Defense in depth The principle of defence in depth suggests that where one control would be reasonable, more controls that approach risks in different fashions are better. Controls, when used in depth, can make severe vulnerabilities extraordinarily difficult to exploit and thus unlikely to occur. With secure coding, this may take the form of tier-based validation, centralized auditing controls, and requiring users to be logged on all pages. For example, a flawed administrative interface is unlikely to be vulnerable to an anonymous attack if it correctly gates access to production management networks, checks for administrative user authorization, and logs all access. Fail securely Applications regularly fail to process transactions for many reasons. How they fail can determine if an application is secure or not. ``` is_admin = true; try { code_which_may_faile(); is_admin = is_user_assigned_role(\"Adminstrator\"); } catch (Exception err) { log.error(err.toString()); } ``` - If either codeWhichMayFail() or isUserInRole fails or throws an exception, the user is an admin by default. This is obviously a security risk. Don\u2019t trust services Many organizations utilize the processing capabilities of third-party partners, who more than likely have different security policies and posture than you. It is unlikely that you can influence or control any external third party, whether they are home users or major suppliers or partners. Therefore, the implicit trust of externally run systems is not warranted. All external systems should be treated similarly. For example, a loyalty program provider provides data that is used by Internet Banking, providing the number of reward points and a small list of potential redemption items. However, the data should be checked to ensure that it is safe to display to end-users and that the reward points are a positive number, and not improbably large. Separation of duties The key to fraud control is the separation of duties. For example, someone who requests a computer cannot also sign for it, nor should they directly receive the computer. This prevents the user from requesting many computers and claiming they never arrived. Certain roles have different levels of trust than normal users. In particular, administrators are different from normal users. In general, administrators should not be users of the application. For example, an administrator should be able to turn the system on or off, set password policy but shouldn\u2019t be able to log on to the storefront as a super privileged user, such as being able to \u201cbuy\u201d goods on behalf of other users. Avoid security by obscurity Security through obscurity is a weak security control, and nearly always fails when it is the only control. This is not to say that keeping secrets is a bad idea, it simply means that the security of systems should not be reliant upon keeping details hidden. For example, the security of an application should not rely upon knowledge of the source code being kept secret. The security should rely upon many other factors, including reasonable password policies, defence in depth, business transaction limits, solid network architecture, and fraud, and audit controls. A practical example is Linux. Linux\u2019s source code is widely available, and yet when properly secured, Linux is a secure and robust operating system. Keep security simple Attack surface area and simplicity go hand in hand. Certain software engineering practices prefer overly complex approaches to what would otherwise be a relatively straightforward and simple design. Developers should avoid the use of double negatives and complex architectures when a simpler approach would be faster and simpler. For example, although it might be fashionable to have a slew of singleton entity beans running on a separate middleware server, it is more secure and faster to simply use global variables with an appropriate mutex mechanism to protect against race conditions. Fix security issues correctly Once a security issue has been identified, it is important to develop a test for it and to understand the root cause of the issue. When design patterns are used, the security issue is likely widespread amongst all codebases, so developing the right fix without introducing regressions is essential. For example, a user has found that they can see another user\u2019s balance by adjusting their cookie. The fix seems to be relatively straightforward, but as the cookie handling code is shared among all applications, a change to just one application will trickle through to all other applications. The fix must, therefore, be tested on all affected applications. Reliability & Security Reliability and security are both crucial components of a truly trustworthy system, but building systems that are both reliable and secure is difficult. While the requirements for reliability and security share many common properties, they also require different design considerations. It is easy to miss the subtle interplay between reliability and security that can cause unexpected outcomes Ex: A password management application failure was triggered by a reliability problem i.e poor load-balancing and load-shedding strategies and its recovery were later complicated by multiple measures (HSM mechanism which needs to be plugged into server racks, which works as an authentication & the HSM token supposedly locked inside a case.. & the problem can be further elongated ) designed to increase the security of the system. Authentication vs Authorization Authentication is the act of validating that users are who they claim to be. Passwords are the most common authentication factor\u2014if a user enters the correct password, the system assumes the identity is valid and grants access. Other technologies such as One-Time Pins, authentication apps, and even biometrics can also be used to authenticate identity. In some instances, systems require the successful verification of more than one factor before granting access. This multi-factor authentication (MFA) requirement is often deployed to increase security beyond what passwords alone can provide. Authorization in system security is the process of giving the user permission to access a specific resource or function. This term is often used interchangeably with access control or client privilege. Giving someone permission to download a particular file on a server or providing individual users with administrative access to an application are good examples. In secure environments, authorization must always follow authentication, users should first prove that their identities are genuine before an organization\u2019s administrators grant them access to the requested resources. Common authentication flow (local authentication) The user registers using an identifier like username/email/mobile The application stores user credentials in the database The application sends a verification email/message to validate the registration Post successful registration, the user enters credentials for logging in On successful authentication, the user is allowed access to specific resources OpenID/OAuth OpenID is an authentication protocol that allows us to authenticate users without using a local auth system. In such a scenario, a user has to be registered with an OpenID Provider and the same provider should be integrated with the authentication flow of your application. To verify the details, we have to forward the authentication requests to the provider. On successful authentication, we receive a success message and/or profile details with which we can execute the necessary flow. OAuth is an authorization mechanism that allows your application user access to a provider(Gmail/Facebook/Instagram/etc). On successful response, we (your application) receive a token with which the application can access certain APIs on behalf of a user. OAuth is convenient in case your business use case requires some certain user-facing APIs like access to Google Drive or sending tweets on your behalf. Most OAuth 2.0 providers can be used for pseudo authentication. Having said that, it can get pretty complicated if you are using multiple OAuth providers to authenticate users on top of the local authentication system. Cryptography It is the science and study of hiding any text in such a way that only the intended recipients or authorized persons can read it and that any text can even use things such as invisible ink or the mechanical cryptography machines of the past. Cryptography is necessary for securing critical or proprietary information and is used to encode private data messages by converting some plain text into ciphertext. At its core, there are two ways of doing this, more advanced methods are all built upon. Ciphers Ciphers are the cornerstone of cryptography. A cipher is a set of algorithms that performs encryption or decryption on a message. An encryption algorithm (E) takes a secret key (k) and a message (m) and produces a ciphertext (c). Similarly, a Decryption algorithm (D) takes a secret key (K) and the previous resulting Ciphertext (C). They are represented as follows: E(k,m) = c D(k,c) = m This also means that for it to be a cipher, it must satisfy the consistency equation as follows, making it possible to decrypt. D(k,E(k,m)) = m Stream Ciphers: The message is broken into characters or bits and enciphered with a key or keystream(should be random and generated independently of the message stream) that is as long as the plaintext bitstream. If the keystream is random, this scheme would be unbreakable unless the keystream was acquired, making it unconditionally secure. The keystream must be provided to both parties in a secure way to prevent its release. Block Ciphers: Block ciphers \u2014 process messages in blocks, each of which is then encrypted or decrypted. A block cipher is a symmetric cipher in which blocks of plaintext are treated as a whole and used to produce ciphertext blocks. The block cipher takes blocks that are b bits long and encrypts them to blocks that are also b bits long. Block sizes are typically 64 or 128 bits long. Encryption Secret Key (Symmetric Key) : the same key is used for encryption and decryption Public Key (Asymmetric Key) in an asymmetric, the encryption and decryption keys are different but related. The encryption key is known as the public key and the decryption key is known as the private key. The public and private keys are known as a key pair. Symmetric Key Encryption DES The Data Encryption Standard (DES) has been the worldwide encryption standard for a long time. IBM developed DES in 1975, and it has held up remarkably well against years of cryptanalysis. DES is a symmetric encryption algorithm with a fixed key length of 56 bits. The algorithm is still good, but because of the short key length, it is susceptible to brute-force attacks that have sufficient resources. DES usually operates in block mode, whereby it encrypts data in 64-bit blocks. The same algorithm and key are used for both encryption and decryption. Because DES is based on simple mathematical functions, it can be easily implemented and accelerated in hardware. Triple DES With advances in computer processing power, the original 56-bit DES key became too short to withstand an attacker with even a limited budget. One way of increasing the effective key length of DES without changing the well-analyzed algorithm itself is to use the same algorithm with different keys several times in a row. The technique of applying DES three times in a row to a plain text block is called Triple DES (3DES). The 3DES technique is shown in Figure. Brute-force attacks on 3DES are considered unfeasible today. Because the basic algorithm has been tested in the field for more than 25 years, it is considered to be more trustworthy than its predecessor. AES On October 2, 2000, The U.S. National Institute of Standards and Technology (NIST) announced the selection of the Rijndael cipher as the AES algorithm. This cipher, developed by Joan Daemen and Vincent Rijmen, has a variable block length and key length. The algorithm currently specifies how to use keys with a length of 128, 192, or 256 bits to encrypt blocks with a length of 128, 192, or 256 bits (all nine combinations of key length and block length are possible). Both block and key lengths can be extended easily to multiples of 32 bits. AES was chosen to replace DES and 3DES because they are either too weak (DES, in terms of key length) or too slow (3DES) to run on modern, efficient hardware. AES is more efficient and much faster, usually by a factor of 5 compared to DES on the same hardware. AES is also more suitable for high throughput, especially if pure software encryption is used. However, AES is a relatively young algorithm, and as the golden rule of cryptography states, \u201cA more mature algorithm is always more trusted.\u201d Asymmetric Key Algorithm In a symmetric key system, Alice first puts the secret message in a box and then padlocks the box using a lock to which she has a key. She then sends the box to Bob through regular mail. When Bob receives the box, he uses an identical copy of Alice's key (which he has obtained previously) to open the box and read the message. In an asymmetric key system, instead of opening the box when he receives it, Bob simply adds his own personal lock to the box and returns the box through public mail to Alice. Alice uses her key to remove her lock and returns the box to Bob, with Bob's lock still in place. Finally, Bob uses his key to remove his lock and reads the message from Alice. The critical advantage in an asymmetric system is that Alice never needs to send a copy of her key to Bob. This reduces the possibility that a third party (for example, an unscrupulous postmaster) can copy the key while it is in transit to Bob, allowing that third party to spy on all future messages sent by Alice. In addition, if Bob is careless and allows someone else to copy his key, Alice's messages to Bob are compromised, but Alice's messages to other people remain secret NOTE : In terms of TLS key exchange, this is the common approach. Diffie-Hellman The protocol has two system parameters, p and g. They are both public and may be used by everybody. Parameter p is a prime number, and parameter g (usually called a generator) is an integer that is smaller than p, but with the following property: For every number n between 1 and p \u2013 1 inclusive, there is a power k of g such that n = gk mod p. Diffie Hellman algorithm is an asymmetric algorithm used to establish a shared secret for a symmetric key algorithm. Nowadays most of the people use hybrid cryptosystem i.e, a combination of symmetric and asymmetric encryption. Asymmetric Encryption is used as a technique in key exchange mechanism to share a secret key and after the key is shared between sender and receiver, the communication will take place using symmetric encryption. The shared secret key will be used to encrypt the communication. Refer: https://medium.com/@akhigbemmanuel/what-is-the-diffie-hellman-key-exchange-algorithm-84d60025a30d RSA The RSA algorithm is very flexible and has a variable key length where, if necessary, speed can be traded for the level of security of the algorithm. The RSA keys are usually 512 to 2048 bits long. RSA has withstood years of extensive cryptanalysis. Although those years neither proved nor disproved RSA's security, they attest to a confidence level in the algorithm. RSA security is based on the difficulty of factoring very large numbers. If an easy method of factoring these large numbers were discovered, the effectiveness of RSA would be destroyed. Refer: https://medium.com/curiositypapers/a-complete-explanation-of-rsa-asymmetric-encryption-742c5971e0f NOTE : RSA Keys can be used for key exchange just like Diffie Hellman Hashing Algorithms Hashing is one of the mechanisms used for data integrity assurance. Hashing is based on a one-way mathematical function, which is relatively easy to compute but significantly harder to reverse. A hash function, which is a one-way function to input data to produce a fixed-length digest (fingerprint) of output data. The digest is cryptographically strong; that is, it is impossible to recover input data from its digest. If the input data changes just a little, the digest (fingerprint) changes substantially in what is called an avalanche effect. More: https://medium.com/@rauljordan/the-state-of-hashing-algorithms-the-why-the-how-and-the-future-b21d5c0440de https://medium.com/@StevieCEllis/the-beautiful-hash-algorithm-f18d9d2b84fb MD5 MD5 is a one-way function with which it is easy to compute the hash from the given input data, but it is unfeasible to compute input data given only a hash. SHA-1 MD5 is considered less secure than SHA-1 because MD5 has some weaknesses. HA-1 also uses a stronger, 160-bit digest, which makes MD5 the second choice as hash methods are concerned. The algorithm takes a message of less than 264 bits in length and produces a 160-bit message digest. This algorithm is slightly slower than MD5. NOTE : SHA-1 is also recently demonstrated to be broken, Minimum current recommendation is SHA-256 Digital Certificates Digital signatures, provide a means to digitally authenticate devices and individual users. In public-key cryptography, such as the RSA encryption system, each user has a key-pair containing both a public key and a private key. The keys act as complements, and anything encrypted with one of the keys can be decrypted with the other. In simple terms, a signature is formed when data is encrypted with a user's private key. The receiver verifies the signature by decrypting the message with the sender's public key. Key management is often considered the most difficult task in designing and implementing cryptographic systems. Businesses can simplify some of the deployment and management issues that are encountered with secured data communications by employing a Public Key Infrastructure (PKI). Because corporations often move security-sensitive communications across the Internet, an effective mechanism must be implemented to protect sensitive information from the threats presented on the Internet. PKI provides a hierarchical framework for managing digital security attributes. Each PKI participant holds a digital certificate that has been issued by a CA (either public or private). The certificate contains several attributes that are used when parties negotiate a secure connection. These attributes must include the certificate validity period, end-host identity information, encryption keys that will be used for secure communications, and the signature of the issuing CA. Optional attributes may be included, depending on the requirements and capability of the PKI. A CA can be a trusted third party, such as VeriSign or Entrust, or a private (in-house) CA that you establish within your organization. The fact that the message could be decrypted using the sender's public key means that the holder of the private key created the message. This process relies on the receiver having a copy of the sender's public key and knowing with a high degree of certainty that it really does belong to the sender and not to someone pretending to be the sender. To validate the CA's signature, the receiver must know the CA's public key. Normally, this is handled out-of-band or through an operation performed during the installation of the certificate. For instance, most web browsers are configured with the root certificates of several CAs by default. CA Enrollment process The end host generates a private-public key pair. The end host generates a certificate request, which it forwards to the CA. Manual human intervention is required to approve the enrollment request, which is received by the CA. After the CA operator approves the request, the CA signs the certificate request with its private key and returns the completed certificate to the end host. The end host writes the certificate into a nonvolatile storage area (PC hard disk or NVRAM on Cisco routers). Refer : https://www.ssh.com/manuals/server-zos-product/55/ch06s03s01.html Login Security SSH SSH, the Secure Shell, is a popular, powerful, software-based approach to network security. Whenever data is sent by a computer to the network, SSH automatically encrypts (scrambles) it. Then, when the data reaches its intended recipient, SSH automatically decrypts (unscrambles) it. The result is transparent encryption: users can work normally, unaware that their communications are safely encrypted on the network. In addition, SSH can use modern, secure encryption algorithms based on how it's being configured and is effective enough to be found within mission-critical applications at major corporations. SSH has a client/server architecture An SSH server program, typically installed and run by a system administrator, accepts or rejects incoming connections to its host computer. Users then run SSH client programs, typically on other computers, to make requests of the SSH server, such as \u201cPlease log me in,\u201d \u201cPlease send me a file,\u201d or \u201cPlease execute this command.\u201d All communications between clients and servers are securely encrypted and protected from modification. What SSH is not: Although SSH stands for Secure Shell, it is not a true shell in the sense of the Unix Bourne shell and C shell. It is not a command interpreter, nor does it provide wildcard expansion, command history, and so forth. Rather, SSH creates a channel for running a shell on a remote computer, with end-to-end encryption between the two systems. The major features and guarantees of the SSH protocol are: Privacy of your data, via strong encryption Integrity of communications, guaranteeing they haven\u2019t been altered Authentication, i.e., proof of identity of senders and receivers Authorization, i.e., access control to accounts Forwarding or tunnelling to encrypt other TCP/IP-based sessions Kerberos According to Greek mythology Kerberos (Cerberus) was the gigantic, three-headed dog that guards the gates of the underworld to prevent the dead from leaving. So when it comes to Computer Science, Kerberos is a network authentication protocol and is currently the default authentication technology used by Microsoft Active Directory to authenticate users to services within a local area network. Kerberos uses symmetric-key cryptography and requires a trusted third-party authentication service to verify user identities. So they used the name of Kerberos for their computer network authentication protocol as the three heads of the Kerberos represent: a client: A user/ a service a server: Kerberos protected hosts reside - a Key Distribution Center (KDC), which acts as the trusted third-party authentication service. The KDC includes the following two servers: Authentication Server (AS) that performs the initial authentication and issues ticket-granting tickets (TGT) for users. Ticket-Granting Server (TGS) that issues service tickets that are based on the initial ticket-granting tickets (TGT). Certificate Chain The first part of the output of the OpenSSL command shows three certificates numbered 0, 1, and 2(not 2 anymore). Each certificate has a subject, s, and an issuer, i. The first certificate, number 0, is called the end-entity certificate. The subject line tells us it\u2019s valid for any subdomain of google.com because its subject is set to *.google.com. $ openssl s_client -connect www.google.com:443 -CApath /etc/ssl/certs CONNECTED(00000005) depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign verify return:1 depth=1 C = US, O = Google Trust Services, CN = GTS CA 1O1 verify return:1 depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = www.google.com verify return:1 --- Certificate chain 0 s:/C=US/ST=California/L=Mountain View/O=Google LLC/CN=www.google.com i:/C=US/O=Google Trust Services/CN=GTS CA 1O1 1 s:/C=US/O=Google Trust Services/CN=GTS CA 1O1 i:/OU=GlobalSign Root CA - R2/O=GlobalSign/CN=GlobalSign --- Server certificate The issuer line indicates it\u2019s issued by Google Internet Authority G2, which also happens to be the subject of the second certificate, number 1 What the OpenSSL command line doesn\u2019t show here is the trust store that contains the list of CA certificates trusted by the system OpenSSL runs on. The public certificate of GlobalSign Authority must be present in the system\u2019s trust store to close the verification chain. This is called a chain of trust, and the figure below summarizes its behaviour at a high level. High-level view of the concept of chain of trust applied to verifying the authenticity of a website. The Root CA in the Firefox trust store provides the initial trust to verify the entire chain and trust the end-entity certificate. TLS Handshake The client sends a HELLO message to the server with a list of protocols and algorithms it supports. The server says HELLO back and sends its chain of certificates. Based on the capabilities of the client, the server picks a cipher suite. If the cipher suite supports ephemeral key exchange, like ECDHE does(ECDHE is an algorithm known as the Elliptic Curve Diffie-Hellman Exchange), the server and the client negotiate a pre-master key with the Diffie-Hellman algorithm. The pre-master key is never sent over the wire. The client and server create a session key that will be used to encrypt the data transiting through the connection. At the end of the handshake, both parties possess a secret session key used to encrypt data for the rest of the connection. This is what OpenSSL refers to as Master-Key NOTE There are 3 versions of TLS , TLS 1.0, 1.1 & 1.2 TLS 1.0 was released in 1999, making it a nearly two-decade-old protocol. It has been known to be vulnerable to attacks\u2014such as BEAST and POODLE\u2014for years, in addition to supporting weak cryptography, which doesn\u2019t keep modern-day connections sufficiently secure. TLS 1.1 is the forgotten \u201cmiddle child.\u201d It also has bad cryptography like its younger sibling. In most software, it was leapfrogged by TLS 1.2 and it\u2019s rare to see TLS 1.1 used. \u201cPerfect\u201d Forward Secrecy The term \u201cephemeral\u201d in the key exchange provides an important security feature mis-named perfect forward secrecy (PFS) or just \u201cForward Secrecy\u201d. In a non-ephemeral key exchange, the client sends the pre-master key to the server by encrypting it with the server\u2019s public key. The server then decrypts the pre-master key with its private key. If at a later point in time, the private key of the server is compromised, an attacker can go back to this handshake, decrypt the pre-master key, obtain the session key, and decrypt the entire traffic. Non-ephemeral key exchanges are vulnerable to attacks that may happen in the future on recorded traffic. And because people seldom change their password, decrypting data from the past may still be valuable for an attacker. An ephemeral key exchange like DHE, or its variant on elliptic curve, ECDHE, solves this problem by not transmitting the pre-master key over the wire. Instead, the pre-master key is computed by both the client and the server in isolation, using nonsensitive information exchanged publicly. Because the pre-master key can\u2019t be decrypted later by an attacker, the session key is safe from future attacks: hence, the term perfect forward secrecy. Keys are changed every X blocks along the stream. That prevents an attacker from simply sniffing the stream and applying brute force to crack the whole thing. \"Forward secrecy\" means that just because I can decrypt block M, does not mean that I can decrypt block Q Downside: The downside to PFS is that all those extra computational steps induce latency on the handshake and slow the user down. To avoid repeating this expensive work at every connection, both sides cache the session key for future use via a technique called session resumption. This is what the session-ID and TLS ticket are for: they allow a client and server that share a session ID to skip over the negotiation of a session key, because they already agreed on one previously, and go directly to exchanging data securely.","title":"Fundamentals of Security"},{"location":"security/fundamentals/#part-i-fundamentals","text":"","title":"Part I: Fundamentals"},{"location":"security/fundamentals/#introduction-to-security-overview-for-sre","text":"If you look closely, both Site Reliability Engineering and Security Engineering are concerned with keeping a system usable. Issues like broken releases, capacity shortages, and misconfigurations can make a system unusable (at least temporarily). Security or privacy incidents that break the trust of users also undermine the usefulness of a system. Consequently, system security should be top of mind for SREs. SREs should be involved in both significant design discussions and actual system changes. They have quite a big role in System design & hence are quite sometimes the first line of defence. SRE\u2019s help in preventing bad design & implementations which can affect the overall security of the infrastructure. Successfully designing, implementing, and maintaining systems requires a commitment to the full system lifecycle . This commitment is possible only when security and reliability are central elements in the architecture of systems. Core Pillars of Information Security : Confidentiality \u2013 only allow access to data for which the user is permitted Integrity \u2013 ensure data is not tampered or altered by unauthorized users Availability \u2013 ensure systems and data are available to authorized users when they need it Thinking like a Security Engineer When starting a new application or re-factoring an existing application, you should consider each functional feature, and consider: Is the process surrounding this feature as safe as possible? In other words, is this a flawed process? If I were evil, how would I abuse this feature? Or more specifically failing to address how a feature can be abused can cause design flaws. Is the feature required to be on by default? If so, are there limits or options that could help reduce the risk from this feature? Security Principles By OWASP (Open Web Application Security Project) Minimize attack surface area : Every feature that is added to an application adds a certain amount of risk to the overall application. The aim of secure development is to reduce the overall risk by reducing the attack surface area. For example, a web application implements online help with a search function. The search function may be vulnerable to SQL injection attacks. If the help feature was limited to authorized users, the attack likelihood is reduced. If the help feature\u2019s search function was gated through centralized data validation routines, the ability to perform SQL injection is dramatically reduced. However, if the help feature was re-written to eliminate the search function (through a better user interface, for example), this almost eliminates the attack surface area, even if the help feature was available to the Internet at large. Establish secure defaults: There are many ways to deliver an \u201cout of the box\u201d experience for users. However, by default, the experience should be secure, and it should be up to the user to reduce their security \u2013 if they are allowed. For example, by default, password ageing and complexity should be enabled. Users might be allowed to turn these two features off to simplify their use of the application and increase their risk. Default Passwords of routers, IoT devices should be changed Principle of Least privilege The principle of least privilege recommends that accounts have the least amount of privilege required to perform their business processes. This encompasses user rights, resource permissions such as CPU limits, memory, network, and file system permissions. For example, if a middleware server only requires access to the network, read access to a database table, and the ability to write to a log, this describes all the permissions that should be granted. Under no circumstances should the middleware be granted administrative privileges. Principle of Defense in depth The principle of defence in depth suggests that where one control would be reasonable, more controls that approach risks in different fashions are better. Controls, when used in depth, can make severe vulnerabilities extraordinarily difficult to exploit and thus unlikely to occur. With secure coding, this may take the form of tier-based validation, centralized auditing controls, and requiring users to be logged on all pages. For example, a flawed administrative interface is unlikely to be vulnerable to an anonymous attack if it correctly gates access to production management networks, checks for administrative user authorization, and logs all access. Fail securely Applications regularly fail to process transactions for many reasons. How they fail can determine if an application is secure or not. ``` is_admin = true; try { code_which_may_faile(); is_admin = is_user_assigned_role(\"Adminstrator\"); } catch (Exception err) { log.error(err.toString()); } ``` - If either codeWhichMayFail() or isUserInRole fails or throws an exception, the user is an admin by default. This is obviously a security risk. Don\u2019t trust services Many organizations utilize the processing capabilities of third-party partners, who more than likely have different security policies and posture than you. It is unlikely that you can influence or control any external third party, whether they are home users or major suppliers or partners. Therefore, the implicit trust of externally run systems is not warranted. All external systems should be treated similarly. For example, a loyalty program provider provides data that is used by Internet Banking, providing the number of reward points and a small list of potential redemption items. However, the data should be checked to ensure that it is safe to display to end-users and that the reward points are a positive number, and not improbably large. Separation of duties The key to fraud control is the separation of duties. For example, someone who requests a computer cannot also sign for it, nor should they directly receive the computer. This prevents the user from requesting many computers and claiming they never arrived. Certain roles have different levels of trust than normal users. In particular, administrators are different from normal users. In general, administrators should not be users of the application. For example, an administrator should be able to turn the system on or off, set password policy but shouldn\u2019t be able to log on to the storefront as a super privileged user, such as being able to \u201cbuy\u201d goods on behalf of other users. Avoid security by obscurity Security through obscurity is a weak security control, and nearly always fails when it is the only control. This is not to say that keeping secrets is a bad idea, it simply means that the security of systems should not be reliant upon keeping details hidden. For example, the security of an application should not rely upon knowledge of the source code being kept secret. The security should rely upon many other factors, including reasonable password policies, defence in depth, business transaction limits, solid network architecture, and fraud, and audit controls. A practical example is Linux. Linux\u2019s source code is widely available, and yet when properly secured, Linux is a secure and robust operating system. Keep security simple Attack surface area and simplicity go hand in hand. Certain software engineering practices prefer overly complex approaches to what would otherwise be a relatively straightforward and simple design. Developers should avoid the use of double negatives and complex architectures when a simpler approach would be faster and simpler. For example, although it might be fashionable to have a slew of singleton entity beans running on a separate middleware server, it is more secure and faster to simply use global variables with an appropriate mutex mechanism to protect against race conditions. Fix security issues correctly Once a security issue has been identified, it is important to develop a test for it and to understand the root cause of the issue. When design patterns are used, the security issue is likely widespread amongst all codebases, so developing the right fix without introducing regressions is essential. For example, a user has found that they can see another user\u2019s balance by adjusting their cookie. The fix seems to be relatively straightforward, but as the cookie handling code is shared among all applications, a change to just one application will trickle through to all other applications. The fix must, therefore, be tested on all affected applications. Reliability & Security Reliability and security are both crucial components of a truly trustworthy system, but building systems that are both reliable and secure is difficult. While the requirements for reliability and security share many common properties, they also require different design considerations. It is easy to miss the subtle interplay between reliability and security that can cause unexpected outcomes Ex: A password management application failure was triggered by a reliability problem i.e poor load-balancing and load-shedding strategies and its recovery were later complicated by multiple measures (HSM mechanism which needs to be plugged into server racks, which works as an authentication & the HSM token supposedly locked inside a case.. & the problem can be further elongated ) designed to increase the security of the system.","title":"Introduction to Security Overview for SRE"},{"location":"security/fundamentals/#authentication-vs-authorization","text":"Authentication is the act of validating that users are who they claim to be. Passwords are the most common authentication factor\u2014if a user enters the correct password, the system assumes the identity is valid and grants access. Other technologies such as One-Time Pins, authentication apps, and even biometrics can also be used to authenticate identity. In some instances, systems require the successful verification of more than one factor before granting access. This multi-factor authentication (MFA) requirement is often deployed to increase security beyond what passwords alone can provide. Authorization in system security is the process of giving the user permission to access a specific resource or function. This term is often used interchangeably with access control or client privilege. Giving someone permission to download a particular file on a server or providing individual users with administrative access to an application are good examples. In secure environments, authorization must always follow authentication, users should first prove that their identities are genuine before an organization\u2019s administrators grant them access to the requested resources.","title":"Authentication vs Authorization"},{"location":"security/fundamentals/#common-authentication-flow-local-authentication","text":"The user registers using an identifier like username/email/mobile The application stores user credentials in the database The application sends a verification email/message to validate the registration Post successful registration, the user enters credentials for logging in On successful authentication, the user is allowed access to specific resources","title":"Common authentication flow (local authentication)"},{"location":"security/fundamentals/#openidoauth","text":"OpenID is an authentication protocol that allows us to authenticate users without using a local auth system. In such a scenario, a user has to be registered with an OpenID Provider and the same provider should be integrated with the authentication flow of your application. To verify the details, we have to forward the authentication requests to the provider. On successful authentication, we receive a success message and/or profile details with which we can execute the necessary flow. OAuth is an authorization mechanism that allows your application user access to a provider(Gmail/Facebook/Instagram/etc). On successful response, we (your application) receive a token with which the application can access certain APIs on behalf of a user. OAuth is convenient in case your business use case requires some certain user-facing APIs like access to Google Drive or sending tweets on your behalf. Most OAuth 2.0 providers can be used for pseudo authentication. Having said that, it can get pretty complicated if you are using multiple OAuth providers to authenticate users on top of the local authentication system.","title":"OpenID/OAuth"},{"location":"security/fundamentals/#cryptography","text":"It is the science and study of hiding any text in such a way that only the intended recipients or authorized persons can read it and that any text can even use things such as invisible ink or the mechanical cryptography machines of the past. Cryptography is necessary for securing critical or proprietary information and is used to encode private data messages by converting some plain text into ciphertext. At its core, there are two ways of doing this, more advanced methods are all built upon.","title":"Cryptography"},{"location":"security/fundamentals/#ciphers","text":"Ciphers are the cornerstone of cryptography. A cipher is a set of algorithms that performs encryption or decryption on a message. An encryption algorithm (E) takes a secret key (k) and a message (m) and produces a ciphertext (c). Similarly, a Decryption algorithm (D) takes a secret key (K) and the previous resulting Ciphertext (C). They are represented as follows: E(k,m) = c D(k,c) = m This also means that for it to be a cipher, it must satisfy the consistency equation as follows, making it possible to decrypt. D(k,E(k,m)) = m Stream Ciphers: The message is broken into characters or bits and enciphered with a key or keystream(should be random and generated independently of the message stream) that is as long as the plaintext bitstream. If the keystream is random, this scheme would be unbreakable unless the keystream was acquired, making it unconditionally secure. The keystream must be provided to both parties in a secure way to prevent its release. Block Ciphers: Block ciphers \u2014 process messages in blocks, each of which is then encrypted or decrypted. A block cipher is a symmetric cipher in which blocks of plaintext are treated as a whole and used to produce ciphertext blocks. The block cipher takes blocks that are b bits long and encrypts them to blocks that are also b bits long. Block sizes are typically 64 or 128 bits long. Encryption Secret Key (Symmetric Key) : the same key is used for encryption and decryption Public Key (Asymmetric Key) in an asymmetric, the encryption and decryption keys are different but related. The encryption key is known as the public key and the decryption key is known as the private key. The public and private keys are known as a key pair. Symmetric Key Encryption DES The Data Encryption Standard (DES) has been the worldwide encryption standard for a long time. IBM developed DES in 1975, and it has held up remarkably well against years of cryptanalysis. DES is a symmetric encryption algorithm with a fixed key length of 56 bits. The algorithm is still good, but because of the short key length, it is susceptible to brute-force attacks that have sufficient resources. DES usually operates in block mode, whereby it encrypts data in 64-bit blocks. The same algorithm and key are used for both encryption and decryption. Because DES is based on simple mathematical functions, it can be easily implemented and accelerated in hardware. Triple DES With advances in computer processing power, the original 56-bit DES key became too short to withstand an attacker with even a limited budget. One way of increasing the effective key length of DES without changing the well-analyzed algorithm itself is to use the same algorithm with different keys several times in a row. The technique of applying DES three times in a row to a plain text block is called Triple DES (3DES). The 3DES technique is shown in Figure. Brute-force attacks on 3DES are considered unfeasible today. Because the basic algorithm has been tested in the field for more than 25 years, it is considered to be more trustworthy than its predecessor. AES On October 2, 2000, The U.S. National Institute of Standards and Technology (NIST) announced the selection of the Rijndael cipher as the AES algorithm. This cipher, developed by Joan Daemen and Vincent Rijmen, has a variable block length and key length. The algorithm currently specifies how to use keys with a length of 128, 192, or 256 bits to encrypt blocks with a length of 128, 192, or 256 bits (all nine combinations of key length and block length are possible). Both block and key lengths can be extended easily to multiples of 32 bits. AES was chosen to replace DES and 3DES because they are either too weak (DES, in terms of key length) or too slow (3DES) to run on modern, efficient hardware. AES is more efficient and much faster, usually by a factor of 5 compared to DES on the same hardware. AES is also more suitable for high throughput, especially if pure software encryption is used. However, AES is a relatively young algorithm, and as the golden rule of cryptography states, \u201cA more mature algorithm is always more trusted.\u201d Asymmetric Key Algorithm In a symmetric key system, Alice first puts the secret message in a box and then padlocks the box using a lock to which she has a key. She then sends the box to Bob through regular mail. When Bob receives the box, he uses an identical copy of Alice's key (which he has obtained previously) to open the box and read the message. In an asymmetric key system, instead of opening the box when he receives it, Bob simply adds his own personal lock to the box and returns the box through public mail to Alice. Alice uses her key to remove her lock and returns the box to Bob, with Bob's lock still in place. Finally, Bob uses his key to remove his lock and reads the message from Alice. The critical advantage in an asymmetric system is that Alice never needs to send a copy of her key to Bob. This reduces the possibility that a third party (for example, an unscrupulous postmaster) can copy the key while it is in transit to Bob, allowing that third party to spy on all future messages sent by Alice. In addition, if Bob is careless and allows someone else to copy his key, Alice's messages to Bob are compromised, but Alice's messages to other people remain secret NOTE : In terms of TLS key exchange, this is the common approach. Diffie-Hellman The protocol has two system parameters, p and g. They are both public and may be used by everybody. Parameter p is a prime number, and parameter g (usually called a generator) is an integer that is smaller than p, but with the following property: For every number n between 1 and p \u2013 1 inclusive, there is a power k of g such that n = gk mod p. Diffie Hellman algorithm is an asymmetric algorithm used to establish a shared secret for a symmetric key algorithm. Nowadays most of the people use hybrid cryptosystem i.e, a combination of symmetric and asymmetric encryption. Asymmetric Encryption is used as a technique in key exchange mechanism to share a secret key and after the key is shared between sender and receiver, the communication will take place using symmetric encryption. The shared secret key will be used to encrypt the communication. Refer: https://medium.com/@akhigbemmanuel/what-is-the-diffie-hellman-key-exchange-algorithm-84d60025a30d RSA The RSA algorithm is very flexible and has a variable key length where, if necessary, speed can be traded for the level of security of the algorithm. The RSA keys are usually 512 to 2048 bits long. RSA has withstood years of extensive cryptanalysis. Although those years neither proved nor disproved RSA's security, they attest to a confidence level in the algorithm. RSA security is based on the difficulty of factoring very large numbers. If an easy method of factoring these large numbers were discovered, the effectiveness of RSA would be destroyed. Refer: https://medium.com/curiositypapers/a-complete-explanation-of-rsa-asymmetric-encryption-742c5971e0f NOTE : RSA Keys can be used for key exchange just like Diffie Hellman Hashing Algorithms Hashing is one of the mechanisms used for data integrity assurance. Hashing is based on a one-way mathematical function, which is relatively easy to compute but significantly harder to reverse. A hash function, which is a one-way function to input data to produce a fixed-length digest (fingerprint) of output data. The digest is cryptographically strong; that is, it is impossible to recover input data from its digest. If the input data changes just a little, the digest (fingerprint) changes substantially in what is called an avalanche effect. More: https://medium.com/@rauljordan/the-state-of-hashing-algorithms-the-why-the-how-and-the-future-b21d5c0440de https://medium.com/@StevieCEllis/the-beautiful-hash-algorithm-f18d9d2b84fb MD5 MD5 is a one-way function with which it is easy to compute the hash from the given input data, but it is unfeasible to compute input data given only a hash. SHA-1 MD5 is considered less secure than SHA-1 because MD5 has some weaknesses. HA-1 also uses a stronger, 160-bit digest, which makes MD5 the second choice as hash methods are concerned. The algorithm takes a message of less than 264 bits in length and produces a 160-bit message digest. This algorithm is slightly slower than MD5. NOTE : SHA-1 is also recently demonstrated to be broken, Minimum current recommendation is SHA-256 Digital Certificates Digital signatures, provide a means to digitally authenticate devices and individual users. In public-key cryptography, such as the RSA encryption system, each user has a key-pair containing both a public key and a private key. The keys act as complements, and anything encrypted with one of the keys can be decrypted with the other. In simple terms, a signature is formed when data is encrypted with a user's private key. The receiver verifies the signature by decrypting the message with the sender's public key. Key management is often considered the most difficult task in designing and implementing cryptographic systems. Businesses can simplify some of the deployment and management issues that are encountered with secured data communications by employing a Public Key Infrastructure (PKI). Because corporations often move security-sensitive communications across the Internet, an effective mechanism must be implemented to protect sensitive information from the threats presented on the Internet. PKI provides a hierarchical framework for managing digital security attributes. Each PKI participant holds a digital certificate that has been issued by a CA (either public or private). The certificate contains several attributes that are used when parties negotiate a secure connection. These attributes must include the certificate validity period, end-host identity information, encryption keys that will be used for secure communications, and the signature of the issuing CA. Optional attributes may be included, depending on the requirements and capability of the PKI. A CA can be a trusted third party, such as VeriSign or Entrust, or a private (in-house) CA that you establish within your organization. The fact that the message could be decrypted using the sender's public key means that the holder of the private key created the message. This process relies on the receiver having a copy of the sender's public key and knowing with a high degree of certainty that it really does belong to the sender and not to someone pretending to be the sender. To validate the CA's signature, the receiver must know the CA's public key. Normally, this is handled out-of-band or through an operation performed during the installation of the certificate. For instance, most web browsers are configured with the root certificates of several CAs by default. CA Enrollment process The end host generates a private-public key pair. The end host generates a certificate request, which it forwards to the CA. Manual human intervention is required to approve the enrollment request, which is received by the CA. After the CA operator approves the request, the CA signs the certificate request with its private key and returns the completed certificate to the end host. The end host writes the certificate into a nonvolatile storage area (PC hard disk or NVRAM on Cisco routers). Refer : https://www.ssh.com/manuals/server-zos-product/55/ch06s03s01.html","title":"Ciphers"},{"location":"security/fundamentals/#login-security","text":"","title":"Login Security"},{"location":"security/fundamentals/#ssh","text":"SSH, the Secure Shell, is a popular, powerful, software-based approach to network security. Whenever data is sent by a computer to the network, SSH automatically encrypts (scrambles) it. Then, when the data reaches its intended recipient, SSH automatically decrypts (unscrambles) it. The result is transparent encryption: users can work normally, unaware that their communications are safely encrypted on the network. In addition, SSH can use modern, secure encryption algorithms based on how it's being configured and is effective enough to be found within mission-critical applications at major corporations. SSH has a client/server architecture An SSH server program, typically installed and run by a system administrator, accepts or rejects incoming connections to its host computer. Users then run SSH client programs, typically on other computers, to make requests of the SSH server, such as \u201cPlease log me in,\u201d \u201cPlease send me a file,\u201d or \u201cPlease execute this command.\u201d All communications between clients and servers are securely encrypted and protected from modification. What SSH is not: Although SSH stands for Secure Shell, it is not a true shell in the sense of the Unix Bourne shell and C shell. It is not a command interpreter, nor does it provide wildcard expansion, command history, and so forth. Rather, SSH creates a channel for running a shell on a remote computer, with end-to-end encryption between the two systems. The major features and guarantees of the SSH protocol are: Privacy of your data, via strong encryption Integrity of communications, guaranteeing they haven\u2019t been altered Authentication, i.e., proof of identity of senders and receivers Authorization, i.e., access control to accounts Forwarding or tunnelling to encrypt other TCP/IP-based sessions","title":"SSH"},{"location":"security/fundamentals/#kerberos","text":"According to Greek mythology Kerberos (Cerberus) was the gigantic, three-headed dog that guards the gates of the underworld to prevent the dead from leaving. So when it comes to Computer Science, Kerberos is a network authentication protocol and is currently the default authentication technology used by Microsoft Active Directory to authenticate users to services within a local area network. Kerberos uses symmetric-key cryptography and requires a trusted third-party authentication service to verify user identities. So they used the name of Kerberos for their computer network authentication protocol as the three heads of the Kerberos represent: a client: A user/ a service a server: Kerberos protected hosts reside - a Key Distribution Center (KDC), which acts as the trusted third-party authentication service. The KDC includes the following two servers: Authentication Server (AS) that performs the initial authentication and issues ticket-granting tickets (TGT) for users. Ticket-Granting Server (TGS) that issues service tickets that are based on the initial ticket-granting tickets (TGT).","title":"Kerberos"},{"location":"security/fundamentals/#certificate-chain","text":"The first part of the output of the OpenSSL command shows three certificates numbered 0, 1, and 2(not 2 anymore). Each certificate has a subject, s, and an issuer, i. The first certificate, number 0, is called the end-entity certificate. The subject line tells us it\u2019s valid for any subdomain of google.com because its subject is set to *.google.com. $ openssl s_client -connect www.google.com:443 -CApath /etc/ssl/certs CONNECTED(00000005) depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign verify return:1 depth=1 C = US, O = Google Trust Services, CN = GTS CA 1O1 verify return:1 depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = www.google.com verify return:1 --- Certificate chain 0 s:/C=US/ST=California/L=Mountain View/O=Google LLC/CN=www.google.com i:/C=US/O=Google Trust Services/CN=GTS CA 1O1 1 s:/C=US/O=Google Trust Services/CN=GTS CA 1O1 i:/OU=GlobalSign Root CA - R2/O=GlobalSign/CN=GlobalSign --- Server certificate The issuer line indicates it\u2019s issued by Google Internet Authority G2, which also happens to be the subject of the second certificate, number 1 What the OpenSSL command line doesn\u2019t show here is the trust store that contains the list of CA certificates trusted by the system OpenSSL runs on. The public certificate of GlobalSign Authority must be present in the system\u2019s trust store to close the verification chain. This is called a chain of trust, and the figure below summarizes its behaviour at a high level. High-level view of the concept of chain of trust applied to verifying the authenticity of a website. The Root CA in the Firefox trust store provides the initial trust to verify the entire chain and trust the end-entity certificate.","title":"Certificate Chain"},{"location":"security/fundamentals/#tls-handshake","text":"The client sends a HELLO message to the server with a list of protocols and algorithms it supports. The server says HELLO back and sends its chain of certificates. Based on the capabilities of the client, the server picks a cipher suite. If the cipher suite supports ephemeral key exchange, like ECDHE does(ECDHE is an algorithm known as the Elliptic Curve Diffie-Hellman Exchange), the server and the client negotiate a pre-master key with the Diffie-Hellman algorithm. The pre-master key is never sent over the wire. The client and server create a session key that will be used to encrypt the data transiting through the connection. At the end of the handshake, both parties possess a secret session key used to encrypt data for the rest of the connection. This is what OpenSSL refers to as Master-Key NOTE There are 3 versions of TLS , TLS 1.0, 1.1 & 1.2 TLS 1.0 was released in 1999, making it a nearly two-decade-old protocol. It has been known to be vulnerable to attacks\u2014such as BEAST and POODLE\u2014for years, in addition to supporting weak cryptography, which doesn\u2019t keep modern-day connections sufficiently secure. TLS 1.1 is the forgotten \u201cmiddle child.\u201d It also has bad cryptography like its younger sibling. In most software, it was leapfrogged by TLS 1.2 and it\u2019s rare to see TLS 1.1 used.","title":"TLS Handshake"},{"location":"security/fundamentals/#perfect-forward-secrecy","text":"The term \u201cephemeral\u201d in the key exchange provides an important security feature mis-named perfect forward secrecy (PFS) or just \u201cForward Secrecy\u201d. In a non-ephemeral key exchange, the client sends the pre-master key to the server by encrypting it with the server\u2019s public key. The server then decrypts the pre-master key with its private key. If at a later point in time, the private key of the server is compromised, an attacker can go back to this handshake, decrypt the pre-master key, obtain the session key, and decrypt the entire traffic. Non-ephemeral key exchanges are vulnerable to attacks that may happen in the future on recorded traffic. And because people seldom change their password, decrypting data from the past may still be valuable for an attacker. An ephemeral key exchange like DHE, or its variant on elliptic curve, ECDHE, solves this problem by not transmitting the pre-master key over the wire. Instead, the pre-master key is computed by both the client and the server in isolation, using nonsensitive information exchanged publicly. Because the pre-master key can\u2019t be decrypted later by an attacker, the session key is safe from future attacks: hence, the term perfect forward secrecy. Keys are changed every X blocks along the stream. That prevents an attacker from simply sniffing the stream and applying brute force to crack the whole thing. \"Forward secrecy\" means that just because I can decrypt block M, does not mean that I can decrypt block Q Downside: The downside to PFS is that all those extra computational steps induce latency on the handshake and slow the user down. To avoid repeating this expensive work at every connection, both sides cache the session key for future use via a technique called session resumption. This is what the session-ID and TLS ticket are for: they allow a client and server that share a session ID to skip over the negotiation of a session key, because they already agreed on one previously, and go directly to exchanging data securely.","title":"\u201cPerfect\u201d Forward Secrecy"},{"location":"security/intro/","text":"Security Prerequisites Linux Basics Linux Networking What to expect from this course The course covers fundamentals of information security along with touching on subjects of system security, network & web security. This course aims to get you familiar with the basics of information security in day to day operations & then as an SRE develop the mindset of ensuring that security takes a front-seat while developing solutions. The course also serves as an introduction to common risks and best practices along with practical ways to find out vulnerable systems and loopholes which might become compromised if not secured. What is not covered under this course The courseware is not an ethical hacking workshop or a very deep dive into the fundamentals of the problems. The course does not deal with hacking or breaking into systems but rather an approach on how to ensure you don\u2019t get into those situations and also to make you aware of different ways a system can be compromised. Course Contents Fundamentals Network Security Threats, Attacks & Defence Writing Secure Code & More Conclusion","title":"Introduction"},{"location":"security/intro/#security","text":"","title":"Security"},{"location":"security/intro/#prerequisites","text":"Linux Basics Linux Networking","title":"Prerequisites"},{"location":"security/intro/#what-to-expect-from-this-course","text":"The course covers fundamentals of information security along with touching on subjects of system security, network & web security. This course aims to get you familiar with the basics of information security in day to day operations & then as an SRE develop the mindset of ensuring that security takes a front-seat while developing solutions. The course also serves as an introduction to common risks and best practices along with practical ways to find out vulnerable systems and loopholes which might become compromised if not secured.","title":"What to expect from this course"},{"location":"security/intro/#what-is-not-covered-under-this-course","text":"The courseware is not an ethical hacking workshop or a very deep dive into the fundamentals of the problems. The course does not deal with hacking or breaking into systems but rather an approach on how to ensure you don\u2019t get into those situations and also to make you aware of different ways a system can be compromised.","title":"What is not covered under this course"},{"location":"security/intro/#course-contents","text":"Fundamentals Network Security Threats, Attacks & Defence Writing Secure Code & More Conclusion","title":"Course Contents"},{"location":"security/network_security/","text":"Part II: Network Security Introduction TCP/IP is the dominant networking technology today. It is a five-layer architecture. These layers are, from top to bottom, the application layer, the transport layer (TCP), the network layer (IP), the data-link layer, and the physical layer. In addition to TCP/IP, there also are other networking technologies. For convenience, we use the OSI network model to represent non-TCP/IP network technologies. Different networks are interconnected using gateways. A gateway can be placed at any layer. The OSI model is a seven-layer architecture. The OSI architecture is similar to the TCP/IP architecture, except that the OSI model specifies two additional layers between the application layer and the transport layer in the TCP/IP architecture. These two layers are the presentation layer and the session layer. Figure 5.1 shows the relationship between the TCP/IP layers and the OSI layers. The application layer in TCP/IP corresponds to the application layer and the presentation layer in OSI. The transport layer in TCP/IP corresponds to the session layer and the transport layer in OSI. The remaining three layers in the TCP/IP architecture are one-to-one correspondent to the remaining three layers in the OSI model. Correspondence between layers of the TCP/IP architecture and the OSI model. Also shown are placements of cryptographic algorithms in network layers, where the dotted arrows indicate actual communications of cryptographic algorithms The functionalities of OSI layers are briefly described as follows: The application layer serves as an interface between applications and network programs. It supports application programs and end-user processing. Common application-layer programs include remote logins, file transfer, email, and Web browsing. The presentation layer is responsible for dealing with data that is formed differently. This protocol layer allows application-layer programs residing on different sides of a communication channel with different platforms to understand each other's data formats regardless of how they are presented. The session layer is responsible for creating, managing, and closing a communication connection. The transport layer is responsible for providing reliable connections, such as packet sequencing, traffic control, and congestion control. The network layer is responsible for routing device-independent data packets from the current hop to the next hop. The data-link layer is responsible for encapsulating device-independent data packets into device-dependent data frames. It has two sublayers: logical link control and media access control. The physical layer is responsible for transmitting device-dependent frames through some physical media. Starting from the application layer, data generated from an application program is passed down layer-by-layer to the physical layer. Data from the previous layer is enclosed in a new envelope at the current layer, where the data from the previous layer is also just an envelope containing the data from the layer before it. This is similar to enclosing a smaller envelope in a larger one. The envelope added at each layer contains sufficient information for handling the packet. Application-layer data are divided into blocks small enough to be encapsulated in an envelope at the next layer. Application data blocks are \u201cdressed up\u201d in the TCP/IP architecture according to the following basic steps. At the sending side, an application data block is encapsulated in a TCP packet when it is passed down to the TCP layer. In other words, a TCP packet consists of a header and a payload, where the header corresponds to the TCP envelope and the payload is the application data block. Likewise, the TCP packet will be encapsulated in an IP packet when it is passed down to the IP layer. An IP packet consists of a header and a payload, which is the TCP packet passed down from the TCP layer. The IP packet will be encapsulated in a device-dependent frame (e.g., an Ethernet frame) when it is passed down to the data-link layer. A frame has a header, and it may also have a trailer. For example, in addition to having a header, an Ethernet frame also has a 32-bit cyclic redundancy check (CRC) trailer. When it is passed down to the physical layer, a frame will be transformed into a sequence of media signals for transmission Flow Diagram of a Packet Generation At the destination side, the medium signals are converted by the physical layer into a frame, which is passed up to the data-link layer. The data-link layer passes the frame payload (i.e., the IP packet encapsulated in the frame) up to the IP layer. The IP layer passes the IP payload, namely, the TCP packet encapsulated in the IP packet, up to the TCP layer. The TCP layer passes the TCP payload, namely, the application data block, up to the application layer. When a packet arrives at a router, it only goes up to the IP layer, where certain fields in the IP header are modified (e.g., the value of TTL is decreased by 1). This modified packet is then passed back down layer-by-layer to the physical layer for further transmission. Public Key Infrastructure To deploy cryptographic algorithms in network applications, we need a way to distribute secret keys using open networks. Public-key cryptography is the best way to distribute these secret keys. To use public-key cryptography, we need to build a public-key infrastructure (PKI) to support and manage public-key certificates and certificate authority (CA) networks. In particular, PKIs are set up to perform the following functions: Determine the legitimacy of users before issuing public-key certificates to them. Issue public-key certificates upon user requests. Extend public-key certificates valid time upon user requests. Revoke public-key certificates upon users' requests or when the corresponding private keys are compromised. Store and manage public-key certificates. Prevent digital signature signers from denying their signatures. Support CA networks to allow different CAs to authenticate public-key certificates issued by other CAs. X.509: https://certificatedecoder.dev/?gclid=EAIaIQobChMI0M731O6G6gIVVSQrCh04bQaAEAAYASAAEgKRkPD_BwE IPsec: A Security Protocol at the Network Layer IPsec is a major security protocol at the network layer IPsec provides a potent platform for constructing virtual private networks (VPN). VPNs are private networks overlayed on public networks. The purpose of deploying cryptographic algorithms at the network layer is to encrypt or authenticate IP packets (either just the payloads or the whole packets). IPsec also specifies how to exchange keys. Thus, IPsec consists of authentication protocols, encryption protocols, and key exchange protocols. They are referred to, respectively, as authentication header (AH), encapsulating security payload (ESP), and Internet key exchange (IKE). PGP & S/MIME : Email Security There are several security protocols at the application layer. The most used of these protocols are email security protocols namely PGP and S/MIME. SMTP (\u201cSimple Mail Transfer Protocol\u201d) is used for sending and delivering from a client to a server via port 25: it\u2019s the outgoing server. On the contrary, POP (\u201cPost Office Protocol\u201d) allows the users to pick up the message and download it into their inbox: it\u2019s the incoming server. The latest version of the Post Office Protocol is named POP3, and it\u2019s been used since 1996; it uses port 110 PGP PGP implements all major cryptographic algorithms, the ZIP compression algorithm, and the Base64 encoding algorithm. It can be used to authenticate a message, encrypt a message, or both. PGP follows the following general process: authentication, ZIP compression, encryption, and Base64 encoding. The Base64 encoding procedure makes the message ready for SMTP transmission GPG (GnuPG) GnuPG is another free encryption standard that companies may use that is based on OpenPGP. GnuPG serves as a replacement for Symantec\u2019s PGP. The main difference is the supported algorithms. However, GnuPG plays nice with PGP by design. Because GnuPG is open, some businesses would prefer the technical support and the user interface that comes with Symantec\u2019s PGP. It is important to note that there are some nuances between the compatibility of GnuPG and PGP, such as the compatibility between certain algorithms, but in most applications such as email, there are workarounds. One such algorithm is the IDEA Module which isn\u2019t included in GnuPG out of the box due to patent issues. S/MIME SMTP can only handle 7-bit ASCII text (You can use UTF-8 extensions to alleviate these limitations, ) messages. While POP can handle other content types besides 7-bit ASCII, POP may, under a common default setting, download all the messages stored in the mail server to the user's local computer. After that, if POP removes these messages from the mail server. This makes it difficult for the users to read their messages from multiple computers. The Multipurpose Internet Mail Extension protocol (MIME) was designed to support sending and receiving email messages in various formats, including nontext files generated by word processors, graphics files, sound files, and video clips. Moreover, MIME allows a single message to include mixed types of data in any combination of these formats. The Internet Mail Access Protocol (IMAP), operated on TCP port 143(only for non-encrypted), stores (Configurable on both server & client just like PoP) incoming email messages in the mail server until the user deletes them deliberately. This allows the users to access their mailbox from multiple machines and download messages to a local machine without deleting it from the mailbox in the mail server. SSL/TLS SSL uses a PKI to decide if a server\u2019s public key is trustworthy by requiring servers to use a security certificate signed by a trusted CA. When Netscape Navigator 1.0 was released, it trusted a single CA operated by the RSA Data Security corporation. The server\u2019s public RSA keys were used to be stored in the security certificate, which can then be used by the browser to establish a secure communication channel. The security certificates we use today still rely on the same standard (named X.509) that Netscape Navigator 1.0 used back then. Netscape intended to train users(though this didn\u2019t work out later) to differentiate secure communications from insecure ones, so they put a lock icon next to the address bar. When the lock is open, the communication is insecure. A closed lock means communication has been secured with SSL, which required the server to provide a signed certificate. You\u2019re obviously familiar with this icon as it\u2019s been in every browser ever since. The engineers at Netscape truly created a standard for secure internet communications. A year after releasing SSL 2.0, Netscape fixed several security issues and released SSL 3.0, a protocol that, albeit being officially deprecated since June 2015, remains in use in certain parts of the world more than 20 years after its introduction. To standardize SSL, the Internet Engineering Task Force (IETF) created a slightly modified SSL 3.0 and, in 1999, unveiled it as Transport Layer Security (TLS) 1.0. The name change between SSL and TLS continues to confuse people today. Officially, TLS is the new SSL, but in practice, people use SSL and TLS interchangeably to talk about any version of the protocol. Must See: https://tls.ulfheim.net/ https://davidwong.fr/tls13/ Network Perimeter Security Let us see how we keep a check on the perimeter i.e the edges, the first layer of protection General Firewall Framework Firewalls are needed because encryption algorithms cannot effectively stop malicious packets from getting into an edge network. This is because IP packets, regardless of whether they are encrypted, can always be forwarded into an edge network. Firewalls that were developed in the 1990s are important instruments to help restrict network access. A firewall may be a hardware device, a software package, or a combination of both. Packets flowing into the internal network from the outside should be evaluated before they are allowed to enter. One of the critical elements of a firewall is its ability to examine packets without imposing a negative impact on communication speed while providing security protections for the internal network. The packet inspection that is carried out by firewalls can be done using several different methods. Based on the particular method used by the firewall, it can be characterized as either a packet filter, circuit gateway, application gateway, or dynamic packet filter. Packet Filters It inspects ingress packets coming to an internal network from outside and inspects egress packets going outside from an internal network Packing filtering only inspects IP headers and TCP headers, not the payloads generated at the application layer A packet-filtering firewall uses a set of rules to determine whether a packet should be allowed or denied to pass through. 2 types: Stateless It treats each packet as an independent object, and it does not keep track of any previously processed packets. In other words, stateless filtering inspects a packet when it arrives and makes a decision without leaving any record of the packet being inspected. Stateful Stateful filtering, also referred to as connection-state filtering, keeps track of connections between an internal host and an external host. A connection state (or state, for short) indicates whether it is a TCP connection or a UDP connection and whether the connection is established. Circuit Gateways Circuit gateways, also referred to as circuit-level gateways, are typically operated at the transportation layer They evaluate the information of the IP addresses and the port numbers contained in TCP (or UDP) headers and use it to determine whether to allow or to disallow an internal host and an external host to establish a connection. It is common practice to combine packet filters and circuit gateways to form a dynamic packet filter (DPF). Application Gateways(ALG) Aka PROXY Servers An Application Level Gateway (ALG) acts as a proxy for internal hosts, processing service requests from external clients. An ALG performs deep inspections on each IP packet (ingress or egress). In particular, an ALG inspects application program formats contained in the packet (e.g., MIME format or SQL format) and examines whether its payload is permitted. Thus, an ALG may be able to detect a computer virus contained in the payload. Because an ALG inspects packet payloads, it may be able to detect malicious code and quarantine suspicious packets, in addition to blocking packets with suspicious IP addresses and TCP ports. On the other hand, an ALG also incurs substantial computation and space overheads. Trusted Systems & Bastion Hosts A Trusted Operating System (TOS) is an operating system that meets a particular set of security requirements. Whether an operating system can be trusted or not depends on several elements. For example, for an operating system on a particular computer to be certified trusted, one needs to validate that, among other things, the following four requirements are satisfied: Its system design contains no defects; Its system software contains no loopholes; Its system is configured properly; and Its system management is appropriate. Bastion Hosts Bastion hosts are computers with strong defence mechanisms. They often serve as host computers for implementing application gateways, circuit gateways, and other types of firewalls. A bastion host is operated on a trusted operating system that must not contain unnecessary functionalities or programs. This measure helps to reduce error probabilities and makes it easier to conduct security checks. Only those network application programs that are necessary, for example, SSH, DNS, SMTP, and authentication programs, are installed on a bastion host. Bastion hosts are also primarily used as controlled ingress points so that the security monitoring can focus more narrowly on actions happening at a single point closely. Common Techniques & Scannings, Packet Capturing Scanning Ports with Nmap Nmap (\"Network Mapper\") is a free and open-source (license) utility for network discovery and security auditing. Many systems and network administrators also find it useful for tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime. The best thing about Nmap is it\u2019s free and open-source and is very flexible and versatile Nmap is often used to determine alive hosts in a network, open ports on those hosts, services running on those open ports, and version identification of that service on that port. More at http://scanme.nmap.org/ nmap [scan type] [options] [target specification] Nmap uses 6 different port states: Open \u2014 An open port is one that is actively accepting TCP, UDP or SCTP connections. Open ports are what interests us the most because they are the ones that are vulnerable to attacks. Open ports also show the available services on a network. Closed \u2014 A port that receives and responds to Nmap probe packets but there is no application listening on that port. Useful for identifying that the host exists and for OS detection. Filtered \u2014 Nmap can\u2019t determine whether the port is open because packet filtering prevents its probes from reaching the port. Filtering could come from firewalls or router rules. Often little information is given from filtered ports during scans as the filters can drop the probes without responding or respond with useless error messages e.g. destination unreachable. Unfiltered \u2014 Port is accessible but Nmap doesn\u2019t know if it is open or closed. Only used in ACK scan which is used to map firewall rulesets. Other scan types can be used to identify whether the port is open. Open/filtered \u2014 Nmap is unable to determine between open and filtered. This happens when an open port gives no response. No response could mean that the probe was dropped by a packet filter or any response is blocked. Closed/filtered \u2014 Nmap is unable to determine whether a port is closed or filtered. Only used in the IP ID idle scan. Types of Nmap Scan: TCP Connect TCP Connect scan completes the 3-way handshake. If a port is open, the operating system completes the TCP three-way handshake and the port scanner immediately closes the connection to avoid DOS. This is \u201cnoisy\u201d because the services can log the sender IP address and might trigger Intrusion Detection Systems. UDP Scan This scan checks to see if any UDP ports are listening. Since UDP does not respond with a positive acknowledgement like TCP and only responds to an incoming UDP packet when the port is closed, SYN Scan SYN scan is another form of TCP scanning. This scan type is also known as \u201chalf-open scanning\u201d because it never actually opens a full TCP connection. The port scanner generates a SYN packet. If the target port is open, it will respond with an SYN-ACK packet. The scanner host responds with an RST packet, closing the connection before the handshake is completed. If the port is closed but unfiltered, the target will instantly respond with an RST packet. SYN scan has the advantage that the individual services never actually receive a connection. FIN Scan This is a stealthy scan, like the SYN scan, but sends a TCP FIN packet instead. ACK Scan Ack scanning determines whether the port is filtered or not. Null Scan Another very stealthy scan that sets all the TCP header flags to off or null. This is not normally a valid packet and some hosts will not know what to do with this. XMAS Scan Similar to the NULL scan except for all the flags in the TCP header is set to on RPC Scan This special type of scan looks for machine answering to RPC (Remote Procedure Call) services IDLE Scan It is a super stealthy method whereby the scan packets are bounced off an external host. You don\u2019t need to have control over the other host but it does have to set up and meet certain requirements. You must input the IP address of our \u201czombie\u201d host and what port number to use. It is one of the more controversial options in Nmap since it only has a use for malicious attacks. Scan Techniques A couple of scan techniques which can be used to gain more information about a system and its ports. You can read more at https://medium.com/infosec-adventures/nmap-cheatsheet-a423fcdda0ca OpenVAS OpenVAS is a full-featured vulnerability scanner. OpenVAS is a framework of services and tools that provides a comprehensive and powerful vulnerability scanning and management package OpenVAS, which is an open-source program, began as a fork of the once-more-popular scanning program, Nessus. OpenVAS is made up of three main parts. These are: a regularly updated feed of Network Vulnerability Tests (NVTs); a scanner, which runs the NVTs; and an SQLite 3 database for storing both your test configurations and the NVTs\u2019 results and configurations. https://www.greenbone.net/en/install_use_gce/ WireShark Wireshark is a protocol analyzer. This means Wireshark is designed to decode not only packet bits and bytes but also the relations between packets and protocols. Wireshark understands protocol sequences. A simple demo of Wireshark Capture only udp packets: Capture filter = \u201cudp\u201d Capture only tcp packets Capture filter = \u201ctcp\u201d TCP/IP 3 way Handshake Filter by IP address: displays all traffic from IP, be it source or destination ip.addr == 192.168.1.1 Filter by source address: display traffic only from IP source ip.src == 192.168.0.1 Filter by destination: display traffic only form IP destination ip.dst == 192.168.0.1 Filter by IP subnet: display traffic from subnet, be it source or destination ip.addr = 192.168.0.1/24 Filter by protocol: filter traffic by protocol name dns http ftp arp ssh telnet icmp Exclude IP address: remove traffic from and to IP address !ip.addr ==192.168.0.1 Display traffic between two specific subnet ip.addr == 192.168.0.1/24 and ip.addr == 192.168.1.1/24 Display traffic between two specific workstations ip.addr == 192.168.0.1 and ip.addr == 192.168.0.2 Filter by MAC eth.addr = 00:50:7f:c5:b6:78 Filter TCP port tcp.port == 80 Filter TCP port source tcp.srcport == 80 Filter TCP port destination tcp.dstport == 80 Find user agents http.user_agent contains Firefox !http.user_agent contains || !http.user_agent contains Chrome Filter broadcast traffic !(arp or icmp or dns) Filter IP address and port tcp.port == 80 && ip.addr == 192.168.0.1 Filter all http get requests http.request Filter all http get requests and responses http.request or http.response Filter three way handshake tcp.flags.syn==1 or (tcp.seq==1 and tcp.ack==1 and tcp.len==0 and tcp.analysis.initial_rtt) Find files by type frame contains \u201c(attachment|tar|exe|zip|pdf)\u201d Find traffic based on keyword tcp contains facebook frame contains facebook Detecting SYN Floods tcp.flags.syn == 1 and tcp.flags.ack == 0 Wireshark Promiscuous Mode - By default, Wireshark only captures packets going to and from the computer where it runs. By checking the box to run Wireshark in Promiscuous Mode in the Capture Settings, you can capture most of the traffic on the LAN. DumpCap Dumpcap is a network traffic dump tool. It captures packet data from a live network and writes the packets to a file. Dumpcap\u2019s native capture file format is pcapng, which is also the format used by Wireshark. By default, Dumpcap uses the pcap library to capture traffic from the first available network interface and writes the received raw packet data, along with the packets\u2019 time stamps into a pcapng file. The capture filter syntax follows the rules of the pcap library. The Wireshark command-line utility called 'dumpcap.exe' can be used to capture LAN traffic over an extended period of time. Wireshark itself can also be used, but dumpcap does not significantly utilize the computer's memory while capturing for long periods. DaemonLogger Daemonlogger is a packet logging application designed specifically for use in Network and Systems Management (NSM) environments. The biggest benefit Daemonlogger provides is that, like Dumpcap, it is simple to use for capturing packets. In order to begin capturing, you need only to invoke the command and specify an interface. daemonlogger \u2013i eth1 This option, by default, will begin capturing packets and logging them to the current working directory. Packets will be collected until the capture file size reaches 2 GB, and then a new file will be created. This will continue indefinitely until the process is halted. NetSniff-NG Netsniff-NG is a high-performance packet capture utility While the utilities we\u2019ve discussed to this point rely on Libpcap for capture, Netsniff-NG utilizes zero-copy mechanisms to capture packets. This is done with the intent to support full packet capture over high throughput links. To begin capturing packets with Netsniff-NG, we have to specify an input and output. In most cases, the input will be a network interface, and the output will be a file or folder on disk. netsniff-ng \u2013i eth1 \u2013o data.pcap Netflow NetFlow is a feature that was introduced on Cisco routers around 1996 that provides the ability to collect IP network traffic as it enters or exits an interface. By analyzing the data provided by NetFlow, a network administrator can determine things such as the source and destination of traffic, class of service, and the causes of congestion. A typical flow monitoring setup (using NetFlow) consists of three main components:[1] Flow exporter: aggregates packets into flows and exports flow records towards one or more flow collectors. Flow collector: responsible for reception, storage and pre-processing of flow data received from a flow exporter. Analysis application: analyzes received flow data in the context of intrusion detection or traffic profiling, for example. Routers and switches that support NetFlow can collect IP traffic statistics on all interfaces where NetFlow is enabled, and later export those statistics as NetFlow records toward at least one NetFlow collector\u2014typically a server that does the actual traffic analysis. IDS A security solution that detects security-related events in your environment but does not block them. IDS sensors can be software and hardware-based used to collect and analyze the network traffic. These sensors are available in two varieties, network IDS and host IDS. A host IDS is a server-specific agent running on a server with a minimum of overhead to monitor the operating system. A network IDS can be embedded in a networking device, a standalone appliance, or a module monitoring the network traffic. Signature Based IDS The signature-based IDS monitors the network traffic or observes the system and sends an alarm if a known malicious event is happening. It does so by comparing the data flow against a database of known attack patterns These signatures explicitly define what traffic or activity should be considered as malicious. Signature-based detection has been the bread and butter of network-based defensive security for over a decade, partially because it is very similar to how malicious activity is detected at the host level with antivirus utilities The formula is fairly simple: an analyst observes a malicious activity, derives indicators from the activity and develops them into signatures, and then those signatures will alert whenever the activity occurs again. ex: SNORT & SURICATA Policy-Based IDS The policy-based IDSs (mainly host IDSs) trigger an alarm whenever a violation occurs against the configured policy. This configured policy is or should be a representation of the security policies. This type of IDS is flexible and can be customized to a company's network requirements because it knows exactly what is permitted and what is not. On the other hand, the signature-based systems rely on vendor specifics and default settings. Anomaly Based IDS The anomaly-based IDS looks for traffic that deviates from the normal, but the definition of what is a normal network traffic pattern is the tricky part Two types of anomaly-based IDS exist: statistical and nonstatistical anomaly detection Statistical anomaly detection learns the traffic patterns interactively over a period of time. In the nonstatistical approach, the IDS has a predefined configuration of the supposedly acceptable and valid traffic patterns. Host-Based IDS & Network-Based IDS A host IDS can be described as a distributed agent residing on each server of the network that needs protection. These distributed agents are tied very closely to the underlying operating system. Network IDSs, on the other hand, can be described as intelligent sniffing devices. Data (raw packets) is captured from the network by a network IDS, whereas host IDSs capture the data from the host on which they are installed. Honeypots The use of decoy machines to direct intruders' attention away from the machines under protection is a major technique to preclude intrusion attacks. Any device, system, directory, or file used as a decoy to lure attackers away from important assets and to collect intrusion or abusive behaviours is referred to as a honeypot. A honeypot may be implemented as a physical device or as an emulation system. The idea is to set up decoy machines in a LAN, or decoy directories/files in a file system and make them appear important, but with several exploitable loopholes, to lure attackers to attack these machines or directories/files, so that other machines, directories, and files can evade intruders' attentions. A decoy machine may be a host computer or a server computer. Likewise, we may also set up decoy routers or even decoy LANs. Chinks In The Armour (TCP/IP Security Issues) IP Spoofing In this type of attack, the attacker replaces the IP address of the sender, or in some rare cases the destination, with a different address. IP spoofing is normally used to exploit a target host. In other cases, it is used to start a denial-of-service (DoS) attack. In a DoS attack, an attacker modifies the IP packet to mislead the target host into accepting the original packet as a packet sourced at a trusted host. The attacker must know the IP address of the trusted host to modify the packet headers (source IP address) so that it appears that the packets are coming from that host. IP Spoofing Detection Techniques Direct TTL Probes In this technique we send a packet to a host of suspect spoofed IP that triggers reply and compares TTL with suspect packet; if the TTL in the reply is not the same as the packet being checked; it is a spoofed packet. This Technique is successful when the attacker is in a different subnet from the victim. IP Identification Number. Send a probe to the host of suspect spoofed traffic that triggers a reply and compares IP ID with suspect traffic. If IP IDs are not in the near value of packet being checked, suspect traffic is spoofed TCP Flow Control Method Attackers sending spoofed TCP packets will not receive the target\u2019s SYN-ACK packets. Attackers cannot, therefore, be responsive to change in the congestion window size When the receiver still receives traffic even after a windows size is exhausted, most probably the packets are spoofed. Covert Channel A covert or clandestine channel can be best described as a pipe or communication channel between two entities that can be exploited by a process or application transferring information in a manner that violates the system's security specifications. More specifically for TCP/IP, in some instances, covert channels are established, and data can be secretly passed between two end systems. Ex: ICMP resides at the Internet layer of the TCP/IP protocol suite and is implemented in all TCP/IP hosts. Based on the specifications of the ICMP Protocol, an ICMP Echo Request message should have an 8-byte header and a 56-byte payload. The ICMP Echo Request packet should not carry any data in the payload. However, these packets are often used to carry secret information. The ICMP packets are altered slightly to carry secret data in the payload. This makes the size of the packet larger, but no control exists in the protocol stack to defeat this behaviour. The alteration of ICMP packets allows intruders to program specialized client-server pairs. These small pieces of code export confidential information without alerting the network administrator. ICMP can be leveraged for more than data exfiltration. For eg. some C&C tools such as Loki used ICMP channel to establish encrypted interactive session back in 1996. Deep packet inspection has since come a long way. A lot of IDS/IPS detect ICMP tunnelling. Check for echo responses that do not contain the same payload as request Check for the volume of ICMP traffic especially for volumes beyond an acceptable threshold IP Fragmentation Attack The TCP/IP protocol suite, or more specifically IP, allows the fragmentation of packets.(this is a feature & not a bug) IP fragmentation offset is used to keep track of the different parts of a datagram. The information or content in this field is used at the destination to reassemble the datagrams All such fragments have the same Identification field value, and the fragmentation offset indicates the position of the current fragment in the context of the original packet. Many access routers and firewalls do not perform packet reassembly. In normal operation, IP fragments do not overlap, but attackers can create artificially fragmented packets to mislead the routers or firewalls. Usually, these packets are small and almost impractical for end systems because of data and computational overhead. A good example of an IP fragmentation attack is the Ping of Death attack. The Ping of Death attack sends fragments that, when reassembled at the end station, create a larger packet than the maximum permissible length. TCP Flags Data exchange using TCP does not happen until a three-way handshake has been completed. This handshake uses different flags to influence the way TCP segments are processed. There are 6 bits in the TCP header that are often called flags. Namely: 6 different flags are part of the TCP header: Urgent pointer field (URG), Acknowledgment field (ACK), Push function (PSH), Reset the connection (RST), Synchronize sequence numbers (SYN), and the sender is finished with this connection (FIN). Abuse of the normal operation or settings of these flags can be used by attackers to launch DoS attacks. This causes network servers or web servers to crash or hang. | SYN | FIN | PSH | RST | Validity| |------|------|-------|------|---------| | 1 |1 |0 |0 |Illegal Combination | 1 |1 |1 |0 |Illegal Combination | 1 |1 |0 |1 |Illegal Combination | 1 |1 |1 |1 |Illegal Combination The attacker's ultimate goal is to write special programs or pieces of code that can construct these illegal combinations resulting in an efficient DoS attack. SYN FLOOD The timers (or lack of certain timers) in 3 way handshake are often used and exploited by attackers to disable services or even to enter systems. After step 2 of the three-way handshake, no limit is set on the time to wait after receiving a SYN. The attacker initiates many connection requests to the webserver of Company XYZ (almost certainly with a spoofed IP address). The SYN+ACK packets (Step 2) sent by the web server back to the originating source IP address are not replied to. This leaves a TCP session half-open on the webserver. Multiple packets cause multiple TCP sessions to stay open. Based on the hardware limitations of the server, a limited number of TCP sessions can stay open, and as a result, the webserver refuses further connection establishments attempts from any host as soon as a certain limit is reached. These half-open connections need to be completed or timed out before new connections can be established. FIN Attack In normal operation, the sender sets the TCP FIN flag indicating that no more data will be transmitted and the connection can be closed down. This is a four-way handshake mechanism, with both sender and receiver expected to send an acknowledgement on a received FIN packet. During an attack that is trying to kill connections, a spoofed FIN packet is constructed. This packet also has the correct sequence number, so the packets are seen as valid by the targeted host. These sequence numbers are easy to predict. This process is referred to as TCP sequence number prediction, whereby the attacker either sniffs the current Sequence and Acknowledgment (SEQ/ACK) numbers of the connection or can algorithmically predict these numbers. Connection Hijacking An authorized user (Employee X) sends HTTP requests over a TCP session with the webserver. The web server accepts the packets from Employee X only when the packet has the correct SEQ/ACK numbers. As seen previously, these numbers are important for the webserver to distinguish between different sessions and to make sure it is still talking to Employee X. Imagine that the cracker starts sending packets to the web server spoofing the IP address of Employee X, using the correct SEQ/ACK combination. The web server accepts the packet and increments the ACK number. In the meantime, Employee X continues to send packets but with incorrect SEQ/ACK numbers. As a result of sending unsynchronized packets, all data from Employee X is discarded when received by the webserver. The attacker pretends to be Employee X using the correct numbers. This finally results in the cracker hijacking the connection, whereby Employee X is completely confused and the webserver replies assuming the cracker is sending correct synchronized data. STEPS: The attacker examines the traffic flows with a network monitor and notices traffic from Employee X to a web server. The web server returns or echoes data back to the origination station (Employee X). Employee X acknowledges the packet. The cracker launches a spoofed packet to the server. The web server responds to the cracker. The cracker starts verifying SEQ/ACK numbers to double-check success. At this time, the cracker takes over the session from Employee X, which results in a session hanging for Employee X. The cracker can start sending traffic to the webserver. The web server returns the requested data to confirm delivery with the correct ACK number. The cracker can continue to send data (keeping track of the correct SEQ/ACK numbers) until eventually setting the FIN flag to terminate the session. Buffer Overflow A buffer is a temporary data storage area used to store program code and data. When a program or process tries to store more data in a buffer than it was originally anticipated to hold, a buffer overflow occurs. Buffers are temporary storage locations in memory (memory or buffer sizes are often measured in bytes) that can store a fixed amount of data in bytes. When more data is retrieved than can be stored in a buffer location, the additional information must go into an adjacent buffer, resulting in overwriting the valid data held in them. Mechanism: Buffer overflow vulnerabilities exist in different types. But the overall goal for all buffer overflow attacks is to take over the control of a privileged program and, if possible, the host. The attacker has two tasks to achieve this goal. First, the dirty code needs to be available in the program's code address space. Second, the privileged program should jump to that particular part of the code, which ensures that the proper parameters are loaded into memory. The first task can be achieved in two ways: by injecting the code in the right address space or by using the existing code and modifying certain parameters slightly. The second task is a little more complex because the program's control flow needs to be modified to make the program jump to the dirty code. CounterMeasure: The most important approach is to have a concerted focus on writing correct code. A second method is to make the data buffers (memory locations) address space of the program code non-executable. This type of address space makes it impossible to execute code, which might be infiltrated in the program's buffers during an attack. More Spoofing Address Resolution Protocol Spoofing The Address Resolution Protocol (ARP) provides a mechanism to resolve, or map, a known IP address to a MAC sublayer address. Using ARP spoofing, the cracker can exploit this hardware address authentication mechanism by spoofing the hardware address of Host B. Basically, the attacker can convince any host or network device on the local network that the cracker's workstation is the host to be trusted. This is a common method used in a switched environment. ARP spoofing can be prevented with the implementation of static ARP tables in all the hosts and routers of your network. Alternatively, you can implement an ARP server that responds to ARP requests on behalf of the target host. DNS Spoofing DNS spoofing is the method whereby the hacker convinces the target machine that the system it wants to connect to is the machine of the cracker. The cracker modifies some records so that name entries of hosts correspond to the attacker's IP address. There have been instances in which the complete DNS server was compromised by an attack. To counter DNS spoofing, the reverse lookup detects these attacks. The reverse lookup is a mechanism to verify the IP address against a name. The IP address and name files are usually kept on different servers to make compromise much more difficult","title":"Network Security"},{"location":"security/network_security/#part-ii-network-security","text":"","title":"Part II: Network Security"},{"location":"security/network_security/#introduction","text":"TCP/IP is the dominant networking technology today. It is a five-layer architecture. These layers are, from top to bottom, the application layer, the transport layer (TCP), the network layer (IP), the data-link layer, and the physical layer. In addition to TCP/IP, there also are other networking technologies. For convenience, we use the OSI network model to represent non-TCP/IP network technologies. Different networks are interconnected using gateways. A gateway can be placed at any layer. The OSI model is a seven-layer architecture. The OSI architecture is similar to the TCP/IP architecture, except that the OSI model specifies two additional layers between the application layer and the transport layer in the TCP/IP architecture. These two layers are the presentation layer and the session layer. Figure 5.1 shows the relationship between the TCP/IP layers and the OSI layers. The application layer in TCP/IP corresponds to the application layer and the presentation layer in OSI. The transport layer in TCP/IP corresponds to the session layer and the transport layer in OSI. The remaining three layers in the TCP/IP architecture are one-to-one correspondent to the remaining three layers in the OSI model. Correspondence between layers of the TCP/IP architecture and the OSI model. Also shown are placements of cryptographic algorithms in network layers, where the dotted arrows indicate actual communications of cryptographic algorithms The functionalities of OSI layers are briefly described as follows: The application layer serves as an interface between applications and network programs. It supports application programs and end-user processing. Common application-layer programs include remote logins, file transfer, email, and Web browsing. The presentation layer is responsible for dealing with data that is formed differently. This protocol layer allows application-layer programs residing on different sides of a communication channel with different platforms to understand each other's data formats regardless of how they are presented. The session layer is responsible for creating, managing, and closing a communication connection. The transport layer is responsible for providing reliable connections, such as packet sequencing, traffic control, and congestion control. The network layer is responsible for routing device-independent data packets from the current hop to the next hop. The data-link layer is responsible for encapsulating device-independent data packets into device-dependent data frames. It has two sublayers: logical link control and media access control. The physical layer is responsible for transmitting device-dependent frames through some physical media. Starting from the application layer, data generated from an application program is passed down layer-by-layer to the physical layer. Data from the previous layer is enclosed in a new envelope at the current layer, where the data from the previous layer is also just an envelope containing the data from the layer before it. This is similar to enclosing a smaller envelope in a larger one. The envelope added at each layer contains sufficient information for handling the packet. Application-layer data are divided into blocks small enough to be encapsulated in an envelope at the next layer. Application data blocks are \u201cdressed up\u201d in the TCP/IP architecture according to the following basic steps. At the sending side, an application data block is encapsulated in a TCP packet when it is passed down to the TCP layer. In other words, a TCP packet consists of a header and a payload, where the header corresponds to the TCP envelope and the payload is the application data block. Likewise, the TCP packet will be encapsulated in an IP packet when it is passed down to the IP layer. An IP packet consists of a header and a payload, which is the TCP packet passed down from the TCP layer. The IP packet will be encapsulated in a device-dependent frame (e.g., an Ethernet frame) when it is passed down to the data-link layer. A frame has a header, and it may also have a trailer. For example, in addition to having a header, an Ethernet frame also has a 32-bit cyclic redundancy check (CRC) trailer. When it is passed down to the physical layer, a frame will be transformed into a sequence of media signals for transmission Flow Diagram of a Packet Generation At the destination side, the medium signals are converted by the physical layer into a frame, which is passed up to the data-link layer. The data-link layer passes the frame payload (i.e., the IP packet encapsulated in the frame) up to the IP layer. The IP layer passes the IP payload, namely, the TCP packet encapsulated in the IP packet, up to the TCP layer. The TCP layer passes the TCP payload, namely, the application data block, up to the application layer. When a packet arrives at a router, it only goes up to the IP layer, where certain fields in the IP header are modified (e.g., the value of TTL is decreased by 1). This modified packet is then passed back down layer-by-layer to the physical layer for further transmission.","title":"Introduction"},{"location":"security/network_security/#public-key-infrastructure","text":"To deploy cryptographic algorithms in network applications, we need a way to distribute secret keys using open networks. Public-key cryptography is the best way to distribute these secret keys. To use public-key cryptography, we need to build a public-key infrastructure (PKI) to support and manage public-key certificates and certificate authority (CA) networks. In particular, PKIs are set up to perform the following functions: Determine the legitimacy of users before issuing public-key certificates to them. Issue public-key certificates upon user requests. Extend public-key certificates valid time upon user requests. Revoke public-key certificates upon users' requests or when the corresponding private keys are compromised. Store and manage public-key certificates. Prevent digital signature signers from denying their signatures. Support CA networks to allow different CAs to authenticate public-key certificates issued by other CAs. X.509: https://certificatedecoder.dev/?gclid=EAIaIQobChMI0M731O6G6gIVVSQrCh04bQaAEAAYASAAEgKRkPD_BwE","title":"Public Key Infrastructure"},{"location":"security/network_security/#ipsec-a-security-protocol-at-the-network-layer","text":"IPsec is a major security protocol at the network layer IPsec provides a potent platform for constructing virtual private networks (VPN). VPNs are private networks overlayed on public networks. The purpose of deploying cryptographic algorithms at the network layer is to encrypt or authenticate IP packets (either just the payloads or the whole packets). IPsec also specifies how to exchange keys. Thus, IPsec consists of authentication protocols, encryption protocols, and key exchange protocols. They are referred to, respectively, as authentication header (AH), encapsulating security payload (ESP), and Internet key exchange (IKE).","title":"IPsec: A Security Protocol at the Network Layer"},{"location":"security/network_security/#pgp-smime-email-security","text":"There are several security protocols at the application layer. The most used of these protocols are email security protocols namely PGP and S/MIME. SMTP (\u201cSimple Mail Transfer Protocol\u201d) is used for sending and delivering from a client to a server via port 25: it\u2019s the outgoing server. On the contrary, POP (\u201cPost Office Protocol\u201d) allows the users to pick up the message and download it into their inbox: it\u2019s the incoming server. The latest version of the Post Office Protocol is named POP3, and it\u2019s been used since 1996; it uses port 110 PGP PGP implements all major cryptographic algorithms, the ZIP compression algorithm, and the Base64 encoding algorithm. It can be used to authenticate a message, encrypt a message, or both. PGP follows the following general process: authentication, ZIP compression, encryption, and Base64 encoding. The Base64 encoding procedure makes the message ready for SMTP transmission GPG (GnuPG) GnuPG is another free encryption standard that companies may use that is based on OpenPGP. GnuPG serves as a replacement for Symantec\u2019s PGP. The main difference is the supported algorithms. However, GnuPG plays nice with PGP by design. Because GnuPG is open, some businesses would prefer the technical support and the user interface that comes with Symantec\u2019s PGP. It is important to note that there are some nuances between the compatibility of GnuPG and PGP, such as the compatibility between certain algorithms, but in most applications such as email, there are workarounds. One such algorithm is the IDEA Module which isn\u2019t included in GnuPG out of the box due to patent issues. S/MIME SMTP can only handle 7-bit ASCII text (You can use UTF-8 extensions to alleviate these limitations, ) messages. While POP can handle other content types besides 7-bit ASCII, POP may, under a common default setting, download all the messages stored in the mail server to the user's local computer. After that, if POP removes these messages from the mail server. This makes it difficult for the users to read their messages from multiple computers. The Multipurpose Internet Mail Extension protocol (MIME) was designed to support sending and receiving email messages in various formats, including nontext files generated by word processors, graphics files, sound files, and video clips. Moreover, MIME allows a single message to include mixed types of data in any combination of these formats. The Internet Mail Access Protocol (IMAP), operated on TCP port 143(only for non-encrypted), stores (Configurable on both server & client just like PoP) incoming email messages in the mail server until the user deletes them deliberately. This allows the users to access their mailbox from multiple machines and download messages to a local machine without deleting it from the mailbox in the mail server. SSL/TLS SSL uses a PKI to decide if a server\u2019s public key is trustworthy by requiring servers to use a security certificate signed by a trusted CA. When Netscape Navigator 1.0 was released, it trusted a single CA operated by the RSA Data Security corporation. The server\u2019s public RSA keys were used to be stored in the security certificate, which can then be used by the browser to establish a secure communication channel. The security certificates we use today still rely on the same standard (named X.509) that Netscape Navigator 1.0 used back then. Netscape intended to train users(though this didn\u2019t work out later) to differentiate secure communications from insecure ones, so they put a lock icon next to the address bar. When the lock is open, the communication is insecure. A closed lock means communication has been secured with SSL, which required the server to provide a signed certificate. You\u2019re obviously familiar with this icon as it\u2019s been in every browser ever since. The engineers at Netscape truly created a standard for secure internet communications. A year after releasing SSL 2.0, Netscape fixed several security issues and released SSL 3.0, a protocol that, albeit being officially deprecated since June 2015, remains in use in certain parts of the world more than 20 years after its introduction. To standardize SSL, the Internet Engineering Task Force (IETF) created a slightly modified SSL 3.0 and, in 1999, unveiled it as Transport Layer Security (TLS) 1.0. The name change between SSL and TLS continues to confuse people today. Officially, TLS is the new SSL, but in practice, people use SSL and TLS interchangeably to talk about any version of the protocol. Must See: https://tls.ulfheim.net/ https://davidwong.fr/tls13/","title":"PGP & S/MIME : Email Security"},{"location":"security/network_security/#network-perimeter-security","text":"Let us see how we keep a check on the perimeter i.e the edges, the first layer of protection","title":"Network Perimeter Security"},{"location":"security/network_security/#general-firewall-framework","text":"Firewalls are needed because encryption algorithms cannot effectively stop malicious packets from getting into an edge network. This is because IP packets, regardless of whether they are encrypted, can always be forwarded into an edge network. Firewalls that were developed in the 1990s are important instruments to help restrict network access. A firewall may be a hardware device, a software package, or a combination of both. Packets flowing into the internal network from the outside should be evaluated before they are allowed to enter. One of the critical elements of a firewall is its ability to examine packets without imposing a negative impact on communication speed while providing security protections for the internal network. The packet inspection that is carried out by firewalls can be done using several different methods. Based on the particular method used by the firewall, it can be characterized as either a packet filter, circuit gateway, application gateway, or dynamic packet filter.","title":"General Firewall Framework"},{"location":"security/network_security/#packet-filters","text":"It inspects ingress packets coming to an internal network from outside and inspects egress packets going outside from an internal network Packing filtering only inspects IP headers and TCP headers, not the payloads generated at the application layer A packet-filtering firewall uses a set of rules to determine whether a packet should be allowed or denied to pass through. 2 types: Stateless It treats each packet as an independent object, and it does not keep track of any previously processed packets. In other words, stateless filtering inspects a packet when it arrives and makes a decision without leaving any record of the packet being inspected. Stateful Stateful filtering, also referred to as connection-state filtering, keeps track of connections between an internal host and an external host. A connection state (or state, for short) indicates whether it is a TCP connection or a UDP connection and whether the connection is established.","title":"Packet Filters"},{"location":"security/network_security/#circuit-gateways","text":"Circuit gateways, also referred to as circuit-level gateways, are typically operated at the transportation layer They evaluate the information of the IP addresses and the port numbers contained in TCP (or UDP) headers and use it to determine whether to allow or to disallow an internal host and an external host to establish a connection. It is common practice to combine packet filters and circuit gateways to form a dynamic packet filter (DPF).","title":"Circuit Gateways"},{"location":"security/network_security/#application-gatewaysalg","text":"Aka PROXY Servers An Application Level Gateway (ALG) acts as a proxy for internal hosts, processing service requests from external clients. An ALG performs deep inspections on each IP packet (ingress or egress). In particular, an ALG inspects application program formats contained in the packet (e.g., MIME format or SQL format) and examines whether its payload is permitted. Thus, an ALG may be able to detect a computer virus contained in the payload. Because an ALG inspects packet payloads, it may be able to detect malicious code and quarantine suspicious packets, in addition to blocking packets with suspicious IP addresses and TCP ports. On the other hand, an ALG also incurs substantial computation and space overheads.","title":"Application Gateways(ALG)"},{"location":"security/network_security/#trusted-systems-bastion-hosts","text":"A Trusted Operating System (TOS) is an operating system that meets a particular set of security requirements. Whether an operating system can be trusted or not depends on several elements. For example, for an operating system on a particular computer to be certified trusted, one needs to validate that, among other things, the following four requirements are satisfied: Its system design contains no defects; Its system software contains no loopholes; Its system is configured properly; and Its system management is appropriate. Bastion Hosts Bastion hosts are computers with strong defence mechanisms. They often serve as host computers for implementing application gateways, circuit gateways, and other types of firewalls. A bastion host is operated on a trusted operating system that must not contain unnecessary functionalities or programs. This measure helps to reduce error probabilities and makes it easier to conduct security checks. Only those network application programs that are necessary, for example, SSH, DNS, SMTP, and authentication programs, are installed on a bastion host. Bastion hosts are also primarily used as controlled ingress points so that the security monitoring can focus more narrowly on actions happening at a single point closely.","title":"Trusted Systems & Bastion Hosts"},{"location":"security/network_security/#common-techniques-scannings-packet-capturing","text":"","title":"Common Techniques & Scannings, Packet Capturing"},{"location":"security/network_security/#scanning-ports-with-nmap","text":"Nmap (\"Network Mapper\") is a free and open-source (license) utility for network discovery and security auditing. Many systems and network administrators also find it useful for tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime. The best thing about Nmap is it\u2019s free and open-source and is very flexible and versatile Nmap is often used to determine alive hosts in a network, open ports on those hosts, services running on those open ports, and version identification of that service on that port. More at http://scanme.nmap.org/ nmap [scan type] [options] [target specification] Nmap uses 6 different port states: Open \u2014 An open port is one that is actively accepting TCP, UDP or SCTP connections. Open ports are what interests us the most because they are the ones that are vulnerable to attacks. Open ports also show the available services on a network. Closed \u2014 A port that receives and responds to Nmap probe packets but there is no application listening on that port. Useful for identifying that the host exists and for OS detection. Filtered \u2014 Nmap can\u2019t determine whether the port is open because packet filtering prevents its probes from reaching the port. Filtering could come from firewalls or router rules. Often little information is given from filtered ports during scans as the filters can drop the probes without responding or respond with useless error messages e.g. destination unreachable. Unfiltered \u2014 Port is accessible but Nmap doesn\u2019t know if it is open or closed. Only used in ACK scan which is used to map firewall rulesets. Other scan types can be used to identify whether the port is open. Open/filtered \u2014 Nmap is unable to determine between open and filtered. This happens when an open port gives no response. No response could mean that the probe was dropped by a packet filter or any response is blocked. Closed/filtered \u2014 Nmap is unable to determine whether a port is closed or filtered. Only used in the IP ID idle scan.","title":"Scanning Ports with Nmap"},{"location":"security/network_security/#types-of-nmap-scan","text":"TCP Connect TCP Connect scan completes the 3-way handshake. If a port is open, the operating system completes the TCP three-way handshake and the port scanner immediately closes the connection to avoid DOS. This is \u201cnoisy\u201d because the services can log the sender IP address and might trigger Intrusion Detection Systems. UDP Scan This scan checks to see if any UDP ports are listening. Since UDP does not respond with a positive acknowledgement like TCP and only responds to an incoming UDP packet when the port is closed, SYN Scan SYN scan is another form of TCP scanning. This scan type is also known as \u201chalf-open scanning\u201d because it never actually opens a full TCP connection. The port scanner generates a SYN packet. If the target port is open, it will respond with an SYN-ACK packet. The scanner host responds with an RST packet, closing the connection before the handshake is completed. If the port is closed but unfiltered, the target will instantly respond with an RST packet. SYN scan has the advantage that the individual services never actually receive a connection. FIN Scan This is a stealthy scan, like the SYN scan, but sends a TCP FIN packet instead. ACK Scan Ack scanning determines whether the port is filtered or not. Null Scan Another very stealthy scan that sets all the TCP header flags to off or null. This is not normally a valid packet and some hosts will not know what to do with this. XMAS Scan Similar to the NULL scan except for all the flags in the TCP header is set to on RPC Scan This special type of scan looks for machine answering to RPC (Remote Procedure Call) services IDLE Scan It is a super stealthy method whereby the scan packets are bounced off an external host. You don\u2019t need to have control over the other host but it does have to set up and meet certain requirements. You must input the IP address of our \u201czombie\u201d host and what port number to use. It is one of the more controversial options in Nmap since it only has a use for malicious attacks. Scan Techniques A couple of scan techniques which can be used to gain more information about a system and its ports. You can read more at https://medium.com/infosec-adventures/nmap-cheatsheet-a423fcdda0ca","title":"Types of Nmap Scan:"},{"location":"security/network_security/#openvas","text":"OpenVAS is a full-featured vulnerability scanner. OpenVAS is a framework of services and tools that provides a comprehensive and powerful vulnerability scanning and management package OpenVAS, which is an open-source program, began as a fork of the once-more-popular scanning program, Nessus. OpenVAS is made up of three main parts. These are: a regularly updated feed of Network Vulnerability Tests (NVTs); a scanner, which runs the NVTs; and an SQLite 3 database for storing both your test configurations and the NVTs\u2019 results and configurations. https://www.greenbone.net/en/install_use_gce/","title":"OpenVAS"},{"location":"security/network_security/#wireshark","text":"Wireshark is a protocol analyzer. This means Wireshark is designed to decode not only packet bits and bytes but also the relations between packets and protocols. Wireshark understands protocol sequences. A simple demo of Wireshark Capture only udp packets: Capture filter = \u201cudp\u201d Capture only tcp packets Capture filter = \u201ctcp\u201d TCP/IP 3 way Handshake Filter by IP address: displays all traffic from IP, be it source or destination ip.addr == 192.168.1.1 Filter by source address: display traffic only from IP source ip.src == 192.168.0.1 Filter by destination: display traffic only form IP destination ip.dst == 192.168.0.1 Filter by IP subnet: display traffic from subnet, be it source or destination ip.addr = 192.168.0.1/24 Filter by protocol: filter traffic by protocol name dns http ftp arp ssh telnet icmp Exclude IP address: remove traffic from and to IP address !ip.addr ==192.168.0.1 Display traffic between two specific subnet ip.addr == 192.168.0.1/24 and ip.addr == 192.168.1.1/24 Display traffic between two specific workstations ip.addr == 192.168.0.1 and ip.addr == 192.168.0.2 Filter by MAC eth.addr = 00:50:7f:c5:b6:78 Filter TCP port tcp.port == 80 Filter TCP port source tcp.srcport == 80 Filter TCP port destination tcp.dstport == 80 Find user agents http.user_agent contains Firefox !http.user_agent contains || !http.user_agent contains Chrome Filter broadcast traffic !(arp or icmp or dns) Filter IP address and port tcp.port == 80 && ip.addr == 192.168.0.1 Filter all http get requests http.request Filter all http get requests and responses http.request or http.response Filter three way handshake tcp.flags.syn==1 or (tcp.seq==1 and tcp.ack==1 and tcp.len==0 and tcp.analysis.initial_rtt) Find files by type frame contains \u201c(attachment|tar|exe|zip|pdf)\u201d Find traffic based on keyword tcp contains facebook frame contains facebook Detecting SYN Floods tcp.flags.syn == 1 and tcp.flags.ack == 0 Wireshark Promiscuous Mode - By default, Wireshark only captures packets going to and from the computer where it runs. By checking the box to run Wireshark in Promiscuous Mode in the Capture Settings, you can capture most of the traffic on the LAN.","title":"WireShark"},{"location":"security/network_security/#dumpcap","text":"Dumpcap is a network traffic dump tool. It captures packet data from a live network and writes the packets to a file. Dumpcap\u2019s native capture file format is pcapng, which is also the format used by Wireshark. By default, Dumpcap uses the pcap library to capture traffic from the first available network interface and writes the received raw packet data, along with the packets\u2019 time stamps into a pcapng file. The capture filter syntax follows the rules of the pcap library. The Wireshark command-line utility called 'dumpcap.exe' can be used to capture LAN traffic over an extended period of time. Wireshark itself can also be used, but dumpcap does not significantly utilize the computer's memory while capturing for long periods.","title":"DumpCap"},{"location":"security/network_security/#daemonlogger","text":"Daemonlogger is a packet logging application designed specifically for use in Network and Systems Management (NSM) environments. The biggest benefit Daemonlogger provides is that, like Dumpcap, it is simple to use for capturing packets. In order to begin capturing, you need only to invoke the command and specify an interface. daemonlogger \u2013i eth1 This option, by default, will begin capturing packets and logging them to the current working directory. Packets will be collected until the capture file size reaches 2 GB, and then a new file will be created. This will continue indefinitely until the process is halted.","title":"DaemonLogger"},{"location":"security/network_security/#netsniff-ng","text":"Netsniff-NG is a high-performance packet capture utility While the utilities we\u2019ve discussed to this point rely on Libpcap for capture, Netsniff-NG utilizes zero-copy mechanisms to capture packets. This is done with the intent to support full packet capture over high throughput links. To begin capturing packets with Netsniff-NG, we have to specify an input and output. In most cases, the input will be a network interface, and the output will be a file or folder on disk. netsniff-ng \u2013i eth1 \u2013o data.pcap","title":"NetSniff-NG"},{"location":"security/network_security/#netflow","text":"NetFlow is a feature that was introduced on Cisco routers around 1996 that provides the ability to collect IP network traffic as it enters or exits an interface. By analyzing the data provided by NetFlow, a network administrator can determine things such as the source and destination of traffic, class of service, and the causes of congestion. A typical flow monitoring setup (using NetFlow) consists of three main components:[1] Flow exporter: aggregates packets into flows and exports flow records towards one or more flow collectors. Flow collector: responsible for reception, storage and pre-processing of flow data received from a flow exporter. Analysis application: analyzes received flow data in the context of intrusion detection or traffic profiling, for example. Routers and switches that support NetFlow can collect IP traffic statistics on all interfaces where NetFlow is enabled, and later export those statistics as NetFlow records toward at least one NetFlow collector\u2014typically a server that does the actual traffic analysis.","title":"Netflow"},{"location":"security/network_security/#ids","text":"A security solution that detects security-related events in your environment but does not block them. IDS sensors can be software and hardware-based used to collect and analyze the network traffic. These sensors are available in two varieties, network IDS and host IDS. A host IDS is a server-specific agent running on a server with a minimum of overhead to monitor the operating system. A network IDS can be embedded in a networking device, a standalone appliance, or a module monitoring the network traffic. Signature Based IDS The signature-based IDS monitors the network traffic or observes the system and sends an alarm if a known malicious event is happening. It does so by comparing the data flow against a database of known attack patterns These signatures explicitly define what traffic or activity should be considered as malicious. Signature-based detection has been the bread and butter of network-based defensive security for over a decade, partially because it is very similar to how malicious activity is detected at the host level with antivirus utilities The formula is fairly simple: an analyst observes a malicious activity, derives indicators from the activity and develops them into signatures, and then those signatures will alert whenever the activity occurs again. ex: SNORT & SURICATA Policy-Based IDS The policy-based IDSs (mainly host IDSs) trigger an alarm whenever a violation occurs against the configured policy. This configured policy is or should be a representation of the security policies. This type of IDS is flexible and can be customized to a company's network requirements because it knows exactly what is permitted and what is not. On the other hand, the signature-based systems rely on vendor specifics and default settings. Anomaly Based IDS The anomaly-based IDS looks for traffic that deviates from the normal, but the definition of what is a normal network traffic pattern is the tricky part Two types of anomaly-based IDS exist: statistical and nonstatistical anomaly detection Statistical anomaly detection learns the traffic patterns interactively over a period of time. In the nonstatistical approach, the IDS has a predefined configuration of the supposedly acceptable and valid traffic patterns. Host-Based IDS & Network-Based IDS A host IDS can be described as a distributed agent residing on each server of the network that needs protection. These distributed agents are tied very closely to the underlying operating system. Network IDSs, on the other hand, can be described as intelligent sniffing devices. Data (raw packets) is captured from the network by a network IDS, whereas host IDSs capture the data from the host on which they are installed. Honeypots The use of decoy machines to direct intruders' attention away from the machines under protection is a major technique to preclude intrusion attacks. Any device, system, directory, or file used as a decoy to lure attackers away from important assets and to collect intrusion or abusive behaviours is referred to as a honeypot. A honeypot may be implemented as a physical device or as an emulation system. The idea is to set up decoy machines in a LAN, or decoy directories/files in a file system and make them appear important, but with several exploitable loopholes, to lure attackers to attack these machines or directories/files, so that other machines, directories, and files can evade intruders' attentions. A decoy machine may be a host computer or a server computer. Likewise, we may also set up decoy routers or even decoy LANs.","title":"IDS"},{"location":"security/network_security/#chinks-in-the-armour-tcpip-security-issues","text":"","title":"Chinks In The Armour (TCP/IP Security Issues)"},{"location":"security/network_security/#ip-spoofing","text":"In this type of attack, the attacker replaces the IP address of the sender, or in some rare cases the destination, with a different address. IP spoofing is normally used to exploit a target host. In other cases, it is used to start a denial-of-service (DoS) attack. In a DoS attack, an attacker modifies the IP packet to mislead the target host into accepting the original packet as a packet sourced at a trusted host. The attacker must know the IP address of the trusted host to modify the packet headers (source IP address) so that it appears that the packets are coming from that host. IP Spoofing Detection Techniques Direct TTL Probes In this technique we send a packet to a host of suspect spoofed IP that triggers reply and compares TTL with suspect packet; if the TTL in the reply is not the same as the packet being checked; it is a spoofed packet. This Technique is successful when the attacker is in a different subnet from the victim. IP Identification Number. Send a probe to the host of suspect spoofed traffic that triggers a reply and compares IP ID with suspect traffic. If IP IDs are not in the near value of packet being checked, suspect traffic is spoofed TCP Flow Control Method Attackers sending spoofed TCP packets will not receive the target\u2019s SYN-ACK packets. Attackers cannot, therefore, be responsive to change in the congestion window size When the receiver still receives traffic even after a windows size is exhausted, most probably the packets are spoofed.","title":"IP Spoofing"},{"location":"security/network_security/#covert-channel","text":"A covert or clandestine channel can be best described as a pipe or communication channel between two entities that can be exploited by a process or application transferring information in a manner that violates the system's security specifications. More specifically for TCP/IP, in some instances, covert channels are established, and data can be secretly passed between two end systems. Ex: ICMP resides at the Internet layer of the TCP/IP protocol suite and is implemented in all TCP/IP hosts. Based on the specifications of the ICMP Protocol, an ICMP Echo Request message should have an 8-byte header and a 56-byte payload. The ICMP Echo Request packet should not carry any data in the payload. However, these packets are often used to carry secret information. The ICMP packets are altered slightly to carry secret data in the payload. This makes the size of the packet larger, but no control exists in the protocol stack to defeat this behaviour. The alteration of ICMP packets allows intruders to program specialized client-server pairs. These small pieces of code export confidential information without alerting the network administrator. ICMP can be leveraged for more than data exfiltration. For eg. some C&C tools such as Loki used ICMP channel to establish encrypted interactive session back in 1996. Deep packet inspection has since come a long way. A lot of IDS/IPS detect ICMP tunnelling. Check for echo responses that do not contain the same payload as request Check for the volume of ICMP traffic especially for volumes beyond an acceptable threshold","title":"Covert Channel"},{"location":"security/network_security/#ip-fragmentation-attack","text":"The TCP/IP protocol suite, or more specifically IP, allows the fragmentation of packets.(this is a feature & not a bug) IP fragmentation offset is used to keep track of the different parts of a datagram. The information or content in this field is used at the destination to reassemble the datagrams All such fragments have the same Identification field value, and the fragmentation offset indicates the position of the current fragment in the context of the original packet. Many access routers and firewalls do not perform packet reassembly. In normal operation, IP fragments do not overlap, but attackers can create artificially fragmented packets to mislead the routers or firewalls. Usually, these packets are small and almost impractical for end systems because of data and computational overhead. A good example of an IP fragmentation attack is the Ping of Death attack. The Ping of Death attack sends fragments that, when reassembled at the end station, create a larger packet than the maximum permissible length. TCP Flags Data exchange using TCP does not happen until a three-way handshake has been completed. This handshake uses different flags to influence the way TCP segments are processed. There are 6 bits in the TCP header that are often called flags. Namely: 6 different flags are part of the TCP header: Urgent pointer field (URG), Acknowledgment field (ACK), Push function (PSH), Reset the connection (RST), Synchronize sequence numbers (SYN), and the sender is finished with this connection (FIN). Abuse of the normal operation or settings of these flags can be used by attackers to launch DoS attacks. This causes network servers or web servers to crash or hang. | SYN | FIN | PSH | RST | Validity| |------|------|-------|------|---------| | 1 |1 |0 |0 |Illegal Combination | 1 |1 |1 |0 |Illegal Combination | 1 |1 |0 |1 |Illegal Combination | 1 |1 |1 |1 |Illegal Combination The attacker's ultimate goal is to write special programs or pieces of code that can construct these illegal combinations resulting in an efficient DoS attack. SYN FLOOD The timers (or lack of certain timers) in 3 way handshake are often used and exploited by attackers to disable services or even to enter systems. After step 2 of the three-way handshake, no limit is set on the time to wait after receiving a SYN. The attacker initiates many connection requests to the webserver of Company XYZ (almost certainly with a spoofed IP address). The SYN+ACK packets (Step 2) sent by the web server back to the originating source IP address are not replied to. This leaves a TCP session half-open on the webserver. Multiple packets cause multiple TCP sessions to stay open. Based on the hardware limitations of the server, a limited number of TCP sessions can stay open, and as a result, the webserver refuses further connection establishments attempts from any host as soon as a certain limit is reached. These half-open connections need to be completed or timed out before new connections can be established. FIN Attack In normal operation, the sender sets the TCP FIN flag indicating that no more data will be transmitted and the connection can be closed down. This is a four-way handshake mechanism, with both sender and receiver expected to send an acknowledgement on a received FIN packet. During an attack that is trying to kill connections, a spoofed FIN packet is constructed. This packet also has the correct sequence number, so the packets are seen as valid by the targeted host. These sequence numbers are easy to predict. This process is referred to as TCP sequence number prediction, whereby the attacker either sniffs the current Sequence and Acknowledgment (SEQ/ACK) numbers of the connection or can algorithmically predict these numbers.","title":"IP Fragmentation Attack"},{"location":"security/network_security/#connection-hijacking","text":"An authorized user (Employee X) sends HTTP requests over a TCP session with the webserver. The web server accepts the packets from Employee X only when the packet has the correct SEQ/ACK numbers. As seen previously, these numbers are important for the webserver to distinguish between different sessions and to make sure it is still talking to Employee X. Imagine that the cracker starts sending packets to the web server spoofing the IP address of Employee X, using the correct SEQ/ACK combination. The web server accepts the packet and increments the ACK number. In the meantime, Employee X continues to send packets but with incorrect SEQ/ACK numbers. As a result of sending unsynchronized packets, all data from Employee X is discarded when received by the webserver. The attacker pretends to be Employee X using the correct numbers. This finally results in the cracker hijacking the connection, whereby Employee X is completely confused and the webserver replies assuming the cracker is sending correct synchronized data. STEPS: The attacker examines the traffic flows with a network monitor and notices traffic from Employee X to a web server. The web server returns or echoes data back to the origination station (Employee X). Employee X acknowledges the packet. The cracker launches a spoofed packet to the server. The web server responds to the cracker. The cracker starts verifying SEQ/ACK numbers to double-check success. At this time, the cracker takes over the session from Employee X, which results in a session hanging for Employee X. The cracker can start sending traffic to the webserver. The web server returns the requested data to confirm delivery with the correct ACK number. The cracker can continue to send data (keeping track of the correct SEQ/ACK numbers) until eventually setting the FIN flag to terminate the session.","title":"Connection Hijacking"},{"location":"security/network_security/#buffer-overflow","text":"A buffer is a temporary data storage area used to store program code and data. When a program or process tries to store more data in a buffer than it was originally anticipated to hold, a buffer overflow occurs. Buffers are temporary storage locations in memory (memory or buffer sizes are often measured in bytes) that can store a fixed amount of data in bytes. When more data is retrieved than can be stored in a buffer location, the additional information must go into an adjacent buffer, resulting in overwriting the valid data held in them. Mechanism: Buffer overflow vulnerabilities exist in different types. But the overall goal for all buffer overflow attacks is to take over the control of a privileged program and, if possible, the host. The attacker has two tasks to achieve this goal. First, the dirty code needs to be available in the program's code address space. Second, the privileged program should jump to that particular part of the code, which ensures that the proper parameters are loaded into memory. The first task can be achieved in two ways: by injecting the code in the right address space or by using the existing code and modifying certain parameters slightly. The second task is a little more complex because the program's control flow needs to be modified to make the program jump to the dirty code. CounterMeasure: The most important approach is to have a concerted focus on writing correct code. A second method is to make the data buffers (memory locations) address space of the program code non-executable. This type of address space makes it impossible to execute code, which might be infiltrated in the program's buffers during an attack.","title":"Buffer Overflow"},{"location":"security/network_security/#more-spoofing","text":"Address Resolution Protocol Spoofing The Address Resolution Protocol (ARP) provides a mechanism to resolve, or map, a known IP address to a MAC sublayer address. Using ARP spoofing, the cracker can exploit this hardware address authentication mechanism by spoofing the hardware address of Host B. Basically, the attacker can convince any host or network device on the local network that the cracker's workstation is the host to be trusted. This is a common method used in a switched environment. ARP spoofing can be prevented with the implementation of static ARP tables in all the hosts and routers of your network. Alternatively, you can implement an ARP server that responds to ARP requests on behalf of the target host. DNS Spoofing DNS spoofing is the method whereby the hacker convinces the target machine that the system it wants to connect to is the machine of the cracker. The cracker modifies some records so that name entries of hosts correspond to the attacker's IP address. There have been instances in which the complete DNS server was compromised by an attack. To counter DNS spoofing, the reverse lookup detects these attacks. The reverse lookup is a mechanism to verify the IP address against a name. The IP address and name files are usually kept on different servers to make compromise much more difficult","title":"More Spoofing"},{"location":"security/threats_attacks_defences/","text":"Part III: Threats, Attacks & Defense DNS Protection Cache Poisoning Attack Since DNS responses are cached, a quick response can be provided for repeated translations. DNS negative queries are also cached, e.g., misspelt words, and all cached data periodically times out. Cache poisoning is an issue in what is known as pharming. This term is used to describe a hacker\u2019s attack in which a website\u2019s traffic is redirected to a bogus website by forging the DNS mapping. In this case, an attacker attempts to insert a fake address record for an Internet domain into the DNS. If the server accepts the fake record, the cache is poisoned and subsequent requests for the address of the domain are answered with the address of a server controlled by the attacker. As long as the fake entry is cached by the server, browsers or e-mail servers will automatically go to the address provided by the compromised DNS server. the typical time to live (TTL) for cached entries is a couple of hours, thereby permitting ample time for numerous users to be affected by the attack. DNSSEC (Security Extension) The long-term solution to these DNS problems is authentication. If a resolver cannot distinguish between valid and invalid data in a response, then add source authentication to verify that the data received in response is equal to the data entered by the zone administrator DNS Security Extensions (DNSSEC) protects against data spoofing and corruption and provides mechanisms to authenticate servers and requests, as well as mechanisms to establish authenticity and integrity. When authenticating DNS responses, each DNS zone signs its data using a private key. It is recommended that this signing be done offline and in advance. The query for a particular record returns the requested resource record set (RRset) and signature (RRSIG) of the requested resource record set. The resolver then authenticates the response using a public key, which is pre-configured or learned via a sequence of key records in the DNS hierarchy. The goals of DNSSEC are to provide authentication and integrity for DNS responses without confidentiality or DDoS protection. BGP BGP stands for border gateway protocol. It is a routing protocol that exchanges routing information among multiple Autonomous Systems (AS) An Autonomous System is a collection of routers or networks with the same network policy usually under single administrative control. BGP tells routers which hop to use in order to reach the destination network. BGP is used for both communicating information among routers in an AS (interior) and between multiple ASes (exterior). How BGP Works BGP is responsible for finding a path to a destination router & the path it chooses should be the shortest and most reliable one. This decision is done through a protocol known as Link state. With the link-state protocol, each router broadcasts to all other routers in the network the state of its links and IP subnets. Each router then receives information from the other routers and constructs a complete topology view of the entire network. The next-hop routing table is based on this topology view. The link-state protocol uses a famous algorithm in the field of computer science, Dijkstra\u2019s shortest path algorithm: We start from our router considering the path cost to all our direct neighbours. The shortest path is then taken We then re-look at all our neighbours that we can reach and update our link state table with the cost information. We then continue taking the shortest path until every router has been visited. BGP Vulnerabilities By corrupting the BGP routing table we are able to influence the direction traffic flows on the internet! This action is known as BGP hijacking. Injecting bogus route advertising information into the BGP-distributed routing database by malicious sources, accidentally or routers can disrupt Internet backbone operations. Blackholing traffic: Blackhole route is a network route, i.e., routing table entry, that goes nowhere and packets matching the route prefix are dropped or ignored. Blackhole routes can only be detected by monitoring the lost traffic. Blackhole routes are the best defence against many common viral attacks where the traffic is dropped from infected machines to/from command & control hosts. Infamous BGP Injection attack on Youtube Ex: In 2008, Pakistan decided to block YouTube by creating a BGP route that led into a black hole. Instead, this routing information got transmitted to a hong kong ISP and from there accidentally got propagated to the rest of the world meaning millions were routed through to this black hole and therefore unable to access YouTube. Potentially, the greatest risk to BGP occurs in a denial of service attack in which a router is flooded with more packets than it can handle. Network overload and router resource exhaustion happen when the network begins carrying an excessive number of BGP messages, overloading the router control processors, memory, routing table and reducing the bandwidth available for data traffic. Refer: https://medium.com/bugbountywriteup/bgp-the-weak-link-in-the-internet-what-is-bgp-and-how-do-hackers-exploit-it-d899a68ba5bb Router flapping is another type of attack. Route flapping refers to repetitive changes to the BGP routing table, often several times a minute. Withdrawing and re-advertising at a high-rate can cause a serious problem for routers since they propagate the announcements of routes. If these route flaps happen fast enough, e.g., 30 to 50 times per second, the router becomes overloaded, which eventually prevents convergence on valid routes. The potential impact for Internet users is a slowdown in message delivery, and in some cases, packets may not be delivered at all. BGP Security Border Gateway Protocol Security recommends the use of BGP peer authentication since it is one of the strongest mechanisms for preventing malicious activity. The authentication mechanisms are Internet Protocol Security (IPsec) or BGP MD5. Another method, known as prefix limits, can be used to avoid filling router tables. In this approach, routers should be configured to disable or terminate a BGP peering session, and issue warning messages to administrators when a neighbour sends in excess of a preset number of prefixes. IETF is currently working on improving this space Web-Based Attacks HTTP Response Splitting Attacks HTTP response splitting attack may happen where the server script embeds user data in HTTP response headers without appropriate sanitation. This typically happens when the script embeds user data in the redirection URL of a redirection response (HTTP status code 3xx), or when the script embeds user data in a cookie value or name when the response sets a cookie. HTTP response splitting attacks can be used to perform web cache poisoning and cross-site scripting attacks. HTTP response splitting is the attacker\u2019s ability to send a single HTTP request that forces the webserver to form an output stream, which is then interpreted by the target as two HTTP responses instead of one response. Cross-Site Request Forgery (CSRF or XSRF) A Cross-Site Request Forgery attack tricks the victim\u2019s browser into issuing a command to a vulnerable web application. Vulnerability is caused by browsers automatically including user authentication data, session ID, IP address, Windows domain credentials, etc. with each request. Attackers typically use CSRF to initiate transactions such as transfer funds, login/logout user, close account, access sensitive data, and change account details. The vulnerability is caused by web browsers that automatically include credentials with each request, even for requests caused by a form, script, or image on another site. CSRF can also be dynamically constructed as part of a payload for a cross-site scripting attack All sites relying on automatic credentials are vulnerable. Popular browsers cannot prevent cross-site request forgery. Logging out of high-value sites as soon as possible can mitigate CSRF risk. It is recommended that a high-value website must require a client to manually provide authentication data in the same HTTP request used to perform any operation with security implications. Limiting the lifetime of session cookies can also reduce the chance of being used by other malicious sites. OWASP recommends website developers include a required security token in HTTP requests associated with sensitive business functions in order to mitigate CSRF attacks Cross-Site Scripting (XSS) Attacks Cross-Site Scripting occurs when dynamically generated web pages display user input, such as login information, that is not properly validated, allowing an attacker to embed malicious scripts into the generated page and then execute the script on the machine of any user that views the site. If successful, Cross-Site Scripting vulnerabilities can be exploited to manipulate or steal cookies, create requests that can be mistaken for those of a valid user, compromise confidential information, or execute malicious code on end-user systems. Cross-Site Scripting (XSS or CSS) attacks involve the execution of malicious scripts on the victim\u2019s browser. The victim is simply a user\u2019s host and not the server. XSS results from a failure to validate user input by a web-based application. Document Object Model (DOM) XSS Attacks The Document Object Model (DOM) based XSS does not require the webserver to receive the XSS payload for a successful attack. The attacker abuses the runtime by embedding their data on the client-side. An attacker can force the client (browser) to render the page with parts of the DOM controlled by the attacker. When the page is rendered and the data is processed by the page, typically by a client-side HTML-embedded script such as JavaScript, the page\u2019s code may insecurely embed the data in the page itself, thus delivering the cross-site scripting payload. There are several DOM objects which can serve as an attack vehicle for delivering malicious script to victims browser. Clickjacking The technique works by hiding malicious link/scripts under the cover of the content of a legitimate site. Buttons on a website actually contain invisible links, placed there by the attacker. So, an individual who clicks on an object they can visually see is actually being duped into visiting a malicious page or executing a malicious script. When mouseover is used together with clickjacking, the outcome is devastating. Facebook users have been hit by a clickjacking attack, which tricks people into \u201cliking\u201d a particular Facebook page, thus enabling the attack to spread since Memorial Day 2010. There is not yet effective defence against clickjacking, and disabling JavaScript is the only viable method DataBase Attacks & Defenses SQL injection Attacks It exploits improper input validation in database queries. A successful exploit will allow attackers to access, modify, or delete information in the database. It permits attackers to steal sensitive information stored within the backend databases of affected websites, which may include such things as user credentials, email addresses, personal information, and credit card numbers SELECT USERNAME,PASSWORD from USERS where USERNAME=' You should be redirected automatically to target URL: https://linkedin.com. If not click the link. \"\"\"","title":"The URL Shortening App"},{"location":"python_web/url-shorten-app/#the-url-shortening-app","text":"Let's build a very simple URL shortening app using flask and try to incorporate all aspects of the development process including the reliability aspects. We will not be building the UI and we will come up with a minimal set of API that will be enough for the app to function well.","title":"The URL Shortening App"},{"location":"python_web/url-shorten-app/#design","text":"We don't jump directly to coding. First thing we do is gather requirements. Come up with an approach. Have the approach/design reviewed by peers. Evolve, iterate, document the decisions and tradeoffs. And then finally implement. While we will not do the full blown design document here, we will raise certain questions here that are important to the design.","title":"Design"},{"location":"python_web/url-shorten-app/#1-high-level-operations-and-api-endpoints","text":"Since it's a URL shortening app, we will need an API for generating the shorten link given an original link. And an API/Endpoint which will accept the shorten link and redirect to original URL. We are not including the user aspect of the app to keep things minimal. These two API should make app functional and usable by anyone.","title":"1. High Level Operations and API Endpoints"},{"location":"python_web/url-shorten-app/#2-how-to-shorten","text":"Given a url, we will need to generate a shortened version of it. One approach could be using random characters for each link. Another thing that can be done is to use some sort of hashing algorithm. The benefit here is we will reuse the same hash for the same link. ie: if lot of people are shortening https://www.linkedin.com they all will have the same value, compared to multiple entries in DB if chosen random characters. What about hash collisions? Even in random characters approach, though there is a less probability, hash collisions can happen. And we need to be mindful of them. In that case we might want to prepend/append the string with some random value to avoid conflict. Also, choice of hash algorithm matters. We will need to analyze algorithms. Their CPU requirements and their characteristics. Choose one that suits the most.","title":"2. How to shorten?"},{"location":"python_web/url-shorten-app/#3-is-url-valid","text":"Given a URL to shorten, how do we verify if the URL is valid? Do we even verify or validate? One basic check that can be done is see if the URL matches a regex of a URL. To go even further we can try opening/visiting the URL. But there are certain gotchas here. We need to define success criteria. ie: HTTP 200 means it is valid. What is the URL is in private network? What if URL is temporarily down?","title":"3. Is URL Valid?"},{"location":"python_web/url-shorten-app/#4-storage","text":"Finally, storage. Where will we store the data that we will generate over time? There are multiple database solutions available and we will need to choose the one that suits this app the most. Relational database like MySQL would be a fair choice but be sure to checkout School of SRE's SQL database section and NoSQL databases section for deeper insights into making a more informed decision.","title":"4. Storage"},{"location":"python_web/url-shorten-app/#5-other","text":"We are not accounting for users into our app and other possible features like rate limiting, customized links etc but it will eventually come up with time. Depending on the requirements, they too might need to get incorporated. The minimal working code is given below for reference but I'd encourage you to come up with your own. from flask import Flask, redirect, request from hashlib import md5 app = Flask(\"url_shortener\") mapping = {} @app.route(\"/shorten\", methods=[\"POST\"]) def shorten(): global mapping payload = request.json if \"url\" not in payload: return \"Missing URL Parameter\", 400 # TODO: check if URL is valid hash_ = md5() hash_.update(payload[\"url\"].encode()) digest = hash_.hexdigest()[:5] # limiting to 5 chars. Less the limit more the chances of collission if digest not in mapping: mapping[digest] = payload[\"url\"] return f\"Shortened: r/{digest}\\n\" else: # TODO: check for hash collission return f\"Already exists: r/{digest}\\n\" @app.route(\"/r/ You should be redirected automatically to target URL: https://linkedin.com. If not click the link. \"\"\"","title":"5. Other"},{"location":"security/conclusion/","text":"Conclusion Now that you have completed this course on Security you are now aware of the possible security threats to computer systems & networks. Not only that, but you are now better able to protect your systems as well as recommend security measures to others. This course provides fundamental everyday knowledge on security domain which will also help you keep security at the top of your priority. Other Resources Some books that would be a great resource Holistic Info-Sec for Web Developers https://holisticinfosecforwebdevelopers.com/ - Free and downloadable book series with very broad and deep coverage of what Web Developers and DevOps Engineers need to know in order to create robust, reliable, maintainable and secure software, networks and other, that are delivered continuously, on time, with no nasty surprises Docker Security - Quick Reference: For DevOps Engineers https://holisticinfosecforwebdevelopers.com/ - A book on understanding the Docker security defaults, how to improve them (theory and practical), along with many tools and techniques. How to Hack Like a Legend https://amzn.to/2uWh1Up - A hacker\u2019s tale breaking into a secretive offshore company, Sparc Flow, 2018 How to Investigate Like a Rockstar https://books2read.com/u/4jDWoZ - Live a real crisis to master the secrets of forensic analysis, Sparc Flow, 2017 Real World Cryptography https://www.manning.com/books/real-world-cryptography - This early-access book teaches you applied cryptographic techniques to understand and apply security at every level of your systems and applications. AWS Security https://www.manning.com/books/aws-security?utm_source=github&utm_medium=organic&utm_campaign=book_shields_aws_1_31_20 - This early-access book covers commong AWS security issues and best practices for access policies, data protection, auditing, continuous monitoring, and incident response. Post Training asks/ Further Reading CTF Events like : https://github.com/apsdehal/awesome-ctf Penetration Testing : https://github.com/enaqx/awesome-pentest Threat Intelligence : https://github.com/hslatman/awesome-threat-intelligence Threat Detection & Hunting : https://github.com/0x4D31/awesome-threat-detection Web Security: https://github.com/qazbnm456/awesome-web-security Building Secure and Reliable Systems : https://landing.google.com/sre/resources/foundationsandprinciples/srs-book/","title":"Conclusion"},{"location":"security/conclusion/#conclusion","text":"Now that you have completed this course on Security you are now aware of the possible security threats to computer systems & networks. Not only that, but you are now better able to protect your systems as well as recommend security measures to others. This course provides fundamental everyday knowledge on security domain which will also help you keep security at the top of your priority.","title":"Conclusion"},{"location":"security/conclusion/#other-resources","text":"Some books that would be a great resource Holistic Info-Sec for Web Developers https://holisticinfosecforwebdevelopers.com/ - Free and downloadable book series with very broad and deep coverage of what Web Developers and DevOps Engineers need to know in order to create robust, reliable, maintainable and secure software, networks and other, that are delivered continuously, on time, with no nasty surprises Docker Security - Quick Reference: For DevOps Engineers https://holisticinfosecforwebdevelopers.com/ - A book on understanding the Docker security defaults, how to improve them (theory and practical), along with many tools and techniques. How to Hack Like a Legend https://amzn.to/2uWh1Up - A hacker\u2019s tale breaking into a secretive offshore company, Sparc Flow, 2018 How to Investigate Like a Rockstar https://books2read.com/u/4jDWoZ - Live a real crisis to master the secrets of forensic analysis, Sparc Flow, 2017 Real World Cryptography https://www.manning.com/books/real-world-cryptography - This early-access book teaches you applied cryptographic techniques to understand and apply security at every level of your systems and applications. AWS Security https://www.manning.com/books/aws-security?utm_source=github&utm_medium=organic&utm_campaign=book_shields_aws_1_31_20 - This early-access book covers commong AWS security issues and best practices for access policies, data protection, auditing, continuous monitoring, and incident response.","title":"Other Resources"},{"location":"security/conclusion/#post-training-asks-further-reading","text":"CTF Events like : https://github.com/apsdehal/awesome-ctf Penetration Testing : https://github.com/enaqx/awesome-pentest Threat Intelligence : https://github.com/hslatman/awesome-threat-intelligence Threat Detection & Hunting : https://github.com/0x4D31/awesome-threat-detection Web Security: https://github.com/qazbnm456/awesome-web-security Building Secure and Reliable Systems : https://landing.google.com/sre/resources/foundationsandprinciples/srs-book/","title":"Post Training asks/ Further Reading"},{"location":"security/fundamentals/","text":"Part I: Fundamentals Introduction to Security Overview for SRE If you look closely, both Site Reliability Engineering and Security Engineering are concerned with keeping a system usable. Issues like broken releases, capacity shortages, and misconfigurations can make a system unusable (at least temporarily). Security or privacy incidents that break the trust of users also undermine the usefulness of a system. Consequently, system security should be top of mind for SREs. SREs should be involved in both significant design discussions and actual system changes. They have quite a big role in System design & hence are quite sometimes the first line of defence. SRE\u2019s help in preventing bad design & implementations which can affect the overall security of the infrastructure. Successfully designing, implementing, and maintaining systems requires a commitment to the full system lifecycle . This commitment is possible only when security and reliability are central elements in the architecture of systems. Core Pillars of Information Security : Confidentiality \u2013 only allow access to data for which the user is permitted Integrity \u2013 ensure data is not tampered or altered by unauthorized users Availability \u2013 ensure systems and data are available to authorized users when they need it Thinking like a Security Engineer When starting a new application or re-factoring an existing application, you should consider each functional feature, and consider: Is the process surrounding this feature as safe as possible? In other words, is this a flawed process? If I were evil, how would I abuse this feature? Or more specifically failing to address how a feature can be abused can cause design flaws. Is the feature required to be on by default? If so, are there limits or options that could help reduce the risk from this feature? Security Principles By OWASP (Open Web Application Security Project) Minimize attack surface area : Every feature that is added to an application adds a certain amount of risk to the overall application. The aim of secure development is to reduce the overall risk by reducing the attack surface area. For example, a web application implements online help with a search function. The search function may be vulnerable to SQL injection attacks. If the help feature was limited to authorized users, the attack likelihood is reduced. If the help feature\u2019s search function was gated through centralized data validation routines, the ability to perform SQL injection is dramatically reduced. However, if the help feature was re-written to eliminate the search function (through a better user interface, for example), this almost eliminates the attack surface area, even if the help feature was available to the Internet at large. Establish secure defaults: There are many ways to deliver an \u201cout of the box\u201d experience for users. However, by default, the experience should be secure, and it should be up to the user to reduce their security \u2013 if they are allowed. For example, by default, password ageing and complexity should be enabled. Users might be allowed to turn these two features off to simplify their use of the application and increase their risk. Default Passwords of routers, IoT devices should be changed Principle of Least privilege The principle of least privilege recommends that accounts have the least amount of privilege required to perform their business processes. This encompasses user rights, resource permissions such as CPU limits, memory, network, and file system permissions. For example, if a middleware server only requires access to the network, read access to a database table, and the ability to write to a log, this describes all the permissions that should be granted. Under no circumstances should the middleware be granted administrative privileges. Principle of Defense in depth The principle of defence in depth suggests that where one control would be reasonable, more controls that approach risks in different fashions are better. Controls, when used in depth, can make severe vulnerabilities extraordinarily difficult to exploit and thus unlikely to occur. With secure coding, this may take the form of tier-based validation, centralized auditing controls, and requiring users to be logged on all pages. For example, a flawed administrative interface is unlikely to be vulnerable to an anonymous attack if it correctly gates access to production management networks, checks for administrative user authorization, and logs all access. Fail securely Applications regularly fail to process transactions for many reasons. How they fail can determine if an application is secure or not. ``` is_admin = true; try { code_which_may_faile(); is_admin = is_user_assigned_role(\"Adminstrator\"); } catch (Exception err) { log.error(err.toString()); } ``` - If either codeWhichMayFail() or isUserInRole fails or throws an exception, the user is an admin by default. This is obviously a security risk. Don\u2019t trust services Many organizations utilize the processing capabilities of third-party partners, who more than likely have different security policies and posture than you. It is unlikely that you can influence or control any external third party, whether they are home users or major suppliers or partners. Therefore, the implicit trust of externally run systems is not warranted. All external systems should be treated similarly. For example, a loyalty program provider provides data that is used by Internet Banking, providing the number of reward points and a small list of potential redemption items. However, the data should be checked to ensure that it is safe to display to end-users and that the reward points are a positive number, and not improbably large. Separation of duties The key to fraud control is the separation of duties. For example, someone who requests a computer cannot also sign for it, nor should they directly receive the computer. This prevents the user from requesting many computers and claiming they never arrived. Certain roles have different levels of trust than normal users. In particular, administrators are different from normal users. In general, administrators should not be users of the application. For example, an administrator should be able to turn the system on or off, set password policy but shouldn\u2019t be able to log on to the storefront as a super privileged user, such as being able to \u201cbuy\u201d goods on behalf of other users. Avoid security by obscurity Security through obscurity is a weak security control, and nearly always fails when it is the only control. This is not to say that keeping secrets is a bad idea, it simply means that the security of systems should not be reliant upon keeping details hidden. For example, the security of an application should not rely upon knowledge of the source code being kept secret. The security should rely upon many other factors, including reasonable password policies, defence in depth, business transaction limits, solid network architecture, and fraud, and audit controls. A practical example is Linux. Linux\u2019s source code is widely available, and yet when properly secured, Linux is a secure and robust operating system. Keep security simple Attack surface area and simplicity go hand in hand. Certain software engineering practices prefer overly complex approaches to what would otherwise be a relatively straightforward and simple design. Developers should avoid the use of double negatives and complex architectures when a simpler approach would be faster and simpler. For example, although it might be fashionable to have a slew of singleton entity beans running on a separate middleware server, it is more secure and faster to simply use global variables with an appropriate mutex mechanism to protect against race conditions. Fix security issues correctly Once a security issue has been identified, it is important to develop a test for it and to understand the root cause of the issue. When design patterns are used, the security issue is likely widespread amongst all codebases, so developing the right fix without introducing regressions is essential. For example, a user has found that they can see another user\u2019s balance by adjusting their cookie. The fix seems to be relatively straightforward, but as the cookie handling code is shared among all applications, a change to just one application will trickle through to all other applications. The fix must, therefore, be tested on all affected applications. Reliability & Security Reliability and security are both crucial components of a truly trustworthy system, but building systems that are both reliable and secure is difficult. While the requirements for reliability and security share many common properties, they also require different design considerations. It is easy to miss the subtle interplay between reliability and security that can cause unexpected outcomes Ex: A password management application failure was triggered by a reliability problem i.e poor load-balancing and load-shedding strategies and its recovery were later complicated by multiple measures (HSM mechanism which needs to be plugged into server racks, which works as an authentication & the HSM token supposedly locked inside a case.. & the problem can be further elongated ) designed to increase the security of the system. Authentication vs Authorization Authentication is the act of validating that users are who they claim to be. Passwords are the most common authentication factor\u2014if a user enters the correct password, the system assumes the identity is valid and grants access. Other technologies such as One-Time Pins, authentication apps, and even biometrics can also be used to authenticate identity. In some instances, systems require the successful verification of more than one factor before granting access. This multi-factor authentication (MFA) requirement is often deployed to increase security beyond what passwords alone can provide. Authorization in system security is the process of giving the user permission to access a specific resource or function. This term is often used interchangeably with access control or client privilege. Giving someone permission to download a particular file on a server or providing individual users with administrative access to an application are good examples. In secure environments, authorization must always follow authentication, users should first prove that their identities are genuine before an organization\u2019s administrators grant them access to the requested resources. Common authentication flow (local authentication) The user registers using an identifier like username/email/mobile The application stores user credentials in the database The application sends a verification email/message to validate the registration Post successful registration, the user enters credentials for logging in On successful authentication, the user is allowed access to specific resources OpenID/OAuth OpenID is an authentication protocol that allows us to authenticate users without using a local auth system. In such a scenario, a user has to be registered with an OpenID Provider and the same provider should be integrated with the authentication flow of your application. To verify the details, we have to forward the authentication requests to the provider. On successful authentication, we receive a success message and/or profile details with which we can execute the necessary flow. OAuth is an authorization mechanism that allows your application user access to a provider(Gmail/Facebook/Instagram/etc). On successful response, we (your application) receive a token with which the application can access certain APIs on behalf of a user. OAuth is convenient in case your business use case requires some certain user-facing APIs like access to Google Drive or sending tweets on your behalf. Most OAuth 2.0 providers can be used for pseudo authentication. Having said that, it can get pretty complicated if you are using multiple OAuth providers to authenticate users on top of the local authentication system. Cryptography It is the science and study of hiding any text in such a way that only the intended recipients or authorized persons can read it and that any text can even use things such as invisible ink or the mechanical cryptography machines of the past. Cryptography is necessary for securing critical or proprietary information and is used to encode private data messages by converting some plain text into ciphertext. At its core, there are two ways of doing this, more advanced methods are all built upon. Ciphers Ciphers are the cornerstone of cryptography. A cipher is a set of algorithms that performs encryption or decryption on a message. An encryption algorithm (E) takes a secret key (k) and a message (m) and produces a ciphertext (c). Similarly, a Decryption algorithm (D) takes a secret key (K) and the previous resulting Ciphertext (C). They are represented as follows: E(k,m) = c D(k,c) = m This also means that for it to be a cipher, it must satisfy the consistency equation as follows, making it possible to decrypt. D(k,E(k,m)) = m Stream Ciphers: The message is broken into characters or bits and enciphered with a key or keystream(should be random and generated independently of the message stream) that is as long as the plaintext bitstream. If the keystream is random, this scheme would be unbreakable unless the keystream was acquired, making it unconditionally secure. The keystream must be provided to both parties in a secure way to prevent its release. Block Ciphers: Block ciphers \u2014 process messages in blocks, each of which is then encrypted or decrypted. A block cipher is a symmetric cipher in which blocks of plaintext are treated as a whole and used to produce ciphertext blocks. The block cipher takes blocks that are b bits long and encrypts them to blocks that are also b bits long. Block sizes are typically 64 or 128 bits long. Encryption Secret Key (Symmetric Key) : the same key is used for encryption and decryption Public Key (Asymmetric Key) in an asymmetric, the encryption and decryption keys are different but related. The encryption key is known as the public key and the decryption key is known as the private key. The public and private keys are known as a key pair. Symmetric Key Encryption DES The Data Encryption Standard (DES) has been the worldwide encryption standard for a long time. IBM developed DES in 1975, and it has held up remarkably well against years of cryptanalysis. DES is a symmetric encryption algorithm with a fixed key length of 56 bits. The algorithm is still good, but because of the short key length, it is susceptible to brute-force attacks that have sufficient resources. DES usually operates in block mode, whereby it encrypts data in 64-bit blocks. The same algorithm and key are used for both encryption and decryption. Because DES is based on simple mathematical functions, it can be easily implemented and accelerated in hardware. Triple DES With advances in computer processing power, the original 56-bit DES key became too short to withstand an attacker with even a limited budget. One way of increasing the effective key length of DES without changing the well-analyzed algorithm itself is to use the same algorithm with different keys several times in a row. The technique of applying DES three times in a row to a plain text block is called Triple DES (3DES). The 3DES technique is shown in Figure. Brute-force attacks on 3DES are considered unfeasible today. Because the basic algorithm has been tested in the field for more than 25 years, it is considered to be more trustworthy than its predecessor. AES On October 2, 2000, The U.S. National Institute of Standards and Technology (NIST) announced the selection of the Rijndael cipher as the AES algorithm. This cipher, developed by Joan Daemen and Vincent Rijmen, has a variable block length and key length. The algorithm currently specifies how to use keys with a length of 128, 192, or 256 bits to encrypt blocks with a length of 128, 192, or 256 bits (all nine combinations of key length and block length are possible). Both block and key lengths can be extended easily to multiples of 32 bits. AES was chosen to replace DES and 3DES because they are either too weak (DES, in terms of key length) or too slow (3DES) to run on modern, efficient hardware. AES is more efficient and much faster, usually by a factor of 5 compared to DES on the same hardware. AES is also more suitable for high throughput, especially if pure software encryption is used. However, AES is a relatively young algorithm, and as the golden rule of cryptography states, \u201cA more mature algorithm is always more trusted.\u201d Asymmetric Key Algorithm In a symmetric key system, Alice first puts the secret message in a box and then padlocks the box using a lock to which she has a key. She then sends the box to Bob through regular mail. When Bob receives the box, he uses an identical copy of Alice's key (which he has obtained previously) to open the box and read the message. In an asymmetric key system, instead of opening the box when he receives it, Bob simply adds his own personal lock to the box and returns the box through public mail to Alice. Alice uses her key to remove her lock and returns the box to Bob, with Bob's lock still in place. Finally, Bob uses his key to remove his lock and reads the message from Alice. The critical advantage in an asymmetric system is that Alice never needs to send a copy of her key to Bob. This reduces the possibility that a third party (for example, an unscrupulous postmaster) can copy the key while it is in transit to Bob, allowing that third party to spy on all future messages sent by Alice. In addition, if Bob is careless and allows someone else to copy his key, Alice's messages to Bob are compromised, but Alice's messages to other people remain secret NOTE : In terms of TLS key exchange, this is the common approach. Diffie-Hellman The protocol has two system parameters, p and g. They are both public and may be used by everybody. Parameter p is a prime number, and parameter g (usually called a generator) is an integer that is smaller than p, but with the following property: For every number n between 1 and p \u2013 1 inclusive, there is a power k of g such that n = gk mod p. Diffie Hellman algorithm is an asymmetric algorithm used to establish a shared secret for a symmetric key algorithm. Nowadays most of the people use hybrid cryptosystem i.e, a combination of symmetric and asymmetric encryption. Asymmetric Encryption is used as a technique in key exchange mechanism to share a secret key and after the key is shared between sender and receiver, the communication will take place using symmetric encryption. The shared secret key will be used to encrypt the communication. Refer: https://medium.com/@akhigbemmanuel/what-is-the-diffie-hellman-key-exchange-algorithm-84d60025a30d RSA The RSA algorithm is very flexible and has a variable key length where, if necessary, speed can be traded for the level of security of the algorithm. The RSA keys are usually 512 to 2048 bits long. RSA has withstood years of extensive cryptanalysis. Although those years neither proved nor disproved RSA's security, they attest to a confidence level in the algorithm. RSA security is based on the difficulty of factoring very large numbers. If an easy method of factoring these large numbers were discovered, the effectiveness of RSA would be destroyed. Refer: https://medium.com/curiositypapers/a-complete-explanation-of-rsa-asymmetric-encryption-742c5971e0f NOTE : RSA Keys can be used for key exchange just like Diffie Hellman Hashing Algorithms Hashing is one of the mechanisms used for data integrity assurance. Hashing is based on a one-way mathematical function, which is relatively easy to compute but significantly harder to reverse. A hash function, which is a one-way function to input data to produce a fixed-length digest (fingerprint) of output data. The digest is cryptographically strong; that is, it is impossible to recover input data from its digest. If the input data changes just a little, the digest (fingerprint) changes substantially in what is called an avalanche effect. More: https://medium.com/@rauljordan/the-state-of-hashing-algorithms-the-why-the-how-and-the-future-b21d5c0440de https://medium.com/@StevieCEllis/the-beautiful-hash-algorithm-f18d9d2b84fb MD5 MD5 is a one-way function with which it is easy to compute the hash from the given input data, but it is unfeasible to compute input data given only a hash. SHA-1 MD5 is considered less secure than SHA-1 because MD5 has some weaknesses. HA-1 also uses a stronger, 160-bit digest, which makes MD5 the second choice as hash methods are concerned. The algorithm takes a message of less than 264 bits in length and produces a 160-bit message digest. This algorithm is slightly slower than MD5. NOTE : SHA-1 is also recently demonstrated to be broken, Minimum current recommendation is SHA-256 Digital Certificates Digital signatures, provide a means to digitally authenticate devices and individual users. In public-key cryptography, such as the RSA encryption system, each user has a key-pair containing both a public key and a private key. The keys act as complements, and anything encrypted with one of the keys can be decrypted with the other. In simple terms, a signature is formed when data is encrypted with a user's private key. The receiver verifies the signature by decrypting the message with the sender's public key. Key management is often considered the most difficult task in designing and implementing cryptographic systems. Businesses can simplify some of the deployment and management issues that are encountered with secured data communications by employing a Public Key Infrastructure (PKI). Because corporations often move security-sensitive communications across the Internet, an effective mechanism must be implemented to protect sensitive information from the threats presented on the Internet. PKI provides a hierarchical framework for managing digital security attributes. Each PKI participant holds a digital certificate that has been issued by a CA (either public or private). The certificate contains several attributes that are used when parties negotiate a secure connection. These attributes must include the certificate validity period, end-host identity information, encryption keys that will be used for secure communications, and the signature of the issuing CA. Optional attributes may be included, depending on the requirements and capability of the PKI. A CA can be a trusted third party, such as VeriSign or Entrust, or a private (in-house) CA that you establish within your organization. The fact that the message could be decrypted using the sender's public key means that the holder of the private key created the message. This process relies on the receiver having a copy of the sender's public key and knowing with a high degree of certainty that it really does belong to the sender and not to someone pretending to be the sender. To validate the CA's signature, the receiver must know the CA's public key. Normally, this is handled out-of-band or through an operation performed during the installation of the certificate. For instance, most web browsers are configured with the root certificates of several CAs by default. CA Enrollment process The end host generates a private-public key pair. The end host generates a certificate request, which it forwards to the CA. Manual human intervention is required to approve the enrollment request, which is received by the CA. After the CA operator approves the request, the CA signs the certificate request with its private key and returns the completed certificate to the end host. The end host writes the certificate into a nonvolatile storage area (PC hard disk or NVRAM on Cisco routers). Refer : https://www.ssh.com/manuals/server-zos-product/55/ch06s03s01.html Login Security SSH SSH, the Secure Shell, is a popular, powerful, software-based approach to network security. Whenever data is sent by a computer to the network, SSH automatically encrypts (scrambles) it. Then, when the data reaches its intended recipient, SSH automatically decrypts (unscrambles) it. The result is transparent encryption: users can work normally, unaware that their communications are safely encrypted on the network. In addition, SSH can use modern, secure encryption algorithms based on how it's being configured and is effective enough to be found within mission-critical applications at major corporations. SSH has a client/server architecture An SSH server program, typically installed and run by a system administrator, accepts or rejects incoming connections to its host computer. Users then run SSH client programs, typically on other computers, to make requests of the SSH server, such as \u201cPlease log me in,\u201d \u201cPlease send me a file,\u201d or \u201cPlease execute this command.\u201d All communications between clients and servers are securely encrypted and protected from modification. What SSH is not: Although SSH stands for Secure Shell, it is not a true shell in the sense of the Unix Bourne shell and C shell. It is not a command interpreter, nor does it provide wildcard expansion, command history, and so forth. Rather, SSH creates a channel for running a shell on a remote computer, with end-to-end encryption between the two systems. The major features and guarantees of the SSH protocol are: Privacy of your data, via strong encryption Integrity of communications, guaranteeing they haven\u2019t been altered Authentication, i.e., proof of identity of senders and receivers Authorization, i.e., access control to accounts Forwarding or tunnelling to encrypt other TCP/IP-based sessions Kerberos According to Greek mythology Kerberos (Cerberus) was the gigantic, three-headed dog that guards the gates of the underworld to prevent the dead from leaving. So when it comes to Computer Science, Kerberos is a network authentication protocol and is currently the default authentication technology used by Microsoft Active Directory to authenticate users to services within a local area network. Kerberos uses symmetric-key cryptography and requires a trusted third-party authentication service to verify user identities. So they used the name of Kerberos for their computer network authentication protocol as the three heads of the Kerberos represent: a client: A user/ a service a server: Kerberos protected hosts reside - a Key Distribution Center (KDC), which acts as the trusted third-party authentication service. The KDC includes the following two servers: Authentication Server (AS) that performs the initial authentication and issues ticket-granting tickets (TGT) for users. Ticket-Granting Server (TGS) that issues service tickets that are based on the initial ticket-granting tickets (TGT). Certificate Chain The first part of the output of the OpenSSL command shows three certificates numbered 0, 1, and 2(not 2 anymore). Each certificate has a subject, s, and an issuer, i. The first certificate, number 0, is called the end-entity certificate. The subject line tells us it\u2019s valid for any subdomain of google.com because its subject is set to *.google.com. $ openssl s_client -connect www.google.com:443 -CApath /etc/ssl/certs CONNECTED(00000005) depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign verify return:1 depth=1 C = US, O = Google Trust Services, CN = GTS CA 1O1 verify return:1 depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = www.google.com verify return:1 --- Certificate chain 0 s:/C=US/ST=California/L=Mountain View/O=Google LLC/CN=www.google.com i:/C=US/O=Google Trust Services/CN=GTS CA 1O1 1 s:/C=US/O=Google Trust Services/CN=GTS CA 1O1 i:/OU=GlobalSign Root CA - R2/O=GlobalSign/CN=GlobalSign --- Server certificate The issuer line indicates it\u2019s issued by Google Internet Authority G2, which also happens to be the subject of the second certificate, number 1 What the OpenSSL command line doesn\u2019t show here is the trust store that contains the list of CA certificates trusted by the system OpenSSL runs on. The public certificate of GlobalSign Authority must be present in the system\u2019s trust store to close the verification chain. This is called a chain of trust, and the figure below summarizes its behaviour at a high level. High-level view of the concept of chain of trust applied to verifying the authenticity of a website. The Root CA in the Firefox trust store provides the initial trust to verify the entire chain and trust the end-entity certificate. TLS Handshake The client sends a HELLO message to the server with a list of protocols and algorithms it supports. The server says HELLO back and sends its chain of certificates. Based on the capabilities of the client, the server picks a cipher suite. If the cipher suite supports ephemeral key exchange, like ECDHE does(ECDHE is an algorithm known as the Elliptic Curve Diffie-Hellman Exchange), the server and the client negotiate a pre-master key with the Diffie-Hellman algorithm. The pre-master key is never sent over the wire. The client and server create a session key that will be used to encrypt the data transiting through the connection. At the end of the handshake, both parties possess a secret session key used to encrypt data for the rest of the connection. This is what OpenSSL refers to as Master-Key NOTE There are 3 versions of TLS , TLS 1.0, 1.1 & 1.2 TLS 1.0 was released in 1999, making it a nearly two-decade-old protocol. It has been known to be vulnerable to attacks\u2014such as BEAST and POODLE\u2014for years, in addition to supporting weak cryptography, which doesn\u2019t keep modern-day connections sufficiently secure. TLS 1.1 is the forgotten \u201cmiddle child.\u201d It also has bad cryptography like its younger sibling. In most software, it was leapfrogged by TLS 1.2 and it\u2019s rare to see TLS 1.1 used. \u201cPerfect\u201d Forward Secrecy The term \u201cephemeral\u201d in the key exchange provides an important security feature mis-named perfect forward secrecy (PFS) or just \u201cForward Secrecy\u201d. In a non-ephemeral key exchange, the client sends the pre-master key to the server by encrypting it with the server\u2019s public key. The server then decrypts the pre-master key with its private key. If at a later point in time, the private key of the server is compromised, an attacker can go back to this handshake, decrypt the pre-master key, obtain the session key, and decrypt the entire traffic. Non-ephemeral key exchanges are vulnerable to attacks that may happen in the future on recorded traffic. And because people seldom change their password, decrypting data from the past may still be valuable for an attacker. An ephemeral key exchange like DHE, or its variant on elliptic curve, ECDHE, solves this problem by not transmitting the pre-master key over the wire. Instead, the pre-master key is computed by both the client and the server in isolation, using nonsensitive information exchanged publicly. Because the pre-master key can\u2019t be decrypted later by an attacker, the session key is safe from future attacks: hence, the term perfect forward secrecy. Keys are changed every X blocks along the stream. That prevents an attacker from simply sniffing the stream and applying brute force to crack the whole thing. \"Forward secrecy\" means that just because I can decrypt block M, does not mean that I can decrypt block Q Downside: The downside to PFS is that all those extra computational steps induce latency on the handshake and slow the user down. To avoid repeating this expensive work at every connection, both sides cache the session key for future use via a technique called session resumption. This is what the session-ID and TLS ticket are for: they allow a client and server that share a session ID to skip over the negotiation of a session key, because they already agreed on one previously, and go directly to exchanging data securely.","title":"Fundamentals of Security"},{"location":"security/fundamentals/#part-i-fundamentals","text":"","title":"Part I: Fundamentals"},{"location":"security/fundamentals/#introduction-to-security-overview-for-sre","text":"If you look closely, both Site Reliability Engineering and Security Engineering are concerned with keeping a system usable. Issues like broken releases, capacity shortages, and misconfigurations can make a system unusable (at least temporarily). Security or privacy incidents that break the trust of users also undermine the usefulness of a system. Consequently, system security should be top of mind for SREs. SREs should be involved in both significant design discussions and actual system changes. They have quite a big role in System design & hence are quite sometimes the first line of defence. SRE\u2019s help in preventing bad design & implementations which can affect the overall security of the infrastructure. Successfully designing, implementing, and maintaining systems requires a commitment to the full system lifecycle . This commitment is possible only when security and reliability are central elements in the architecture of systems. Core Pillars of Information Security : Confidentiality \u2013 only allow access to data for which the user is permitted Integrity \u2013 ensure data is not tampered or altered by unauthorized users Availability \u2013 ensure systems and data are available to authorized users when they need it Thinking like a Security Engineer When starting a new application or re-factoring an existing application, you should consider each functional feature, and consider: Is the process surrounding this feature as safe as possible? In other words, is this a flawed process? If I were evil, how would I abuse this feature? Or more specifically failing to address how a feature can be abused can cause design flaws. Is the feature required to be on by default? If so, are there limits or options that could help reduce the risk from this feature? Security Principles By OWASP (Open Web Application Security Project) Minimize attack surface area : Every feature that is added to an application adds a certain amount of risk to the overall application. The aim of secure development is to reduce the overall risk by reducing the attack surface area. For example, a web application implements online help with a search function. The search function may be vulnerable to SQL injection attacks. If the help feature was limited to authorized users, the attack likelihood is reduced. If the help feature\u2019s search function was gated through centralized data validation routines, the ability to perform SQL injection is dramatically reduced. However, if the help feature was re-written to eliminate the search function (through a better user interface, for example), this almost eliminates the attack surface area, even if the help feature was available to the Internet at large. Establish secure defaults: There are many ways to deliver an \u201cout of the box\u201d experience for users. However, by default, the experience should be secure, and it should be up to the user to reduce their security \u2013 if they are allowed. For example, by default, password ageing and complexity should be enabled. Users might be allowed to turn these two features off to simplify their use of the application and increase their risk. Default Passwords of routers, IoT devices should be changed Principle of Least privilege The principle of least privilege recommends that accounts have the least amount of privilege required to perform their business processes. This encompasses user rights, resource permissions such as CPU limits, memory, network, and file system permissions. For example, if a middleware server only requires access to the network, read access to a database table, and the ability to write to a log, this describes all the permissions that should be granted. Under no circumstances should the middleware be granted administrative privileges. Principle of Defense in depth The principle of defence in depth suggests that where one control would be reasonable, more controls that approach risks in different fashions are better. Controls, when used in depth, can make severe vulnerabilities extraordinarily difficult to exploit and thus unlikely to occur. With secure coding, this may take the form of tier-based validation, centralized auditing controls, and requiring users to be logged on all pages. For example, a flawed administrative interface is unlikely to be vulnerable to an anonymous attack if it correctly gates access to production management networks, checks for administrative user authorization, and logs all access. Fail securely Applications regularly fail to process transactions for many reasons. How they fail can determine if an application is secure or not. ``` is_admin = true; try { code_which_may_faile(); is_admin = is_user_assigned_role(\"Adminstrator\"); } catch (Exception err) { log.error(err.toString()); } ``` - If either codeWhichMayFail() or isUserInRole fails or throws an exception, the user is an admin by default. This is obviously a security risk. Don\u2019t trust services Many organizations utilize the processing capabilities of third-party partners, who more than likely have different security policies and posture than you. It is unlikely that you can influence or control any external third party, whether they are home users or major suppliers or partners. Therefore, the implicit trust of externally run systems is not warranted. All external systems should be treated similarly. For example, a loyalty program provider provides data that is used by Internet Banking, providing the number of reward points and a small list of potential redemption items. However, the data should be checked to ensure that it is safe to display to end-users and that the reward points are a positive number, and not improbably large. Separation of duties The key to fraud control is the separation of duties. For example, someone who requests a computer cannot also sign for it, nor should they directly receive the computer. This prevents the user from requesting many computers and claiming they never arrived. Certain roles have different levels of trust than normal users. In particular, administrators are different from normal users. In general, administrators should not be users of the application. For example, an administrator should be able to turn the system on or off, set password policy but shouldn\u2019t be able to log on to the storefront as a super privileged user, such as being able to \u201cbuy\u201d goods on behalf of other users. Avoid security by obscurity Security through obscurity is a weak security control, and nearly always fails when it is the only control. This is not to say that keeping secrets is a bad idea, it simply means that the security of systems should not be reliant upon keeping details hidden. For example, the security of an application should not rely upon knowledge of the source code being kept secret. The security should rely upon many other factors, including reasonable password policies, defence in depth, business transaction limits, solid network architecture, and fraud, and audit controls. A practical example is Linux. Linux\u2019s source code is widely available, and yet when properly secured, Linux is a secure and robust operating system. Keep security simple Attack surface area and simplicity go hand in hand. Certain software engineering practices prefer overly complex approaches to what would otherwise be a relatively straightforward and simple design. Developers should avoid the use of double negatives and complex architectures when a simpler approach would be faster and simpler. For example, although it might be fashionable to have a slew of singleton entity beans running on a separate middleware server, it is more secure and faster to simply use global variables with an appropriate mutex mechanism to protect against race conditions. Fix security issues correctly Once a security issue has been identified, it is important to develop a test for it and to understand the root cause of the issue. When design patterns are used, the security issue is likely widespread amongst all codebases, so developing the right fix without introducing regressions is essential. For example, a user has found that they can see another user\u2019s balance by adjusting their cookie. The fix seems to be relatively straightforward, but as the cookie handling code is shared among all applications, a change to just one application will trickle through to all other applications. The fix must, therefore, be tested on all affected applications. Reliability & Security Reliability and security are both crucial components of a truly trustworthy system, but building systems that are both reliable and secure is difficult. While the requirements for reliability and security share many common properties, they also require different design considerations. It is easy to miss the subtle interplay between reliability and security that can cause unexpected outcomes Ex: A password management application failure was triggered by a reliability problem i.e poor load-balancing and load-shedding strategies and its recovery were later complicated by multiple measures (HSM mechanism which needs to be plugged into server racks, which works as an authentication & the HSM token supposedly locked inside a case.. & the problem can be further elongated ) designed to increase the security of the system.","title":"Introduction to Security Overview for SRE"},{"location":"security/fundamentals/#authentication-vs-authorization","text":"Authentication is the act of validating that users are who they claim to be. Passwords are the most common authentication factor\u2014if a user enters the correct password, the system assumes the identity is valid and grants access. Other technologies such as One-Time Pins, authentication apps, and even biometrics can also be used to authenticate identity. In some instances, systems require the successful verification of more than one factor before granting access. This multi-factor authentication (MFA) requirement is often deployed to increase security beyond what passwords alone can provide. Authorization in system security is the process of giving the user permission to access a specific resource or function. This term is often used interchangeably with access control or client privilege. Giving someone permission to download a particular file on a server or providing individual users with administrative access to an application are good examples. In secure environments, authorization must always follow authentication, users should first prove that their identities are genuine before an organization\u2019s administrators grant them access to the requested resources.","title":"Authentication vs Authorization"},{"location":"security/fundamentals/#common-authentication-flow-local-authentication","text":"The user registers using an identifier like username/email/mobile The application stores user credentials in the database The application sends a verification email/message to validate the registration Post successful registration, the user enters credentials for logging in On successful authentication, the user is allowed access to specific resources","title":"Common authentication flow (local authentication)"},{"location":"security/fundamentals/#openidoauth","text":"OpenID is an authentication protocol that allows us to authenticate users without using a local auth system. In such a scenario, a user has to be registered with an OpenID Provider and the same provider should be integrated with the authentication flow of your application. To verify the details, we have to forward the authentication requests to the provider. On successful authentication, we receive a success message and/or profile details with which we can execute the necessary flow. OAuth is an authorization mechanism that allows your application user access to a provider(Gmail/Facebook/Instagram/etc). On successful response, we (your application) receive a token with which the application can access certain APIs on behalf of a user. OAuth is convenient in case your business use case requires some certain user-facing APIs like access to Google Drive or sending tweets on your behalf. Most OAuth 2.0 providers can be used for pseudo authentication. Having said that, it can get pretty complicated if you are using multiple OAuth providers to authenticate users on top of the local authentication system.","title":"OpenID/OAuth"},{"location":"security/fundamentals/#cryptography","text":"It is the science and study of hiding any text in such a way that only the intended recipients or authorized persons can read it and that any text can even use things such as invisible ink or the mechanical cryptography machines of the past. Cryptography is necessary for securing critical or proprietary information and is used to encode private data messages by converting some plain text into ciphertext. At its core, there are two ways of doing this, more advanced methods are all built upon.","title":"Cryptography"},{"location":"security/fundamentals/#ciphers","text":"Ciphers are the cornerstone of cryptography. A cipher is a set of algorithms that performs encryption or decryption on a message. An encryption algorithm (E) takes a secret key (k) and a message (m) and produces a ciphertext (c). Similarly, a Decryption algorithm (D) takes a secret key (K) and the previous resulting Ciphertext (C). They are represented as follows: E(k,m) = c D(k,c) = m This also means that for it to be a cipher, it must satisfy the consistency equation as follows, making it possible to decrypt. D(k,E(k,m)) = m Stream Ciphers: The message is broken into characters or bits and enciphered with a key or keystream(should be random and generated independently of the message stream) that is as long as the plaintext bitstream. If the keystream is random, this scheme would be unbreakable unless the keystream was acquired, making it unconditionally secure. The keystream must be provided to both parties in a secure way to prevent its release. Block Ciphers: Block ciphers \u2014 process messages in blocks, each of which is then encrypted or decrypted. A block cipher is a symmetric cipher in which blocks of plaintext are treated as a whole and used to produce ciphertext blocks. The block cipher takes blocks that are b bits long and encrypts them to blocks that are also b bits long. Block sizes are typically 64 or 128 bits long. Encryption Secret Key (Symmetric Key) : the same key is used for encryption and decryption Public Key (Asymmetric Key) in an asymmetric, the encryption and decryption keys are different but related. The encryption key is known as the public key and the decryption key is known as the private key. The public and private keys are known as a key pair. Symmetric Key Encryption DES The Data Encryption Standard (DES) has been the worldwide encryption standard for a long time. IBM developed DES in 1975, and it has held up remarkably well against years of cryptanalysis. DES is a symmetric encryption algorithm with a fixed key length of 56 bits. The algorithm is still good, but because of the short key length, it is susceptible to brute-force attacks that have sufficient resources. DES usually operates in block mode, whereby it encrypts data in 64-bit blocks. The same algorithm and key are used for both encryption and decryption. Because DES is based on simple mathematical functions, it can be easily implemented and accelerated in hardware. Triple DES With advances in computer processing power, the original 56-bit DES key became too short to withstand an attacker with even a limited budget. One way of increasing the effective key length of DES without changing the well-analyzed algorithm itself is to use the same algorithm with different keys several times in a row. The technique of applying DES three times in a row to a plain text block is called Triple DES (3DES). The 3DES technique is shown in Figure. Brute-force attacks on 3DES are considered unfeasible today. Because the basic algorithm has been tested in the field for more than 25 years, it is considered to be more trustworthy than its predecessor. AES On October 2, 2000, The U.S. National Institute of Standards and Technology (NIST) announced the selection of the Rijndael cipher as the AES algorithm. This cipher, developed by Joan Daemen and Vincent Rijmen, has a variable block length and key length. The algorithm currently specifies how to use keys with a length of 128, 192, or 256 bits to encrypt blocks with a length of 128, 192, or 256 bits (all nine combinations of key length and block length are possible). Both block and key lengths can be extended easily to multiples of 32 bits. AES was chosen to replace DES and 3DES because they are either too weak (DES, in terms of key length) or too slow (3DES) to run on modern, efficient hardware. AES is more efficient and much faster, usually by a factor of 5 compared to DES on the same hardware. AES is also more suitable for high throughput, especially if pure software encryption is used. However, AES is a relatively young algorithm, and as the golden rule of cryptography states, \u201cA more mature algorithm is always more trusted.\u201d Asymmetric Key Algorithm In a symmetric key system, Alice first puts the secret message in a box and then padlocks the box using a lock to which she has a key. She then sends the box to Bob through regular mail. When Bob receives the box, he uses an identical copy of Alice's key (which he has obtained previously) to open the box and read the message. In an asymmetric key system, instead of opening the box when he receives it, Bob simply adds his own personal lock to the box and returns the box through public mail to Alice. Alice uses her key to remove her lock and returns the box to Bob, with Bob's lock still in place. Finally, Bob uses his key to remove his lock and reads the message from Alice. The critical advantage in an asymmetric system is that Alice never needs to send a copy of her key to Bob. This reduces the possibility that a third party (for example, an unscrupulous postmaster) can copy the key while it is in transit to Bob, allowing that third party to spy on all future messages sent by Alice. In addition, if Bob is careless and allows someone else to copy his key, Alice's messages to Bob are compromised, but Alice's messages to other people remain secret NOTE : In terms of TLS key exchange, this is the common approach. Diffie-Hellman The protocol has two system parameters, p and g. They are both public and may be used by everybody. Parameter p is a prime number, and parameter g (usually called a generator) is an integer that is smaller than p, but with the following property: For every number n between 1 and p \u2013 1 inclusive, there is a power k of g such that n = gk mod p. Diffie Hellman algorithm is an asymmetric algorithm used to establish a shared secret for a symmetric key algorithm. Nowadays most of the people use hybrid cryptosystem i.e, a combination of symmetric and asymmetric encryption. Asymmetric Encryption is used as a technique in key exchange mechanism to share a secret key and after the key is shared between sender and receiver, the communication will take place using symmetric encryption. The shared secret key will be used to encrypt the communication. Refer: https://medium.com/@akhigbemmanuel/what-is-the-diffie-hellman-key-exchange-algorithm-84d60025a30d RSA The RSA algorithm is very flexible and has a variable key length where, if necessary, speed can be traded for the level of security of the algorithm. The RSA keys are usually 512 to 2048 bits long. RSA has withstood years of extensive cryptanalysis. Although those years neither proved nor disproved RSA's security, they attest to a confidence level in the algorithm. RSA security is based on the difficulty of factoring very large numbers. If an easy method of factoring these large numbers were discovered, the effectiveness of RSA would be destroyed. Refer: https://medium.com/curiositypapers/a-complete-explanation-of-rsa-asymmetric-encryption-742c5971e0f NOTE : RSA Keys can be used for key exchange just like Diffie Hellman Hashing Algorithms Hashing is one of the mechanisms used for data integrity assurance. Hashing is based on a one-way mathematical function, which is relatively easy to compute but significantly harder to reverse. A hash function, which is a one-way function to input data to produce a fixed-length digest (fingerprint) of output data. The digest is cryptographically strong; that is, it is impossible to recover input data from its digest. If the input data changes just a little, the digest (fingerprint) changes substantially in what is called an avalanche effect. More: https://medium.com/@rauljordan/the-state-of-hashing-algorithms-the-why-the-how-and-the-future-b21d5c0440de https://medium.com/@StevieCEllis/the-beautiful-hash-algorithm-f18d9d2b84fb MD5 MD5 is a one-way function with which it is easy to compute the hash from the given input data, but it is unfeasible to compute input data given only a hash. SHA-1 MD5 is considered less secure than SHA-1 because MD5 has some weaknesses. HA-1 also uses a stronger, 160-bit digest, which makes MD5 the second choice as hash methods are concerned. The algorithm takes a message of less than 264 bits in length and produces a 160-bit message digest. This algorithm is slightly slower than MD5. NOTE : SHA-1 is also recently demonstrated to be broken, Minimum current recommendation is SHA-256 Digital Certificates Digital signatures, provide a means to digitally authenticate devices and individual users. In public-key cryptography, such as the RSA encryption system, each user has a key-pair containing both a public key and a private key. The keys act as complements, and anything encrypted with one of the keys can be decrypted with the other. In simple terms, a signature is formed when data is encrypted with a user's private key. The receiver verifies the signature by decrypting the message with the sender's public key. Key management is often considered the most difficult task in designing and implementing cryptographic systems. Businesses can simplify some of the deployment and management issues that are encountered with secured data communications by employing a Public Key Infrastructure (PKI). Because corporations often move security-sensitive communications across the Internet, an effective mechanism must be implemented to protect sensitive information from the threats presented on the Internet. PKI provides a hierarchical framework for managing digital security attributes. Each PKI participant holds a digital certificate that has been issued by a CA (either public or private). The certificate contains several attributes that are used when parties negotiate a secure connection. These attributes must include the certificate validity period, end-host identity information, encryption keys that will be used for secure communications, and the signature of the issuing CA. Optional attributes may be included, depending on the requirements and capability of the PKI. A CA can be a trusted third party, such as VeriSign or Entrust, or a private (in-house) CA that you establish within your organization. The fact that the message could be decrypted using the sender's public key means that the holder of the private key created the message. This process relies on the receiver having a copy of the sender's public key and knowing with a high degree of certainty that it really does belong to the sender and not to someone pretending to be the sender. To validate the CA's signature, the receiver must know the CA's public key. Normally, this is handled out-of-band or through an operation performed during the installation of the certificate. For instance, most web browsers are configured with the root certificates of several CAs by default. CA Enrollment process The end host generates a private-public key pair. The end host generates a certificate request, which it forwards to the CA. Manual human intervention is required to approve the enrollment request, which is received by the CA. After the CA operator approves the request, the CA signs the certificate request with its private key and returns the completed certificate to the end host. The end host writes the certificate into a nonvolatile storage area (PC hard disk or NVRAM on Cisco routers). Refer : https://www.ssh.com/manuals/server-zos-product/55/ch06s03s01.html","title":"Ciphers"},{"location":"security/fundamentals/#login-security","text":"","title":"Login Security"},{"location":"security/fundamentals/#ssh","text":"SSH, the Secure Shell, is a popular, powerful, software-based approach to network security. Whenever data is sent by a computer to the network, SSH automatically encrypts (scrambles) it. Then, when the data reaches its intended recipient, SSH automatically decrypts (unscrambles) it. The result is transparent encryption: users can work normally, unaware that their communications are safely encrypted on the network. In addition, SSH can use modern, secure encryption algorithms based on how it's being configured and is effective enough to be found within mission-critical applications at major corporations. SSH has a client/server architecture An SSH server program, typically installed and run by a system administrator, accepts or rejects incoming connections to its host computer. Users then run SSH client programs, typically on other computers, to make requests of the SSH server, such as \u201cPlease log me in,\u201d \u201cPlease send me a file,\u201d or \u201cPlease execute this command.\u201d All communications between clients and servers are securely encrypted and protected from modification. What SSH is not: Although SSH stands for Secure Shell, it is not a true shell in the sense of the Unix Bourne shell and C shell. It is not a command interpreter, nor does it provide wildcard expansion, command history, and so forth. Rather, SSH creates a channel for running a shell on a remote computer, with end-to-end encryption between the two systems. The major features and guarantees of the SSH protocol are: Privacy of your data, via strong encryption Integrity of communications, guaranteeing they haven\u2019t been altered Authentication, i.e., proof of identity of senders and receivers Authorization, i.e., access control to accounts Forwarding or tunnelling to encrypt other TCP/IP-based sessions","title":"SSH"},{"location":"security/fundamentals/#kerberos","text":"According to Greek mythology Kerberos (Cerberus) was the gigantic, three-headed dog that guards the gates of the underworld to prevent the dead from leaving. So when it comes to Computer Science, Kerberos is a network authentication protocol and is currently the default authentication technology used by Microsoft Active Directory to authenticate users to services within a local area network. Kerberos uses symmetric-key cryptography and requires a trusted third-party authentication service to verify user identities. So they used the name of Kerberos for their computer network authentication protocol as the three heads of the Kerberos represent: a client: A user/ a service a server: Kerberos protected hosts reside - a Key Distribution Center (KDC), which acts as the trusted third-party authentication service. The KDC includes the following two servers: Authentication Server (AS) that performs the initial authentication and issues ticket-granting tickets (TGT) for users. Ticket-Granting Server (TGS) that issues service tickets that are based on the initial ticket-granting tickets (TGT).","title":"Kerberos"},{"location":"security/fundamentals/#certificate-chain","text":"The first part of the output of the OpenSSL command shows three certificates numbered 0, 1, and 2(not 2 anymore). Each certificate has a subject, s, and an issuer, i. The first certificate, number 0, is called the end-entity certificate. The subject line tells us it\u2019s valid for any subdomain of google.com because its subject is set to *.google.com. $ openssl s_client -connect www.google.com:443 -CApath /etc/ssl/certs CONNECTED(00000005) depth=2 OU = GlobalSign Root CA - R2, O = GlobalSign, CN = GlobalSign verify return:1 depth=1 C = US, O = Google Trust Services, CN = GTS CA 1O1 verify return:1 depth=0 C = US, ST = California, L = Mountain View, O = Google LLC, CN = www.google.com verify return:1 --- Certificate chain 0 s:/C=US/ST=California/L=Mountain View/O=Google LLC/CN=www.google.com i:/C=US/O=Google Trust Services/CN=GTS CA 1O1 1 s:/C=US/O=Google Trust Services/CN=GTS CA 1O1 i:/OU=GlobalSign Root CA - R2/O=GlobalSign/CN=GlobalSign --- Server certificate The issuer line indicates it\u2019s issued by Google Internet Authority G2, which also happens to be the subject of the second certificate, number 1 What the OpenSSL command line doesn\u2019t show here is the trust store that contains the list of CA certificates trusted by the system OpenSSL runs on. The public certificate of GlobalSign Authority must be present in the system\u2019s trust store to close the verification chain. This is called a chain of trust, and the figure below summarizes its behaviour at a high level. High-level view of the concept of chain of trust applied to verifying the authenticity of a website. The Root CA in the Firefox trust store provides the initial trust to verify the entire chain and trust the end-entity certificate.","title":"Certificate Chain"},{"location":"security/fundamentals/#tls-handshake","text":"The client sends a HELLO message to the server with a list of protocols and algorithms it supports. The server says HELLO back and sends its chain of certificates. Based on the capabilities of the client, the server picks a cipher suite. If the cipher suite supports ephemeral key exchange, like ECDHE does(ECDHE is an algorithm known as the Elliptic Curve Diffie-Hellman Exchange), the server and the client negotiate a pre-master key with the Diffie-Hellman algorithm. The pre-master key is never sent over the wire. The client and server create a session key that will be used to encrypt the data transiting through the connection. At the end of the handshake, both parties possess a secret session key used to encrypt data for the rest of the connection. This is what OpenSSL refers to as Master-Key NOTE There are 3 versions of TLS , TLS 1.0, 1.1 & 1.2 TLS 1.0 was released in 1999, making it a nearly two-decade-old protocol. It has been known to be vulnerable to attacks\u2014such as BEAST and POODLE\u2014for years, in addition to supporting weak cryptography, which doesn\u2019t keep modern-day connections sufficiently secure. TLS 1.1 is the forgotten \u201cmiddle child.\u201d It also has bad cryptography like its younger sibling. In most software, it was leapfrogged by TLS 1.2 and it\u2019s rare to see TLS 1.1 used.","title":"TLS Handshake"},{"location":"security/fundamentals/#perfect-forward-secrecy","text":"The term \u201cephemeral\u201d in the key exchange provides an important security feature mis-named perfect forward secrecy (PFS) or just \u201cForward Secrecy\u201d. In a non-ephemeral key exchange, the client sends the pre-master key to the server by encrypting it with the server\u2019s public key. The server then decrypts the pre-master key with its private key. If at a later point in time, the private key of the server is compromised, an attacker can go back to this handshake, decrypt the pre-master key, obtain the session key, and decrypt the entire traffic. Non-ephemeral key exchanges are vulnerable to attacks that may happen in the future on recorded traffic. And because people seldom change their password, decrypting data from the past may still be valuable for an attacker. An ephemeral key exchange like DHE, or its variant on elliptic curve, ECDHE, solves this problem by not transmitting the pre-master key over the wire. Instead, the pre-master key is computed by both the client and the server in isolation, using nonsensitive information exchanged publicly. Because the pre-master key can\u2019t be decrypted later by an attacker, the session key is safe from future attacks: hence, the term perfect forward secrecy. Keys are changed every X blocks along the stream. That prevents an attacker from simply sniffing the stream and applying brute force to crack the whole thing. \"Forward secrecy\" means that just because I can decrypt block M, does not mean that I can decrypt block Q Downside: The downside to PFS is that all those extra computational steps induce latency on the handshake and slow the user down. To avoid repeating this expensive work at every connection, both sides cache the session key for future use via a technique called session resumption. This is what the session-ID and TLS ticket are for: they allow a client and server that share a session ID to skip over the negotiation of a session key, because they already agreed on one previously, and go directly to exchanging data securely.","title":"\u201cPerfect\u201d Forward Secrecy"},{"location":"security/intro/","text":"Security Prerequisites Linux Basics Linux Networking What to expect from this course The course covers fundamentals of information security along with touching on subjects of system security, network & web security. This course aims to get you familiar with the basics of information security in day to day operations & then as an SRE develop the mindset of ensuring that security takes a front-seat while developing solutions. The course also serves as an introduction to common risks and best practices along with practical ways to find out vulnerable systems and loopholes which might become compromised if not secured. What is not covered under this course The courseware is not an ethical hacking workshop or a very deep dive into the fundamentals of the problems. The course does not deal with hacking or breaking into systems but rather an approach on how to ensure you don\u2019t get into those situations and also to make you aware of different ways a system can be compromised. Course Contents Fundamentals Network Security Threats, Attacks & Defence Writing Secure Code & More Conclusion","title":"Introduction"},{"location":"security/intro/#security","text":"","title":"Security"},{"location":"security/intro/#prerequisites","text":"Linux Basics Linux Networking","title":"Prerequisites"},{"location":"security/intro/#what-to-expect-from-this-course","text":"The course covers fundamentals of information security along with touching on subjects of system security, network & web security. This course aims to get you familiar with the basics of information security in day to day operations & then as an SRE develop the mindset of ensuring that security takes a front-seat while developing solutions. The course also serves as an introduction to common risks and best practices along with practical ways to find out vulnerable systems and loopholes which might become compromised if not secured.","title":"What to expect from this course"},{"location":"security/intro/#what-is-not-covered-under-this-course","text":"The courseware is not an ethical hacking workshop or a very deep dive into the fundamentals of the problems. The course does not deal with hacking or breaking into systems but rather an approach on how to ensure you don\u2019t get into those situations and also to make you aware of different ways a system can be compromised.","title":"What is not covered under this course"},{"location":"security/intro/#course-contents","text":"Fundamentals Network Security Threats, Attacks & Defence Writing Secure Code & More Conclusion","title":"Course Contents"},{"location":"security/network_security/","text":"Part II: Network Security Introduction TCP/IP is the dominant networking technology today. It is a five-layer architecture. These layers are, from top to bottom, the application layer, the transport layer (TCP), the network layer (IP), the data-link layer, and the physical layer. In addition to TCP/IP, there also are other networking technologies. For convenience, we use the OSI network model to represent non-TCP/IP network technologies. Different networks are interconnected using gateways. A gateway can be placed at any layer. The OSI model is a seven-layer architecture. The OSI architecture is similar to the TCP/IP architecture, except that the OSI model specifies two additional layers between the application layer and the transport layer in the TCP/IP architecture. These two layers are the presentation layer and the session layer. Figure 5.1 shows the relationship between the TCP/IP layers and the OSI layers. The application layer in TCP/IP corresponds to the application layer and the presentation layer in OSI. The transport layer in TCP/IP corresponds to the session layer and the transport layer in OSI. The remaining three layers in the TCP/IP architecture are one-to-one correspondent to the remaining three layers in the OSI model. Correspondence between layers of the TCP/IP architecture and the OSI model. Also shown are placements of cryptographic algorithms in network layers, where the dotted arrows indicate actual communications of cryptographic algorithms The functionalities of OSI layers are briefly described as follows: The application layer serves as an interface between applications and network programs. It supports application programs and end-user processing. Common application-layer programs include remote logins, file transfer, email, and Web browsing. The presentation layer is responsible for dealing with data that is formed differently. This protocol layer allows application-layer programs residing on different sides of a communication channel with different platforms to understand each other's data formats regardless of how they are presented. The session layer is responsible for creating, managing, and closing a communication connection. The transport layer is responsible for providing reliable connections, such as packet sequencing, traffic control, and congestion control. The network layer is responsible for routing device-independent data packets from the current hop to the next hop. The data-link layer is responsible for encapsulating device-independent data packets into device-dependent data frames. It has two sublayers: logical link control and media access control. The physical layer is responsible for transmitting device-dependent frames through some physical media. Starting from the application layer, data generated from an application program is passed down layer-by-layer to the physical layer. Data from the previous layer is enclosed in a new envelope at the current layer, where the data from the previous layer is also just an envelope containing the data from the layer before it. This is similar to enclosing a smaller envelope in a larger one. The envelope added at each layer contains sufficient information for handling the packet. Application-layer data are divided into blocks small enough to be encapsulated in an envelope at the next layer. Application data blocks are \u201cdressed up\u201d in the TCP/IP architecture according to the following basic steps. At the sending side, an application data block is encapsulated in a TCP packet when it is passed down to the TCP layer. In other words, a TCP packet consists of a header and a payload, where the header corresponds to the TCP envelope and the payload is the application data block. Likewise, the TCP packet will be encapsulated in an IP packet when it is passed down to the IP layer. An IP packet consists of a header and a payload, which is the TCP packet passed down from the TCP layer. The IP packet will be encapsulated in a device-dependent frame (e.g., an Ethernet frame) when it is passed down to the data-link layer. A frame has a header, and it may also have a trailer. For example, in addition to having a header, an Ethernet frame also has a 32-bit cyclic redundancy check (CRC) trailer. When it is passed down to the physical layer, a frame will be transformed into a sequence of media signals for transmission Flow Diagram of a Packet Generation At the destination side, the medium signals are converted by the physical layer into a frame, which is passed up to the data-link layer. The data-link layer passes the frame payload (i.e., the IP packet encapsulated in the frame) up to the IP layer. The IP layer passes the IP payload, namely, the TCP packet encapsulated in the IP packet, up to the TCP layer. The TCP layer passes the TCP payload, namely, the application data block, up to the application layer. When a packet arrives at a router, it only goes up to the IP layer, where certain fields in the IP header are modified (e.g., the value of TTL is decreased by 1). This modified packet is then passed back down layer-by-layer to the physical layer for further transmission. Public Key Infrastructure To deploy cryptographic algorithms in network applications, we need a way to distribute secret keys using open networks. Public-key cryptography is the best way to distribute these secret keys. To use public-key cryptography, we need to build a public-key infrastructure (PKI) to support and manage public-key certificates and certificate authority (CA) networks. In particular, PKIs are set up to perform the following functions: Determine the legitimacy of users before issuing public-key certificates to them. Issue public-key certificates upon user requests. Extend public-key certificates valid time upon user requests. Revoke public-key certificates upon users' requests or when the corresponding private keys are compromised. Store and manage public-key certificates. Prevent digital signature signers from denying their signatures. Support CA networks to allow different CAs to authenticate public-key certificates issued by other CAs. X.509: https://certificatedecoder.dev/?gclid=EAIaIQobChMI0M731O6G6gIVVSQrCh04bQaAEAAYASAAEgKRkPD_BwE IPsec: A Security Protocol at the Network Layer IPsec is a major security protocol at the network layer IPsec provides a potent platform for constructing virtual private networks (VPN). VPNs are private networks overlayed on public networks. The purpose of deploying cryptographic algorithms at the network layer is to encrypt or authenticate IP packets (either just the payloads or the whole packets). IPsec also specifies how to exchange keys. Thus, IPsec consists of authentication protocols, encryption protocols, and key exchange protocols. They are referred to, respectively, as authentication header (AH), encapsulating security payload (ESP), and Internet key exchange (IKE). PGP & S/MIME : Email Security There are several security protocols at the application layer. The most used of these protocols are email security protocols namely PGP and S/MIME. SMTP (\u201cSimple Mail Transfer Protocol\u201d) is used for sending and delivering from a client to a server via port 25: it\u2019s the outgoing server. On the contrary, POP (\u201cPost Office Protocol\u201d) allows the users to pick up the message and download it into their inbox: it\u2019s the incoming server. The latest version of the Post Office Protocol is named POP3, and it\u2019s been used since 1996; it uses port 110 PGP PGP implements all major cryptographic algorithms, the ZIP compression algorithm, and the Base64 encoding algorithm. It can be used to authenticate a message, encrypt a message, or both. PGP follows the following general process: authentication, ZIP compression, encryption, and Base64 encoding. The Base64 encoding procedure makes the message ready for SMTP transmission GPG (GnuPG) GnuPG is another free encryption standard that companies may use that is based on OpenPGP. GnuPG serves as a replacement for Symantec\u2019s PGP. The main difference is the supported algorithms. However, GnuPG plays nice with PGP by design. Because GnuPG is open, some businesses would prefer the technical support and the user interface that comes with Symantec\u2019s PGP. It is important to note that there are some nuances between the compatibility of GnuPG and PGP, such as the compatibility between certain algorithms, but in most applications such as email, there are workarounds. One such algorithm is the IDEA Module which isn\u2019t included in GnuPG out of the box due to patent issues. S/MIME SMTP can only handle 7-bit ASCII text (You can use UTF-8 extensions to alleviate these limitations, ) messages. While POP can handle other content types besides 7-bit ASCII, POP may, under a common default setting, download all the messages stored in the mail server to the user's local computer. After that, if POP removes these messages from the mail server. This makes it difficult for the users to read their messages from multiple computers. The Multipurpose Internet Mail Extension protocol (MIME) was designed to support sending and receiving email messages in various formats, including nontext files generated by word processors, graphics files, sound files, and video clips. Moreover, MIME allows a single message to include mixed types of data in any combination of these formats. The Internet Mail Access Protocol (IMAP), operated on TCP port 143(only for non-encrypted), stores (Configurable on both server & client just like PoP) incoming email messages in the mail server until the user deletes them deliberately. This allows the users to access their mailbox from multiple machines and download messages to a local machine without deleting it from the mailbox in the mail server. SSL/TLS SSL uses a PKI to decide if a server\u2019s public key is trustworthy by requiring servers to use a security certificate signed by a trusted CA. When Netscape Navigator 1.0 was released, it trusted a single CA operated by the RSA Data Security corporation. The server\u2019s public RSA keys were used to be stored in the security certificate, which can then be used by the browser to establish a secure communication channel. The security certificates we use today still rely on the same standard (named X.509) that Netscape Navigator 1.0 used back then. Netscape intended to train users(though this didn\u2019t work out later) to differentiate secure communications from insecure ones, so they put a lock icon next to the address bar. When the lock is open, the communication is insecure. A closed lock means communication has been secured with SSL, which required the server to provide a signed certificate. You\u2019re obviously familiar with this icon as it\u2019s been in every browser ever since. The engineers at Netscape truly created a standard for secure internet communications. A year after releasing SSL 2.0, Netscape fixed several security issues and released SSL 3.0, a protocol that, albeit being officially deprecated since June 2015, remains in use in certain parts of the world more than 20 years after its introduction. To standardize SSL, the Internet Engineering Task Force (IETF) created a slightly modified SSL 3.0 and, in 1999, unveiled it as Transport Layer Security (TLS) 1.0. The name change between SSL and TLS continues to confuse people today. Officially, TLS is the new SSL, but in practice, people use SSL and TLS interchangeably to talk about any version of the protocol. Must See: https://tls.ulfheim.net/ https://davidwong.fr/tls13/ Network Perimeter Security Let us see how we keep a check on the perimeter i.e the edges, the first layer of protection General Firewall Framework Firewalls are needed because encryption algorithms cannot effectively stop malicious packets from getting into an edge network. This is because IP packets, regardless of whether they are encrypted, can always be forwarded into an edge network. Firewalls that were developed in the 1990s are important instruments to help restrict network access. A firewall may be a hardware device, a software package, or a combination of both. Packets flowing into the internal network from the outside should be evaluated before they are allowed to enter. One of the critical elements of a firewall is its ability to examine packets without imposing a negative impact on communication speed while providing security protections for the internal network. The packet inspection that is carried out by firewalls can be done using several different methods. Based on the particular method used by the firewall, it can be characterized as either a packet filter, circuit gateway, application gateway, or dynamic packet filter. Packet Filters It inspects ingress packets coming to an internal network from outside and inspects egress packets going outside from an internal network Packing filtering only inspects IP headers and TCP headers, not the payloads generated at the application layer A packet-filtering firewall uses a set of rules to determine whether a packet should be allowed or denied to pass through. 2 types: Stateless It treats each packet as an independent object, and it does not keep track of any previously processed packets. In other words, stateless filtering inspects a packet when it arrives and makes a decision without leaving any record of the packet being inspected. Stateful Stateful filtering, also referred to as connection-state filtering, keeps track of connections between an internal host and an external host. A connection state (or state, for short) indicates whether it is a TCP connection or a UDP connection and whether the connection is established. Circuit Gateways Circuit gateways, also referred to as circuit-level gateways, are typically operated at the transportation layer They evaluate the information of the IP addresses and the port numbers contained in TCP (or UDP) headers and use it to determine whether to allow or to disallow an internal host and an external host to establish a connection. It is common practice to combine packet filters and circuit gateways to form a dynamic packet filter (DPF). Application Gateways(ALG) Aka PROXY Servers An Application Level Gateway (ALG) acts as a proxy for internal hosts, processing service requests from external clients. An ALG performs deep inspections on each IP packet (ingress or egress). In particular, an ALG inspects application program formats contained in the packet (e.g., MIME format or SQL format) and examines whether its payload is permitted. Thus, an ALG may be able to detect a computer virus contained in the payload. Because an ALG inspects packet payloads, it may be able to detect malicious code and quarantine suspicious packets, in addition to blocking packets with suspicious IP addresses and TCP ports. On the other hand, an ALG also incurs substantial computation and space overheads. Trusted Systems & Bastion Hosts A Trusted Operating System (TOS) is an operating system that meets a particular set of security requirements. Whether an operating system can be trusted or not depends on several elements. For example, for an operating system on a particular computer to be certified trusted, one needs to validate that, among other things, the following four requirements are satisfied: Its system design contains no defects; Its system software contains no loopholes; Its system is configured properly; and Its system management is appropriate. Bastion Hosts Bastion hosts are computers with strong defence mechanisms. They often serve as host computers for implementing application gateways, circuit gateways, and other types of firewalls. A bastion host is operated on a trusted operating system that must not contain unnecessary functionalities or programs. This measure helps to reduce error probabilities and makes it easier to conduct security checks. Only those network application programs that are necessary, for example, SSH, DNS, SMTP, and authentication programs, are installed on a bastion host. Bastion hosts are also primarily used as controlled ingress points so that the security monitoring can focus more narrowly on actions happening at a single point closely. Common Techniques & Scannings, Packet Capturing Scanning Ports with Nmap Nmap (\"Network Mapper\") is a free and open-source (license) utility for network discovery and security auditing. Many systems and network administrators also find it useful for tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime. The best thing about Nmap is it\u2019s free and open-source and is very flexible and versatile Nmap is often used to determine alive hosts in a network, open ports on those hosts, services running on those open ports, and version identification of that service on that port. More at http://scanme.nmap.org/ nmap [scan type] [options] [target specification] Nmap uses 6 different port states: Open \u2014 An open port is one that is actively accepting TCP, UDP or SCTP connections. Open ports are what interests us the most because they are the ones that are vulnerable to attacks. Open ports also show the available services on a network. Closed \u2014 A port that receives and responds to Nmap probe packets but there is no application listening on that port. Useful for identifying that the host exists and for OS detection. Filtered \u2014 Nmap can\u2019t determine whether the port is open because packet filtering prevents its probes from reaching the port. Filtering could come from firewalls or router rules. Often little information is given from filtered ports during scans as the filters can drop the probes without responding or respond with useless error messages e.g. destination unreachable. Unfiltered \u2014 Port is accessible but Nmap doesn\u2019t know if it is open or closed. Only used in ACK scan which is used to map firewall rulesets. Other scan types can be used to identify whether the port is open. Open/filtered \u2014 Nmap is unable to determine between open and filtered. This happens when an open port gives no response. No response could mean that the probe was dropped by a packet filter or any response is blocked. Closed/filtered \u2014 Nmap is unable to determine whether a port is closed or filtered. Only used in the IP ID idle scan. Types of Nmap Scan: TCP Connect TCP Connect scan completes the 3-way handshake. If a port is open, the operating system completes the TCP three-way handshake and the port scanner immediately closes the connection to avoid DOS. This is \u201cnoisy\u201d because the services can log the sender IP address and might trigger Intrusion Detection Systems. UDP Scan This scan checks to see if any UDP ports are listening. Since UDP does not respond with a positive acknowledgement like TCP and only responds to an incoming UDP packet when the port is closed, SYN Scan SYN scan is another form of TCP scanning. This scan type is also known as \u201chalf-open scanning\u201d because it never actually opens a full TCP connection. The port scanner generates a SYN packet. If the target port is open, it will respond with an SYN-ACK packet. The scanner host responds with an RST packet, closing the connection before the handshake is completed. If the port is closed but unfiltered, the target will instantly respond with an RST packet. SYN scan has the advantage that the individual services never actually receive a connection. FIN Scan This is a stealthy scan, like the SYN scan, but sends a TCP FIN packet instead. ACK Scan Ack scanning determines whether the port is filtered or not. Null Scan Another very stealthy scan that sets all the TCP header flags to off or null. This is not normally a valid packet and some hosts will not know what to do with this. XMAS Scan Similar to the NULL scan except for all the flags in the TCP header is set to on RPC Scan This special type of scan looks for machine answering to RPC (Remote Procedure Call) services IDLE Scan It is a super stealthy method whereby the scan packets are bounced off an external host. You don\u2019t need to have control over the other host but it does have to set up and meet certain requirements. You must input the IP address of our \u201czombie\u201d host and what port number to use. It is one of the more controversial options in Nmap since it only has a use for malicious attacks. Scan Techniques A couple of scan techniques which can be used to gain more information about a system and its ports. You can read more at https://medium.com/infosec-adventures/nmap-cheatsheet-a423fcdda0ca OpenVAS OpenVAS is a full-featured vulnerability scanner. OpenVAS is a framework of services and tools that provides a comprehensive and powerful vulnerability scanning and management package OpenVAS, which is an open-source program, began as a fork of the once-more-popular scanning program, Nessus. OpenVAS is made up of three main parts. These are: a regularly updated feed of Network Vulnerability Tests (NVTs); a scanner, which runs the NVTs; and an SQLite 3 database for storing both your test configurations and the NVTs\u2019 results and configurations. https://www.greenbone.net/en/install_use_gce/ WireShark Wireshark is a protocol analyzer. This means Wireshark is designed to decode not only packet bits and bytes but also the relations between packets and protocols. Wireshark understands protocol sequences. A simple demo of Wireshark Capture only udp packets: Capture filter = \u201cudp\u201d Capture only tcp packets Capture filter = \u201ctcp\u201d TCP/IP 3 way Handshake Filter by IP address: displays all traffic from IP, be it source or destination ip.addr == 192.168.1.1 Filter by source address: display traffic only from IP source ip.src == 192.168.0.1 Filter by destination: display traffic only form IP destination ip.dst == 192.168.0.1 Filter by IP subnet: display traffic from subnet, be it source or destination ip.addr = 192.168.0.1/24 Filter by protocol: filter traffic by protocol name dns http ftp arp ssh telnet icmp Exclude IP address: remove traffic from and to IP address !ip.addr ==192.168.0.1 Display traffic between two specific subnet ip.addr == 192.168.0.1/24 and ip.addr == 192.168.1.1/24 Display traffic between two specific workstations ip.addr == 192.168.0.1 and ip.addr == 192.168.0.2 Filter by MAC eth.addr = 00:50:7f:c5:b6:78 Filter TCP port tcp.port == 80 Filter TCP port source tcp.srcport == 80 Filter TCP port destination tcp.dstport == 80 Find user agents http.user_agent contains Firefox !http.user_agent contains || !http.user_agent contains Chrome Filter broadcast traffic !(arp or icmp or dns) Filter IP address and port tcp.port == 80 && ip.addr == 192.168.0.1 Filter all http get requests http.request Filter all http get requests and responses http.request or http.response Filter three way handshake tcp.flags.syn==1 or (tcp.seq==1 and tcp.ack==1 and tcp.len==0 and tcp.analysis.initial_rtt) Find files by type frame contains \u201c(attachment|tar|exe|zip|pdf)\u201d Find traffic based on keyword tcp contains facebook frame contains facebook Detecting SYN Floods tcp.flags.syn == 1 and tcp.flags.ack == 0 Wireshark Promiscuous Mode - By default, Wireshark only captures packets going to and from the computer where it runs. By checking the box to run Wireshark in Promiscuous Mode in the Capture Settings, you can capture most of the traffic on the LAN. DumpCap Dumpcap is a network traffic dump tool. It captures packet data from a live network and writes the packets to a file. Dumpcap\u2019s native capture file format is pcapng, which is also the format used by Wireshark. By default, Dumpcap uses the pcap library to capture traffic from the first available network interface and writes the received raw packet data, along with the packets\u2019 time stamps into a pcapng file. The capture filter syntax follows the rules of the pcap library. The Wireshark command-line utility called 'dumpcap.exe' can be used to capture LAN traffic over an extended period of time. Wireshark itself can also be used, but dumpcap does not significantly utilize the computer's memory while capturing for long periods. DaemonLogger Daemonlogger is a packet logging application designed specifically for use in Network and Systems Management (NSM) environments. The biggest benefit Daemonlogger provides is that, like Dumpcap, it is simple to use for capturing packets. In order to begin capturing, you need only to invoke the command and specify an interface. daemonlogger \u2013i eth1 This option, by default, will begin capturing packets and logging them to the current working directory. Packets will be collected until the capture file size reaches 2 GB, and then a new file will be created. This will continue indefinitely until the process is halted. NetSniff-NG Netsniff-NG is a high-performance packet capture utility While the utilities we\u2019ve discussed to this point rely on Libpcap for capture, Netsniff-NG utilizes zero-copy mechanisms to capture packets. This is done with the intent to support full packet capture over high throughput links. To begin capturing packets with Netsniff-NG, we have to specify an input and output. In most cases, the input will be a network interface, and the output will be a file or folder on disk. netsniff-ng \u2013i eth1 \u2013o data.pcap Netflow NetFlow is a feature that was introduced on Cisco routers around 1996 that provides the ability to collect IP network traffic as it enters or exits an interface. By analyzing the data provided by NetFlow, a network administrator can determine things such as the source and destination of traffic, class of service, and the causes of congestion. A typical flow monitoring setup (using NetFlow) consists of three main components:[1] Flow exporter: aggregates packets into flows and exports flow records towards one or more flow collectors. Flow collector: responsible for reception, storage and pre-processing of flow data received from a flow exporter. Analysis application: analyzes received flow data in the context of intrusion detection or traffic profiling, for example. Routers and switches that support NetFlow can collect IP traffic statistics on all interfaces where NetFlow is enabled, and later export those statistics as NetFlow records toward at least one NetFlow collector\u2014typically a server that does the actual traffic analysis. IDS A security solution that detects security-related events in your environment but does not block them. IDS sensors can be software and hardware-based used to collect and analyze the network traffic. These sensors are available in two varieties, network IDS and host IDS. A host IDS is a server-specific agent running on a server with a minimum of overhead to monitor the operating system. A network IDS can be embedded in a networking device, a standalone appliance, or a module monitoring the network traffic. Signature Based IDS The signature-based IDS monitors the network traffic or observes the system and sends an alarm if a known malicious event is happening. It does so by comparing the data flow against a database of known attack patterns These signatures explicitly define what traffic or activity should be considered as malicious. Signature-based detection has been the bread and butter of network-based defensive security for over a decade, partially because it is very similar to how malicious activity is detected at the host level with antivirus utilities The formula is fairly simple: an analyst observes a malicious activity, derives indicators from the activity and develops them into signatures, and then those signatures will alert whenever the activity occurs again. ex: SNORT & SURICATA Policy-Based IDS The policy-based IDSs (mainly host IDSs) trigger an alarm whenever a violation occurs against the configured policy. This configured policy is or should be a representation of the security policies. This type of IDS is flexible and can be customized to a company's network requirements because it knows exactly what is permitted and what is not. On the other hand, the signature-based systems rely on vendor specifics and default settings. Anomaly Based IDS The anomaly-based IDS looks for traffic that deviates from the normal, but the definition of what is a normal network traffic pattern is the tricky part Two types of anomaly-based IDS exist: statistical and nonstatistical anomaly detection Statistical anomaly detection learns the traffic patterns interactively over a period of time. In the nonstatistical approach, the IDS has a predefined configuration of the supposedly acceptable and valid traffic patterns. Host-Based IDS & Network-Based IDS A host IDS can be described as a distributed agent residing on each server of the network that needs protection. These distributed agents are tied very closely to the underlying operating system. Network IDSs, on the other hand, can be described as intelligent sniffing devices. Data (raw packets) is captured from the network by a network IDS, whereas host IDSs capture the data from the host on which they are installed. Honeypots The use of decoy machines to direct intruders' attention away from the machines under protection is a major technique to preclude intrusion attacks. Any device, system, directory, or file used as a decoy to lure attackers away from important assets and to collect intrusion or abusive behaviours is referred to as a honeypot. A honeypot may be implemented as a physical device or as an emulation system. The idea is to set up decoy machines in a LAN, or decoy directories/files in a file system and make them appear important, but with several exploitable loopholes, to lure attackers to attack these machines or directories/files, so that other machines, directories, and files can evade intruders' attentions. A decoy machine may be a host computer or a server computer. Likewise, we may also set up decoy routers or even decoy LANs. Chinks In The Armour (TCP/IP Security Issues) IP Spoofing In this type of attack, the attacker replaces the IP address of the sender, or in some rare cases the destination, with a different address. IP spoofing is normally used to exploit a target host. In other cases, it is used to start a denial-of-service (DoS) attack. In a DoS attack, an attacker modifies the IP packet to mislead the target host into accepting the original packet as a packet sourced at a trusted host. The attacker must know the IP address of the trusted host to modify the packet headers (source IP address) so that it appears that the packets are coming from that host. IP Spoofing Detection Techniques Direct TTL Probes In this technique we send a packet to a host of suspect spoofed IP that triggers reply and compares TTL with suspect packet; if the TTL in the reply is not the same as the packet being checked; it is a spoofed packet. This Technique is successful when the attacker is in a different subnet from the victim. IP Identification Number. Send a probe to the host of suspect spoofed traffic that triggers a reply and compares IP ID with suspect traffic. If IP IDs are not in the near value of packet being checked, suspect traffic is spoofed TCP Flow Control Method Attackers sending spoofed TCP packets will not receive the target\u2019s SYN-ACK packets. Attackers cannot, therefore, be responsive to change in the congestion window size When the receiver still receives traffic even after a windows size is exhausted, most probably the packets are spoofed. Covert Channel A covert or clandestine channel can be best described as a pipe or communication channel between two entities that can be exploited by a process or application transferring information in a manner that violates the system's security specifications. More specifically for TCP/IP, in some instances, covert channels are established, and data can be secretly passed between two end systems. Ex: ICMP resides at the Internet layer of the TCP/IP protocol suite and is implemented in all TCP/IP hosts. Based on the specifications of the ICMP Protocol, an ICMP Echo Request message should have an 8-byte header and a 56-byte payload. The ICMP Echo Request packet should not carry any data in the payload. However, these packets are often used to carry secret information. The ICMP packets are altered slightly to carry secret data in the payload. This makes the size of the packet larger, but no control exists in the protocol stack to defeat this behaviour. The alteration of ICMP packets allows intruders to program specialized client-server pairs. These small pieces of code export confidential information without alerting the network administrator. ICMP can be leveraged for more than data exfiltration. For eg. some C&C tools such as Loki used ICMP channel to establish encrypted interactive session back in 1996. Deep packet inspection has since come a long way. A lot of IDS/IPS detect ICMP tunnelling. Check for echo responses that do not contain the same payload as request Check for the volume of ICMP traffic especially for volumes beyond an acceptable threshold IP Fragmentation Attack The TCP/IP protocol suite, or more specifically IP, allows the fragmentation of packets.(this is a feature & not a bug) IP fragmentation offset is used to keep track of the different parts of a datagram. The information or content in this field is used at the destination to reassemble the datagrams All such fragments have the same Identification field value, and the fragmentation offset indicates the position of the current fragment in the context of the original packet. Many access routers and firewalls do not perform packet reassembly. In normal operation, IP fragments do not overlap, but attackers can create artificially fragmented packets to mislead the routers or firewalls. Usually, these packets are small and almost impractical for end systems because of data and computational overhead. A good example of an IP fragmentation attack is the Ping of Death attack. The Ping of Death attack sends fragments that, when reassembled at the end station, create a larger packet than the maximum permissible length. TCP Flags Data exchange using TCP does not happen until a three-way handshake has been completed. This handshake uses different flags to influence the way TCP segments are processed. There are 6 bits in the TCP header that are often called flags. Namely: 6 different flags are part of the TCP header: Urgent pointer field (URG), Acknowledgment field (ACK), Push function (PSH), Reset the connection (RST), Synchronize sequence numbers (SYN), and the sender is finished with this connection (FIN). Abuse of the normal operation or settings of these flags can be used by attackers to launch DoS attacks. This causes network servers or web servers to crash or hang. | SYN | FIN | PSH | RST | Validity| |------|------|-------|------|---------| | 1 |1 |0 |0 |Illegal Combination | 1 |1 |1 |0 |Illegal Combination | 1 |1 |0 |1 |Illegal Combination | 1 |1 |1 |1 |Illegal Combination The attacker's ultimate goal is to write special programs or pieces of code that can construct these illegal combinations resulting in an efficient DoS attack. SYN FLOOD The timers (or lack of certain timers) in 3 way handshake are often used and exploited by attackers to disable services or even to enter systems. After step 2 of the three-way handshake, no limit is set on the time to wait after receiving a SYN. The attacker initiates many connection requests to the webserver of Company XYZ (almost certainly with a spoofed IP address). The SYN+ACK packets (Step 2) sent by the web server back to the originating source IP address are not replied to. This leaves a TCP session half-open on the webserver. Multiple packets cause multiple TCP sessions to stay open. Based on the hardware limitations of the server, a limited number of TCP sessions can stay open, and as a result, the webserver refuses further connection establishments attempts from any host as soon as a certain limit is reached. These half-open connections need to be completed or timed out before new connections can be established. FIN Attack In normal operation, the sender sets the TCP FIN flag indicating that no more data will be transmitted and the connection can be closed down. This is a four-way handshake mechanism, with both sender and receiver expected to send an acknowledgement on a received FIN packet. During an attack that is trying to kill connections, a spoofed FIN packet is constructed. This packet also has the correct sequence number, so the packets are seen as valid by the targeted host. These sequence numbers are easy to predict. This process is referred to as TCP sequence number prediction, whereby the attacker either sniffs the current Sequence and Acknowledgment (SEQ/ACK) numbers of the connection or can algorithmically predict these numbers. Connection Hijacking An authorized user (Employee X) sends HTTP requests over a TCP session with the webserver. The web server accepts the packets from Employee X only when the packet has the correct SEQ/ACK numbers. As seen previously, these numbers are important for the webserver to distinguish between different sessions and to make sure it is still talking to Employee X. Imagine that the cracker starts sending packets to the web server spoofing the IP address of Employee X, using the correct SEQ/ACK combination. The web server accepts the packet and increments the ACK number. In the meantime, Employee X continues to send packets but with incorrect SEQ/ACK numbers. As a result of sending unsynchronized packets, all data from Employee X is discarded when received by the webserver. The attacker pretends to be Employee X using the correct numbers. This finally results in the cracker hijacking the connection, whereby Employee X is completely confused and the webserver replies assuming the cracker is sending correct synchronized data. STEPS: The attacker examines the traffic flows with a network monitor and notices traffic from Employee X to a web server. The web server returns or echoes data back to the origination station (Employee X). Employee X acknowledges the packet. The cracker launches a spoofed packet to the server. The web server responds to the cracker. The cracker starts verifying SEQ/ACK numbers to double-check success. At this time, the cracker takes over the session from Employee X, which results in a session hanging for Employee X. The cracker can start sending traffic to the webserver. The web server returns the requested data to confirm delivery with the correct ACK number. The cracker can continue to send data (keeping track of the correct SEQ/ACK numbers) until eventually setting the FIN flag to terminate the session. Buffer Overflow A buffer is a temporary data storage area used to store program code and data. When a program or process tries to store more data in a buffer than it was originally anticipated to hold, a buffer overflow occurs. Buffers are temporary storage locations in memory (memory or buffer sizes are often measured in bytes) that can store a fixed amount of data in bytes. When more data is retrieved than can be stored in a buffer location, the additional information must go into an adjacent buffer, resulting in overwriting the valid data held in them. Mechanism: Buffer overflow vulnerabilities exist in different types. But the overall goal for all buffer overflow attacks is to take over the control of a privileged program and, if possible, the host. The attacker has two tasks to achieve this goal. First, the dirty code needs to be available in the program's code address space. Second, the privileged program should jump to that particular part of the code, which ensures that the proper parameters are loaded into memory. The first task can be achieved in two ways: by injecting the code in the right address space or by using the existing code and modifying certain parameters slightly. The second task is a little more complex because the program's control flow needs to be modified to make the program jump to the dirty code. CounterMeasure: The most important approach is to have a concerted focus on writing correct code. A second method is to make the data buffers (memory locations) address space of the program code non-executable. This type of address space makes it impossible to execute code, which might be infiltrated in the program's buffers during an attack. More Spoofing Address Resolution Protocol Spoofing The Address Resolution Protocol (ARP) provides a mechanism to resolve, or map, a known IP address to a MAC sublayer address. Using ARP spoofing, the cracker can exploit this hardware address authentication mechanism by spoofing the hardware address of Host B. Basically, the attacker can convince any host or network device on the local network that the cracker's workstation is the host to be trusted. This is a common method used in a switched environment. ARP spoofing can be prevented with the implementation of static ARP tables in all the hosts and routers of your network. Alternatively, you can implement an ARP server that responds to ARP requests on behalf of the target host. DNS Spoofing DNS spoofing is the method whereby the hacker convinces the target machine that the system it wants to connect to is the machine of the cracker. The cracker modifies some records so that name entries of hosts correspond to the attacker's IP address. There have been instances in which the complete DNS server was compromised by an attack. To counter DNS spoofing, the reverse lookup detects these attacks. The reverse lookup is a mechanism to verify the IP address against a name. The IP address and name files are usually kept on different servers to make compromise much more difficult","title":"Network Security"},{"location":"security/network_security/#part-ii-network-security","text":"","title":"Part II: Network Security"},{"location":"security/network_security/#introduction","text":"TCP/IP is the dominant networking technology today. It is a five-layer architecture. These layers are, from top to bottom, the application layer, the transport layer (TCP), the network layer (IP), the data-link layer, and the physical layer. In addition to TCP/IP, there also are other networking technologies. For convenience, we use the OSI network model to represent non-TCP/IP network technologies. Different networks are interconnected using gateways. A gateway can be placed at any layer. The OSI model is a seven-layer architecture. The OSI architecture is similar to the TCP/IP architecture, except that the OSI model specifies two additional layers between the application layer and the transport layer in the TCP/IP architecture. These two layers are the presentation layer and the session layer. Figure 5.1 shows the relationship between the TCP/IP layers and the OSI layers. The application layer in TCP/IP corresponds to the application layer and the presentation layer in OSI. The transport layer in TCP/IP corresponds to the session layer and the transport layer in OSI. The remaining three layers in the TCP/IP architecture are one-to-one correspondent to the remaining three layers in the OSI model. Correspondence between layers of the TCP/IP architecture and the OSI model. Also shown are placements of cryptographic algorithms in network layers, where the dotted arrows indicate actual communications of cryptographic algorithms The functionalities of OSI layers are briefly described as follows: The application layer serves as an interface between applications and network programs. It supports application programs and end-user processing. Common application-layer programs include remote logins, file transfer, email, and Web browsing. The presentation layer is responsible for dealing with data that is formed differently. This protocol layer allows application-layer programs residing on different sides of a communication channel with different platforms to understand each other's data formats regardless of how they are presented. The session layer is responsible for creating, managing, and closing a communication connection. The transport layer is responsible for providing reliable connections, such as packet sequencing, traffic control, and congestion control. The network layer is responsible for routing device-independent data packets from the current hop to the next hop. The data-link layer is responsible for encapsulating device-independent data packets into device-dependent data frames. It has two sublayers: logical link control and media access control. The physical layer is responsible for transmitting device-dependent frames through some physical media. Starting from the application layer, data generated from an application program is passed down layer-by-layer to the physical layer. Data from the previous layer is enclosed in a new envelope at the current layer, where the data from the previous layer is also just an envelope containing the data from the layer before it. This is similar to enclosing a smaller envelope in a larger one. The envelope added at each layer contains sufficient information for handling the packet. Application-layer data are divided into blocks small enough to be encapsulated in an envelope at the next layer. Application data blocks are \u201cdressed up\u201d in the TCP/IP architecture according to the following basic steps. At the sending side, an application data block is encapsulated in a TCP packet when it is passed down to the TCP layer. In other words, a TCP packet consists of a header and a payload, where the header corresponds to the TCP envelope and the payload is the application data block. Likewise, the TCP packet will be encapsulated in an IP packet when it is passed down to the IP layer. An IP packet consists of a header and a payload, which is the TCP packet passed down from the TCP layer. The IP packet will be encapsulated in a device-dependent frame (e.g., an Ethernet frame) when it is passed down to the data-link layer. A frame has a header, and it may also have a trailer. For example, in addition to having a header, an Ethernet frame also has a 32-bit cyclic redundancy check (CRC) trailer. When it is passed down to the physical layer, a frame will be transformed into a sequence of media signals for transmission Flow Diagram of a Packet Generation At the destination side, the medium signals are converted by the physical layer into a frame, which is passed up to the data-link layer. The data-link layer passes the frame payload (i.e., the IP packet encapsulated in the frame) up to the IP layer. The IP layer passes the IP payload, namely, the TCP packet encapsulated in the IP packet, up to the TCP layer. The TCP layer passes the TCP payload, namely, the application data block, up to the application layer. When a packet arrives at a router, it only goes up to the IP layer, where certain fields in the IP header are modified (e.g., the value of TTL is decreased by 1). This modified packet is then passed back down layer-by-layer to the physical layer for further transmission.","title":"Introduction"},{"location":"security/network_security/#public-key-infrastructure","text":"To deploy cryptographic algorithms in network applications, we need a way to distribute secret keys using open networks. Public-key cryptography is the best way to distribute these secret keys. To use public-key cryptography, we need to build a public-key infrastructure (PKI) to support and manage public-key certificates and certificate authority (CA) networks. In particular, PKIs are set up to perform the following functions: Determine the legitimacy of users before issuing public-key certificates to them. Issue public-key certificates upon user requests. Extend public-key certificates valid time upon user requests. Revoke public-key certificates upon users' requests or when the corresponding private keys are compromised. Store and manage public-key certificates. Prevent digital signature signers from denying their signatures. Support CA networks to allow different CAs to authenticate public-key certificates issued by other CAs. X.509: https://certificatedecoder.dev/?gclid=EAIaIQobChMI0M731O6G6gIVVSQrCh04bQaAEAAYASAAEgKRkPD_BwE","title":"Public Key Infrastructure"},{"location":"security/network_security/#ipsec-a-security-protocol-at-the-network-layer","text":"IPsec is a major security protocol at the network layer IPsec provides a potent platform for constructing virtual private networks (VPN). VPNs are private networks overlayed on public networks. The purpose of deploying cryptographic algorithms at the network layer is to encrypt or authenticate IP packets (either just the payloads or the whole packets). IPsec also specifies how to exchange keys. Thus, IPsec consists of authentication protocols, encryption protocols, and key exchange protocols. They are referred to, respectively, as authentication header (AH), encapsulating security payload (ESP), and Internet key exchange (IKE).","title":"IPsec: A Security Protocol at the Network Layer"},{"location":"security/network_security/#pgp-smime-email-security","text":"There are several security protocols at the application layer. The most used of these protocols are email security protocols namely PGP and S/MIME. SMTP (\u201cSimple Mail Transfer Protocol\u201d) is used for sending and delivering from a client to a server via port 25: it\u2019s the outgoing server. On the contrary, POP (\u201cPost Office Protocol\u201d) allows the users to pick up the message and download it into their inbox: it\u2019s the incoming server. The latest version of the Post Office Protocol is named POP3, and it\u2019s been used since 1996; it uses port 110 PGP PGP implements all major cryptographic algorithms, the ZIP compression algorithm, and the Base64 encoding algorithm. It can be used to authenticate a message, encrypt a message, or both. PGP follows the following general process: authentication, ZIP compression, encryption, and Base64 encoding. The Base64 encoding procedure makes the message ready for SMTP transmission GPG (GnuPG) GnuPG is another free encryption standard that companies may use that is based on OpenPGP. GnuPG serves as a replacement for Symantec\u2019s PGP. The main difference is the supported algorithms. However, GnuPG plays nice with PGP by design. Because GnuPG is open, some businesses would prefer the technical support and the user interface that comes with Symantec\u2019s PGP. It is important to note that there are some nuances between the compatibility of GnuPG and PGP, such as the compatibility between certain algorithms, but in most applications such as email, there are workarounds. One such algorithm is the IDEA Module which isn\u2019t included in GnuPG out of the box due to patent issues. S/MIME SMTP can only handle 7-bit ASCII text (You can use UTF-8 extensions to alleviate these limitations, ) messages. While POP can handle other content types besides 7-bit ASCII, POP may, under a common default setting, download all the messages stored in the mail server to the user's local computer. After that, if POP removes these messages from the mail server. This makes it difficult for the users to read their messages from multiple computers. The Multipurpose Internet Mail Extension protocol (MIME) was designed to support sending and receiving email messages in various formats, including nontext files generated by word processors, graphics files, sound files, and video clips. Moreover, MIME allows a single message to include mixed types of data in any combination of these formats. The Internet Mail Access Protocol (IMAP), operated on TCP port 143(only for non-encrypted), stores (Configurable on both server & client just like PoP) incoming email messages in the mail server until the user deletes them deliberately. This allows the users to access their mailbox from multiple machines and download messages to a local machine without deleting it from the mailbox in the mail server. SSL/TLS SSL uses a PKI to decide if a server\u2019s public key is trustworthy by requiring servers to use a security certificate signed by a trusted CA. When Netscape Navigator 1.0 was released, it trusted a single CA operated by the RSA Data Security corporation. The server\u2019s public RSA keys were used to be stored in the security certificate, which can then be used by the browser to establish a secure communication channel. The security certificates we use today still rely on the same standard (named X.509) that Netscape Navigator 1.0 used back then. Netscape intended to train users(though this didn\u2019t work out later) to differentiate secure communications from insecure ones, so they put a lock icon next to the address bar. When the lock is open, the communication is insecure. A closed lock means communication has been secured with SSL, which required the server to provide a signed certificate. You\u2019re obviously familiar with this icon as it\u2019s been in every browser ever since. The engineers at Netscape truly created a standard for secure internet communications. A year after releasing SSL 2.0, Netscape fixed several security issues and released SSL 3.0, a protocol that, albeit being officially deprecated since June 2015, remains in use in certain parts of the world more than 20 years after its introduction. To standardize SSL, the Internet Engineering Task Force (IETF) created a slightly modified SSL 3.0 and, in 1999, unveiled it as Transport Layer Security (TLS) 1.0. The name change between SSL and TLS continues to confuse people today. Officially, TLS is the new SSL, but in practice, people use SSL and TLS interchangeably to talk about any version of the protocol. Must See: https://tls.ulfheim.net/ https://davidwong.fr/tls13/","title":"PGP & S/MIME : Email Security"},{"location":"security/network_security/#network-perimeter-security","text":"Let us see how we keep a check on the perimeter i.e the edges, the first layer of protection","title":"Network Perimeter Security"},{"location":"security/network_security/#general-firewall-framework","text":"Firewalls are needed because encryption algorithms cannot effectively stop malicious packets from getting into an edge network. This is because IP packets, regardless of whether they are encrypted, can always be forwarded into an edge network. Firewalls that were developed in the 1990s are important instruments to help restrict network access. A firewall may be a hardware device, a software package, or a combination of both. Packets flowing into the internal network from the outside should be evaluated before they are allowed to enter. One of the critical elements of a firewall is its ability to examine packets without imposing a negative impact on communication speed while providing security protections for the internal network. The packet inspection that is carried out by firewalls can be done using several different methods. Based on the particular method used by the firewall, it can be characterized as either a packet filter, circuit gateway, application gateway, or dynamic packet filter.","title":"General Firewall Framework"},{"location":"security/network_security/#packet-filters","text":"It inspects ingress packets coming to an internal network from outside and inspects egress packets going outside from an internal network Packing filtering only inspects IP headers and TCP headers, not the payloads generated at the application layer A packet-filtering firewall uses a set of rules to determine whether a packet should be allowed or denied to pass through. 2 types: Stateless It treats each packet as an independent object, and it does not keep track of any previously processed packets. In other words, stateless filtering inspects a packet when it arrives and makes a decision without leaving any record of the packet being inspected. Stateful Stateful filtering, also referred to as connection-state filtering, keeps track of connections between an internal host and an external host. A connection state (or state, for short) indicates whether it is a TCP connection or a UDP connection and whether the connection is established.","title":"Packet Filters"},{"location":"security/network_security/#circuit-gateways","text":"Circuit gateways, also referred to as circuit-level gateways, are typically operated at the transportation layer They evaluate the information of the IP addresses and the port numbers contained in TCP (or UDP) headers and use it to determine whether to allow or to disallow an internal host and an external host to establish a connection. It is common practice to combine packet filters and circuit gateways to form a dynamic packet filter (DPF).","title":"Circuit Gateways"},{"location":"security/network_security/#application-gatewaysalg","text":"Aka PROXY Servers An Application Level Gateway (ALG) acts as a proxy for internal hosts, processing service requests from external clients. An ALG performs deep inspections on each IP packet (ingress or egress). In particular, an ALG inspects application program formats contained in the packet (e.g., MIME format or SQL format) and examines whether its payload is permitted. Thus, an ALG may be able to detect a computer virus contained in the payload. Because an ALG inspects packet payloads, it may be able to detect malicious code and quarantine suspicious packets, in addition to blocking packets with suspicious IP addresses and TCP ports. On the other hand, an ALG also incurs substantial computation and space overheads.","title":"Application Gateways(ALG)"},{"location":"security/network_security/#trusted-systems-bastion-hosts","text":"A Trusted Operating System (TOS) is an operating system that meets a particular set of security requirements. Whether an operating system can be trusted or not depends on several elements. For example, for an operating system on a particular computer to be certified trusted, one needs to validate that, among other things, the following four requirements are satisfied: Its system design contains no defects; Its system software contains no loopholes; Its system is configured properly; and Its system management is appropriate. Bastion Hosts Bastion hosts are computers with strong defence mechanisms. They often serve as host computers for implementing application gateways, circuit gateways, and other types of firewalls. A bastion host is operated on a trusted operating system that must not contain unnecessary functionalities or programs. This measure helps to reduce error probabilities and makes it easier to conduct security checks. Only those network application programs that are necessary, for example, SSH, DNS, SMTP, and authentication programs, are installed on a bastion host. Bastion hosts are also primarily used as controlled ingress points so that the security monitoring can focus more narrowly on actions happening at a single point closely.","title":"Trusted Systems & Bastion Hosts"},{"location":"security/network_security/#common-techniques-scannings-packet-capturing","text":"","title":"Common Techniques & Scannings, Packet Capturing"},{"location":"security/network_security/#scanning-ports-with-nmap","text":"Nmap (\"Network Mapper\") is a free and open-source (license) utility for network discovery and security auditing. Many systems and network administrators also find it useful for tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime. The best thing about Nmap is it\u2019s free and open-source and is very flexible and versatile Nmap is often used to determine alive hosts in a network, open ports on those hosts, services running on those open ports, and version identification of that service on that port. More at http://scanme.nmap.org/ nmap [scan type] [options] [target specification] Nmap uses 6 different port states: Open \u2014 An open port is one that is actively accepting TCP, UDP or SCTP connections. Open ports are what interests us the most because they are the ones that are vulnerable to attacks. Open ports also show the available services on a network. Closed \u2014 A port that receives and responds to Nmap probe packets but there is no application listening on that port. Useful for identifying that the host exists and for OS detection. Filtered \u2014 Nmap can\u2019t determine whether the port is open because packet filtering prevents its probes from reaching the port. Filtering could come from firewalls or router rules. Often little information is given from filtered ports during scans as the filters can drop the probes without responding or respond with useless error messages e.g. destination unreachable. Unfiltered \u2014 Port is accessible but Nmap doesn\u2019t know if it is open or closed. Only used in ACK scan which is used to map firewall rulesets. Other scan types can be used to identify whether the port is open. Open/filtered \u2014 Nmap is unable to determine between open and filtered. This happens when an open port gives no response. No response could mean that the probe was dropped by a packet filter or any response is blocked. Closed/filtered \u2014 Nmap is unable to determine whether a port is closed or filtered. Only used in the IP ID idle scan.","title":"Scanning Ports with Nmap"},{"location":"security/network_security/#types-of-nmap-scan","text":"TCP Connect TCP Connect scan completes the 3-way handshake. If a port is open, the operating system completes the TCP three-way handshake and the port scanner immediately closes the connection to avoid DOS. This is \u201cnoisy\u201d because the services can log the sender IP address and might trigger Intrusion Detection Systems. UDP Scan This scan checks to see if any UDP ports are listening. Since UDP does not respond with a positive acknowledgement like TCP and only responds to an incoming UDP packet when the port is closed, SYN Scan SYN scan is another form of TCP scanning. This scan type is also known as \u201chalf-open scanning\u201d because it never actually opens a full TCP connection. The port scanner generates a SYN packet. If the target port is open, it will respond with an SYN-ACK packet. The scanner host responds with an RST packet, closing the connection before the handshake is completed. If the port is closed but unfiltered, the target will instantly respond with an RST packet. SYN scan has the advantage that the individual services never actually receive a connection. FIN Scan This is a stealthy scan, like the SYN scan, but sends a TCP FIN packet instead. ACK Scan Ack scanning determines whether the port is filtered or not. Null Scan Another very stealthy scan that sets all the TCP header flags to off or null. This is not normally a valid packet and some hosts will not know what to do with this. XMAS Scan Similar to the NULL scan except for all the flags in the TCP header is set to on RPC Scan This special type of scan looks for machine answering to RPC (Remote Procedure Call) services IDLE Scan It is a super stealthy method whereby the scan packets are bounced off an external host. You don\u2019t need to have control over the other host but it does have to set up and meet certain requirements. You must input the IP address of our \u201czombie\u201d host and what port number to use. It is one of the more controversial options in Nmap since it only has a use for malicious attacks. Scan Techniques A couple of scan techniques which can be used to gain more information about a system and its ports. You can read more at https://medium.com/infosec-adventures/nmap-cheatsheet-a423fcdda0ca","title":"Types of Nmap Scan:"},{"location":"security/network_security/#openvas","text":"OpenVAS is a full-featured vulnerability scanner. OpenVAS is a framework of services and tools that provides a comprehensive and powerful vulnerability scanning and management package OpenVAS, which is an open-source program, began as a fork of the once-more-popular scanning program, Nessus. OpenVAS is made up of three main parts. These are: a regularly updated feed of Network Vulnerability Tests (NVTs); a scanner, which runs the NVTs; and an SQLite 3 database for storing both your test configurations and the NVTs\u2019 results and configurations. https://www.greenbone.net/en/install_use_gce/","title":"OpenVAS"},{"location":"security/network_security/#wireshark","text":"Wireshark is a protocol analyzer. This means Wireshark is designed to decode not only packet bits and bytes but also the relations between packets and protocols. Wireshark understands protocol sequences. A simple demo of Wireshark Capture only udp packets: Capture filter = \u201cudp\u201d Capture only tcp packets Capture filter = \u201ctcp\u201d TCP/IP 3 way Handshake Filter by IP address: displays all traffic from IP, be it source or destination ip.addr == 192.168.1.1 Filter by source address: display traffic only from IP source ip.src == 192.168.0.1 Filter by destination: display traffic only form IP destination ip.dst == 192.168.0.1 Filter by IP subnet: display traffic from subnet, be it source or destination ip.addr = 192.168.0.1/24 Filter by protocol: filter traffic by protocol name dns http ftp arp ssh telnet icmp Exclude IP address: remove traffic from and to IP address !ip.addr ==192.168.0.1 Display traffic between two specific subnet ip.addr == 192.168.0.1/24 and ip.addr == 192.168.1.1/24 Display traffic between two specific workstations ip.addr == 192.168.0.1 and ip.addr == 192.168.0.2 Filter by MAC eth.addr = 00:50:7f:c5:b6:78 Filter TCP port tcp.port == 80 Filter TCP port source tcp.srcport == 80 Filter TCP port destination tcp.dstport == 80 Find user agents http.user_agent contains Firefox !http.user_agent contains || !http.user_agent contains Chrome Filter broadcast traffic !(arp or icmp or dns) Filter IP address and port tcp.port == 80 && ip.addr == 192.168.0.1 Filter all http get requests http.request Filter all http get requests and responses http.request or http.response Filter three way handshake tcp.flags.syn==1 or (tcp.seq==1 and tcp.ack==1 and tcp.len==0 and tcp.analysis.initial_rtt) Find files by type frame contains \u201c(attachment|tar|exe|zip|pdf)\u201d Find traffic based on keyword tcp contains facebook frame contains facebook Detecting SYN Floods tcp.flags.syn == 1 and tcp.flags.ack == 0 Wireshark Promiscuous Mode - By default, Wireshark only captures packets going to and from the computer where it runs. By checking the box to run Wireshark in Promiscuous Mode in the Capture Settings, you can capture most of the traffic on the LAN.","title":"WireShark"},{"location":"security/network_security/#dumpcap","text":"Dumpcap is a network traffic dump tool. It captures packet data from a live network and writes the packets to a file. Dumpcap\u2019s native capture file format is pcapng, which is also the format used by Wireshark. By default, Dumpcap uses the pcap library to capture traffic from the first available network interface and writes the received raw packet data, along with the packets\u2019 time stamps into a pcapng file. The capture filter syntax follows the rules of the pcap library. The Wireshark command-line utility called 'dumpcap.exe' can be used to capture LAN traffic over an extended period of time. Wireshark itself can also be used, but dumpcap does not significantly utilize the computer's memory while capturing for long periods.","title":"DumpCap"},{"location":"security/network_security/#daemonlogger","text":"Daemonlogger is a packet logging application designed specifically for use in Network and Systems Management (NSM) environments. The biggest benefit Daemonlogger provides is that, like Dumpcap, it is simple to use for capturing packets. In order to begin capturing, you need only to invoke the command and specify an interface. daemonlogger \u2013i eth1 This option, by default, will begin capturing packets and logging them to the current working directory. Packets will be collected until the capture file size reaches 2 GB, and then a new file will be created. This will continue indefinitely until the process is halted.","title":"DaemonLogger"},{"location":"security/network_security/#netsniff-ng","text":"Netsniff-NG is a high-performance packet capture utility While the utilities we\u2019ve discussed to this point rely on Libpcap for capture, Netsniff-NG utilizes zero-copy mechanisms to capture packets. This is done with the intent to support full packet capture over high throughput links. To begin capturing packets with Netsniff-NG, we have to specify an input and output. In most cases, the input will be a network interface, and the output will be a file or folder on disk. netsniff-ng \u2013i eth1 \u2013o data.pcap","title":"NetSniff-NG"},{"location":"security/network_security/#netflow","text":"NetFlow is a feature that was introduced on Cisco routers around 1996 that provides the ability to collect IP network traffic as it enters or exits an interface. By analyzing the data provided by NetFlow, a network administrator can determine things such as the source and destination of traffic, class of service, and the causes of congestion. A typical flow monitoring setup (using NetFlow) consists of three main components:[1] Flow exporter: aggregates packets into flows and exports flow records towards one or more flow collectors. Flow collector: responsible for reception, storage and pre-processing of flow data received from a flow exporter. Analysis application: analyzes received flow data in the context of intrusion detection or traffic profiling, for example. Routers and switches that support NetFlow can collect IP traffic statistics on all interfaces where NetFlow is enabled, and later export those statistics as NetFlow records toward at least one NetFlow collector\u2014typically a server that does the actual traffic analysis.","title":"Netflow"},{"location":"security/network_security/#ids","text":"A security solution that detects security-related events in your environment but does not block them. IDS sensors can be software and hardware-based used to collect and analyze the network traffic. These sensors are available in two varieties, network IDS and host IDS. A host IDS is a server-specific agent running on a server with a minimum of overhead to monitor the operating system. A network IDS can be embedded in a networking device, a standalone appliance, or a module monitoring the network traffic. Signature Based IDS The signature-based IDS monitors the network traffic or observes the system and sends an alarm if a known malicious event is happening. It does so by comparing the data flow against a database of known attack patterns These signatures explicitly define what traffic or activity should be considered as malicious. Signature-based detection has been the bread and butter of network-based defensive security for over a decade, partially because it is very similar to how malicious activity is detected at the host level with antivirus utilities The formula is fairly simple: an analyst observes a malicious activity, derives indicators from the activity and develops them into signatures, and then those signatures will alert whenever the activity occurs again. ex: SNORT & SURICATA Policy-Based IDS The policy-based IDSs (mainly host IDSs) trigger an alarm whenever a violation occurs against the configured policy. This configured policy is or should be a representation of the security policies. This type of IDS is flexible and can be customized to a company's network requirements because it knows exactly what is permitted and what is not. On the other hand, the signature-based systems rely on vendor specifics and default settings. Anomaly Based IDS The anomaly-based IDS looks for traffic that deviates from the normal, but the definition of what is a normal network traffic pattern is the tricky part Two types of anomaly-based IDS exist: statistical and nonstatistical anomaly detection Statistical anomaly detection learns the traffic patterns interactively over a period of time. In the nonstatistical approach, the IDS has a predefined configuration of the supposedly acceptable and valid traffic patterns. Host-Based IDS & Network-Based IDS A host IDS can be described as a distributed agent residing on each server of the network that needs protection. These distributed agents are tied very closely to the underlying operating system. Network IDSs, on the other hand, can be described as intelligent sniffing devices. Data (raw packets) is captured from the network by a network IDS, whereas host IDSs capture the data from the host on which they are installed. Honeypots The use of decoy machines to direct intruders' attention away from the machines under protection is a major technique to preclude intrusion attacks. Any device, system, directory, or file used as a decoy to lure attackers away from important assets and to collect intrusion or abusive behaviours is referred to as a honeypot. A honeypot may be implemented as a physical device or as an emulation system. The idea is to set up decoy machines in a LAN, or decoy directories/files in a file system and make them appear important, but with several exploitable loopholes, to lure attackers to attack these machines or directories/files, so that other machines, directories, and files can evade intruders' attentions. A decoy machine may be a host computer or a server computer. Likewise, we may also set up decoy routers or even decoy LANs.","title":"IDS"},{"location":"security/network_security/#chinks-in-the-armour-tcpip-security-issues","text":"","title":"Chinks In The Armour (TCP/IP Security Issues)"},{"location":"security/network_security/#ip-spoofing","text":"In this type of attack, the attacker replaces the IP address of the sender, or in some rare cases the destination, with a different address. IP spoofing is normally used to exploit a target host. In other cases, it is used to start a denial-of-service (DoS) attack. In a DoS attack, an attacker modifies the IP packet to mislead the target host into accepting the original packet as a packet sourced at a trusted host. The attacker must know the IP address of the trusted host to modify the packet headers (source IP address) so that it appears that the packets are coming from that host. IP Spoofing Detection Techniques Direct TTL Probes In this technique we send a packet to a host of suspect spoofed IP that triggers reply and compares TTL with suspect packet; if the TTL in the reply is not the same as the packet being checked; it is a spoofed packet. This Technique is successful when the attacker is in a different subnet from the victim. IP Identification Number. Send a probe to the host of suspect spoofed traffic that triggers a reply and compares IP ID with suspect traffic. If IP IDs are not in the near value of packet being checked, suspect traffic is spoofed TCP Flow Control Method Attackers sending spoofed TCP packets will not receive the target\u2019s SYN-ACK packets. Attackers cannot, therefore, be responsive to change in the congestion window size When the receiver still receives traffic even after a windows size is exhausted, most probably the packets are spoofed.","title":"IP Spoofing"},{"location":"security/network_security/#covert-channel","text":"A covert or clandestine channel can be best described as a pipe or communication channel between two entities that can be exploited by a process or application transferring information in a manner that violates the system's security specifications. More specifically for TCP/IP, in some instances, covert channels are established, and data can be secretly passed between two end systems. Ex: ICMP resides at the Internet layer of the TCP/IP protocol suite and is implemented in all TCP/IP hosts. Based on the specifications of the ICMP Protocol, an ICMP Echo Request message should have an 8-byte header and a 56-byte payload. The ICMP Echo Request packet should not carry any data in the payload. However, these packets are often used to carry secret information. The ICMP packets are altered slightly to carry secret data in the payload. This makes the size of the packet larger, but no control exists in the protocol stack to defeat this behaviour. The alteration of ICMP packets allows intruders to program specialized client-server pairs. These small pieces of code export confidential information without alerting the network administrator. ICMP can be leveraged for more than data exfiltration. For eg. some C&C tools such as Loki used ICMP channel to establish encrypted interactive session back in 1996. Deep packet inspection has since come a long way. A lot of IDS/IPS detect ICMP tunnelling. Check for echo responses that do not contain the same payload as request Check for the volume of ICMP traffic especially for volumes beyond an acceptable threshold","title":"Covert Channel"},{"location":"security/network_security/#ip-fragmentation-attack","text":"The TCP/IP protocol suite, or more specifically IP, allows the fragmentation of packets.(this is a feature & not a bug) IP fragmentation offset is used to keep track of the different parts of a datagram. The information or content in this field is used at the destination to reassemble the datagrams All such fragments have the same Identification field value, and the fragmentation offset indicates the position of the current fragment in the context of the original packet. Many access routers and firewalls do not perform packet reassembly. In normal operation, IP fragments do not overlap, but attackers can create artificially fragmented packets to mislead the routers or firewalls. Usually, these packets are small and almost impractical for end systems because of data and computational overhead. A good example of an IP fragmentation attack is the Ping of Death attack. The Ping of Death attack sends fragments that, when reassembled at the end station, create a larger packet than the maximum permissible length. TCP Flags Data exchange using TCP does not happen until a three-way handshake has been completed. This handshake uses different flags to influence the way TCP segments are processed. There are 6 bits in the TCP header that are often called flags. Namely: 6 different flags are part of the TCP header: Urgent pointer field (URG), Acknowledgment field (ACK), Push function (PSH), Reset the connection (RST), Synchronize sequence numbers (SYN), and the sender is finished with this connection (FIN). Abuse of the normal operation or settings of these flags can be used by attackers to launch DoS attacks. This causes network servers or web servers to crash or hang. | SYN | FIN | PSH | RST | Validity| |------|------|-------|------|---------| | 1 |1 |0 |0 |Illegal Combination | 1 |1 |1 |0 |Illegal Combination | 1 |1 |0 |1 |Illegal Combination | 1 |1 |1 |1 |Illegal Combination The attacker's ultimate goal is to write special programs or pieces of code that can construct these illegal combinations resulting in an efficient DoS attack. SYN FLOOD The timers (or lack of certain timers) in 3 way handshake are often used and exploited by attackers to disable services or even to enter systems. After step 2 of the three-way handshake, no limit is set on the time to wait after receiving a SYN. The attacker initiates many connection requests to the webserver of Company XYZ (almost certainly with a spoofed IP address). The SYN+ACK packets (Step 2) sent by the web server back to the originating source IP address are not replied to. This leaves a TCP session half-open on the webserver. Multiple packets cause multiple TCP sessions to stay open. Based on the hardware limitations of the server, a limited number of TCP sessions can stay open, and as a result, the webserver refuses further connection establishments attempts from any host as soon as a certain limit is reached. These half-open connections need to be completed or timed out before new connections can be established. FIN Attack In normal operation, the sender sets the TCP FIN flag indicating that no more data will be transmitted and the connection can be closed down. This is a four-way handshake mechanism, with both sender and receiver expected to send an acknowledgement on a received FIN packet. During an attack that is trying to kill connections, a spoofed FIN packet is constructed. This packet also has the correct sequence number, so the packets are seen as valid by the targeted host. These sequence numbers are easy to predict. This process is referred to as TCP sequence number prediction, whereby the attacker either sniffs the current Sequence and Acknowledgment (SEQ/ACK) numbers of the connection or can algorithmically predict these numbers.","title":"IP Fragmentation Attack"},{"location":"security/network_security/#connection-hijacking","text":"An authorized user (Employee X) sends HTTP requests over a TCP session with the webserver. The web server accepts the packets from Employee X only when the packet has the correct SEQ/ACK numbers. As seen previously, these numbers are important for the webserver to distinguish between different sessions and to make sure it is still talking to Employee X. Imagine that the cracker starts sending packets to the web server spoofing the IP address of Employee X, using the correct SEQ/ACK combination. The web server accepts the packet and increments the ACK number. In the meantime, Employee X continues to send packets but with incorrect SEQ/ACK numbers. As a result of sending unsynchronized packets, all data from Employee X is discarded when received by the webserver. The attacker pretends to be Employee X using the correct numbers. This finally results in the cracker hijacking the connection, whereby Employee X is completely confused and the webserver replies assuming the cracker is sending correct synchronized data. STEPS: The attacker examines the traffic flows with a network monitor and notices traffic from Employee X to a web server. The web server returns or echoes data back to the origination station (Employee X). Employee X acknowledges the packet. The cracker launches a spoofed packet to the server. The web server responds to the cracker. The cracker starts verifying SEQ/ACK numbers to double-check success. At this time, the cracker takes over the session from Employee X, which results in a session hanging for Employee X. The cracker can start sending traffic to the webserver. The web server returns the requested data to confirm delivery with the correct ACK number. The cracker can continue to send data (keeping track of the correct SEQ/ACK numbers) until eventually setting the FIN flag to terminate the session.","title":"Connection Hijacking"},{"location":"security/network_security/#buffer-overflow","text":"A buffer is a temporary data storage area used to store program code and data. When a program or process tries to store more data in a buffer than it was originally anticipated to hold, a buffer overflow occurs. Buffers are temporary storage locations in memory (memory or buffer sizes are often measured in bytes) that can store a fixed amount of data in bytes. When more data is retrieved than can be stored in a buffer location, the additional information must go into an adjacent buffer, resulting in overwriting the valid data held in them. Mechanism: Buffer overflow vulnerabilities exist in different types. But the overall goal for all buffer overflow attacks is to take over the control of a privileged program and, if possible, the host. The attacker has two tasks to achieve this goal. First, the dirty code needs to be available in the program's code address space. Second, the privileged program should jump to that particular part of the code, which ensures that the proper parameters are loaded into memory. The first task can be achieved in two ways: by injecting the code in the right address space or by using the existing code and modifying certain parameters slightly. The second task is a little more complex because the program's control flow needs to be modified to make the program jump to the dirty code. CounterMeasure: The most important approach is to have a concerted focus on writing correct code. A second method is to make the data buffers (memory locations) address space of the program code non-executable. This type of address space makes it impossible to execute code, which might be infiltrated in the program's buffers during an attack.","title":"Buffer Overflow"},{"location":"security/network_security/#more-spoofing","text":"Address Resolution Protocol Spoofing The Address Resolution Protocol (ARP) provides a mechanism to resolve, or map, a known IP address to a MAC sublayer address. Using ARP spoofing, the cracker can exploit this hardware address authentication mechanism by spoofing the hardware address of Host B. Basically, the attacker can convince any host or network device on the local network that the cracker's workstation is the host to be trusted. This is a common method used in a switched environment. ARP spoofing can be prevented with the implementation of static ARP tables in all the hosts and routers of your network. Alternatively, you can implement an ARP server that responds to ARP requests on behalf of the target host. DNS Spoofing DNS spoofing is the method whereby the hacker convinces the target machine that the system it wants to connect to is the machine of the cracker. The cracker modifies some records so that name entries of hosts correspond to the attacker's IP address. There have been instances in which the complete DNS server was compromised by an attack. To counter DNS spoofing, the reverse lookup detects these attacks. The reverse lookup is a mechanism to verify the IP address against a name. The IP address and name files are usually kept on different servers to make compromise much more difficult","title":"More Spoofing"},{"location":"security/threats_attacks_defences/","text":"Part III: Threats, Attacks & Defense DNS Protection Cache Poisoning Attack Since DNS responses are cached, a quick response can be provided for repeated translations. DNS negative queries are also cached, e.g., misspelt words, and all cached data periodically times out. Cache poisoning is an issue in what is known as pharming. This term is used to describe a hacker\u2019s attack in which a website\u2019s traffic is redirected to a bogus website by forging the DNS mapping. In this case, an attacker attempts to insert a fake address record for an Internet domain into the DNS. If the server accepts the fake record, the cache is poisoned and subsequent requests for the address of the domain are answered with the address of a server controlled by the attacker. As long as the fake entry is cached by the server, browsers or e-mail servers will automatically go to the address provided by the compromised DNS server. the typical time to live (TTL) for cached entries is a couple of hours, thereby permitting ample time for numerous users to be affected by the attack. DNSSEC (Security Extension) The long-term solution to these DNS problems is authentication. If a resolver cannot distinguish between valid and invalid data in a response, then add source authentication to verify that the data received in response is equal to the data entered by the zone administrator DNS Security Extensions (DNSSEC) protects against data spoofing and corruption and provides mechanisms to authenticate servers and requests, as well as mechanisms to establish authenticity and integrity. When authenticating DNS responses, each DNS zone signs its data using a private key. It is recommended that this signing be done offline and in advance. The query for a particular record returns the requested resource record set (RRset) and signature (RRSIG) of the requested resource record set. The resolver then authenticates the response using a public key, which is pre-configured or learned via a sequence of key records in the DNS hierarchy. The goals of DNSSEC are to provide authentication and integrity for DNS responses without confidentiality or DDoS protection. BGP BGP stands for border gateway protocol. It is a routing protocol that exchanges routing information among multiple Autonomous Systems (AS) An Autonomous System is a collection of routers or networks with the same network policy usually under single administrative control. BGP tells routers which hop to use in order to reach the destination network. BGP is used for both communicating information among routers in an AS (interior) and between multiple ASes (exterior). How BGP Works BGP is responsible for finding a path to a destination router & the path it chooses should be the shortest and most reliable one. This decision is done through a protocol known as Link state. With the link-state protocol, each router broadcasts to all other routers in the network the state of its links and IP subnets. Each router then receives information from the other routers and constructs a complete topology view of the entire network. The next-hop routing table is based on this topology view. The link-state protocol uses a famous algorithm in the field of computer science, Dijkstra\u2019s shortest path algorithm: We start from our router considering the path cost to all our direct neighbours. The shortest path is then taken We then re-look at all our neighbours that we can reach and update our link state table with the cost information. We then continue taking the shortest path until every router has been visited. BGP Vulnerabilities By corrupting the BGP routing table we are able to influence the direction traffic flows on the internet! This action is known as BGP hijacking. Injecting bogus route advertising information into the BGP-distributed routing database by malicious sources, accidentally or routers can disrupt Internet backbone operations. Blackholing traffic: Blackhole route is a network route, i.e., routing table entry, that goes nowhere and packets matching the route prefix are dropped or ignored. Blackhole routes can only be detected by monitoring the lost traffic. Blackhole routes are the best defence against many common viral attacks where the traffic is dropped from infected machines to/from command & control hosts. Infamous BGP Injection attack on Youtube Ex: In 2008, Pakistan decided to block YouTube by creating a BGP route that led into a black hole. Instead, this routing information got transmitted to a hong kong ISP and from there accidentally got propagated to the rest of the world meaning millions were routed through to this black hole and therefore unable to access YouTube. Potentially, the greatest risk to BGP occurs in a denial of service attack in which a router is flooded with more packets than it can handle. Network overload and router resource exhaustion happen when the network begins carrying an excessive number of BGP messages, overloading the router control processors, memory, routing table and reducing the bandwidth available for data traffic. Refer: https://medium.com/bugbountywriteup/bgp-the-weak-link-in-the-internet-what-is-bgp-and-how-do-hackers-exploit-it-d899a68ba5bb Router flapping is another type of attack. Route flapping refers to repetitive changes to the BGP routing table, often several times a minute. Withdrawing and re-advertising at a high-rate can cause a serious problem for routers since they propagate the announcements of routes. If these route flaps happen fast enough, e.g., 30 to 50 times per second, the router becomes overloaded, which eventually prevents convergence on valid routes. The potential impact for Internet users is a slowdown in message delivery, and in some cases, packets may not be delivered at all. BGP Security Border Gateway Protocol Security recommends the use of BGP peer authentication since it is one of the strongest mechanisms for preventing malicious activity. The authentication mechanisms are Internet Protocol Security (IPsec) or BGP MD5. Another method, known as prefix limits, can be used to avoid filling router tables. In this approach, routers should be configured to disable or terminate a BGP peering session, and issue warning messages to administrators when a neighbour sends in excess of a preset number of prefixes. IETF is currently working on improving this space Web-Based Attacks HTTP Response Splitting Attacks HTTP response splitting attack may happen where the server script embeds user data in HTTP response headers without appropriate sanitation. This typically happens when the script embeds user data in the redirection URL of a redirection response (HTTP status code 3xx), or when the script embeds user data in a cookie value or name when the response sets a cookie. HTTP response splitting attacks can be used to perform web cache poisoning and cross-site scripting attacks. HTTP response splitting is the attacker\u2019s ability to send a single HTTP request that forces the webserver to form an output stream, which is then interpreted by the target as two HTTP responses instead of one response. Cross-Site Request Forgery (CSRF or XSRF) A Cross-Site Request Forgery attack tricks the victim\u2019s browser into issuing a command to a vulnerable web application. Vulnerability is caused by browsers automatically including user authentication data, session ID, IP address, Windows domain credentials, etc. with each request. Attackers typically use CSRF to initiate transactions such as transfer funds, login/logout user, close account, access sensitive data, and change account details. The vulnerability is caused by web browsers that automatically include credentials with each request, even for requests caused by a form, script, or image on another site. CSRF can also be dynamically constructed as part of a payload for a cross-site scripting attack All sites relying on automatic credentials are vulnerable. Popular browsers cannot prevent cross-site request forgery. Logging out of high-value sites as soon as possible can mitigate CSRF risk. It is recommended that a high-value website must require a client to manually provide authentication data in the same HTTP request used to perform any operation with security implications. Limiting the lifetime of session cookies can also reduce the chance of being used by other malicious sites. OWASP recommends website developers include a required security token in HTTP requests associated with sensitive business functions in order to mitigate CSRF attacks Cross-Site Scripting (XSS) Attacks Cross-Site Scripting occurs when dynamically generated web pages display user input, such as login information, that is not properly validated, allowing an attacker to embed malicious scripts into the generated page and then execute the script on the machine of any user that views the site. If successful, Cross-Site Scripting vulnerabilities can be exploited to manipulate or steal cookies, create requests that can be mistaken for those of a valid user, compromise confidential information, or execute malicious code on end-user systems. Cross-Site Scripting (XSS or CSS) attacks involve the execution of malicious scripts on the victim\u2019s browser. The victim is simply a user\u2019s host and not the server. XSS results from a failure to validate user input by a web-based application. Document Object Model (DOM) XSS Attacks The Document Object Model (DOM) based XSS does not require the webserver to receive the XSS payload for a successful attack. The attacker abuses the runtime by embedding their data on the client-side. An attacker can force the client (browser) to render the page with parts of the DOM controlled by the attacker. When the page is rendered and the data is processed by the page, typically by a client-side HTML-embedded script such as JavaScript, the page\u2019s code may insecurely embed the data in the page itself, thus delivering the cross-site scripting payload. There are several DOM objects which can serve as an attack vehicle for delivering malicious script to victims browser. Clickjacking The technique works by hiding malicious link/scripts under the cover of the content of a legitimate site. Buttons on a website actually contain invisible links, placed there by the attacker. So, an individual who clicks on an object they can visually see is actually being duped into visiting a malicious page or executing a malicious script. When mouseover is used together with clickjacking, the outcome is devastating. Facebook users have been hit by a clickjacking attack, which tricks people into \u201cliking\u201d a particular Facebook page, thus enabling the attack to spread since Memorial Day 2010. There is not yet effective defence against clickjacking, and disabling JavaScript is the only viable method DataBase Attacks & Defenses SQL injection Attacks It exploits improper input validation in database queries. A successful exploit will allow attackers to access, modify, or delete information in the database. It permits attackers to steal sensitive information stored within the backend databases of affected websites, which may include such things as user credentials, email addresses, personal information, and credit card numbers SELECT USERNAME,PASSWORD from USERS where USERNAME='Redirecting...
* Closing connection 0 Redirecting...
* Closing connection 0 Redirecting...
* Closing connection 0