GitHub/school-of-sre
1
0
Fork 0
mirror of https://github.com/linkedin/school-of-sre synced 2026-01-07 09:08:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
0fb7d395b1fb1e8bd9ad4b8f22488d29248fa74a
school-of-sre/python_web/python-concepts/index.html
Kalyanasundaram Somasundaram 0fb7d395b1 Deployed 4eb864e with MkDocs version: 1.1.2
2020-11-17 13:49:14 +05:50

1062 lines
27 KiB
HTML
Raw Blame History

<!doctype html>
<html lang="en" class="no-js">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<link rel="shortcut icon" href="../../assets/images/favicon.png">
<meta name="generator" content="mkdocs-1.1.2, mkdocs-material-6.1.5">
<title>Some Python Concepts - SchoolOfSRE</title>
<link rel="stylesheet" href="../../assets/stylesheets/main.21aed14c.min.css">
<link rel="stylesheet" href="../../assets/stylesheets/palette.196e0c26.min.css">
<link href="https://fonts.gstatic.com" rel="preconnect" crossorigin>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,400i,700%7CRoboto+Mono&display=fallback">
<style>body,input{font-family:"Roboto",-apple-system,BlinkMacSystemFont,Helvetica,Arial,sans-serif}code,kbd,pre{font-family:"Roboto Mono",SFMono-Regular,Consolas,Menlo,monospace}</style>
</head>
<body dir="ltr" data-md-color-scheme="" data-md-color-primary="none" data-md-color-accent="none">
<input class="md-toggle" data-md-toggle="drawer" type="checkbox" id="__drawer" autocomplete="off">
<input class="md-toggle" data-md-toggle="search" type="checkbox" id="__search" autocomplete="off">
<label class="md-overlay" for="__drawer"></label>
<div data-md-component="skip">
<a href="#some-python-concepts" class="md-skip">
Skip to content
</a>
</div>
<div data-md-component="announce">
</div>
<header class="md-header" data-md-component="header">
<nav class="md-header-nav md-grid" aria-label="Header">
<a href="../.." title="SchoolOfSRE" class="md-header-nav__button md-logo" aria-label="SchoolOfSRE">
<img src="../../img/sos.png" alt="logo">
</a>
<label class="md-header-nav__button md-icon" for="__drawer">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M3 6h18v2H3V6m0 5h18v2H3v-2m0 5h18v2H3v-2z"/></svg>
</label>
<div class="md-header-nav__title" data-md-component="header-title">
<div class="md-header-nav__ellipsis">
<span class="md-header-nav__topic md-ellipsis">
SchoolOfSRE
</span>
<span class="md-header-nav__topic md-ellipsis">
Some Python Concepts
</span>
</div>
</div>
</nav>
</header>
<div class="md-container" data-md-component="container">
<main class="md-main" data-md-component="main">
<div class="md-main__inner md-grid">
<div class="md-sidebar md-sidebar--primary" data-md-component="navigation">
<div class="md-sidebar__scrollwrap">
<div class="md-sidebar__inner">
<nav class="md-nav md-nav--primary" aria-label="Navigation" data-md-level="0">
<label class="md-nav__title" for="__drawer">
<a href="../.." title="SchoolOfSRE" class="md-nav__button md-logo" aria-label="SchoolOfSRE">
<img src="../../img/sos.png" alt="logo">
</a>
SchoolOfSRE
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../.." class="md-nav__link">
Home
</a>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle" data-md-toggle="nav-2" type="checkbox" id="nav-2" >
<label class="md-nav__link" for="nav-2">
Fundamentals Series
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" aria-label="Fundamentals Series" data-md-level="1">
<label class="md-nav__title" for="nav-2">
<span class="md-nav__icon md-icon"></span>
Fundamentals Series
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle" data-md-toggle="nav-2-1" type="checkbox" id="nav-2-1" >
<label class="md-nav__link" for="nav-2-1">
Git
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" aria-label="Git" data-md-level="2">
<label class="md-nav__title" for="nav-2-1">
<span class="md-nav__icon md-icon"></span>
Git
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../git/git-basics/" class="md-nav__link">
Git Basics
</a>
</li>
<li class="md-nav__item">
<a href="../../git/branches/" class="md-nav__link">
Working With Branches
</a>
</li>
<li class="md-nav__item">
<a href="../../git/github-hooks/" class="md-nav__link">
Github and Hooks
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle" data-md-toggle="nav-2-2" type="checkbox" id="nav-2-2" >
<label class="md-nav__link" for="nav-2-2">
Linux Networking
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" aria-label="Linux Networking" data-md-level="2">
<label class="md-nav__title" for="nav-2-2">
<span class="md-nav__icon md-icon"></span>
Linux Networking
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../linux_networking/intro/" class="md-nav__link">
Introduction
</a>
</li>
<li class="md-nav__item">
<a href="../../linux_networking/dns/" class="md-nav__link">
DNS
</a>
</li>
<li class="md-nav__item">
<a href="../../linux_networking/udp/" class="md-nav__link">
UDP
</a>
</li>
<li class="md-nav__item">
<a href="../../linux_networking/http/" class="md-nav__link">
HTTP
</a>
</li>
<li class="md-nav__item">
<a href="../../linux_networking/tcp/" class="md-nav__link">
TCP
</a>
</li>
<li class="md-nav__item">
<a href="../../linux_networking/ipr/" class="md-nav__link">
Routing and Conclusion
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--active md-nav__item--nested">
<input class="md-nav__toggle md-toggle" data-md-toggle="nav-3" type="checkbox" id="nav-3" checked>
<label class="md-nav__link" for="nav-3">
Python and Web
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" aria-label="Python and Web" data-md-level="1">
<label class="md-nav__title" for="nav-3">
<span class="md-nav__icon md-icon"></span>
Python and Web
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../intro/" class="md-nav__link">
Introduction
</a>
</li>
<li class="md-nav__item md-nav__item--active">
<input class="md-nav__toggle md-toggle" data-md-toggle="toc" type="checkbox" id="__toc">
<label class="md-nav__link md-nav__link--active" for="__toc">
Some Python Concepts
<span class="md-nav__icon md-icon"></span>
</label>
<a href="./" class="md-nav__link md-nav__link--active">
Some Python Concepts
</a>
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
<label class="md-nav__title" for="__toc">
<span class="md-nav__icon md-icon"></span>
Table of contents
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#python-functions" class="md-nav__link">
Python Functions
</a>
<nav class="md-nav" aria-label="Python Functions">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#globals" class="md-nav__link">
globals
</a>
</li>
<li class="md-nav__item">
<a href="#code" class="md-nav__link">
code
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#decorators" class="md-nav__link">
Decorators
</a>
</li>
<li class="md-nav__item">
<a href="#some-gotchas" class="md-nav__link">
Some Gotchas
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="../python-web-flask/" class="md-nav__link">
Python, Web and Flask
</a>
</li>
<li class="md-nav__item">
<a href="../url-shorten-app/" class="md-nav__link">
The URL Shortening App
</a>
</li>
<li class="md-nav__item">
<a href="../sre-conclusion/" class="md-nav__link">
SRE Aspects of The App and Conclusion
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle" data-md-toggle="nav-4" type="checkbox" id="nav-4" >
<label class="md-nav__link" for="nav-4">
Systems Design
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" aria-label="Systems Design" data-md-level="1">
<label class="md-nav__title" for="nav-4">
<span class="md-nav__icon md-icon"></span>
Systems Design
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../systems_design/intro/" class="md-nav__link">
Introduction
</a>
</li>
<li class="md-nav__item">
<a href="../../systems_design/scalability/" class="md-nav__link">
Scalability
</a>
</li>
<li class="md-nav__item">
<a href="../../systems_design/availability/" class="md-nav__link">
Availability
</a>
</li>
<li class="md-nav__item">
<a href="../../systems_design/fault-tolerance/" class="md-nav__link">
Fault Tolerance
</a>
</li>
<li class="md-nav__item">
<a href="../../systems_design/conclusion/" class="md-nav__link">
Conclusion
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle" data-md-toggle="nav-5" type="checkbox" id="nav-5" >
<label class="md-nav__link" for="nav-5">
Data
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" aria-label="Data" data-md-level="1">
<label class="md-nav__title" for="nav-5">
<span class="md-nav__icon md-icon"></span>
Data
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle" data-md-toggle="nav-5-1" type="checkbox" id="nav-5-1" >
<label class="md-nav__link" for="nav-5-1">
Big Data
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" aria-label="Big Data" data-md-level="2">
<label class="md-nav__title" for="nav-5-1">
<span class="md-nav__icon md-icon"></span>
Big Data
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../big_data/intro/" class="md-nav__link">
Introduction
</a>
</li>
<li class="md-nav__item">
<a href="../../big_data/overview/" class="md-nav__link">
Overview of Big Data
</a>
</li>
<li class="md-nav__item">
<a href="../../big_data/usage/" class="md-nav__link">
Usage of Big Data techniques
</a>
</li>
<li class="md-nav__item">
<a href="../../big_data/evolution/" class="md-nav__link">
Evolution of Hadoop
</a>
</li>
<li class="md-nav__item">
<a href="../../big_data/architecture/" class="md-nav__link">
Architecture of Hadoop
</a>
</li>
<li class="md-nav__item">
<a href="../../big_data/tasks/" class="md-nav__link">
Tasks and conclusion
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item md-nav__item--nested">
<input class="md-nav__toggle md-toggle" data-md-toggle="nav-6" type="checkbox" id="nav-6" >
<label class="md-nav__link" for="nav-6">
Security
<span class="md-nav__icon md-icon"></span>
</label>
<nav class="md-nav" aria-label="Security" data-md-level="1">
<label class="md-nav__title" for="nav-6">
<span class="md-nav__icon md-icon"></span>
Security
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="../../security/intro/" class="md-nav__link">
Introduction
</a>
</li>
<li class="md-nav__item">
<a href="../../security/fundamentals/" class="md-nav__link">
Fundamentals of Security
</a>
</li>
<li class="md-nav__item">
<a href="../../security/network_security/" class="md-nav__link">
Network Security
</a>
</li>
<li class="md-nav__item">
<a href="../../security/threats_attacks_defences/" class="md-nav__link">
Threat, Attacks & Defences
</a>
</li>
<li class="md-nav__item">
<a href="../../security/writing_secure_code/" class="md-nav__link">
Writing Secure code
</a>
</li>
</ul>
</nav>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div class="md-sidebar md-sidebar--secondary" data-md-component="toc">
<div class="md-sidebar__scrollwrap">
<div class="md-sidebar__inner">
<nav class="md-nav md-nav--secondary" aria-label="Table of contents">
<label class="md-nav__title" for="__toc">
<span class="md-nav__icon md-icon"></span>
Table of contents
</label>
<ul class="md-nav__list" data-md-scrollfix>
<li class="md-nav__item">
<a href="#python-functions" class="md-nav__link">
Python Functions
</a>
<nav class="md-nav" aria-label="Python Functions">
<ul class="md-nav__list">
<li class="md-nav__item">
<a href="#globals" class="md-nav__link">
globals
</a>
</li>
<li class="md-nav__item">
<a href="#code" class="md-nav__link">
code
</a>
</li>
</ul>
</nav>
</li>
<li class="md-nav__item">
<a href="#decorators" class="md-nav__link">
Decorators
</a>
</li>
<li class="md-nav__item">
<a href="#some-gotchas" class="md-nav__link">
Some Gotchas
</a>
</li>
</ul>
</nav>
</div>
</div>
</div>
<div class="md-content">
<article class="md-content__inner md-typeset">
<h1 id="some-python-concepts">Some Python Concepts</h1>
<p>Though you are expected to know python and its syntax at basic level, let us discuss some fundamental concepts that will help you understand the python language better.</p>
<p><strong>Everything in Python is an object.</strong></p>
<p>That includes the functions, lists, dicts, classes, modules, a running function (instance of function definition), everything. In the CPython, it would mean there is an underlying struct variable for each object.</p>
<p>In python's current execution context, all the variables are stored in a dict. It'd be a string to object mapping. If you have a function and a float variable defined in the current context, here is how it is handled internally.</p>
<pre><code class="language-python">&gt;&gt;&gt; float_number=42.0
&gt;&gt;&gt; def foo_func():
... pass
...
# NOTICE HOW VARIABLE NAMES ARE STRINGS, stored in a dict
&gt;&gt;&gt; locals()
{'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': &lt;class '_frozen_importlib.BuiltinImporter'&gt;, '__spec__': None, '__annotations__': {}, '__builtins__': &lt;module 'builtins' (built-in)&gt;, 'float_number': 42.0, 'foo_func': &lt;function foo_func at 0x1055847a0&gt;}
</code></pre>
<h2 id="python-functions">Python Functions</h2>
<p>Since functions too are objects, we can see what all attributes a function contains as following</p>
<pre><code class="language-python">&gt;&gt;&gt; def hello(name):
... print(f&quot;Hello, {name}!&quot;)
...
&gt;&gt;&gt; dir(hello)
['__annotations__', '__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__',
'__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__get__', '__getattribute__', '__globals__', '__gt__',
'__hash__', '__init__', '__init_subclass__', '__kwdefaults__', '__le__', '__lt__', '__module__', '__name__',
'__ne__', '__new__', '__qualname__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__',
'__subclasshook__']
</code></pre>
<p>While there are a lot of them, let's look at some interesting ones</p>
<h4 id="globals"><strong>globals</strong></h4>
<p>This attribute, as the name suggests, has references of global variables. If you ever need to know what all global variables are in the scope of this function, this will tell you. See how the function start seeing the new variable in globals</p>
<pre><code class="language-python">&gt;&gt;&gt; hello.__globals__
{'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': &lt;class '_frozen_importlib.BuiltinImporter'&gt;, '__spec__': None, '__annotations__': {}, '__builtins__': &lt;module 'builtins' (built-in)&gt;, 'hello': &lt;function hello at 0x7fe4e82554c0&gt;}
# adding new global variable
&gt;&gt;&gt; GLOBAL=&quot;g_val&quot;
&gt;&gt;&gt; hello.__globals__
{'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': &lt;class '_frozen_importlib.BuiltinImporter'&gt;, '__spec__': None, '__annotations__': {}, '__builtins__': &lt;module 'builtins' (built-in)&gt;, 'hello': &lt;function hello at 0x7fe4e82554c0&gt;, 'GLOBAL': 'g_val'}
</code></pre>
<h3 id="code"><strong>code</strong></h3>
<p>This is an interesting one! As everything in python is an object, this includes the bytecode too. The compiled python bytecode is a python code object. Which is accessible via <code>__code__</code> attribute here. A function has an associated code object which carries some interesting information.</p>
<pre><code class="language-python"># the file in which function is defined
# stdin here since this is run in an interpreter
&gt;&gt;&gt; hello.__code__.co_filename
'&lt;stdin&gt;'
# number of arguments the function takes
&gt;&gt;&gt; hello.__code__.co_argcount
1
# local variable names
&gt;&gt;&gt; hello.__code__.co_varnames
('name',)
# the function code's compiled bytecode
&gt;&gt;&gt; hello.__code__.co_code
b't\x00d\x01|\x00\x9b\x00d\x02\x9d\x03\x83\x01\x01\x00d\x00S\x00'
</code></pre>
<p>There are more code attributes which you can enlist by <code>&gt;&gt;&gt; dir(hello.__code__)</code></p>
<h2 id="decorators">Decorators</h2>
<p>Related to functions, python has another feature called decorators. Let's see how that works, keeping <code>everything is an object</code> in mind.</p>
<p>Here is a sample decorator:</p>
<pre><code class="language-python">&gt;&gt;&gt; def deco(func):
... def inner():
... print(&quot;before&quot;)
... func()
... print(&quot;after&quot;)
... return inner
...
&gt;&gt;&gt; @deco
... def hello_world():
... print(&quot;hello world&quot;)
...
&gt;&gt;&gt;
&gt;&gt;&gt; hello_world()
before
hello world
after
</code></pre>
<p>Here <code>@deco</code> syntax is used to decorate the <code>hello_world</code> function. It is essentially same as doing</p>
<pre><code class="language-python">&gt;&gt;&gt; def hello_world():
... print(&quot;hello world&quot;)
...
&gt;&gt;&gt; hello_world = deco(hello_world)
</code></pre>
<p>What goes inside the <code>deco</code> function might seem complex. Let's try to uncover it.</p>
<ol>
<li>Function <code>hello_world</code> is created</li>
<li>It is passed to <code>deco</code> function</li>
<li><code>deco</code> create a new function<ol>
<li>This new function is calls <code>hello_world</code> function</li>
<li>And does a couple other things</li>
</ol>
</li>
<li><code>deco</code> returns the newly created function</li>
<li><code>hello_world</code> is replaced with above function</li>
</ol>
<p>Let's visualize it for better understanding</p>
<pre><code> BEFORE function_object (ID: 100)
&quot;hello_world&quot; +--------------------+
+ |print(&quot;hello_world&quot;)|
| | |
+--------------&gt; | |
| |
+--------------------+
WHAT DECORATOR DOES
creates a new function (ID: 101)
+---------------------------------+
|input arg: function with id: 100 |
| |
|print(&quot;before&quot;) |
|call function object with id 100 |
|print(&quot;after&quot;) |
| |
+---------------------------^-----+
|
|
AFTER |
|
|
&quot;hello_world&quot; +-------------+
</code></pre>
<p>Note how the <code>hello_world</code> name points to a new function object but that new function object knows the reference (ID) of the original function.</p>
<h2 id="some-gotchas">Some Gotchas</h2>
<ul>
<li>While it is very quick to build prototypes in python and there are tons of libraries available, as the codebase complexity increases, type errors become more common and will get hard to deal with. (There are solutions to that problem like type annotations in python. Checkout <a href="http://mypy-lang.org/">mypy</a>.)</li>
<li>Because python is dynamically typed language, that means all types are determined at runtime. And that makes python run very slow compared to other statically typed languages.</li>
<li>Python has something called <a href="https://www.dabeaz.com/python/UnderstandingGIL.pdf">GIL</a> (global interpreter lock) which is a limiting factor for utilizing multiple CPI cores for parallel computation.</li>
<li>Some weird things that python does: https://github.com/satwikkansal/wtfpython</li>
</ul>
</article>
</div>
</div>
</main>
<footer class="md-footer">
<div class="md-footer-nav">
<nav class="md-footer-nav__inner md-grid" aria-label="Footer">
<a href="../intro/" class="md-footer-nav__link md-footer-nav__link--prev" rel="prev">
<div class="md-footer-nav__button md-icon">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M20 11v2H8l5.5 5.5-1.42 1.42L4.16 12l7.92-7.92L13.5 5.5 8 11h12z"/></svg>
</div>
<div class="md-footer-nav__title">
<div class="md-ellipsis">
<span class="md-footer-nav__direction">
Previous
</span>
Introduction
</div>
</div>
</a>
<a href="../python-web-flask/" class="md-footer-nav__link md-footer-nav__link--next" rel="next">
<div class="md-footer-nav__title">
<div class="md-ellipsis">
<span class="md-footer-nav__direction">
Next
</span>
Python, Web and Flask
</div>
</div>
<div class="md-footer-nav__button md-icon">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path d="M4 11v2h12l-5.5 5.5 1.42 1.42L19.84 12l-7.92-7.92L10.5 5.5 16 11H4z"/></svg>
</div>
</a>
</nav>
</div>
<div class="md-footer-meta md-typeset">
<div class="md-footer-meta__inner md-grid">
<div class="md-footer-copyright">
Made with
<a href="https://squidfunk.github.io/mkdocs-material/" target="_blank" rel="noopener">
Material for MkDocs
</a>
</div>
</div>
</div>
</footer>
</div>
<script src="../../assets/javascripts/vendor.7e0ee788.min.js"></script>
<script src="../../assets/javascripts/bundle.b3a72adc.min.js"></script><script id="__lang" type="application/json">{"clipboard.copy": "Copy to clipboard", "clipboard.copied": "Copied to clipboard", "search.config.lang": "en", "search.config.pipeline": "trimmer, stopWordFilter", "search.config.separator": "[\\s\\-]+", "search.placeholder": "Search", "search.result.placeholder": "Type to start searching", "search.result.none": "No matching documents", "search.result.one": "1 matching document", "search.result.other": "# matching documents", "search.result.more.one": "1 more on this page", "search.result.more.other": "# more on this page", "search.result.term.missing": "Missing"}</script>
<script>
app = initialize({
base: "../..",
features: [],
search: Object.assign({
worker: "../../assets/javascripts/worker/search.4ac00218.min.js"
}, typeof search !== "undefined" && search)
})
</script>
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink