CS50 2018 Notes

CS50 HarvardX Edx 2018 Course Notes


Last Updated: May 18, 2018 by Pepe Sandoval



Want to show support?

If you find the information in this page useful and want to show your support, you can make a donation

Use PayPal

This will help me to create more stuff and fix the existent content... or probably your money will be used to buy beer


CS50 Notes

Index

Intro

  • Computational thinking follows this general logic to solve a problem:
    • input -> process (algorithms) -> output
  • Computers always use a finite amount of information to represent something in the case of number you use a certain amount of finite number of bits
  • Computer doesn't know how to make assumptions it will only do precisely what you tell it to do
  • Everything on the SW world ends up taking up much longer than anticipated

Basic Elements of Programming language

  • Statements
  • Loops
  • Conditionals
    • Boolean expressions
    • Fork / branches
  • Data types
    • primitives
    • modifiers / qualifiers

void is a type but not a data type

  • scope refers to the context in which a variable exist or is usable

precedence

  • precedence answers the question what operation should we do first. Rules:
    • innermost parentheses to outwards
    • if operators have equal priority you solve from left to right
    • dereference operator has priority over math operators but not over suffix increment or decrement operators

Typecasting

  • It is an easy way to alter computer interpretation by implicitly (you let compiler do the casting) or explicitly (you specify the type with a parentheses) changing its data type
  • The data type determines how the computer and user interprets a variable
  • In typecasting you can make a variable lose precision but never gain precision (truncation)

Compiler / Compilation

  • source code: code that is on a programming languages and it can be understood by humans who know that programming languages
  • object code: Fancy way of saying binary output or "code that can be understand by a machine"
  • Compile in the most general sense means transforming code from one programming language into another language or into another format
  • Compiler: Piece of SW that takes source code as input and produces object code as output

Compilation main steps (Considering C/C++) CS50 Compilers Short

  • Compile is the general term to describe the entire process needed to generate an executable, which precisely involves other processes
  • Preprocessing
    • Replaces all pre-processor directives (#define, #include...) which basically means copy and paste all the included .h files or libraries and replace all the occurrences of a #define with the correspondent value
    • This step is done by the C pre-processor
  • Compilation
    • Transform C language code to assembly code
  • Assembler
    • Translates Assembly code into object code or machine code which is the binary code or ones and zeros a CPU can actually understand
    • This step is considered a translation and NOT a transformation since is a very direct/easy to go from assembly to object code
  • Linking
    • Combines object files into an executable file or application
    • To link multiple files you specify all of the when linking with the only restriction that only one can have a main function
    • Adding libraries flags is the same that specifying the .o with the compiled code you use on your program

      put all -l options at the end

make

  • make is an utility mostly used for compiling source code
  • It is a program the sequences other operations to produce a successful compilation output (usually an application or a library)
  • It is a program that organizes invocations of the compiler on source code files so that things get compiled in the proper order
  • Makefiles are the configuration files that tell make exactly what to do.

Library

  • A Library is a collection of related prewritten code, and its a way a programmer can share common code
  • Technically it is just binary or compiled file that has been produced by linking together a collection of object files
  • A Library is usually composed of two parts
    • Header files (*.h) include only declaration not implementation of functionality, this is an example of information hiding or encapsulation
    • implementation files (*.c) include the actual code implementation

      when sharing a library we usually don't share the implementation files (*.c) we just share a compiled binary file which is represent all the .c files

Computational and Algorithm complexity

  • Computational complexity asks the questions how an algorithm handles larger and larger data set or how does this algorithm scale as we throw more and more data at it
  • Its a more uniform way of measuring algorithm efficiency
  • asymptotic complexity: analyze how fast a programs run time grows asymptotically, so the input tend to infinity

Big O

  • Big O notation is a notation that computer scientist uses to describe an upper bound on an algorithm's running time
  • Describes the maximum number of steps or maximum amount of time an algorithm will take to complete
  • It gives an idea of how much time an algorithm takes to complete in the the worst case scenario
    • for example we may say an algorithm takes at worst n squared (n^2) steps

Omega

  • Omega notation is a notation that computer scientist use to describe a lower bound on an algorithm's running time
  • It gives an idea of how much time an algorithm takes to complete in the best case scenario

Sorting and Searching Algorithms

Selection Sort - O(n^2) and Om(n^2)
  • Find the smallest unsorted element and swap it with the first unsorted element of the list (put it a the beginning of the list)
  • Pick the criteria, find the first element that meets that criteria and order that element.
  • Build a sorted list one element at a time
Bubble Sort - O(n^2) and Om(n)
  • Look at adjacent pair elements on the list and if they are out of order swap them, then move to the next pair and repeat this process until no swaps are needed
  • The general idea is to swap elements to move the bigger elements to end smaller elements to the beginning
Insertion Sort - O(n^2) and Om(n)
  • For the first element state a one element array is sorted, find the next smallest element and insert it in the proper position making sure all elements behind are sorted
  • It basically find element and insert it where belongs, shifting elements as necessary
Merge Sort - O(n*log(n)) and Om(n*log(n))
  • The idea is to sort smaller arrays and combine them together in sorted array using recursion
  • sort left side of the array, sort right side of the array and merge the two halves together
Linear search - O(n) and Om(1)
  • Iterate across the array from left to right looking for the specified element
Binary search - O(log(n)) and Om(1)
  • It makes the assumption the data is sorted, it requires the condition that the data is sorted in order to work
  • Divide the amount of data by halve each time we are trying to find an element until the sub-list is zero
  • It makes a binary decision

Data structures

  • A data structure is just a container for data
  • Something that provides a way to store data in a specific format

Binary search tree

It's a data structure with the following properties:

  1. If you look at any node its left child is smaller and its right child is bigger
  2. The left subtree of a node only contains values that are less than or equal to the node's value
  3. The right subtree of a node only contains values that are greater than or equal to the node's value
  4. Both left and right subtrees are also binary search tree

Hash table

  • Combine the random access ability of an array with the dynamism of a linked list
  • Insert, delete and lookup start to tend toward average of O(1)
  • The data itself will produce an identifier that can be used to add/remove/update/search it
  • They don't solve the problem of sorting/ordering data
  • Elements:
    • Hash Function math function returns a non-neg integer called hash code
      • It should use all the data and only the data, not incorporated or add anything extra without reason
      • Be deterministic. same input -> same output
      • Uniformly distribute data
      • Generate very different hash codes for similar data
    • Array capable of storing the data we want to add
  • It runs the data through the hash function, this return a hash code which is used as the index of the array in which we are going to store the data
  • collision occurs when two pieces of data that are put through the same hash function produce the same hash code
  • chaining technique of using linked list instead of simple arrays to store data

Tries

  • They are a specific type of tree
  • Use the data as a roadmap, no two different pieces of data cant have exactly the same roadmap
  • It uses the idea that keys should be unique to avoid collisions
  • to insert we need to build the correct path from the root to the leaf

Data Structures Summary

Data structure Pros Cons
Arrays Lookup is great due to random access instant time Random insertion and deletion is bad because it requires shifting
| Small memory usage and Relatively easy to sort Fixed size, no flexibility or scalability
Linked List Small memory usage and Insertion/deletion is easy Difficult to sort and lookup can be slow
Hash tables Deletion is easy and lookup is relatively fast Hard to sort and insertion can become difficult (bad hash function)
Tries Deletion is easy, lookup is fast, it sorts as you build Insertion is complex and uses a lot of memory in the general case

Web Fundamentals

  • We use/get a public IP from the ISP and then use a router and network address translation to assign unique/private addresses (ranges: 10.#.#.#, 172.16.#.#, 192.168.#.#)
  • Router allows data requests from all of the devices on a local network to be processed through a single IP address
  • Web server is a software which purpose is to serve web pages
  • DHCP protocol used to dynamically assign IP addresses
  • DNS protocol that translates domain names to IP addresses and vice versa, the DNS servers are a decentralized network in which each node holds local names and large
  • The internet is just interconnected networks, connection of multiple smaller networks which create a bigger network, to communicate with one an other
    • Not all the networks are directly connected to each other, we rely on routers to distribute the communications

Internet Simplified

TCP/IP

  • IP deals with getting packets to the right device (IP address) and TCP deals with getting all the information to the correct service/app (port)

  • IP

    • Ensures every device has an IP address handles the getting/sending of data (the passing of information) to some destination, but doesn't care if the full data actually is received correctly
    • Responsible for transmitting from point to point on the internet and splitting data into packets
  • TCP
    • Ensures or guarantees that the data actually gets to the destination, to the correct service, application or program. This means it deals with ordering and merging of packets
    • It is the protocol used to specify what type of web service is being requested for example (SSH = 22, DNS = 53, HTTP = 80, HTTPS = 443, SMTP = 585...) this is what the port is used for.
    • port
      • A port number is how a specific service, or program, is identified on a machine.
      • Identifies the type of service and the specific application that is waiting for response or sending requests

HTTP

  • It is a application layer protocol, set of conventions/rules that dictates how client talks to a server
  • Dictates the format by which clients request webpages from servers and the format servers use to return information to clients
  • Specifies how to make requests to webpages and how the response will be formatted
  • get sends the request parameters directly in the URL publicly, post sends the parameters privately

HTML

  • Markup language used to create webpages. It is not a programming language (no loops, no functions, no conditions, no logic, no variables, etc.)
  • Consists of tags that tell a browser what to do or how to create/render? a web page and attributes which modify the behavior of tags
  • Every html page contains a head and a body which is the actual content of a web page

CSS

  • Language used to create style sheets which are used to customize the looks and style of a website
  • Style sheets are used to cascade properties to all the child tags of an html parent tag
  • Allows to define style (a set of properties) for tags
  • A CSS file of style sheet is constructed with declarations that consist of:
    • selectors: Identifier of the tag that will be modified
    • key-value pairs of properties and value of that property
  • Selectors don't have to apply only to HTML generic tags, you can define:
    • An id selector (#) which will apply only to an HTML tag with an unique ID
    • A class (.) selector which allows you to define a name for a set/group of tags and it will apply only to HTML tags that are given the same class attribute
    • Other CSS selectors CSS Selectors
  • Media query: A specific way of assigning CSS to a particular HTML on a particular type of media (specific window size on a PC, resolution size, for printing)
    • Uses the @media on .css files
  • SASS: Extension to CSS that let us programmatically generate CSS. Uses .scss extension instead of usual .css
    • Allows to define variables using $, nesting and inheritance using templates (something that other things are going to inherit), uses % to define templates
    • Web browsers usually don't understand SASS natively so we need a pre-processor to covert it to CSS

Dynamic programming

  • It is a way of looking to a certain kinds of problems
  • Category of algorithms that store the results of past computations and use those to calculate new values
  • Use of look-up tables which are the ones that store results of the previous calculations

Python

  • Scope is more flexible in python because is tide to sequence but not necessarily to indentation
  • from <lib> import * imports all functions and variables, using only import <lib> requires to prefix function calls with library name
  • Named parameters is supported so to pass parameters to function without relying on the order
  • In Python classes are capitalized and file names (modules) are not by convention
  • A decorator (@) is used to modify the behavior of a function without actually changing the code of the function.
    • It modifies the behavior of the original function by defining a function that takes as input the function we want to modify and returning a new function

Web-server

Flask:

  • Web server software. Its a Web Micro framework
  • You define routes for the web server. This means you tell what to do when a request for a certain source is received
  • digital stamp or sign given by a webpage to a browser
  • the browser stores digital sign and is constantly reminding the server of this through the http header in the http requests its sending to the server
    • cookie: session=value

Session

  • Whatever it is the server is remembering on that users behalf per his cookie
  • Box filled with values related to the user

setup

  1. install git: sudo apt-get install git
  2. install python: sudo apt-get install python && sudo apt-get install python3
  3. install sqlite3: sudo apt-get install sqlite3
  4. install virtualenv:
    • sudo pip3 install virtualenv
    • sudo apt-get install python-virtualenv
  5. change to project directory, create a virtualenv and activate environment:
    • cd ~/my_project
    • virtualenv -v python3 myvenv
    • source myvenv/bin/activate

you can go back to the "real world" by deactivating your virtual environment with the deactivate command

  1. install your project requirements
    • pip install -r requirements.txt

verify pip --version returns that pip3 is being used and make sure to add Flask and Flask-Session to the requirements.txt file

  1. export Flask env variables
    export FLASK_APP=application.py
    export FLASK_DEBUG=1

MVC (Model view Controller)

  • Paradigm (practices) to structure web server software or web projects. Its primary motivation is security to prevent the user to have a direct connection with the model (data)
  • Controller is the programing logic that responds to inputs and updates/changes content (Code in java, python, etc.)
  • View aesthetics, view or user interface the user actually see (html, CSS, jinja, templates, layouts, etc.)
  • Model Refers to the data or where all the important data for your site lives (Database). Also refers to the logic to read/write/store data

MVC

Database and Database Design

  • Database Piece of software or database program stores a hierarchical organized set of tables and listens for connections, gets requests and serves responses
  • In a database it's usually better to separate data into smaller pieces like First name, Last name, Street address, City, Country, etc.
  • Separating data creates the problem of unnecessary redundancy and waste of storage space
  • Relational databases solve the problem of unnecessary redundancy, the idea is to factor out common information and create relation between tables using IDs/keys
    • For this we use foreign key, which is the method used to connect multiple tables together
  • Relational databases is not user friendly for humans is good for machines (computers)
  • All queries to a database refers to the rows of a certain table
  • Transactions: is the method used to solve race condition problems when accessing a database, the idea is to lock the database while running a transaction (sequence of commands)

SQL (Structure Query Language) and SQLite

  • Language to interact with a database (query a database)
  • phpLiteAdmin/phpMyAdmin: GUI tool to visualize/edit a binary file that contains a database
  • Fundamental SQL Data types: BLOB (Binary Large OObject), INTEGER, NULL, REAL, TEXT, DATETIME, NUMERIC
  • SQL uses BEGIN and COMMIT to lock the database and complete transaction
  • Modifiers:
    • PRIMARY KEY: There can only be one primary key per Table and signals that the data in this column will be used as Identifier
    • UNIQUE: make a column unique and make sure we don't have data duplicates
    • INDEX: indicate the database to index the data in this column to make it easily/efficiently searchable
    • NOT NULL: Don't let NULL be inserted in this column, it cant be blank
    • FOREIGN KEY: This is the primary key in another table, use it to link data in different tables
  • Built-in Functions: date, datetime, time
  • Basic Operations:
Operation Canonical Example
CREATE CREATE TABLE <table_name> ('<column_name>' <SQL Type> <SQL MOdifiers>, ... ) CREATE TABLE 'registrants' ('id' INTEGER PRIMRAY KEY, 'name' TEXT, 'dorm' TEXT)
INSERT INSERT INTO "<table_name>" ('<columns>') VALUES (<values>) INSERT INTO "registrants" ("id", "name", "dorm") VALUES (1, 'Pepe', 'GDL')
SELECT SELECT <columns> FROM "<table>" WHERE <condition> ORDER BY <column> SELECT * FROM "registrants"
- SELECT <columns> FROM "<table1>" JOIN "<table2>" ON <condition> SELECT * FROM "Album" JOIN "Artist" ON Album.ArtistId = Artist.ArtistId
UPDATE UPDATE "<table>" SET <column> = <value> WHERE <condition> UPDATE "registrants" SET "name" = 'Pepe Ch' WHERE id = 1
DELETE DELETE FROM "<table>" WHERE <condition> DELETE FROM "registrants" WHERE id = 1
  • sqlite commands
    • sqlite database_file_name.db starts and creates a database with the given name
    • Commands that start with . is SQLite command line program specific not part of SQL
      ~/workspace/ $ sqlite3 database_file_name.db
      SQLite version 3.8.2 2013-12-06 14:53:30
      Enter ".help" for instructions
      Enter SQL statements terminated with a ";"
      sqlite> .help
      ...
      sqlite> CREATE TABLE 'registrants' ('id' INTEGER PRIMRAY KEY, 'name' TEXT, 'dorm' TEXT);
      ...
      sqlite> .schema
      ...
      sqlite> .quit

SQLAlchemy

  • It is a Python library used to connect Python and SQL, it allows python code to run SQL queries

ORM (Object-Relational Mapping)

  • Allows us to tie OOP with databases
  • Uses method and classes to interact with databases
  • For any table in our database we will have a class

JavaScript

  • Event based programming: paradigm in which programs listen and react to events
  • The language, built into the browser (JS), essentially, just itself has this infinite loop that it's constantly listening for events and reacting accordingly as told by a programmer
  • Language that is commonly used in the confines of a browser (on the client side)
  • Recently it has been used on the server side in something called node.js
  • Everything is an object and all variables are JSON (JavaScript Object Notation) objects which are list of key-value pairs
  • Supports anonymous functions also known as lambda functions
  • The document object is a special global variable/object that refers to the web document, it's the object that organizes the entire content of a web page so we can manipulate the page's elements programmatically
    • DOM common properties: innerHTML, nodeName, id, parentNode, childNodes, attributes, style
    • DOM common methods: getElementById(id), getElementsByTagName(tag), appendChild(node), removeChild(node)
  • navigator is another special global variable from the browser that has geo functionality
  • variables can be declared with const (a constant with local scope), let (variable exists only in the scope of innermost curly braces) or var (variable exists in the whole scope of function where it was declared)
  • Arrow Functions (=>) are defined with the list of parameters taken as input and without the use of the word function
  • this refers to the specific object that calls a function but only if it was defined in the traditional way using the keyword function()
    • This means that using arrow notation => will bound this to whatever it was found in the code that was enclosing that arrow function

jQuery

  • It is an open-source library that is designed to "simplify" common usages of JavaScript like DOM manipulation
  • Bootstrap requires jQuery to work
  • jQuery defines a global variable called jQuery which alias is $
  • jQuery Documentation

AJAX

  • AJAX is a feature in browsers that lets them execute HTTP request after the first/primary request is done (the primary request will be visiting the web page)
    • Extra HTTP requests usually return application/json in response instead of html/text like when visiting a page
  • AJAX allows to interact with the server side without having to actually reload the whole page, so it allows us to update the webpage dynamically
  • This means we can asynchronously refresh sections of the page
  • To get asynchronous update functionality we use a special JavaScript object called XMLHttpRequest, this is the general process:
    1. Create XMLHttpRequest object
    2. Define its onreadystatechange or onload behavior, which is just the callback that will be called when the HTTP request has completed
      • (optional) In this callback use the readyState property to check status of the request
    3. Make async requests
      • Use the open() method to define the request
      • Use the send() to send the request
  • AJAX jQuery Documentation

Web Sockets

  • Web sockets allows full duplex real-time communication between client and server
  • For uses cases where the request-response paradigm of a client-server is not enough and real-time communication between client and server is needed like for messaging apps, streaming...
  • socket.io is the JS library to work with sockets
  • The idea behind sockets is that the web client and server are constantly sending/emitting events which are broadcasted and also listening/receiving events
  • The general process of using sockets is the following
    • Server
      • Inits websocket socketio = SocketIO(app)
      • Define routes for events emitted by the client `@socketio.on("event name")`
      • In case is necessary broadcast an event to notify other clients of the received event emit("event name", data, broadcast = True)
    • Client
      • Connects to the websocket var socket = io.connect(location.protocol + '//' + document.domain + ':' + location.port);
      • Defines what happens when the socket connects to the server socket.on('connect', () => {...};);
      • Defines how to emit events using the configured socket, give a name to the event and the data to be emitted socket.emit('event name', data);
      • Defines what to do when receiving an event socket.on('event name', data => {...};);

DevOps

  • Devs or Developers are who create SW, they code functionality, implement features, etc.
  • Ops or Operators are the ones that deploy the SW, they are in charge of knowing how to setup a system to deploy/install and make the SW work
  • DevOps goal is to integrate developers and operators, so both share responsibility of building and deploying a product
  • DevOps:
    • Cultural philosophy
    • Practices: Refers to continuous integration, delivery and deployment, Infrastructure config as code (YAML), monitor and logging
    • Tools to implement devops practices
  • Faster deployments
  • Accelerate hoe SW development works

DevOps AWS Services

  • CodeBuild: Helps to compile SW, run tests and produce software packages. Manage by a text config files. Works with CodePipeline and Jenkins tools
  • CodeDeploy: Automates code deployments to instances, update your application, supports rollback and avoids downtime during deployment
  • CodePipeline: Handles continuous delivery, manages the build (uses CodeBuild), test and deploy (uses CodeDeploy) of code every time there is change
  • CodeStar: Allows to develop, build and deploy SW
  • AWSEducate:

React

  • JavaScript library for building user interfaces
  • JS library that helps develop responsive web pages
  • React JS was developed by Facebook and it rival is Angular JS which was developed by Google
  • JS standalone is mutational because it first fetches an HTML element and then change it, the paradigm of React is abstract the mutational elements
  • into data or variables (React states) and then use a render() function to paint/draw/render the screen like in game development

Install and Setup

  • Install Node and npm
    • Node Allows to run JS in the systems is installed
    • npm Node Package Manager, it is the package manager for Node
  • Install create-react-app which is a tool that help create React projects faster
    • npm install -g create-react-app
  • Create a React project and go to localhost:3000
    ~/ $ create-react-app react_app_name
    ~/ $ cd react_app_name
    ~/react_app_name/ $ npm start

JSX

  • JavaScript Extension (JSX) allows us to easily mix JavaScript and HTML
  • It allows to easily return HTML tags so it can be displayed properly

Component

  • Fundamental building block for React, which have internal state and external prop
    • State: internal information about the component, use this.setState() to change it
    • Props: external quantities handed to the component
  • Contained view of the project which can be changed by altering its state and prop
  • Can be nested inside each other and used across multiple files and projects
  • Every component must have the render() method
    • Called when the state changes, although is called every time the state changes not everything is reloaded on the page
    • The method where we decide what the user should see based on the state and props
    • This edits a Virtual DOM which React uses as input to compute the most efficient way to actually update the actual web page DOM
    • It returns the HTML we want to be rendered

G Suite

  • A platform of all the Google applications
  • Google APIs that let you interact with Google applications
  • Google Apps Script
    • JavaScript platform by which you can interact with the Google APIs
  • Python Google API examples: [Core Python Programming Blog] ()

Tkinter

  • Light-weight GUI, useful for simple GUI building
  • Install: sudo pip3 install tkinter
  • General steps when creating a GUI in Tkinter
    1. Import Tkinter
    2. Create Tk object
    3. Set title and geometry
    4. Create elements and add them to the grid
    5. connect elements through functions
    6. Start window main loop
import tkinter as tk

def on_button_clicked():
    greeting = "Hello " + str(entry1.get())
    display = tk.Text(master = window, height =10, width = 30)
    display.grid(column = 0, row = 3)
    display.insert(tk.END, greeting)

window = tk.Tk()
window.title("My App")
window.geometry("400x400)

# label
label_title = tk.Label(text = "Hello World")
label_title.grid(column = 0, row = 0)

# entry
entry1 = tk.Entry()
entry1.grid(column = 0, row = 1)

# button
button1 = tk.Button(text = "clickme", command = on_button_clicked)
button1.grid(column = 0, row = 2)

# start window
window.mainloop()

Django:

  • Web server software. Its a ** Full Web Server framework**
  • Django 2 is the latest version
  • Django divides all its web application into projects (a project is the whole web application)
  • Django is based on the idea of integrating multiple Python packages to create a website
  • A Django project refers to a entire web application and it consists of one or more Django apps, each app serves a particular purpose in the project
    • Django has some built-in apps

a Python package is collection of python files that are grouped together for some purpose a __init__.py tells you that the folder storing this file is a python module

  • Main Django project files:
    • manage.py:
    • settings.py: has all the settings of the web application like time zone, dependencies, databases
    • urls.py: takes care what routes or URLs the user can visit
    • wsgi.py: used to deploy the Django app in a web server
  • Main Django App files:
    • views.py: this file contains the views the user will see when accessing a certain route
    • models.py: in this file is where the classes that are going to define the types of data that will be stored in the database are declared
    • tests.py: For unit test purposes you can inherit from Django TestCase and run them with the command: python manage.py test

Setup a Django project

  1. Creating a project: django-admin startproject project_name
    • Creates the main python package
  1. Creating a Django app inside a Django project:

    1. Make sure you have the python Django package: python -m pip install django or pip3 install django
    2. Make sure you are inside the Django project folder where the manage.py file is located
    3. Run: python manage.py startapp app_name
    4. Edit views.py file that was created under the app_name folder, this file contains the views the user will see when accessing a certain route. So a common practice is to have a function for each route which must receive a request object as argument/parameter. For example it should look something like this:

      from django.http import HttpResponse
      from django.shortcuts import render
      
      # Create your views here.
      def index(request):
          return HttpResponse("Hello World")
    5. Create urls.py inside the app folder (should be something like this:.../project_name/app_name)

      from django.urls import path
      from . import views
      
      urlpatterns = [
          path("", views.index)
      ]
  1. Link project's urls.py with app's urls.py, this means edit urls.py inside project folder, which is the global routes file. It should look something like this

    from django.conf.urls import url
    from django.contrib import admin
    from django.urls import include, path
    
    urlpatterns = [
        path('', include('app_name.urls')),
        url(r'^admin/', admin.site.urls),
            ]
  1. Run Django app: python manage.py runserver
    • You may need to give a specific IP and port like python manage.py runserver 172.17.0.15:8080 and add hostname to ALLOWED_HOSTS list in settings.py in case you receive some permission errors when accessing the site
Django migrations and models
  • Migrations allows that every time we make a change to the models.py file Django generates SQL needed to update the database tables

  • If you are using the models (models.py) in your Django app you will need to add them in the INSTALLED_APP list in the Django project settings.py and run the python manage.py makemigrations and python manage.py migrate commands

    • For example, assuming we have a flights app the models.py of this app may look something like this:

      # Create your models here.
      class Flight(models.Model):
        origin = models.CharField(max_length = 64)
        destination = models.CharField(max_length = 64)
        duration = models.IntegerField()
      
        def __str__ (self):
            return f"{self.id} - {self.origin} to {self.destination}"
    • And for this same example the settings.py may have a line like this: INSTALLED_APPS = ['flights.apps.FlightsConfig', 'django.contrib.admin', 'django.contrib.auth', ... ]

    • The python manage.py makemigrations command will take the changes in the models file and generate a migration (a 000X_initial.py file) which represents the changes that need to be made to the database so the changes in the model work

      ~/ $ python manage.py makemigrations
      Migrations for 'flights':
      flights/migrations/0001_initial.py
        - Create model Flight

      python manage.py sqlmigrate flights 0001 will let you examine the SQL code (usually create table commands) that is needed to store the model in the database

    • The python manage.py migrate command will apply a migration to the database, actually creates the tables inside the database that is defined in the DATABASES variable in the settings.py file

    • Python Django shell example run with python manage.py shell:

      Python 3.6.0 (default, Apr  9 2018, 15:01:03)
      [GCC 4.8.4] on linux
      Type "help", "copyright", "credits" or "license" for more information.
      (InteractiveConsole)
      >>> from flights.models import Flight
      >>> f = Flight(origin="NYC", destination="LDN", duration=415)
      >>> f.save()
      >>> Flight.objects.all()
      <QuerySet [<Flight: 1 - NYC to LDN>]>
      >>> f = Flight.objects.first()
      >>> f
      <Flight: 1 - NYC to LDN>
      >>> f.delete()
      (1, {'flights.Flight': 1})

Django Render Template

  1. Create a file to render, usually the html files are organized like this in a Django App: .../project_name/app_name/templates/app_name/index.html
  1. Modify views.py to return/render an html file and use id needed.

    • views.py example

      from django.http import HttpResponse
      from django.shortcuts import render
      
      # Create your views here.
      def index(request):
          context = {
              "stuff": ["Hello", "World"] # Flight.objects.all()
          }
          return render(request, "app_name/index.html", context)
      
    • index.html example
      <!DOCTYPE HTML>
      <html lang="en-US">
          <head>
              <title>Index</title>
          </head>
          <body>
              <h1>My Page title</h1>
              <ul>
                  {% for s in stuff %}
                      <li>
                          {{ s }}
                      </li>
                  {% endfor %}
              </ul>
          </body>
      </html>

Django admin app

  • It's a Django built-in app that allows us to create, add, modify and remove existing data
Django admin app Setup
  1. Register your models in your app admin.py file so the Admin site can manipulate these types of data. For example it may look something like this:

    from django.contrib import admin
    from .models import Airport, Flight
    
    # Register your models here.
    admin.site.register(Airport)
    admin.site.register(Flight)
  2. Create a user account with command: python manage.py createsuperuser, you will need to provide: username, email and password (at least 8 characters)

    • This creates an entry in a user table stored in the database that is managed automatically by Django admin app
    • You can access the admin app by going to the app_name/admin route to manipulate your database

Testing

  • pull-request let you start a conversation about a change that you intend to be included in a certain branch (usually master)
  • Functional Testing tests a specific functionality, test that certain function produces the expected outputs (meets the requirements it's supposed to)
    • Unit tests test single functionality of a component
    • Components test test component as an entity
    • Integration tests test the components interacting and integrated together are working correctly
    • end-to-end tests test entire flow or how user uses the application
  • Non-Functional Testing test reliability, security, operational functionality

Selenium

  • Selenium is a browser testing tool that allows to test things that are happening inside the web browser, it uses a webdriver that allows us to use Python Code to control what's happening in a browser, in other words pretend to be the user manipulating the page. Install: pip install selenium

CI/CD (Continuous Integration / Continuous Delivery)

  • Continuous Integration: Refers to consistently and continuously integrating code and automatically testing it
    • Request merging of code between different development efforts (different people and/or branches)
    • This aspect can be solved using a version control tool like git
    • Automated unit testing - This aspect can be solved using a unit test framework for example unittest in Python and an automation tool like Travis CI or Jenkins
  • Continuous Delivery: Refers to continuously updating code of an application, making continuous deliveries to the ultimate application a user or client use
    • Continuous deployment of code - This aspect can be solved using
  • CI-CD flow:
    1. Push code to repo. E.g. push to repo in GitHub
    2. Host of repo notifies CI tool. E.g. GitHub notifies Travis
    3. CI tool pull code from repo. E.g Travis pull new code from GitHub
    4. CI tool runs tests. E.g Travis runs defined unit test
    5. CD tool notifies or triggers deployment of application in other tool. E.g. Travis sends emails or triggers deployment using AWS, Azure or other

Travis CI

  • Continuous integration tool that uses files in YAML format for configuration
  • It is GitHub-based so GitHub repos can talk to Travis directly

  • The basic config is usually a .travis.yaml file inside the repo with contents:

    language: python
    python:
        - 3.6
    install:
        - pup install -r requirements.txt
    script:
        - python manage.py test

Containers

  • A container is an isolated environment that contains all the needed dependencies to run an application

  • They are not complete operating systems, they are isolated environments (containers) that have just the things we want installed, so we define the image which is what should go into a particular container and have that same image in other systems

Docker

  • Docker is a tool to create an manage containers
  • A Docker image contains the instructions to create a Docker container where the application will live
  • Docker creates a link between the local folder where you have your application and the Docker container where you actually run you application. In other words it copies files from the local folder to the container
  • Deployment tools can use Docker images and configuration files to setup the right environment to run an application
  • Docker commands:
    • docker-compose command to start the Docker container and run the application there. For this you will need Dockerfile and docker-compose.yml files
    • docker ps check the containers running usually there is a container for each service
    • docker -it <docker_container_id> -l start a shell on the container

Docker

  • Dockerfile is the file that defines a Docker image. For example:
    FROM python:3
    WORKDIR /usr/src/app
    ADD requirements.txt /usr/src/app
    RUN pip install -r requirements.txt
    ADD . /usr/src/app
  • docker-compose.yaml is a file that defines all the different services that make up the application that will we run in the Docker container. For example a database

    version: '3'
    
    services:
        db:
            image: postgres
        migration:
            build: .
            command: python3 manage.py migrate
            volumes:
                - .:/usr/src/app
            depends_on:
                - db
        web:
            build: .
            command: python3 manage.py runserver 0.0.0.0:8000
            volumes:
                - .:/usr/src/app
            ports:
                - "8000:8000"
            depends_on:
                - db
                - migration

Scalability

  • Vertical scaling: Refers to the process of scaling or increasing the resources of a server, so adding processing power, memory and storage are examples of vertical scaling
  • Horizontal scaling: Refers to the process of adding more servers that work together handling requests
  • Load Balancer: Refers to the system that decides how the work loads and requests are distributed between servers
  • Session-Aware Load Balancing: Refers to to the idea of making sure the load balancer is session-aware, knows about users session to send same user requests to the same servers, this can be implemented with sticky sessions, sessions in database, client-side session, etc.
  • Auto-scaling: Refers to the idea that the number of servers can scale depending upon traffic coming in
  • Database Partitioning: Refers to the idea of partitioning data tables to operate on more manageable units, have queries run on tables more efficiently and quickly
    • Vertical Database Partitioning : refers to separating data into multiple tables to reduce number of columns in each table
    • Horizontal Database Partitioning: refers to separating data into multiple tables to reduce the number of rows in a table
  • Database Sharding Refers to the idea of splitting a database among different servers with the possibility of gaining concurrent access
  • Database Replication Refer to having multiple copies of the database in different servers to distribute load between the databases
    • Single-Primary Replication has a unique primary database to write/update data to it and other secondary databases to only read data from it and data on the secondary databases gets updated every time the primary database changes
    • Multi-primary Replication has multiple primary databases
  • Caching refers to the idea of storing data in some temporary place where is easy and fast to access it at a later time
    • Client-Side Caching: http response can send a cache-control header to indicate to a browser that some client-side files wont cache for a certain amount of time so it can be cached (stored) in local memory for that time
    • Server-Side Caching: Connect all servers to the cache that can cache information from database

Security

  • Open source:
    • + More eyes in code to find and fix bugs
    • - Code can be seen by anybody so people can exploit vulnerabilities they have learned from the code without reporting them
  • Version control:

    • - Pushing some access token or password to repo makes it permanent since you can access the history
  • HTML:

    • - Most HTML code of a page is shared so it can be used to impersonate a site (trick user into believe you represent another site)
  • HTTP

    • - intermediary routing devices have access to the full contents of the requests and reponses between two devices so it can alter the information or use it for it's benefit
  • HTTPS

    • + Http request and response are encrypted using public key cryptography
  • WebServer and Webserver Framework (E.g. Flask, Django)
    • + use of environment variables for password, secret keys or tokens
    • - leaking information indirectly about whether user exists for example a password reset where you enter email and return error if user doesn't exists
    • -** _Cross-Site Request Forgery (CSRF)**_ Trying to forge a request to some other website (Usually a website where the user is already log in). For example a user already logged in into his bank website clicks a button in another website that makes a post request (knowing the format of the bank's post request) to the bank to attempt to transfer some money
    • -** _DDoS (Distributed Denial Of Service)**_ Attacks
  • Database
    • - storing plaintext user information
    • + store encrypted user information (hash password)
    • - knowing the hash function and database can make passwords vulnerable because someone can try random passwords and compare them with the ones stored in the database
    • -** _SQL Injection**_ (Run arbitrary queries on database) Injecting SQL logic
  • APIs

    • - Expose information, need to implement some access control mechanism (authentication -> API keys)
  • JavaScript

    • - Malicious code running on the web browser
    • -** _Cross Site Scripting**_ make some arbitrary code to run on the users web browser to steal cookies, redirect to other page, display other content in a page, etc. for example by sending a link to a page that when rendered will have injected random JS that will run in your browser
    • +

References

Want to show support?

If you find the information in this page useful and want to show your support, you can make a donation

Use PayPal

This will help me to create more stuff and fix the existent content... or probably your money will be used to buy beer