JosephFerano/Notes-TheAlgorithmDesignManual
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Notes-TheAlgorithmDesignManual/TheAlgorithmDesignManual.org

627 lines
17 KiB
Org Mode
Raw Blame History

#+TITLE: Notes & Exercises: The Algorith Design Manual
#+AUTHOR: Joseph Ferano
#+STARTUP: overview
#+OPTIONS: ^:{}
* Chapter 1
** 1.1 Robots
An /algorithm/ is a procedure that takes any of the possible input instances
and transforms it to the desired output.
The Robot arm problem is presented where it is trying to solder contact points
and visit all points in the shortest path possible.
The first algorithm considered is ~NearestNeighbor~. However this is a naïve, and
the arm hopscotches around
Next we consider ~ClosestPair~, but that too misses in certain instances.
Next is ~OptimalTSP~, which will always give the correct result because it
enumerates all possible combinations and return the one with the shortest
path. For 20 points however, the algorithm grows at a right of O(n!). TSP stands
for Traveling Salesman Problem.
*** TODO Implement NearestNeighbor
*** TODO Implement ClosestPair
*** TODO Implement OptimalTSP for N < 8
** 1.2 Right Jobs
Here we are introduced to the Movie Scheduling Problem where we try to pick the
largest amount of mutually non-overlapping movies an actor can pick to maximize
their time. Algorithms considered are ~EarliestJobFirst~ to start as soon as
possible, and then ~ShortestJobFirst~ to be done with it the quickest, but both
fail to find optimal solutions.
~ExhaustiveScheduling~ grows at a rate of O(2^{n}) which is much better than O(n!) as
in the previous problem. Finally ~OptimalScheduling~ improves efficiency by first
removing candidates that are overlapping such that it doesn't even compare them.
** 1.3 Correctness
It's important to be clear about the steps in pseudocode when designing
algorithms on paper. There are important things to consider about algorithm
correctness;
- Verifiability
- Simplicity
- Think small
- Think exhaustively
- Hunt for the weakness
- Go for a tie
- Seek extremes
Other tecniques include *Induction*, *Recursion*, and *Summations*.
** 1.4 Modeling
Most algorithms are designed to work on rigorously defined abstract
structures. These fundamental structures include;
- Permutations
- Subsets
- Trees
- Graphs
- Points
- Polygons
- Strings
** 1.5-1.6 War Story about Psychics
* Chapter 2
** 2.1 RAM Model of Computation
This is a simpler kind of Big Oh where
- Each simple operation is 1 step
- Loops and Subroutines are composition of simple operations
- Each memory access is one time step
Like flat earth theory, in practice we use it when engineering certain
structures because we don't take into account the curvature of the Earth.
We can already apply the concept of worst, average, and best case to this model.
** 2.2 Big Oh
The previous model often requires concrete implementations to actually measure
correctly, so instead Big Oh gives us a better, simpler framework for discussing
the relative performance between algorithms. It ignores factors that don't
impact how algorithms scale.
** 2.3 Growth Rates and Dominance Relations
These are the functions that occur in algorithm analyses;
- *Constant O(1)*
Hashtable look up, array look up, consing a list
- *Logarithmic O(log n)*
Binary Search
- *Linear O(n)*
Iterating over a list
- *Superlinear O(n log n)*
Quicksort and Mergesort
- *Quadratic* O(n^{2})
Insertion Sort and Selection Sort
- *Cubic* O(n^{3})
Some dynamic programming problems
- *Exponential* O(C^{n}^{}) *c for any constant c > 1*
Enumerate all subsets
- *Factorial O(n!)*
Generating all permutations or orderings
*Notes*:
- O(n!) algorithms become useless for anything n >= 20
- O(2^{n}) algorithms become impractical for anything n > 40
- O(n^{2}^{}) algorithms start deteriorating after n > 10,000, a million is hopeless
- O(n^{2}^{}) and O(n log n) Are fine up to 1 billion
** 2.4 Working with Big Oh
Apparently you can do arithmetic on the Big Oh functions
** 2.5 Efficiency
*** Selection Sort
**** C
#+begin_src C :includes stdio.h
void print_nums(int *nums, int length) {
for (int i = 0; i < length; i++) {
printf("%d,", nums[i]);
}
printf("\n");
}
void selection_sort(int *nums, int length) {
int i, j;
int min_idx;
for (i = 0; i < length; i++) {
print_nums(nums, length);
min_idx = i;
for (j = i+1; j < length; j++) {
if (nums[j] < nums[min_idx]) {
min_idx = j;
}
}
int temp = nums[min_idx];
nums[min_idx] = nums[i];
nums[i] = temp;
}
}
int nums[9] = { 2, 4, 9, 1, 3, 8, 5, 7, 6 };
selection_sort(nums, 9);
#+end_src
#+RESULTS:
| 2 | 4 | 9 | 1 | 3 | 8 | 5 | 7 | 6 | |
| 1 | 4 | 9 | 2 | 3 | 8 | 5 | 7 | 6 | |
| 1 | 2 | 9 | 4 | 3 | 8 | 5 | 7 | 6 | |
| 1 | 2 | 3 | 4 | 9 | 8 | 5 | 7 | 6 | |
| 1 | 2 | 3 | 4 | 9 | 8 | 5 | 7 | 6 | |
| 1 | 2 | 3 | 4 | 5 | 8 | 9 | 7 | 6 | |
| 1 | 2 | 3 | 4 | 5 | 6 | 9 | 7 | 8 | |
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 9 | 8 | |
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | |
*** Insertion Sort
**** C
#+begin_src C :includes stdio.h
void insertion_sort(int *nums, int len) {
int i, j;
for (i = 1; i < len; i++) {
j = i;
while (nums[j] < nums[j -1] && j > 0) {
int temp = nums[j];
nums[j] = nums[j - 1];
nums[j - 1] = temp;
j--;
}
}
}
int nums[8] = {1,4,5,2,8,3,7,9};
insertion_sort(nums, 8);
for (int i = 0; i < 8; i++) {
printf("%d", nums[i]);
}
#+end_src
#+RESULTS:
: 12345789
*** TODO String Pattern Matching
*** TODO Matrix Multiplication
** 2.6 Logarithms
Logarithms are the inverse of exponents. Binary search is great for sorted
lists. There are applications related to fast exponentiation, binary trees,
harmonic numbers, and criminal sentencing.
** 2.7 Properties of Logarithms
Common bases for logarithms include 2, /e/, and 10. The base of the logarithm has
no real impact on the growth rate; log_{2} and log_{3} are roughly equivalent.
** 2.8 War Story Pyramids
Cool story bro
** 2.9 Advanced Analysis
Some advanced stuff
- *Inverse Ackerman's Function*
Union-Find data structure
- *log log n*
Binary search on a sorted array of only log n items
- *log n / log log n*
- log^{2} n
- \sqrt{,}n
There are also limits and dominance relations
* Chapter 3
** 3.1 Contiguous vs Linked Data Structures
Advantages of Arrays
- Constant-time access given the index
- Space efficiency
- Memory locality
Downsides is that they don't grow but dynamic arrays fix this by allocating a
new bigger array when needed.
Advantages of Linked Structures
- No overflow, can keep growing
- Insertions/Deletions are simpler
- A collection of pointers are lighter than contiguous data
However, pointers require extra space for storing pointer fields
** 3.2 Stacks and Queues
*** Stacks
/(PUSH, /POP/) LIFO, useful in executing recursive algorithms.
*** Queues
(/ENQUEUE/, /DEQUEUE/) FIFO, useful for breadth-first searches in graphs.
** 3.3 Dictionaries
Not just hashtables but anything that can provide access to data by
content. Some dictionaries implement trees instead of hashing. Both contiguous
and linked structures can be used with tradeoffs between them.
** 3.4 Binary Search Trees
BSTs have a parent and two child nodes; left and right. They support insertion,
deletion, traversal. Interestingly, Min and Max can be calculated by seeking the
leftmost and rightmost node respectively, provided the tree is balanced. BSTs
can have good performance for most cases so long as they remain balanced. O(h)
refers to the time being the height of the BST.
** 3.5 Priority Queues
They allow new elements to enter a system at arbitrary intervals.
** 3.6 War Story
Rather than storing all of the vertices of a mesh, you can share them between
the different triangles, but connecting all vertices requires visiting each
vertice once, a Hamiltonian path, but that's NP-Complete. Using a greedy
heuristic where it tries to always grab the best possible thing first. Then
using a priority queue, they were able to reduce the running time by several
orders of magnitude compared to the naïve approach.
** 3.7 Hasing and Strings
Take a map to a big int, use modulo to spin around, and if /m/ is a large prime
you'll get fairly uniform distribution. The two main ways to solve collisions
are /Chaining/ and /Open Addressing/. Chaining is where each bucket has a linked
list and collisions are appended. Open addressing looks for adjacent empty
buckets.
Hashing is also useful when dealing strings, in particular, substring pattern
matching. Overlaying pattern /p/ over every position in text /t/ would result in
O(m*n). With hashing, you can hash the slices of /t/ and compare them to /p/, and
get slower growth. This is called the *Rabin-Karp algorithm*. While
false-positives may occur, a good hashing function would avoid this.
Hashing is so important Yahoo! employs them extensively.
** 3.8 Specialized Data Structures
These include;
- String
Characters in an array
- Geometric
Collection of points and regions/polygons
- Graph
Using adjacency matrices
- Set
Dicionaries and bit vectors
** 3.9 War Story
They were trying to implement sequencing by hybridization (SBH), but ran into
issues when they used a BST. Then they tried a hashtable, then a trie. Finally
what worked was a compressed suffix tree.
** Exercises
*** 3.42
Reverse the words in a sentence—that is, “My name is Chris” becomes “Chris is
name My.” Optimize for time and space.
#+begin_src C :includes stdlib.h stdio.h string.h
void reverse_word(char *string, int length) {
for (int i = 0; i < length / 2; i++) {
char temp = string[i];
string[i] = string[length - 1 - i];
string[length - 1 - i] = temp;
}
}
void reverse_words(char *string, int length) {
printf("Before: %s\n", string);
reverse_word(string, length);
printf("After: %s\n", string);
int start = 0;
for (int i = 0; i < length; i++) {
if (string[i] == ' ' || i == length - 1) {
if (i == length - 1) i++;
reverse_word(&string[start], i - start);
start = i + 1;
}
}
}
char str[] = "My name is Chris";
reverse_words(str, strlen(str));
printf("Final: %s\n", str);
#+end_src
#+RESULTS:
| Before: | My | name | is | Chris |
| After: | sirhC | si | eman | yM |
| Final: | Chris | is | name | My |
* Chapter 4
** 4.1 Applications of Sorting
Apparently sorting is a big deal. There are a lot of problems that can be solved by using a sorted
list, for example; closest pair, searching, frequency distribution, convex hulls, etc;
The fastest sorting algorithms are ~n log n~, here is how it scales compared to an algorithm with
quadratic growth. Even crazier is that this table divides it by 4 so it's not that ridiculous.
| n | n^{2}/4 | n lg n |
|---------+---------------+-----------|
| 10 | 25 | 33 |
| 100 | 2,500 | 664 |
| 1,000 | 250,000 | 9,965 |
| 10,000 | 25,000,000 | 132,877 |
| 100,000 | 2,500,000,000 | 1,660,960 |
Whenever you have an algorithmic problem, don't be afraid to use sorting, because if it results in
the ability to then do a linear scan, at worst is becomes ~2n log n~, which in the end is just ~n log n~
** 4.2 Pragmatics of Sorting
There are some important things to consider when sorting; the order, keys and their data, equality,
and non-numerical data. For resolving these, we would use /comparison functions/. Here is the
signature of the C function for quicksort
#+begin_src C :include stdlib.h
void qsort(void *base, size_t nitems, size_t size, int (*compar)(const void *, const void*))
#+end_src
It takes a comparison function that might look like this;
#+begin_src C :include stdlib.h
int compare(int *i, int *j) {
if (*i > *j) return 1;
if (*i < *j) return -1;
return 0;
}
#+end_src
** 4.3 Heapsort
An O(n^{2}) sorting algorithm like Selection Sort can be made faster by using the right data structure,
in this case either a heap or a balanced binary tree. The first initial construction will take
one O(n), but subsequent operations within the loop will now take O(log n) rather than O(n), giving
a final complexity of O(n log n).
*** Heaps
They can either be min or max heaps and the root node will dominate its children in min-ness or
max-ness. It's also different in that it can be implemented in an array and still be reasonably
conservative with it's space complexity. However it should be noted that searching isn't efficient
because the nodes aren't guaranteed to be ordered, only the relationship between the parent/child.
Here's the full heap implementation;
#+begin_src C :includes stdio.h stdlib.h
#define PQ_SIZE 256
#define item_type int
struct priority_queue {
item_type *q;
int len;
};
struct priority_queue *pq_create() {
item_type *q = calloc(PQ_SIZE, sizeof(item_type));
struct priority_queue *pq = malloc(sizeof(struct priority_queue));
pq->q = q;
pq->len = 0;
}
int pq_parent(int n) {
if (n == 1) return -1;
else return ((int) n/2);
}
int pq_young_child(int n) {
return 2 * n;
}
// I mean technically we shouldn't need to provide the parent, since we can
// just call that ourselves
void pq_swap(struct priority_queue *pq, int n, int parent) {
item_type item = pq->q[n];
pq->q[n] = pq->q[parent];
pq->q[parent] = item;
}
void pq_bubble_up(struct priority_queue *pq, int n) {
int pidx = pq_parent(n);
if (pidx == -1) {
return;
}
item_type parent = pq->q[pidx];
item_type node = pq->q[n];
if (parent > node) {
pq_swap(pq, n, pidx);
pq_bubble_up(pq, pidx);
}
}
void pq_bubble_down(struct priority_queue *pq, int n) {
int cidx = pq_young_child(n);
if (cidx > pq->len) {
return;
}
item_type child = pq->q[cidx];
item_type node = pq->q[n];
int min_idx = n;
if (cidx <= pq->len && node > child) {
min_idx = cidx;
}
if (cidx + 1 <= pq->len && pq->q[min_idx] > pq->q[cidx + 1]) {
min_idx = cidx + 1;
}
if (node > child) {
pq_swap(pq, n, min_idx);
pq_bubble_down(pq, min_idx);
}
}
void pq_insert(struct priority_queue *pq, item_type x) {
if (pq->len >= PQ_SIZE) {
printf("Error: Priority Queue Overflow");
return;
}
pq->q[++pq->len] = x;
pq_bubble_up(pq, pq->len);
}
item_type pq_pop_top(struct priority_queue *pq) {
if (pq->len == 0) {
printf("Error: No elements in Priority Qeueu");
return -1;
}
item_type top = pq->q[1];
pq->q[1] = pq->q[pq->len--];
pq_bubble_down(pq, 1);
return top;
}
struct priority_queue* pq = pq_create();
#define ELEMENTS 8
item_type arr[ELEMENTS];
for (int i = 0; i < ELEMENTS; i++) {
arr[i] = i + 1;
}
void reverse(item_type *arr) {
for (int i = 0; i < ELEMENTS / 2; i++) {
item_type temp = arr[i];
arr[i] = arr[ELEMENTS - i - 1];
arr[ELEMENTS - i - 1] = temp;
}
}
void shuffle(item_type *arr) {
for (int i = 0; i < ELEMENTS - 1; i++) {
float r = (float)rand() / RAND_MAX;
int idx = (int)(ELEMENTS * r);
item_type temp = arr[idx];
arr[idx] = arr[i];
arr[i] = temp;
}
}
void print_elems(item_type *arr) {
for (int i = 0; i < ELEMENTS; i++) {
printf("%d,", arr[i]);
if (i == ELEMENTS - 1) {
printf("\n");
}
}
}
void print_pq(struct priority_queue *pq) {
for (int i = 1; i <= pq->len; i++) {
printf("%d,", pq->q[i]);
if (i == pq->len) {
printf("\n");
}
}
}
reverse(arr);
for (int i = 0; i < ELEMENTS; i++) {
pq_insert(pq, arr[i]);
}
print_pq(pq);
pq_pop_top(pq);
print_pq(pq);
#+end_src
** 4.4 War Story
Apparently calculating airline tickets is hard
** 4.5 Mergesort
#+begin_src C :includes stdio.h stdlib.h
int *merge_sort(int *array, int start, int len) {
int *sorted = malloc(sizeof(int) * len);
if (len <= 1) {
sorted[0] = array[start];
return sorted;
}
int half = (len + 1) / 2;
int *sorted_l = merge_sort(array, start, half);
int *sorted_r = merge_sort(array, start + half, len / 2);
int size_r = len / 2;
int size_l = half;
int i = 0, ir = 0, il = 0;
for (; i < len; i++) {
if ((il >= size_l && ir < size_r) || (ir < size_r && sorted_l[il] > sorted_r[ir])) {
sorted[i] = sorted_r[ir++];
} else if (il < size_l) {
sorted[i] = sorted_l[il++];
}
}
free(sorted_l);
free(sorted_r);
return sorted;
}
#define AL 10
int array[AL] = { 'y','z','x','n','k','m','d','a','b','c' };
/* int array[AL] = { 9,8,6,7,4,5,2,3,1,0 }; */
/* int array[AL] = { 0,1,2,3,4,5,6,7,8,9 }; */
for (int i = 0; i < AL; i++) {
printf("%c-", array[i]);
} printf("\n");
int *sorted = merge_sort(array, 0, AL);
for (int i = 0; i < AL; i++) {
printf("%c-", sorted[i]);
}
#+end_src
#+RESULTS:
| y-z-x-n-k-m-d-a-b-c- |
| a-b-c-d-k-m-n-x-y-z- |
** 4.6 Quicksort
** 4.7 Distribution Sort: Bucketing
** 4.8 War Story
** 4.9 Binary Search and Related Algorithms
** 4.10 Divide-and-Conquer
Reference in New Issue View Git Blame Copy Permalink