CS 201: Running Time of Programs

Running time of programs I.

Summary.

• Introduction to the idea of running time and what we can count as constant time operations.
• We analyzed the insertion sort pseudocode from the Wikipedia article Insertion sort.

Youtube insertion sort

The pseudocode for insertion sort from the Wikipedia article:

for i ← 1 to length(A)
    x ← A[i]
    j ← i
    while j > 0 and A[j-1] > x
        A[j] ← A[j-1]
        j ← j - 1
    A[j] ← x

This is (slightly) incorrect: the array A has 0-based indexing, so when i = length(A), the array reference A[i] will be out of bounds; imagine the first line corrected to have length(A)-1 instead of length(A). We analyzed this program by thinking of it translated into TC-201 assembly language and estimating the number of TC-201 assembly language instructions that would be executed to sort an array of n numbers.

The new element here is how to account for the time to access the array A. In this pseudocode, the array A is a finite sequence of mutable elements (like a vector in Racket) each of which is accessed by its index, a number from 0 to the length of the array minus 1.

Here are examples of Racket vectors.

(define v (make-vector 5 1))

v

'#(1 1 1 1 1)

(vector? v)

#t

(length v)

length: contract violation
  expected: list?
  given: '#(1 1 1 1 1)

(vector-length v)

5

(vector-ref v 0)

1

(vector-set! v 0 8)

(vector-ref v 0)

8

The vector-set! bang (!) suffix indicates that the function has a side-effect. It can alter the contents of the vector.

The important point here is that vector-length and vector-ref are constant time operations, $O(1)$, unlike length and list-ref for lists which are $O(n)$. The list operations must traverse the list from left to right. The vector operations know exactly where to go, based on the index.

TC-201 Sort

If we consider a TC-201 implementation of an array A of numbers we could allocate a number of contiguous memory registers equal to the length of the array, and store the numbers in order in those memory locations. In addition, we would store the address of the start of the array in another memory location, call it astart. Then the TC-201 instructions to find the value of $A[i]$ would load astart into the accumulator, add the value of i to it, and store it in a temporary location, say temp, and then execute loadi temp to get the value of $A[i]$ into the accumulator. This is a constant number of TC-201 instructions to access any of the elements of the array. To change the value of $A[i]$, we could use a similar address calculation to get the memory address of the i-th element of $A$ into temp, and then execute load value and storei temp to change the value of $A[i]$ to the number in the memory register value. This is similarly a constant number of instructions. Thus it makes sense to count array references as constant time operations.

With that understanding of array references, a straightforward translation of a for-loop into TC-201 instructions, and the observation that assignment, addition, subtraction and comparison are constant time operations, our conclusions about the running time of this implementation of insertion sort were: best case time of Theta(n) (Θ(n)) and worst case time of Theta(n²) (Θ(n²)).

See tc201sort.rkt. We also modified simulate-lite to return how many steps were executed.

Please see the notes: Running time of a program for a simpler example.

Running time of programs II.

Summary.

• Definitions and examples for asymptotic notation: big-Oh, Big Omega, Big Theta.
• An (insert item lst) procedure in Racket.
• Analyzing running time of Racket programs: what are the constant time operations?
• A representation of lists in TC-201 memory, and analysis to show that car (first), cdr (rest), and cons are constant time operations.
• Box and pointer representation of lists.

Please see the notes: Running time of a program and List representation. For the insert procedure, see Sorting.

Closures

See Lexical Scope and Function Closures

Closures are a way of providing a local environment for a function definition that is distinct from the global environment. Here is an example - first with a global scope and then with a local, lexical scope.

(define (make-global-adder)
    (lambda (y) (+ x y)))

(define x 10)

(define addx (make-global-adder))

x

10

(addx 3)

13

(define x 20)

(addx 3)

23

The function addx depends on the current value of x, rather than the value of x at the time the function was defined. Here is another (better) way.

(define (make-adder x)
    (lambda (y) (+ x y)))

(define add2 (make-adder 2))
(define add3 (make-adder 3))

(add2 8)

10

(add3 8)

11

In make-adder the variable x is local and in lexical scope for the lambda expression. When the lambda expression executes, it uses the value of x that we in effect when make-adder was called.

make-adder creates a closure that combines a local environment, defining x, with a function definition. We will see that closures provide a way to implement object oriented programming, where an object contains both data and methods (procedures).

Running time of programs III.

Summary. Running times of insertion sort and merge sort.

Please see the notes: Sorting.

Above, we used a modified version of simulate-lite to measure how many steps were executed by our TC201 program.

Racket has procedures to measure the execution time of your code. First, we define three procedures to sum up the first n integers. The first is the vanilla recursive definition.

(define (sum1 n)
  (if (zero? n)
      0
      (+ n (sum1 (- n 1)))))

The second is the tail-recursive version.

(define (sum2 n [result 0])
  (if (zero? n)
      result
      (sum2 (- n 1) (+ n result))))

The third is Gauss's closed form definition.

(define (sum3 n)
  (/ (* n (+ n 1))
     2))

We use the Racket procedure time-apply to measure the execution time.

(time-apply sum1 '(1000000))

'(500000500000)

159

160

73

time-apply returns four values: a list containing the result(s) of applying proc to the arguments in lst, the number of milliseconds of CPU time required to obtain this result, the number of “real” milliseconds required for the result, and the number of milliseconds of CPU time (included in the first result) spent on garbage collection.

(time-apply sum2 '(1000000))

'(500000500000)

6

6

0

(time-apply sum3 '(1000000))

'(500000500000)

0

0

0

Tail-recursion is considerably faster than plain recursion, and require no garbage collection. That means that it does not run out of free memory.

The closed form solution is virtually instantaneous. We need a more fine grained measure of time. We will use Racket's current-inexact-milliseconds function.

(define (timeit func args)
  (let ((start (current-inexact-milliseconds))
        (val (apply func args))
        (end (current-inexact-milliseconds)))
    (list val (- end start))))

(timeit sum1 '(1000000))

'(500000500000 150.05908203125)

(timeit sum2 '(1000000))

'(500000500000 6.009033203125)

(timeit sum3 '(1000000))

'(500000500000 0.0)

We will revisit these methods below in the section on plotting the results.

Mutators (and a little bit about objects).

Summary.

Representing an "object" as a procedure with local data.

The Racket mutator set! and how to use it to implement a counter object.

This topic is useful for solving problem 7 in hw 8.

Please see the notes: Environments in Racket and the Mutator set!.

counter.rkt

In a setting of pure functional programming, calling a procedure with the same arguments will always produce the same result. This is not the case with Racket's library procedure random. Calling (random 10) might return 6 one time and 0 the next. The random procedure has some local state that changes with each call and allows it to produce different responses to the same argument. An "object" in the sense of object oriented programming can be considered to be a bundle of data and procedures (or "methods") to operate on the data. We can represent an object in Racket as a procedure with local state.

Before we do that, we look at one of Racket's mutators, namely set! - pronounced set-bang. The exclamation point is part of the name of the procedure, and is a convention to indicate that the procedure is a mutator, that is, changes the value of a variable or other data. The form of set! is

    (set! variable expression)

The expression is evaluated to get a value, and then the variable is looked up in the relevant environment (just as we would look it up to find its value) -- in that environment its binding is changed to the value of the expression. Note that the variable must already be defined in order to use it in a set! expression. Attempting to set! a variable that is not defined is an error. This is analogous to having to declare a variable before it can have a value assigned to it.

Example of the use of set! in the top-level environment.

(define count 0)

count

0

(set! count (+ 1 count))

count

1

Evaluating the expression (define count 0) adds count to the top-level environment with its value bound to 0. Then evaluating the expression (set! count (+ 1 count)) evaluates the expression (+ 1 count) in the usual way to get $1$, and looks up count, finding it in the top-level environment, where its value is currently $0$. The value of count is changed to $1$ in the top-level environment; now when its value is looked up, it will be $1$.

This behavior is enough to define a simple counter procedure which will return different values for the same arguments, depending on the value of count. We write the following procedure.

(define (counter cmd)
    (case cmd
      [(increment!) (set! count (+ 1 count))]
      [(zero!) (set! count 0)])
    count)

(counter 'increment!)

2

(counter 'increment!)

3

(counter 'increment!)

4

count

4

(counter 'zero!)

0

count

0

Note that when this procedure refers to count, the search to find the relevant environment finds it in the top-level environment, (where we assume we previously defined it and incremented it to 1). Thus in this case the variable count functions as a "global" variable, accessible everywhere in the file. (We could also have written counter without the case statement, using a cond expression and equal? tests.)

One property we might want for an object is that its data is not global, but local and private to the object, so that it cannot be read or changed without invoking the "methods" (or procedures) associated with the object. We can achieve this goal with the following definition. Assume that we have re-entered the Racket interpreter afresh, so that the preceding definition of count and counter are gone.

(define counter1
    (let ((count 0))
      (lambda (cmd)
        (case cmd
          [(increment!) (set! count (+ 1 count))]
          [(zero!) (set! count 0)])
        count)))

(counter1 'increment!)

1

(counter1 'increment!)

2

count

0

This creates a procedure named counter1 with a private local variable count, whose value can only be inspected and changed by calls to the procedure. This protects against other procedures accidentally or deliberately changing the value of count other than through the interface provided by this procedure. For an analysis of (a variant of) this procedure, and a higher-level counter-creation procedure, in terms of Racket environments, please see the notes: Environments in Racket and the Mutator set!. </p>

If we arrange the lets and lambdas in a different fashion, we get a counter procedure that UTTERLY FAILS at its task.

(define not-a-counter
    (lambda (cmd)
        (let ((count 0))
          (case cmd
            [(increment!) (set! count (+ 1 count)) count]
            [(zero!) (set! count 0) count]))))

As examples of the behavior of this procedure, we have the following.

(not-a-counter 'increment!)

1

(not-a-counter 'zero!)

0

(not-a-counter 'increment!)

1

(not-a-counter 'increment!)

1

Think about what is happening in terms of environments to keep this from behaving like a counter.

(define counter2
  (let ((count 0))
    (lambda cmd
      (if (null? cmd)
          count
          (begin
            (case (car cmd)
              [(increment!) (set! count (+ 1 count))]
              [(zero!) (set! count 0)])
            count)))))

(counter2)

0

(counter2 'increment!)

1

(counter2 'increment!)

2

(counter2)

2

(counter2 'zero!)

0

(counter2)

0

make-counter

Now we will define a function that creates objects like counter2, but with the option of specifying any initial value, not just $0$.

(define make-counter
  (lambda (count)
    (lambda (command)
      (case command
        ((zero!) (set! count 0) count)
        ((increment!) (set! count (+ 1 count)) count)
        ((value) count)
    (else 'error)))))

(define c1 (make-counter 10))

(c1 'value)

10

(c1 'increment!)

11

(c1 'zero!)

0

(c1 'x)

'error

Using a Closure to Create a Stack

Object-oriented programming creates entities that combine data with procedures.

In racket, we can use closures for a similar effect.

Below we create a Racket procedure (make-stack name) that takes a symbol name and returns a Racket procedure that implements a stack object with local storage including a list representing a push down stack - a last-in, first-out (LIFO) data structure, which can process the following commands:

• name -- returns the name of the stack.
• empty? -- return #t if the stack is empty and #f otherwise.
• copy name -- create a new stack which is a copy of this one with the given name.
• show -- return the contents of the stack.
• equal? otherstack -- return #t if the otherstack has the same content as the given stack.
• push element - adds the given element to the top of the stack. Returns an error if element is absent.
• size -- returns the number of elements in the stack.
• peek -- returns the most recent element pushed on the stack, aka, the top of the stack.
• pop -- returns the most recent element pushed on the stack. Removes that element from the stack. Returns an error if the stack is empty.

(require racket) ;; need to define procedure first

(define (make-stack name (data empty))
  (let ((stack data)
        (size (length data)))
    (lambda (cmd . args)
      (case cmd
        ((name) name)
        ((empty?)
         (null? stack))
        ((copy)
         (if (null? args)
             'Error:usage:copy_stack
         (make-stack (first args) stack)))
        ((show)
        stack)
        ((equal?)
         (if (null? args)
             'Error:usage:equal_stack
             (equal? stack ((first args) 'show))))

        ((push)
         (if (null? args)
             'Error:usage:push_element
             (begin
               (set! stack (cons (first args) stack))
               (set! size (+ size 1))
               (first args))))
        ((size) size)

        ((peek)
         (if (null? stack)
             'Error:stack-empty
         (car stack)))
        ((pop)
         (if (null? stack)
             'Error:stack-empty
             (let ((result (car stack)))
               (set! stack (cdr stack))
               (set! size (- size 1))
               result)))
        (else 'invalid-method)
            ))))

Examples of using (make-stack name):

(define s1 (make-stack 'stack1))

(s1 'empty?)

#t

(s1 'size)

0

(s1 'name)

'stack1

(s1 'push 5)

5

(s1 'empty?)

#f

(s1 'push 7)

7

(s1 'push 9)

9

(s1 'size)

3

(s1 'pop)

9

(s1 'size)

2

(define s2 (make-stack 'stack2))

(s1 'name)

'stack1

(s2 'name)

'stack2

(s2 'pop)

'Error:stack-empty

(s2 'push)

'Error:usage:push_element

(s1 'equal? s2)

#f

(s1 'equal? s1)

#t

(s1 'show)

'(7 5)

(s1 'peek)

7

(define s3 (s1 'copy 'stack3))

(s3 'name)

'stack3

(s3 'show)

'(7 5)

(s3 'empty?)

#f

(s1 'equal? s3)

#t

What is the run time complexity of the following stack operations, using big-O notation?

• empty?
• size
• pop
• push

Plotting Data in Racket

See Graphs and Plots in Racket and Plot: Graph Plotting.

Need to use Dr Racket. Does not seem to work in Jupyter notebooks. In Python, we can Plot Function Runtimes.

See also Big-O Cheat Sheet

(require plot)

instantiate-linklet: mismatch;
 reference to a variable that is uninitialized;
 possibly, bytecode file needs re-compile because dependencies changed
  name: has-x-selection?
  exporting instance: "/usr/share/racket/pkgs/gui-lib/mred/private/wx/platform.rkt"
  importing instance: "/usr/share/racket/pkgs/gui-lib/mred/private/wx/common/clipboard.rkt"
  context...:
   temp37_0
   for-loop
   run-module-instance!125
   for-loop
   [repeats 1 more time]
   run-module-instance!125
   for-loop
   [repeats 1 more time]
   run-module-instance!125
   for-loop
   [repeats 1 more time]
   run-module-instance!125
   for-loop
   [repeats 1 more time]
   run-module-instance!125
   for-loop
   ...