CS 470 - Spring 2020.

• Widely used, including British cabinet meeting.
• Bad Privacy Practices: selling data, sending data to Facebook, treating home pages as marketing websites.
• Bad security practices: third party can turn on Mac user's camera overriding local security setting, can steal users' Windows credentials, display LinkedIn data, sloppy encryption: claim AES-256, but use AES-128, in ECB mode (electronic code book - not a good idea), use servers in China, have 700 developers in China, who could slip in backdoors a Chinese authorities request.
• Bad user configurations: zoombombing, meeting id's are too short - hackers can randomly try them - there are tools for this!

hw7 has been posted: hw7.html reinforcement learning.
Extension to Sunday April 19th at 11:59pm.

From piazza: a student asked if it is OK simply to copy and paste pages of RL code for the blackjack problem.

My goal is that you understand what is going on. You are right. There is lots of code out there. Some of it uses TensorFlow or PyTorch. You are welcome to use that code.

You need to include narrative in your jupyter notebook that explains what is going on. It should be easier to write a coherent explanation if you first understand the code yourself. Imagine that you were given this assignment at work and had to present your result to your manager. Generally, your boss does not care where the code came from (assuming you didn't steal it). Your boss wants to be convinced that you know what you are doing in case the code needs to be adapted or maintained.

Try to take that perspective.

• http://www.marynel.net/static/pdfs/mvazquez-IROS15.pdf
• https://arxiv.org/abs/1907.12910
• http://www.marynel.net/static/pdfs/mvazquez-HRI17.pdf

Final Exam: Friday May 1, 2pm

In light of the Universal Pass policy, there will be no final exam.

• The primary function of the final exam is to be able to distinguish among the students in the class.
• Contrary to accepted wisdom, final exams (and studying for final exams) is not an effective learning method. At this point, you have some theoretical appreciation for how learning works. The human brain is a highly adapted learning machine. As we discussed, it is wired to learn from mistakes and similary, with reinforcement learning (or operant conditioning) to learn from reward and punishment.
• When you are studying for an exam, your brain actually knows that you are studying for an exam. Your brain knows when the exam is going to occur. Your brain also knows that after the exam, you have little use for the facts and knowledge you have crammed into your head. Therefore, being a well-oiled machine, your brain puts the crammed knowledge in a short term memory location which is readily accessible during the exam, but can be freed up after the exam.
• To put it in the technical language of memory management, your brain will garbage collect most of what you prepared.
• So, what's the point? Well the stuff that I really want you to learn has been reinforced through the problem sets, and the lectures. I try to tell stories. The human brain has evolved to process episodic memory more effectively than dry semantic facts.
• Also, there are practical skills that I have tried to inculcate in this course, including:
  • Python programming and object oriented programming. The AIMA code provide very good examples.
  • Jupyter notebooks. Comments and documentation are a positive force in the universe. Jupyter notebooks facilitate the combination of text and code. This idea was pioneered years ago by none other than Donald Knuth. He called it literate programming in 1984.
  • X Windows. For many of you, using jupyter notebooks remotely from the zoo required you to learn a bit about X Windows, developed at MIT in 1984 as well.
  • LaTeX, the text formatter offshoot of TeX, also developed by Knuth.
  • mathplotlib. A powerful python module for plotting.
  • github. You may not realize it, but in tackling the current homework assignments, you will often find yourself in the vicinity of github. Make yourself welcome there.
  • openai and gym. It is good to have these in your quiver, along with scikit learn, which was just mentioned in passing.
• The point is, I was not going to test you about any of this stuff. My nefarious plan was to convince (not coerce) you that you would be a better person for learning these tools and techniques.
• Finally, as stated above, cognitive science research indicates that people learn from failure. Good judgment comes from experience. Experience comes from bad judgment. Thus, like it or not, I try to make you fail. I give you vague directions. I point you at code that might not work. (Frankly, I was surprised that the AIMA perceptron code was buggy. It looks like they removed it from the next release.)
• Here is the big reveal: my role is not to get you to learn computer science or artificial intelligence. My role is to get you to love CS and AI. I decided that giving a final exam under the present circumstances was no longer instrumental to that goal.

Lecture: Reinforcement Learning

• Reinforcement Learning
• Reinforcement Learning II

gym.html

FrozenLake.html