CS 370 - Spring 2025. 4/7/2025


[Home]

Welcome to CS 370!

Video of the Day

Reinforcement Learning Tutorial using OpenAI gym FrozenLake environment.

I hereby solicit suggestions for the video of the day. Please email me your ideas with explanations. Selected entries will win 5 homework points. If your video is played at the beginning of class, you must also briefly explain something about the video and something about yourself - in person.

Canvas Quiz of the Day (need daily password)

Most days, there will be a simple canvas quiz related to the lecture. You need a password to activate the quiz, which I will provide in class. These quizzes will count toward your class participation grade. The quiz is available only during class. You get full credit for class participation by completing half of the quizzes.

Click for today's quiz.

Lecture 20: Reinforcement Learning

Announcements

  • Information Society Project Yale Law School. Weekly Events

  • Digital Ethics Center - Call for Director's Fellows.
    The Digital Ethics Center at Yale University is now hiring current/continuing students for our Director's Fellows research assistant program, as well as full-time postgraduate or postdoctoral research scholar positions. Applications are due by 21 April 2025 for Director's Fellows, and 30 April 2025 for all other positions, with an anticipated start date in Fall 2025.

    Please contact Joanna Carmona, Program Manager, with any questions.

    Director's Fellows

    The Director’s Fellowships program at the Digital Ethics Center (DEC) offers an exciting opportunity for Yale students to engage in groundbreaking research on digital ethics and AI governance. With two distinct tracks - Junior Director's Fellows (Undergraduates) and Senior Director's Fellows (Graduate and Professional School Students) - this program is designed to support students at different academic stages, while fostering a shared mission of advancing digital ethics scholarship.

    Each fellow completes a semester-long research project, participates in regular cohort meetings, and receives a monetary award per semester, disbursed monthly. Fellows are expected to be on-site for 6-8 hours weekly. The program is structured around regular meetings during the semester, led by Professor Floridi and other researchers at the DEC. These meetings enable fellows to discuss current topics in digital ethics, exchange feedback on their research projects as a cohort, and acquire research skills practically. The goal is for each fellow to research, write, and revise an original paper of publishable quality by the end of the semester, on a topic relevant to the activities of the Center, with the support of Professor Floridi and usually in collaboration with other researchers. The fellowship may complement the work that students are already doing through their coursework at Yale. For example, a fellow’s paper may evolve from or contribute to their bachelor’s thesis, master’s thesis, or a doctoral dissertation chapter.

    The Director’s Fellowship program is interdisciplinary and inclusive of students from all backgrounds. The only requirement is a demonstrated interest in researching the governance, ethical, legal, and social implications (GELSI) of digital innovation and technologies. Applications from all disciplines are encouraged, with special priority given to applications from graduate students and individuals with prior research experience. Past Fellows include engineering students Tyler Schroder '25 (AY24), Grant Shanklin (Spring 2025), Andrew West '25, and more students from other backgrounds and disciplines.

    Director's Fellow Application and Information

    Administrivia

  • I have office hours Wednesdays from 4-6 p.m. via Zoom, meeting ID 459 434 2854.

  • The TF's office hours are posted on Ed Discussion.

  • I am available for lunch on Mondays at 1 pm in Morse.

  • Homework assignments: [Assignments]. hw7 is now available. Note: there will be no hw8. You can concentrate on the paper and possibly the project.

    I have reviewed all the project proposals. Let me know if you did not get an OK, aka, Complete.

    Asides from previous lectures

    AI in the news

  • Meta Unveils Llama 4: The Next Generation of AI Models Substack, April 5, 2025. 17 billion parameters and 288 billion parameters. Mixture of experts: MoE. Used online reinforcement learning.
  • OpenAI’s Ghibli Moment Spins Out of Control, Bloomberg, April 4, 2025.
  • Google Is Searching for an Answer to ChatGPT, Bloomberg, March 24, 2025. My daughter Alexandra is on the AI Mode team. I have asked her to speak to the class.

  • Access to the Atlantic
  • Access to Economist (Economist.com)
  • Access to Financial Times
  • Access to Wall Street Journal from Yale.
  • Q and AI Bloomberg.
  • Access to Bloomberg.com from Yale.

    Lecture: Reinforcement Learning

  • Deep Learning Nature. LeCun, Bengio, and Hinton. 2015. pdf These are the guys who won the Nobel Prize in Physics last year.

  • Reinforcement Learning: An Introduction Sutton and Barto. 2020.

  • Readings: AIMA chapter 22 (chapter 21 in 3rd edition)

  • 2019 Scassellati Slides:
  • AIMA Slides: rl.html

    gym.html (gymnasium) [problem with installing swig and Box2D]

    FrozenLake.html

    Future topics: NLP and LLMs

  • Moore's Law vs the More Law.
  • Hands-on large language models : language understanding and generation Jay Alammar, O'Reilly, 2024. (Yale library online book).
  • NLP Progress Repository to track the progress in Natural Language Processing (NLP), including the datasets and the current state-of-the-art for the most common NLP tasks.
  • C4 (Colossal Clean Crawled Corpus)
  • Efficient Estimation of Word Representations in Vector Space the word2vec paper, by Jeff Dean and the Google guys. 2011.
    [Home]