CS 201 - Spring 2025. 2/24/2025.

Announcements

Speaker: Greg Durrett, University of Texas, Austin.

Host: Arman Cohan

Title: Specializing LLMs for Reliability

Abstract:

Large language models (LLMs) have advanced the frontiers of AI reasoning: they can synthesize information from multiple sources, derive new conclusions, and explain those conclusions to their users. However, LLMs do not do this reliably. They hallucinate facts, convincingly state incorrect deductions, and exhibit logical fallacies like confirmation bias. In this talk, I will describe my lab’s work on making LLM systems reliable by introspecting their behavior. First, I will demonstrate that better understanding of LLMs helps us train them to be more reliable reasoners. Our work shows that model interpretation techniques can advance training methodology and dataset curation for reasoning models. Second, I will argue that automating fine-grained evaluation of LLM output provides a level of understanding necessary for further progress. I will describe the ingredients of effective automated evaluators and a state-of-the-art factuality evaluation system, MiniCheck, showing that analyzing the nature of hallucinations can help reduce them. Finally, I will describe how deeper understanding of LLMs will let us tackle their most fundamental limitations, such as their inconsistency when given different inputs. I will propose how these pieces might soon be combined to form reliable AI systems.

Bio:

Greg Durrett is an associate professor of Computer Science at UT Austin. His research is broadly in the areas of natural language processing and machine learning. His group develops techniques for reasoning about knowledge in text, verifying factuality of LLM generations, and specializing LLMs to make them more reliable. He is a 2023 Sloan Research Fellow and a recipient of a 2022 NSF CAREER award. His work has been recognized by paper awards at EMNLP 2024 and EMNLP 2013. He was a founding organizer of the Workshop on Natural Language Reasoning and Structured Explanations at ACL 2023 and ACL 2024 and is a current member of the NAACL board. He received his BS in Computer Science and Mathematics from MIT and his PhD in Computer Science from UC Berkeley, where he was advised by Dan Klein.

Website: https://www.cs.utexas.edu/~gdurrett/

Refreshments from Koffee Katering will be available.

Midterm Exam

The midterm will be Tuesday February 25th at 7pm in Davies Auditorium. It will be a 2 hour hand written exam. No computers. No notes. No books. No kidding. Students registered with Student Accessibility Services will take the exam at Becton C031 next door.

Sample Midterm Exam available . (solutions) The midterm will not have a boolean function question. Instead, it will have a struct question. The actual exam will also include UNIX questions (Principles 1 and 2). I will give you a transcript with some of the commands X'd out. You will have to deduce those commands (solutions).

You should be familiar with the recursion and tail recursion examples from the recursion.rkt and Recursion.html For more details on the wonders of tail recursion, see TailRecursion.html and this tail recursion article.

Also, the paper Music and Computation, discussed below, is also in scope, up to but not including Music. There will be true/false questions about binary encodings of numbers, text, images, and sound. No questions about music.

See Point.html for a sample struct question.

Owen Prem held a review session Sunday, February 23rd, from 1-3pm, at DL 220. Check Ed Discussios for details, namely slides and recording.

Using Yale's Clarity for review.

• GPT-4o: please give me practice problems for recursion in racket. just factorial.
• Claude 3.5: please give me practice problems for recursion in racket. Bingo! It came up with 8 examples.
• Claude 3.5: please give me practice problems for tail recursion in racket. Super Bingo! It came up with 8 examples.
• Bonus question: Why is the Anthropic chat bot named Claude?

  I don't actually know why I was named Claude. The reasoning behind my name wasn't part of the information I was given during my training. If you're curious about the origin of my name, I'd recommend checking Anthropic's website or reaching out to them directly for more information. I can say that Claude is a name with French origins, but I don't know if that played any role in the choice. Sorry I can't give you a more definitive answer on this!

• please give me practice problems for recursion in racket. Bingo! Several examples and follow up. Also better graphics.
• please give me practice problems for tail recursion in racket. Bingo! Several examples and follow up.
• please give me practice problems for UNIX commands. Bingo! Several examples. Many go beyond principles 1 and 2.

Lecture: Computability.

Everything is a string of bits

• Computer number formats binary, octal, hexadecimal, integer, negative, fixed point, floating point.
• ASCII Roman alphabet.
• Unicode Every alphabet.
• Digital Audio analog to digial sampling.
• 44,100 Hertz sampling rate. Twice the upper limit of human hearing, per Claude Shannon.
• Compact Disc See Beethoven.
• Pixels See color depth.
• Digital Video

Computability.html (jupyter) Part 2. What is computable?

Getting to know UNIX