Models of language based on large Neural Networks — otherwise known as Deep Learning — are revolutionizing the way we work, learn, and communicate. But how do these models work on a fundamental level? What enables their (seemingly) human-like command of language? In this course, we will focus on building an understanding of neural language models from the ground up, starting with mathematical fundamentals, introducing crucial topics from Computational Linguistics and Machine Learning, surveying current and historical approaches to building neural LMs, and gaining hands-on experience training and analyzing such models on UR's BlueHive computing cluster.
| Days | Time | Location |
|---|---|---|
| Monday and Wednesday | 10:25 - 11:40 AM | Lattimore 513 |
| Role | Name | Office | Office Hours |
|---|---|---|---|
| Instructor | C.M. Downey | Lattimore 507 | TBD |
Required readings are posted in the schedule below, drawn mostly from the following online textbook (abbreviated JM in the schedule), which is a very good general resource:
Class attendance and participation are expected and count towards your grade. I will keep track of attendance. Students are allowed to be absent from up to four sessions for any reason, without needing to contact me, and without penalty. These excused absences may be used for travel, illness, catching up with other courses, etc. However, unexecused absences beyond these four sessions will count against the student's final attendance grade, with the exception of important obligations listed below.
Short quizzes will be held at the beginning of class on most Mondays, and occasionally on Wednesdays if there is no class on Monday. All quiz dates and topics are noted on the course calendar (see below). The quizzes are primarily meant to promote active engagement with the material, and grades will be adjusted so that a score of 85% is the median, within the undergraduate and graduate sections separately. Grades will ONLY be adjusted upward, never downward. For example, if the median grade on a quiz for undergraduates is 75%, all undergraduate grades will be shifted 10% higher (absolute). HOWEVER, quiz grades will be capped at 100%. If the median grade is 85% or higher, grades will not be adjusted.
Students will complete 6-8 homeworks, comprised of both written and (Python) programming assignments. Unless noted otherwise on the schedule, homeworks will be released on Wednesdays, and due at 11pm on the following Wednesday. All homework will be submitted via Blackboard.
Students will work in assigned groups to complete a substantial term project focused on answering a question about language or linguistic theory with deep learning methods. This will minimally involve training or fine-tuning a neural model of language (though not necessarily a Language Model in the technical sense). This project will be scientifically-oriented, i.e. going beyond simply engineering a model to solve an NLP task, and seeking to extend scientific understanding of Language or Language Models. Within these parameters, student groups are encouraged to creatively pursue a topic of interest to them.
Project milestones will be assigned throughout the semester to ensure timely progress and feasible goals (more details to be given as the semester semester progresses). At the end of the semester, each project group will present their work and results to the class, submit a Github repository containing project software, and submit a final writeup in the style of a scientific research paper.
All deadlines and meeting times for this class are in "Eastern Time". Please note: on Sunday November 2, this will change from Eastern Daylight Time (EDT/UTC-4) to Eastern Standard Time (EST/UTC-5). All work should be submitted by 11:00pm the day it is due. Work that is received late will incur the following penalties:
Extensions (without penalty) may be offered if they are requested within a reasonable amount of time (relative to the reason for the extension) before the work is due. Please don't hesitate to ask for an extension if you need one.
Students will not be penalized because of important civic, ethnic, family or religious obligations, or university service. You will have a chance, whenever feasible, to make up within a reasonable time any assignment that is missed for these reasons. Absences for these reasons will count as excused for the sake of the participation grade. But it is your job to inform me of any expected missed work in advance, as soon as possible.
All assignments and activities associated with this course must be performed in accordance with the University of Rochester's Academic Honesty Policy. More information is available here. Please note: The use of Generative AI to produce any part of the written or programming homeworks is not allowed. Generative AI is allowed for programming work on the Term Project only (the final writeup must be your own work).