Advanced Topics in Reinforcement Learning and Planning

Course type

Seminar

Study programs

• Master Computer Science
• Master Data Science
• Master Software Systems Engineering

Content

Students will present papers from a list of important, recent works in machine learning compiled by the professor. Topics include deep and reinforcement learning, and transformer and GNN architectures. The focus will be on the use of these techniques in simple applications that are crisp and easy to understand. The list of papers will also include those underlying well-known learning systems like ChaptGPT, AlphaZero, Gato, etc. The final list of papers and topics to be covered will be defined after the first meeting.

Recommended prior knowledge

Bachelor degree in CS or equivalent and basic AI and ML courses, with some previous exposure to deep learning and reinforcement learning.

Topics

RL and Planning

• Behavior From the Void: Unsupervised Active Pre-Training
  H Liu, P Abbeel
  2021, NeurIPS

• Exploration via Elliptical Episodic Bonuses
  M Henaff, R Raileanu, M Jiang, T Rocktäschel
  2022, NeurIPS

• Optimize Planning Heuristics to Rank, not to Estimate Cost-to-Goal
  L Chrestien, S Edelkamp, A Komenda, T Pevny
  2024, NeurIPS

• Evidence of Learned Look-Ahead in a Chess-Playing Neural Network
  E Jenner, S Kapur, V Georgiev, C Allen, S Emmons, S Russell
  2024, arXiv

• HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learning
  Q Delfosse, J Blüml, B Gregori, K Kersting
  2024, RLC Workshop InterpPol

• Bridging Reinforcement Learning Theory and Practice with the Effective Horizon
  C Laidlaw, SJ Russell, A Dragan
  2023, NeurIPS

Symbolic, Neurosymbolic Learning, Theory

• Mobius: Synthesizing Relational Queries with Recursive and Invented Predicates
  A Thakkar, N Sands, G Petrou, R Alur, M Naik, M Raghothaman
  2023, PACMPL

• Example-Guided Synthesis of Relational Queries
  A Thakkar, A Naik, N Sands, R Alur, M Naik, M Raghothaman
  2021, PLDI

• Diffusion On Syntax Trees For Program Synthesis
  S Kapur, E Jenner, S Russell
  2024, arXiv

• What relations are reliably embeddable in Euclidean space?
  R Bhattacharjee, S Dasgupta
  2020, ALT

• Scallop: A Language for Neurosymbolic Programming
  Z Li, J Huang, M Naik
  2023, PACMPL

Robotics

• CORN: Contact-based Object Representation for Nonprehensile Manipulation of General Unseen Objects
  Y Cho, J Han, Y Cho, B Kim
  2024, ICLR

• M2T2: Multi-Task Masked Transformer for Object-centric Pick and Place
  W Yuan, A Murali, A Mousavian, D Fox
  2023, CoRL

• Learning value functions with relational state representations for guiding task-and-motion planning
  B Kim, L Shimanuki
  2020, CoRL

Graph and Relational Learning, GNNs

• Towards Foundation Models for Knowledge Graph Reasoning
  M Galkin, X Yuan, H Mostafa, J Tang, Z Zhu
  2024, ICLR

• Graph Neural Networks are Dynamic Programmers
  AJ Dudzik, P Veličković
  2022, NeurIPS

• The CLRS Algorithmic Reasoning Benchmark
  P Veličković, AP Badia, D Budden, R Pascanu, A Banino, M Dashevskiy, R Hadsell, C Blundell
  2022, ICML

• A*Net: A Scalable Path-based Reasoning Approach for Knowledge Graphs
  Z Zhu, X Yuan, M Galkin, LP Xhonneux, M Zhang, M Gazeau, J Tang
  2024, NeurIPS

Object-Oriented Perception and Learning

• Grounded Object-Centric Learning
  A Kori, F Locatello, FDS Ribeiro, F Toni, B Glocker
  2024, ICLR

• DDLP: Unsupervised Object-centric Video Prediction with Deep Dynamic Latent Particles
  T Daniel, A Tamar
  2024, TMLR

Transformers

• Learning Transformer Programs
  D Friedman, A Wettig, D Chen
  2024, NeurIPS

• What Algorithms can Transformers Learn? A Study in Length Generalization
  H Zhou, A Bradley, E Littwin, N Razin, O Saremi, J Susskind, S Bengio, P Nakkiran
  2024, ICLR

• Inverted-Attention Transformers can Learn Object Representations: Insights from Slot Attention
  YF Wu, K Greff, GF Elsayed, MC Mozer, T Kipf, S van Steenkiste
  2023, NeurIPS Workshop Causal Representation Learning

• An Image is Worth More Than 16×16 Patches: Exploring Transformers on Individual Pixels
  DK Nguyen, M Assran, U Jain, MR Oswald, CGM Snoek, X Chen
  2024, arXiv

• What Formal Languages Can Transformers Express? A Survey
  L Strobl, W Merrill, G Weiss, D Chiang, D Angluin
  2024, TACL

• Transformers meet Neural Algorithmic Reasoners
  W Bounsi, B Ibarz, A Dudzik, JB Hamrick, L Markeeva, A Vitvitskyi, R Pascanu, P Veličković
  2024, arXiv

• Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languages
  D Angluin, D Chiang, A Yang
  2023, arXiv

• Structured World Representations in Maze-Solving Transformers
  MI Ivanitskiy, AF Spies, T Räuker, G Corlouer, C Mathwin, L Quirke, C Rager, R Shah, D Valentine, CD Behn, K Inoue, SW Fung 2023, arXiv

Registration

Please register via the SUPRA platform offered by the department of CS.

Questions?

If you have any questions, please contact Jonas Reiher.