About the project
This project explores how Large Language Models (LLMs) enhance lifelong robot learning by incrementally acquiring and refining visuomotor skills. It investigates leveraging textual and spatio-temporal data to improve skill adaptation while addressing challenges in continual learning. The methods will be tested in household environments for real-world, open-ended robotic learning scenarios.
The integration of Large Language Models (LLMs) into robot learning has recently led to significant advancements. By leveraging LLMs to generate sequences of natural language commands linked to pretrained skills, robots can efficiently perform long-horizon tasks in physical environments. Additionally, LLMs have enabled few-shot adaptation to novel tools by generating task-agnostic tool descriptions, facilitating language-conditioned learning of new manipulation skills. Long-horizon task learning and rapid adaptation, powered by LLMs, are crucial for lifelong robot learning.
However, current approaches remain incompatible with continual learning settings, where robots must acquire skills sequentially without prior knowledge of task distributions. Furthermore, existing methods do not adequately address the challenge of retaining previously learned knowledge while adapting to new tasks.
This project aims to advance the role of LLMs in lifelong robot learning by developing methods for incrementally acquiring and refining skill representations. Specifically, it will explore how textual descriptions and spatio-temporal action data can be jointly leveraged to learn and adapt visuomotor skills over time. The proposed methods will be evaluated in household environments, focusing on real-world, open-ended robotic learning scenarios.
You will join the School of Electronics and Computer Science within the University of Southampton which is ranked in the top 1% of universities worldwide. The successful candidate must have a strong background in machine learning. Prior knowledge of reinforcement learning, robotics and training and finetuning LLMs is highly desirable.
