Title: Semantics-aware Learning in Autonomous Systems: From Robot Manipulators to Autonomous Vehicles
Abstract:
Semantic scene understanding is a fundamental cognitive faculty which can help artificial robots gain a high degree of invariance and generalization against variations in scene contexts, object types, and motion profiles. In my research, I have been investigating how to derive a more descriptive and structural encoding of scene semantics to get a step closer to the intelligent robots with the greater autonomy.
My talk consists of two parts. In the first part of the talk, I will promote a new holistic view on manipulation semantics, which combines the perception and execution of manipulation actions in one unique framework called “Semantic Event Chain” (SEC). The SEC concept, which I developed during PhD, is an implicit spatio-temporal formulation that encodes actions by coupling the observed effect with the exhibited roles of manipulated objects. I will explain how semantic action encoding can enable robots to link continuous visual sensory signals (e.g., image sequences) to their symbolic descriptions (e.g., action primitives). To highlight the scalability of manipulation semantics, I will introduce various applications of SECs in learning object affordances, coupling language and vision, and memorizing episodic experiences.
In the second part of the talk, I will focus on how to employ scene semantics to solve the cross-modal domain translation problem within a generative multitask learning framework. I will continue with practical applications of this framework in the context of autonomous driving by showing, for instance, how to generate photorealistic RGB-D images from 3D LiDAR point clouds by relying solely on the sensor-agnostic scene semantics. I will conclude my talk by showing how to boost the downstream object detection task through the use of these synthesized semantically rich RGB-D image streams.
Bio:
Eren Aksoy is an Associate Professor (Docent) at Halmstad University in Sweden. He coordinates the Horizon Europe project ROADVIEW, which focuses on robust automated driving in extreme weather. He obtained his Ph.D. in computer science from the University of Göttingen, Germany, in 2012. During his Ph.D. studies, he invented the concept of Semantic Event Chains to encode, learn, and execute human manipulation actions in the context of robot imitation learning. His framework has been used as a technical robot perception-action interface in several EU projects such as IntellACT, Xperience, and ACAT. Prior to relocating to Sweden, he spent three years as a postdoctoral research fellow in the H2T group of Prof. Dr. Tamim Asfour at the Karlsruhe Institute of Technology. Recently, as a visiting scholar, he conducted research on AI-based perception algorithms for autonomous vehicles at Volvo GTT and Zenseact AB in Sweden. He serves as an Associate Editor for various high-ranking robotics journals and conferences including RA-L, IROS, Humanoids, and ITSC. His research interests include scene semantics, computer vision, AI, and cognitive robotics. He actively works on creating semantic representations of visual experiences to improve the environment and action understanding of autonomous systems, such as robots and unmanned vehicles.