站点网址：https://nv-tlabs.github.io
NVIDIA Toronto AI Lab Search NVIDIA Toronto AI Lab NVIDIA Toronto AI Lab Home News Members Research Projects Publications Contact Light Dark Automatic NVIDIA Toronto AI Lab IntroductionWelcome to the homepage of the NVIDIA Toronto Artificial Intelligence Lab led by Professor Sanja Fidler. Our research group was founded in 2018, and is primarily based in Toronto.
The research interests of our lab lie at the intersection of computer vision, machine learning and computer graphics. Our group members are also part of or closely collaborate with academic labs such as the University of Toronto and Vector Institute.
We invite applications for the following positions:
full time research scientist full time research engineer research scientist intern research engineer internGraduate and senior undergraduate students interested in doing an internship in the NVIDIA Toronto AI Lab can directly fill this form. See this link for open positions, or contact our members for more details.
NewsDecember 2023 - Adaptive Shells for Efficient Neural Radiance Field Rendering received the Best Paper Award at SIGGRAPH Asia 2023.
August 2023 - Interactive AI Material Generation and Editing in NVIDIA Omniverse wins Real-Time Live at SIGGRAPH 2023.
July 2023 - Learning Physically Simulated Tennis Skills From Broadcast Videos was awarded an Honorable Mention at SIGGRAPH 2023.
June 2022 - Karsten Kreis co-organized a workshop on diffusion-based generative modeling at CVPR 2022. Or Litani and Towaki Takikawa organized the neural fields tutorial at CVPR 2022.
April 2021 - Our work was presented at GTC 2021.
December 2020 - New version of the website.
May 2020 - 40 Years on, PAC-MAN Recreated with AI by NVIDIA Researchers
December 2019 - 2D or Not 2D: NVIDIA Researchers Bring Images to Life with AI
November 2019 - NVIDIA Makes 3D Deep Learning Research Easy with Kaolin PyTorch Library
November 2019 - NVIDIA Research at ICCV: Generating New City Road Layouts with AI
October 2019 - NVIDIA Research at ICCV: Meta-Sim: Learning to Generate Synthetic Datasets
June 2019 - NVIDIA Research Released at CVPR Helps Developers Create Better Visual Datasets
Members Sanja Fidler Hassan Abu Alhaija David Acuna Matan Atzmon Wenzheng Chen Riccardo de Lutio Clement Fuji Tsang Jun Gao Zan Gojcic Anita Hu Jiahui Huang Amlan Kar Seung Wook Kim Karsten Kreis Marc Law Huan Ling Or Litany James Lucas Rafid Mahmood Janick Martinez Esturo Jason Peng Or Perel Jonah Philion Davis Rempe Nicholas Sharp Frank Shen Masha Shugrina Zian Wang Francis Williams Kevin Xie Kangxue Yin Xiaohui Zeng Haotian Zhang Current interns Arthur Allshire Ruilong Li Ruofan Liang Derek Lim Jonathan Lorraine Ashkan Mirzaei Mathis Petrovich Xuanchi Ren Jonathan Tseng Chenfeng Xu Yilun Xu Hongwei Yi Research TopicsOur research combines machine learning, computer vision and computer graphics for innovative applications such as video games, simulations, film industry, data servers, medical imaging and self-driving cars. A non-exhaustive list of research topics studied in our group include:
Machine Learning for Computer Graphics: Differentiable rendering, 3D deep learning, computational shape analysis, manipulation of color distributions
Generative modeling: Synthetic data generation, dynamic environment simulation, 3D models.
Representation Learning: Semantic segmentation, road layout modeling, complex continuous data structures to represent hierarchies and graphs, motion planning, planner-centric metrics, numerical Optimization, federated simulation
Machine Learning with limited supervision: active learning, self-supervised learning, few-shot learning, domain adaptation, learning with noisy labels
Long-Term Projects Quickly discover relevant content by filtering publications. Omniverse AI ToyboxInteractive 3D Tools as Extensions in NVIDIA Omniverse
NVIDIA is using AI to improve the Content Creation process, giving the power to create virtual worlds to creators everywhere. AI Toybox … Code Kaolin Wisp: A PyTorch Library and Engine for Neural Fields ResearchPytorch Library
NVIDIA Kaolin Wisp is a PyTorch library powered by NVIDIA Kaolin Core to work with neural fields (including NeRFs, NGLOD, instant-ngp … Towaki Takikawa, Or Perel, Clement Fuji Tsang, Charles Loop, Joey Litalien, Jonathan Tremblay, Sanja Fidler, Maria Shugrina Code Project Source Document Kaolin: A PyTorch Library for Accelerating 3D Deep Learning ResearchPytorch Library
Kaolin is a PyTorch library aiming to accelerate 3D deep learning research. Kaolin provides efficient implementations of differentiable … Krishna Murthy Jatavallabhula, Edward Smith, Jean-Francois Lafleche, Clement Fuji Tsang, Artem Rozantsev, Wenzheng Chen, Tommy Xiang, Rev Lebaredian, Sanja Fidler PDF Code Project Source Document Publications Quickly discover relevant content by filtering publications. * Adaptive Shells for Efficient Neural Radiance Field RenderingSIGGRAPH Asia 2023 (Best Paper Award)
Neural radiance fields achieve unprecedented quality for novel view synthesis, but their volumetric formulation remains expensive, … Zian Wang, Tianchang Shen, Merlin Nimier-David, Nicholas Sharp, Jun Gao, Alexander Keller, Sanja Fidler, Thomas Müller, Zan Gojcic PDF Project Video Compact Neural Graphics Primitives with Learned Hash ProbingSIGGRAPH Asia 2023
Compact neural graphics primitives (Ours) have an inherently small size across a variety of use cases with automatically chosen … Towaki Takikawa, Thomas Müller, Merlin Nimier-David, Alex Evans, Sanja Fidler, Alec Jacobson, Alexander Keller PDF Project Language-Guided Traffic Simulation via Scene-Level DiffusionCoRL 2023
Realistic and controllable traffic simulation is a core capability that is necessary to accelerate autonomous vehicle (AV) development. … Ziyuan Zhong, Davis Rempe, Yuxiao Chen, Boris Ivanovic, Yulong Cao, Danfei Xu, Marco Pavone, Baishakhi Ray PDF TexFusion: Synthesizing 3D Textures with Text-Guided Image Diffusion ModelsICCV 2023 (Oral)
We present TexFusion(Texture Diffusion), a new method to synthesize textures for given 3D geometries, using only large-scale … Tianshi Cao, Karsten Kreis, Sanja Fidler, Nicholas Sharp, Kangxue Yin PDF ATT3D: Amortized Text-to-3D Object synthesisICCV 2023
Text-to-3D modeling has seen exciting progress by combining generative text-to-image models with image-to-3D methods like Neural … Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung-Yi Lin, Ming-Yu Liu, Sanja Fidler, James Lucas PDF Project Video DreamTeacher: Pretraining Image Backbones with Deep Generative ModelsICCV 2023
In this work, we introduce a self-supervised feature representation learning framework DreamTeacher that utilizes generative networks … Daiqing Li, Huan Ling, Amlan Kar, David Acuna, Seung Wook Kim, Karsten Kreis, Antonio Torralba, Sanja Fidler PDF Project Learning Human Dynamics in Autonomous Driving ScenariosICCV 2023
Simulation has emerged as an indispensable tool for scaling and accelerating the development of self-driving systems. A critical aspect … Jingbo Wang, Ye Yuan, Zhengyi Luo, Kevin Xie, Dahua Lin, Umar Iqbal, Sanja Fidler, Sameh Khamis PDF Neural LiDAR Fields for Novel View SynthesisICCV 2023
We present Neural Fields for LiDAR (NFL), a method to optimise a neural field scene representation from LiDAR measurements, with the … Shengyu Huang, Zan Gojcic, Zian Wang, Francis Williams, Yoni Kasten, Sanja Fidler, Konrad Schindler, Or Litany PDF Project SKED: Sketch-guided Text-based 3D EditingICCV 2023
Text-to-image diffusion models are gradually introduced into computer graphics, recently enabling the development of Text-to-3D … Aryan Mikaeili, Or Perel, Mehdi Safaee, Daniel Cohen-Or, Ali Mahdavi-Amiri PDF Project Towards Viewpoint Robustness in Bird's Eye View SegmentationICCV 2023
Autonomous vehicles (AV) require that neural networks used for perception be robust to different viewpoints if they are to be deployed … Tzofi Klinghoffer, Jonah Philion, Wenzheng Chen, Or Litany, Zan Gojcic, Jungseock Joo, Ramesh Raskar, Sanja Fidler, Jose M. Alvarez PDF Project Bridging the Sim2Real gap with CARE: Supervised Detection Adaptation with Conditional Alignment and ReweightingTMLR 2023
Sim2Real domain adaptation (DA) research focuses on the constrained setting of adapting from a labeled synthetic source domain to an … Viraj Prabhu, David Acuna, Andrew Liao, Rafid Mahmood, Marc T. Law, Judy Hoffman, Sanja Fidler, James Lucas PDF Differentially Private Diffusion ModelsTMLR 2023
While modern machine learning models rely on increasingly large training datasets, data is often limited in privacy-sensitive domains. … Tim Dockhorn, Tianshi Cao, Arash Vahdat, Karsten Kreis PDF Code CALM: Conditional Adversarial Latent Models for Directable Virtual CharactersSIGGRAPH 2023
In this work, we present Conditional Adversarial Latent Models (CALM), an approach for generating diverse and directable behaviors for … Chen Tessler, Yoni Kasten, Yunrong Guo, Shie Mannor, Gal Chechik, Xue Bin Peng PDF Code Project Data-Free Learning of Reduced-Order KinematicSIGGRAPH 2023
Physical systems ranging from elastic bodies to kinematic linkages are defined on high-dimensional configuration spaces, yet their … Nicholas Sharp, Cristian Romero, Alec Jacobson, Etienne Vouga, Paul Kry, David I.W. Levin, Justin Solomon PDF Code Project Video Flexible Isosurface Extraction for Gradient-Based Mesh OptimizationSIGGRAPH 2023
This work considers gradient-based mesh optimization, where we iteratively optimize for a 3D surface mesh by representing it as the … Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojcic, Sanja Fidler, Nicholas Sharp, Jun Gao PDF Code Project Video Learning Physically Simulated Tennis Skills from Broadcast VideosSIGGRAPH 2023 (Honorable Mention)
We present a system that learns diverse, physically simulated tennis skills from large-scale demonstrations of tennis play harvested … Haotian Zhang, Ye Yuan, Viktor Makoviychuk, Yunrong Guo, Sanja Fidler, Xue Bin Peng, Kayvon Fatahalian PDF Project Synthesizing Physical Character-Scene InteractionsSIGGRAPH 2023
Movement is how people interact with and affect their environment. For realistic character animation, it is necessary to synthesize … Mohamed Hassan, Yunrong Guo, Tingwu Wang, Michael Black, Sanja Fidler, Xue Bin Peng PDF Project Align your Latents: High-Resolution Video Synthesis with Latent Diffusion ModelsCVPR 2023
Latent Diffusion Models (LDMs) enable high-quality image synthesis while avoiding excessive compute demands by training a diffusion … Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, Karsten Kreis PDF Project Fast Monocular Scene Reconstruction with Global-Sparse Local-Dense GridsCVPR 2023
Indoor scene reconstruction from monocular images has long been sought after by augmented reality and robotics developers. Recent … Wei Dong, Chris Choy, Charles Loop, Yuke Zhu, Or Litany, Anima Anandkumar PDF Project Video Magic3D: High-Resolution Text-to-3D Content CreationCVPR 2023
DreamFusion has recently demonstrated the utility of a pre-trained text-to-image diffusion model to optimize Neural Radiance Fields … Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang , Karsten Kreis, Sanja Fidler, Ming-Yu Liu, Tsung-Yi Lin PDF Project Neural Fields meet Explicit Geometric Representations for Inverse Rendering of Urban ScenesCVPR 2023
Reconstruction and intrinsic decomposition of scenes from captured imagery would enable many applications such as relighting and … Zian Wang, Tianchang Shen, Jun Gao, Shengyu Huang, Jacob Munkberg, Jon Hasselgren, Zan Gojcic, Wenzheng Chen, Sanja Fidler PDF Project Video Neural Kernel Surface ReconstructionCVPR 2023 (Highlight)
We present a novel method for reconstructing a 3D im- plicit surface from a large-scale, sparse, and noisy point cloud. Our approach … Jiahui Huang, Zan Gojcic, Matan Atzmon, Or Litany, Sanja Fidler, Francis Williams PDF Project NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion ModelsCVPR 2023
Automatically generating high-quality real world 3D scenes is of enormous interest for applications such as virtual reality and … Seung Wook Kim, Bradley Brown, Kangxue Yin, Karsten Kreis, Katja Schwarz, Daiqing Li, Robin Rombach, Antonio Torralba, Sanja Fidler PDF Project Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory DiffusionCVPR 2023
We introduce a method for generating realistic pedestrian trajectories and full-body animations that can be controlled to meet … Davis Rempe, Zhengyi Luo, Xue Bin Peng, Ye Yuan, Kris Kitani, Karsten Kreis, Sanja Fidler, Or Litany PDF Project VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene CompletionCVPR 2023
Humans can easily imagine the complete 3D geometry of occluded objects and scenes. This appealing ability is vital for recognition and … Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M. Alvarez, Sanja Fidler, Chen Feng, Anima Anandkumar PDF Code Spacetime Representation LearningICLR 2023
Much of the data we encounter in the real world can be represented as directed graphs. In this work, we introduce a general family of … Marc T. Law, James Lucas PDF Mask3D: Mask Transformer for 3D Instance SegmentationICRA 2023
Modern 3D semantic instance segmentation approaches predominantly rely on specialized voting mechanisms followed by carefully designed … Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, Bastian Leibe PDF Code Project Source Document Omniverse AI ToyboxInteractive 3D Tools as Extensions in NVIDIA Omniverse
NVIDIA is using AI to improve the Content Creation process, giving the power to create virtual worlds to creators everywhere. AI Toybox … Code Neural Brushstroke Engine: Learning a Latent Style Space of Interactive Drawing ToolsSIGGRAPH Asia (TOG) 2022
Neural Brushstroke Engine (NeuBE) includes a GAN model that learns to mimic many drawing media styles by learning from unlabeled … Masha Shugrina, Chin-Ying Li, Sanja Fidler PDF Code Project GENIE: Higher-Order Denoising Diffusion SolversNeurIPS 2022
Denoising diffusion models (DDMs) have emerged as a powerful class of generative models. A forward diffusion process slowly perturbs … Tim Dockhorn, Arash Vahdat, Karsten Kreis PDF Code Project GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from ImagesNeurIPS 2022
As several industries are moving towards modeling massive 3D virtual worlds, the need for content creation tools that can scale in … Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, Sanja Fidler PDF Code Project LION: Latent Point Diffusion Models for 3D Shape GenerationNeurIPS 2022
Denoising diffusion models (DDMs) have shown promising results in 3D point cloud synthesis. To advance 3D DDMs and make them useful for … Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojcic, Or Litany, Sanja Fidler, Karsten Kreis PDF Code Project Optimizing data collection for machine learningNeurIPS 2022
Modern deep learning systems require huge data sets to achieve impressive performance, but there is little guidance on how much or what … Rafid Mahmood, James Lucas, Jose M. Alvarez, Sanja Fidler, Marc T. Law PDF Project PADL: Language-Directed Physics-Based Character ControlSIGGRAPH Asia 2022
Developing systems that can synthesize natural and life-like motions for simulated characters has long been a focus for computer … Jordan Juravsky, Yunrong Guo, Sanja Fidler, Xue Bin Peng PDF Project XDGAN: Multi-Modal 3D Shape Generation in 2D SpaceBMVC 2022
Generative models for 2D images has recently seen tremendous progress in quality, resolution and speed as a result of the efficiency of … Hassan Abu Alhaija, Alara Dirik, André Knörig, Sanja Fidler, Masha Shugrina PDF Project Kaolin Wisp: A PyTorch Library and Engine for Neural Fields ResearchPytorch Library
NVIDIA Kaolin Wisp is a PyTorch library powered by NVIDIA Kaolin Core to work with neural fields (including NeRFs, NGLOD, instant-ngp … Towaki Takikawa, Or Perel, Clement Fuji Tsang, Charles Loop, Joey Litalien, Jonathan Tremblay, Sanja Fidler, Maria Shugrina Code Project Source Document Language-Grounded Indoor 3D Semantic Segmentation in the WildECCV 2022
Recent advances in 3D semantic segmentation with deep neural networks have shown remarkable success, with rapid performance increase on … Dávid Rozenberszki, Or Litany, Angela Dai PDF Code Project MvDeCor: Multi-view Dense Correspondence Learning for Fine-grained 3D SegmentationECCV 2022
We propose to utilize self-supervised techniques in the 2D domain for fine-grained 3D shape segmentation tasks. This is inspired by the … Gopal Sharma, Kangxue Yin, Subhransu Maji, Evangelos Kalogerakis, Or Litany, Sanja Fidler PDF Project Learning Smooth Neural Functions via Lipschitz RegularizationSIGGRAPH 2022
Neural implicit fields have recently emerged as a useful representation for 3D shapes. These fields are commonly represented as neural … Derek Liu, Francis Williams, Alec Jacobson, Sanja Fidler, Or Litany PDF Project ASE: Large-Scale Reusable Adversarial Skill Embeddings for Physically Simulated CharactersSIGGRAPH 2022
The incredible feats of athleticism demonstrated by humans are made possible in part by a vast repertoire of general-purpose motor … Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, Sanja Fidler PDF Project Variable Bitrate Neural FieldsSIGGRAPH 2022
Neural approximations of scalar and vector fields, such as signed distance functions and radiance fields, have emerged as accurate, … Towaki Takikawa, Alex Evans, Jonathan Tremblay, Thomas Müller, Morgan McGuire, Alec Jacobson, Sanja Fidler PDF Code Project Video Polymorphic-GAN: Generating Aligned Samples across Multiple Domains with Learned Morph MapsCVPR 2022 (Oral)
Modern image generative models show remarkable sample quality when trained on a single domain or class of objects. In this work, we … Seung Wook Kim, Karsten Kreis, Daiqing Li, Antonio Torralba, Sanja Fidler PDF Project Extracting Triangular 3D Models, Materials, and Lighting From ImagesCVPR 2022 (Oral)
We present an efficient method for joint optimization of topology, materials and lighting from multi-view image observations. Unlike … Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, Sanja Fidler PDF Code Project AUV-Net: Learning Aligned UV Maps for Texture Transfer and SynthesisCVPR 2022
In this paper, we address the problem of texture representation for 3D shapes for the challenging and underexplored tasks of texture … Zhiqin Chen, Kangxue Yin, Sanja Fidler PDF Code Project BigDatasetGAN: Synthesizing ImageNet with Pixel-wise AnnotationsCVPR 2022
Annotating images with pixel-wise labels is a time-consuming and costly process. Recently, DatasetGAN showcased a promising alternative … Daiqing Li, Huan Ling, Seung Wook Kim, Karsten Kreis, Adela Barriuso, Sanja Fidler, Antonio Torralba PDF Code Project Frame Averaging for Equivariant Shape Space LearningCVPR 2022
The task of shape space learning involves mapping a train set of shapes to and from a latent representation space with good … Matan Atzmon, Koki Nagano, Sanja Fidler, Sameh Khamis, Yaron Lipman PDF Project Generating Useful Accident-Prone Driving Scenarios via a Learned Traffic PriorCVPR 2022
Evaluating and improving planning for autonomous vehicles requires scalable generation of long-tail traffic scenarios. To be useful, … Davis Rempe, Jonah Philion, Leonidas Guibas, Sanja Fidler, Or Litany PDF Project How Much More Data Do I Need? Estimating Requirements for Downstream TasksCVPR 2022
Given a small training data set and a learning algorithm, how much more data is necessary to reach a target validation or test … Rafid Mahmood, James Lucas, David Acuna, Daiqing Li, Jonah Philion, Jose M. Alvarez, Zhiding Yu, Sanja Fidler, Marc T. Law PDF Project Neural Fields as Learnable Kernels for 3D ReconstructionCVPR 2022
We present Neural Kernel Fields: a novel method for reconstructing implicit 3D shapes based on a learned kernel ridge regression. Our … Francis Williams, Zan Gojcic, Sameh Khamis, Denis Zorin, Joan Bruna, Sanja Fidler, Or Litany PDF Project Poster Video Score-Based Generative Modeling with Critically-Damped Langevin DiffusionICLR 2022 (Spotlight)
Score-based generative models (SGMs) have demonstrated remarkable synthesis quality. SGMs rely on a diffusion process that gradually … Tim Dockhorn, Arash Vahdat, Karsten Kreis PDF Code Project Tackling the Generative Learning Trilemma with Denoising Diffusion GANsICLR 2022 (Spotlight)
A wide variety of deep generative models has been developed in the past decade. Yet, these models often struggle with simultaneously … Zhisheng Xiao, Karsten Kreis, Arash Vahdat PDF Code Project Domain Adversarial Training: A Game PerspectiveICLR 2022
The dominant line of work in domain adaptation has focused on learning invariant representations using domain-adversarial training. In … David Acuna, Marc T. Law, Guojun Zhang, Sanja Fidler PDF Low-Budget Active Learning via Wasserstein Distance: An Integer Programming ApproachICLR 2022
Active learning is the process of training a model with limited labeled data by selecting a core subset of an unlabeled data pool to … Rafid Mahmood, Sanja Fidler, Marc T. Law PDF Spectral Unions of Partial Deformable 3D ShapesEurographics 2022
Spectral geometric methods have brought revolutionary changes to the field of geometry processing – however, when the data to be … Luca Moschella, Simone Melzi, Luca Cosmo, Filippo Maggioli, Or Litany, Maks Ovsjanikov, Leonidas Guibas, Emanuele Rodolà PDF Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy LabelsWACV 2022
The success of learning with noisy labels (LNL) methods relies heavily on the success of a warm-up stage where standard supervised … Evgenii Zheltonozhskii, Chaim Baskin, Avi Mendelson, Alex M. Bronstein, Or Litany PDF Code ATISS: Autoregressive Transformers for Indoor Scene SynthesisNeurIPS 2021
The ability to synthesize realistic and diverse indoor furniture layouts automatically or based on partial input, unlocks many … Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, Sanja Fidler PDF Code Project Poster Video Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape SynthesisNeurIPS 2021
We introduce DMTet, a deep 3D conditional generative model that can synthesize high-resolution 3D shapes using simple user guides such … Tianchang Shen, Jun Gao, Kangxue Yin, Ming-Yu Liu, Sanja Fidler PDF Project Video Score-based Generative Modeling in Latent SpaceNeurIPS 2021
Score-based generative models (SGMs) have recently demonstrated impressive results in terms of both sample quality and distribution … Arash Vahdat, Karsten Kreis, Jan Kautz PDF Code Project Towards Optimal Strategies for Training Self-Driving Perception Models in SimulationNeurIPS 2021
Autonomous driving relies on a huge volume of real-world data to be labeled to high precision. Alternative solutions seek to exploit … David Acuna, Jonah Philion, Sanja Fidler PDF Project Ultrahyperbolic Neural NetworksNeurIPS 2021 (Spotlight)
Riemannian space forms, such as the Euclidean space, sphere and hyperbolic space, are popular and powerful representation spaces in … Marc T. Law PDF DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable RendererNeurIPS 2021
We consider the challenging problem of predicting intrinsic object properties from a single image by exploiting differentiable … Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khamis, Or Litany, Sanja Fidler PDF Project Don't Generate Me: Training Differentially Private Generative Models with Sinkhorn DivergenceNeurIPS 2021
Although machine learning models trained on massive data have led to breakthroughs in several areas, their deployment in … Tianshi Cao, Alex Bie, Arash Vahdat, Sanja Fidler, Karsten Kreis PDF Project EditGAN: High-Precision Semantic Image EditingNeurIPS 2021
Generative adversarial networks (GANs) have recently found applications in image editing. However, most GAN based image editing methods … Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, Sanja Fidler PDF Project Mix3D: Out-of-Context Data Augmentation for 3D Scenes3DV 2021 (Oral)
We present Mix3D, a data augmentation technique for segmenting large-scale 3D scenes. Since scene context helps reasoning about object … Alexey Nekrasov, Jonas Schult, Or Litany, Bastian Leibe, Francis Engelmann PDF Code Project Learning Indoor Inverse Rendering with 3D Spatially-Varying LightingICCV 2021 (Oral)
In this work, we address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single … Zian Wang, Jonah Philion, Sanja Fidler, Jan Kautz PDF Project 3DStyleNet: Creating 3D Shapes with Geometric and Texture Style VariationsICCV 2021 (Oral)
We propose a method to create plausible geometric and texture style variations of 3D objects in the quest to democratize 3D content … Kangxue Yin, Jun Gao, Maria Shugrina, Sameh Khamis, Sanja Fidler PDF Project Vector Neurons: A General Framework for SO(3)-Equivariant NetworksICCV 2021 (Oral)
Invariance and equivariance to the rotation group have been widely discussed in the 3D deep learning community for pointclouds. Yet … Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, Leonidas Guibas PDF Code Project Physics-based Human Motion Estimation and Synthesis from VideosICCV 2021
Human motion synthesis is an important problem with applications in graphics, gaming and simulation environments for robotics. Existing … Kevin Xie, Tingwu Wang, Umar Iqbal, Yunrong Guo, Sanja Fidler, Florian Shkurti PDF Project Self-Supervised Real-to-Sim Scene GenerationICCV 2021
Synthetic data is emerging as a promising solution to the scalability issue of supervised deep learning, especially when real data are … Aayush Prakash, Shoubhik Debnath, Jean-Francois Lafleche, Eric Cameracci, Gavriel State, Stan Birchfield, Marc T. Law PDF Project f-Domain-Adversarial Learning: Theory and AlgorithmsICML 2021
Unsupervised domain adaptation is used in many machine learning applications where, during training, a model has access to unlabeled … David Acuna, Guojun Zhang, Marc T. Law, Sanja Fidler PDF Code Project Image-Level or Object-Level? A Tale of Two Resampling Strategies for Long-Tailed DetectionICML 2021
Training on datasets with long-tailed distributions has been challenging for major recognition tasks such as classification and … Nadine Chang, Zhiding Yu, Yu-Xiong Wang, Anima Anandkumar, Sanja Fidler, Jose M. Alvarez PDF DatasetGAN: Efficient Labeled Data Factory with Minimal Human EffortCVPR 2021 (Oral)
We introduce DatasetGAN: an automatic procedure to generate massive datasets of high-quality semantically segmented images requiring … Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, Sanja Fidler Project DriveGAN: Towards a Controllable High-Quality Neural SimulationCVPR 2021 (Oral)
Realistic simulators are critical for training and verifying robotics systems. While most of the contemporary simulators are … Seung Wook Kim, Jonah Philion, Antonio Torralba, Sanja Fidler PDF Project Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D SurfacesCVPR 2021 (Oral)
Neural signed distance functions (SDFs) are emerging as an effective representation for 3D shapes. State-of-the-art methods typically … Towaki Takikawa, Joey Litalien, Kangxue Yin, Karsten Kreis, Charles Loop, Derek Nowrouzezahrai, Alec Jacobson, Morgan McGuire, Sanja Fidler PDF Code Project Video Weakly Supervised Learning of Rigid 3D Scene FlowCVPR 2021 (Oral)
We propose a data-driven scene flow estimation algorithm exploiting the observation that many 3D scenes can be explained by a … Zan Gojcic, Or Litany, Andreas Wieser, Leonidas J Guibas, Tolga Birdal PDF Project Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain GeneralizationCVPR 2021
Training deep networks with limited labeled data while achieving a strong generalization ability is key in the quest to reduce human … Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, Sanja Fidler PDF Project 3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object DetectionCVPR 2021
3D object detection is an important yet demanding task that heavily relies on difficult to obtain 3D annotations. To reduce the … He Wang, Yezhen Cong, Or Litany, Yue Gao, Leonidas J. Guibas PDF Project Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural RenderingICLR 2021 (Oral)
Differentiable rendering has paved the way to training neural networks to perform ‘inverse graphics’ tasks such as … Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, Sanja Fidler PDF Project VAEBM: A Symbiosis between Variational Autoencoders and Energy-based ModelsICLR 2021 (Spotlight)
Energy-based models (EBMs) have recently been successful in representing complex distributions of small images. However, sampling from … Zhisheng Xiao, Karsten Kreis, Jan Kautz, Arash Vahdat PDF Code gradSim: Differentiable simulation for system identification and visuomotor controlICLR 2021
We consider the problem of estimating an object’s physical properties such as mass, friction, and elasticity directly from video … Krishna Murthy Jatavallabhula, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jérôme Parent-Lévesque, Kevin Xie, Kenny Erleben, Liam Paull, Florian Shkurti, Derek Nowrouzezahrai, Sanja Fidler PDF Code Project Video ReLMoGen: Integrating Reinforcement Learning and Motion Generation for Interactive NavigationICRA 2021
Many Reinforcement Learning (RL) approaches use joint control signals (positions, velocities, torques) as action space for continuous … Fei Xia, Chengshu Li, Roberto Martin-Martin, Alexander Toshev, Or Litany, Silvio Savarese PDF Project UniCon: Universal Neural Controller For Physics-based Character MotionUniCon
The field of physics-based animation is gaining importance due to the increasing demand for realism in video games and films, and has … Tingwu Wang, Yunrong Guo, Maria Shugrina, Sanja Fidler PDF Project Video Kaolin: A PyTorch Library for Accelerating 3D Deep Learning ResearchPytorch Library
Kaolin is a PyTorch library aiming to accelerate 3D deep learning research. Kaolin provides efficient implementations of differentiable … Krishna Murthy Jatavallabhula, Edward Smith, Jean-Francois Lafleche, Clement Fuji Tsang, Artem Rozantsev, Wenzheng Chen, Tommy Xiang, Rev Lebaredian, Sanja Fidler PDF Code Project Source Document Variational Amodal Object CompletionNeurIPS 2020
Huan Ling, David Acuna, Karsten Kreis, Seung Wook Kim, Sanja Fidler PDF Learning Deformable Tetrahedral Meshes for 3D ReconstructionNeurIPS 2020
3D shape representations that accommodate learning-based 3D reconstruction are an open problem in machine learning and computer … Jun Gao, Wenzheng Chen, Tommy Xiang, Alec Jacobson, Morgan McGuire, Sanja Fidler PDF Code Project Video Ultrahyperbolic Representation LearningNeurIPS 2020
In machine learning, data is usually represented in a (flat) Euclidean space where distances between points are along straight lines. … Marc T. Law, Jos Stam PDF Code Federated Simulation or Medical ImagingMICCAI 2020 (Nominated for the Young Scientist Award)
Labelling data is expensive and time consuming especially for domains such as medical imaging that contain volumetric imaging data and … Daiqing Li, Amlan Kar, Nishant Ravikumar, Alejandro Frangi, Sanja Fidler PDF Project Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data GenerationECCV 2020
Procedural models are being widely used to synthesize scenes for graphics, gaming, and to create (labeled) synthetic datasets for ML. … Jeevan Devaranjan*, Amlan Kar, Sanja Fidler PDF Project Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3DECCV 2020
The goal of perception for autonomous vehicles is to extract semantic representations from multiple sensors and fuse these … Jonah Philion, Sanja Fidler PDF Code Project Video Learning to Simulate Dynamic Environments with GameGANCVPR 2020
Simulation is a crucial component of any robotic system. In order to simulate correctly, we need to write complex rules of the … Seung Wook Kim, Yuhao Zhou, Jonah Philion, Antonio Torralba, Sanja Fidler PDF Project Video Learning to Predict 3D Objects with an Interpolation-based Differentiable RendererNeurIPS 2019
Many machine learning models operate on images, but ignore the fact that images are 2D projections formed by 3D geometry interacting … Wenzheng Chen, Jun Gao, Huan Ling, Edward J. Smith, Jaakko Lehtinen, Alec Jacobson, Sanja Fidler PDF Code Project Video Neural Turtle Graphics for Modeling City Road LayoutsICCV 2019 (Oral)
We propose Neural Turtle Graphics (NTG), a novel gen- erative model for spatial graphs, and demonstrate its ap- plications in modeling … Hang Chu, Daiqing Li, David Acuna, Amlan Kar, Maria Shugrina, Xinkai Wei, Ming-Yu Liu, Antonio Torralba, Sanja Fidler PDF Project Video Meta Sim: Learning to Generate Synthetic DatasetsICCV 2019 (Oral)
Training models to high-end performance requires availability of large labeled datasets, which are expensive to get. The goal of our … Amlan Kar, Aayush Prakash, Ming-Yu Liu, Eric Cameracci, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, Sanja Fidler PDF Code Project Gated-SCNN Gated Shape CNNs for Semantic SegmentationICCV 2019
Current state-of-the-art methods for image segmentation form a dense image representation where the color, shape and texture … Towaki Takikawa, David Acuna, Varun Jampani, Sanja Fidler PDF Code Project Devil is in the Edges: Learning Semantic Boundaries from Noisy AnnotationsCVPR 2019 (Oral)
We tackle the problem of semantic boundary prediction, which aims to identify pixels that belong to object(class) boundaries. We notice … David Acuna, Amlan Kar, Sanja Fidler PDF Code Project ContactOur lab is located in Downtown Toronto (20 minutes away from the St-George campus of the University of Toronto) and hosts many students for their co-op programs. Motivated candidates can contact our members to apply for internships and research positions.
Unauthorized visitors are not permitted in the Toronto office.
431 King St W, 6th floor, Toronto, ON M5V 3M4 Monday-Friday 9:00 to 18:00 +−Leaflet | © OpenStreetMap
Published with Wowchemy — the free, open source website builder that empowers creators.
Cite × Copy Download