publications

2022

1. EMQA

   Episodic Memory Question Answering 🤖

   Samyak Datta, Sameer Dharur, Vincent Cartillier, Ruta Desai, Mukul Khanna, Dhruv Batra, and Devi Parikh

   Accepted at CVPR 2022

   arXiv HTML
2. DeepHS-HDRV

   DeepHS-HDRVideo: Deep High Speed High Dynamic Range Video Reconstruction 📸

   Zeeshan Khan, Parth Shettiwar, Mukul Khanna, and Shanmuganathan Raman

   Accepted at ICPR 2022

   Abs

   Due to hardware constraints, standard off-the-shelf digital cameras suffers from low dynamic range (LDR) and low frame per second (FPS) outputs. Previous works in high dynamic range (HDR) video reconstruction uses sequence of alternating exposure LDR frames as input, and align the neighbouring frames using optical fow based networks. However, these methods often result in motion artifacts in challenging situations. This is because, the alternate exposure frames have to be exposure matched in order to apply alignment using optical fow. Hence, over-saturation and noise in the LDR frames results in inaccurate alignment. To this end, we propose to align the input LDR frames using a pre-trained video frame interpolation network. This results in better alignment of LDR frames, since we circumvent the error-prone exposure matching step, and directly generate intermediate missing frames from the same exposure inputs. Furthermore, it allows us to generate high FPS HDR videos by recursively interpolating the intermediate frames. Through this work, we propose to use video frame interpolation for HDR video reconstruction, and present the frst method to generate high FPS HDR videos. Experimental results demonstrate the effcacy of the proposed framework against optical fow based alignment methods and validate the idea of using video frame interpolation for HDR video reconstruction.

2021

1. BF2NormalNet

   Building Facades to Normal Maps: Adversarial Learning from Single View Images 🏢

   Mukul Khanna, Tanu Sharma, Ayyappa Swamy Thatavarthy, and K. Madhava Krishna

   Conference on Robots and Vision (CRV) 2021

   Abs HTML Code

   Surface normal estimation is an essential component of several computer and robot vision pipelines. While this problem has been extensively studied, most approaches are geared towards indoor scenes and often rely on multiple modalities (depth, multiple views) for accurate estimation of normal maps. Outdoor scenes pose a greater challenge as they exhibit significant lighting variation, often contain occluders, and structures like building facades are often ridden with numerous windows and protrusions. Conventional supervised learning schemes excel in indoor scenes, but do not exhibit competitive performance when trained and deployed in outdoor environments. Furthermore, they involve complex network architectures and require many more trainable parameters. To tackle these challenges, we present an adversarial learning scheme that regularizes the output normal maps from a neural network to appear more realistic, by using a small number of precisely annotated examples. Our method presents a lightweight and simpler architecture, while improving performance by at least 1.5x across most metrics. We evaluate our approaches against the state-of-the-art on normal map estimation, on a synthetic and a real outdoor dataset, and observe significant performance enhancements.

2. Multi-view Planarity Constraints for Skyline Estimation from UAV Images in City Scale Urban Environments 🏙️

   Ayyappa Swamy Thatavarthy, Tanu Sharma, Harshit Sankhla, Mukul Khanna, and K. Madhava Krishna

   VISAPP 2021

   PDF

2019

1. FHDR

   FHDR: HDR Image Reconstruction from a Single LDR Image using Feedback Network 📸

   Zeeshan Khan, Mukul Khanna, and Shanmuganathan Raman

   GlobalSIP 2019

   Abs arXiv Code

   High dynamic range (HDR) image generation from a single exposure low dynamic range (LDR) image has been made possible due to the recent advances in Deep Learning. Various feed-forward Convolutional Neural Networks (CNNs) have been proposed for learning LDR to HDR representations. To better utilize the power of CNNs, we exploit the idea of feedback, where the initial low level features are guided by the high level features using a hidden state of a Recurrent Neural Network. Unlike a single forward pass in a conventional feed-forward network, the reconstruction from LDR to HDR in a feedback network is learned over multiple iterations. This enables us to create a coarse-to-fine representation, leading to an improved reconstruction at every iteration. Various advantages over standard feed-forward networks include early reconstruction ability and better reconstruction quality with fewer network parameters. We design a dense feedback block and propose an end-to-end feedback network-FHDR for HDR image generation from a single exposure LDR image. Qualitative and quantitative evaluations show the superiority of our approach over the state-of-the-art methods.

2. URSIM

   Open Source Simulator for Unmanned Underwater Vehicles using ROS and Unity3D 🐟

   Pushkal Katara, Mukul Khanna, Harshit Nagar, and A. Panaiyappan

   Underwater Technology (UT) 2019

   Abs HTML Code

   The paper presents URSim: an open source 3D underwater simulation framework for Unmanned Underwater Vehicles (UUVs) developed using Robotics Operating System (ROS) and a real-time game engine called Unity3D. Simulation systems like these enable to implement, test, study and analyze complex systems while minimizing cost and disruption to the environment. URSim provides the user an intuitive way to simulate underwater vehicles and robots. It is capable of simulating feedback control systems, dynamic model, underwater vision and mission planning for underwater vehicles and robots. The simulation provides support for underwater sensor modules, underwater physics, collision kinematics and is highly configurable to simulate a realistic underwater environment. The software architecture is adaptive to algorithms for control systems, image processing, navigation and manipulation.