Using Class Activations to Investigate Semantic Segmentation

Semantic segmentation is one of the most popular tasks in computer vision. Its applications span from medical image analysis to self driving cars and beyond. For a given image, in semantic segmentation, we generate masks of image segments corresponding to each type or class. However, these segmented maps may either segment a region properly but assign it to a different class or they may have a possibly poor segmented region identification. Hence there is a need to visualize the regions of importance in an image for a given class. Class Activation Maps (CAMs) are popularly used in the classification task to understand the correlation of a class and the regions in an image that correspond to it. We propose a new framework to model the semantic segmentation task as an end to end classification task. This can be used with any deep learning based segmentation network. Using this, we visualize the gradient based CAMs (GradCAM) for the task of semantic segmentation. We also validate our results by using sanity checks for saliency maps and correlate them to those found for the classification task.