Measuring Invariances

name: inverse
layout: true
class:
---
class: middle, inverse

.right[
[github](https://github.com/facundoq/rotational_variance)
]

***

.frontimage[ ![](img/sample.png "mnist") ]

.left[

#### Facundo Quiroga¹, Jordina Torrents-Barrena²,
#### Laura Lanzarini¹, Dómenec Puig-Valls²

# Measuring transformation (in)variances
# in Neural Networks

***
.foot[
[.logo[![III-LIDI, Universidad Nacional de La Plata](img/logos/lidi.png "III-LIDI, Universidad Nacional de La Plata")]](http://weblidi.info.unlp.edu.ar/)
¹ **III-LIDI** Instituto de Investigación y Desarrrollo en Informática, **Universidad Nacional de La Plata**, *Argentina*

[.logo[![IRCV, Universitat Rovira i Virgili](img/logos/urv.png "IRCV, Universitat Rovira i Virgili")]](http://deim.urv.cat/~rivi)
² **IRCV** Intelligent Robotics and Computer Vision Group, **Universitat Rovira i Virgili**, *Spain*
]
]

---
class: inverse,center
# Brain vs Rotation (180°)
![](img/rotated/mercedes180.jpg)
---
class: inverse,center
# Brain vs Rotation (90°)
![](img/rotated/mercedes90.jpg)

---
class: inverse,center
# Brain vs Rotation (0°)
![](img/rotated/mercedes0.jpg)

Mercedes Sosa
---
class: inverse,center
# How do we encode

.left[
* In our brains?
* In a Neural Network?
]
---
class: inverse,center
# Neural Networks

.mainimage90[![activations](img/activations_image.png)]
.left[
* Properties
  * Structured by layers/activations
  * Complex, distributed representations
]
---
class: inverse,center
# Neural Networks vs Rotations
.mainimage80[
![](img/model/variant_nn.png)
]
---
# *Invariant* Neural Networks

* A network $f$ is invariant to a set of transformations $T=  [t_1,t_2,\dots,t_n]$ if  $ \forall x$
$$f(t_1(x))=f(t_2(x))= \dots =f(t_n(x))$$

* If $T= [R0,R90,R180]$

.mainimage50[
![](img/model/invariant_nn.png)
]

---
# *Invariant* Neural Networks - Current approaches

1. Correct rotation with another model  .imagemodels[![rotation model](img/model/rotation_model.png)]
2. Invariant by *design* (30+ models proposed in the last few years) .imagemodels[![invariant designed model](img/model/invariant_designed_model.png)]
3. Invariant by *training* - with data augmentation .imagemodels[![invariant trained model](img/model/invariant_trained_model.png)]
4. Hybrids

---
# *Invariant* Neural Networks - Evaluation

.rotationvsaccuracy[
![accuracy vs rotation](img/accuracy_vs_rotation.png)
.greenbox[] *Variant* model,
.graybox[] *Invariant* model
]

1. Accuracy vs Rotation Angle (image)
  * **Black box** measure
  * Standard
2. [Lenc and Vedaldi 2014](https://arxiv.org/abs/1411.5908) measure *equivariance*
  * Generalization of *invariance*.
  * Hard to use, various assumptions.
  * Not used (8 citations).

---
.floatright[
  [tf playground](https://playground.tensorflow.org)
]

# Our approach - Motivation

.mainimage90[![activations](img/activations_image_comparison.png)]

* Limitations of past approaches
  * Disregard internal representations
  * No insight on efficiency of encoding

---
class: inverse
layout: true
---
# Our approach - Motivation
.heatmap_sample[

]
.mainimage60[![activations](img/sample_heatmap_rotated_label.png)]
1. Measure invariance in **each activation**
2. Invariance is hard to measure
  * Measure **variance** instead

---
# Proposed measure $V$ - Setup

* Given
  * Activation function $a(x)$
  * Samples $ X = \[ x\_1, \dots, x_{n} \] $
  * Transformations $ T = \[ t\_1, \dots, t_{m} \] $
* Measure variance $V$ for single activation $a$
--

.matrix[
  ![matrix A](img/rotated/matrix_annotated.png)
]
* Activations value matrix $A$:
$$
A = \begin{bmatrix}
a(t\_1(x\_1)) & \cdots & a(t\_m(x\_1)) \\\
\vdots & \ddots & \vdots \\\
a(t\_1(x\_n)) & \cdots & a(t\_m(x\_n))
\end{bmatrix}
$$
--

* Two sources of variance
      * Samples
      * Transformations

---
# Proposed measure $V$ - Definition
.matrix2[
  ![matrix A](img/rotated/matrix_annotated.png)
]
$$
A = \begin{bmatrix}
a(t\_1(x\_1)) & \cdots & a(t\_m(x\_1)) \\\
\vdots & \ddots & \vdots \\\
a(t\_1(x\_n)) & \cdots & a(t\_m(x\_n))
\end{bmatrix}
$$
--

* Variance over transformations (desired)
$$V\_t = \text{mean variance over columns of A} $$
--

* Variance over samples (normalization) $$V\_x = \text{mean variance over rows of A} $$
--

* Normalized variance (single activation): $$ V = \frac{V\_t}{V\_x} $$

---
class:
layout: true
---
# Results - Datasets and Networks

* Measured various datasets/networks
* Show only:
  * MNIST dataset

.sampleimage[ ![](img/mnist.png "mnist") ]

.sampleimage[ ![](img/mnist_rot.png "mnist_rot") ]

* Simple CNN

.image70[![](img/nn.png "nn topology") ]

---

# Results - Invariance per layer - mnist/simple cnn
.center[.image60[![](img/results/collapsed_global.png) ]
* Lower layers have more invariance
* Variant models are similar except for last layers
]

---
# Results - Invariance per class - mnist/simple cnn

* Class conditional invariance

---
class: inverse
layout: true
---
.pull-left[
# Conclusion
* $V$ is a low-granularity (in)variance measure
* Works on arbitrary transformations
* Works on Arbitrary networks
* Insight into Neural Networks internal invariance
]
.pull-right[
# Future work
* Variants of $V$
  * Other normalizations (WIP)
  * ANOVA (WIP)
  * Information theory
* $V$ for specific layers
  * Convolutional Layers (not shown)
  * Recurrent Networks
* $V$ during training (WIP)
  * Transfer learning
  * Random networks
* $V$ as a library
  * Pytorch/Tensorflow/Tensorboard (WIP)
]