I started this post thinking that I would write all the rules for evaluating Jacobians of neural network parameters in specific cases. But while this would certainly be useful for grokking deep learning papers, frankly it's difficult to write that in Latex and the people who have written The Matrix Calculus You Need For Deep Learning paper have already done it much better than I can do.

Rather, I consider my comparative advantage here to provide some expansion on why we should use Jacobians in the first place. If you were to just read the paper above, you might start to think that Jacobians are just notational perks. I hope to convince you that they are much more than that. In at least one setting, Jacobians provide a mathematical framework for analyzing the input-output behavior of deep neural networks, which can help us see things which we might have missed without this framework. A specific case of this phenomenon is a recently discovered technique which was even more recently put into a practical implementation: Jacobian regularization. Here we will see some fruits of our matrix calculus labor.

Deep learning techniques require us to train a neural network by slowly modifying parameters of some function until the function begins returning something close to the intended output. These parameters are often represented in the form of matrices. There are a few reasons for this representation: the matrix form is compact, and it allows us to use the tools of linear algebra directly. Matrix computations can also be processed in parallel, and this standardization allows programmers to build efficient libraries for the training of deep neural networks.

One quite important matrix in deep learning is the Jacobian.

In one sense, the Jacobian matrix is just a way of organizing gradient vectors. Gradient vectors, in turn, are just ways of organizing partial derivatives of an expression. Therefore, the Jacobian matrix is just a big matrix which allows us to organize a lot of partial derivatives. Formally, the Jacobian of $f : R^{n} \to R^{m}$ is defined by the following matrix. We denote $f_{i} (x)$ as the mapping from $R^{n} \to R_{i}$ , where $R_{i}$ is the real number line in the $i$ th coordinate of the output vector $R^{m}$ . Then the Jacobian is simply

\begin{matrix} ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ \begin{matrix} \nabla f_{1}^{⊺} \nabla f_{2}^{⊺} ⋮ \nabla f_{m}^{⊺} \end{matrix} ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ \end{matrix}

But describing the Jacobian as just some big rectangular box of partial derivatives of vector-valued functions hides the... ,

Daily Insights