Interesting, it certainly wouldn’t take up much additional space, but I wonder if it would have any real impact, since it seems somewhat orthogonal to finding a faithful low-dimensional encoding of the activations.