The Abstraction and Reasoning Corpus: The LATFORMER Architecture

cover
11 Mar 2024

This paper is available on arxiv under CC 4.0 license.

Authors:

(1) Mattia Atzeni, EPFL, Switzerland and [email protected];

(2) Mrinmaya Sachan, ETH Zurich, Switzerland;

(3) Andreas Loukas, Prescient Design, Switzerland.

4. The LATFORMER Architecture

While in principle the problem of inferring group actions from input-output pairs can be solved via search over finite groups, in practice the size of the group for lattice symmetry actions makes this approach unfeasible1 . Moreover, we are interested in learning unknown functions jointly with the transformation, which cannot be solved by searching on the space of group actions. Using a neural agent to search the space of possible actions would be a viable alternative, but this would make the problem non-differentiable and we would need to resort to reinforcement learning methods.

In this work, we aim to solve the problem in a differentiable way. Inspired by the observations above, we introduce LATFORMER, which incorporates the insights of Section 3 into a neural architecture. We propose to use gated CNNs to parameterize the masks and we introduce an additional smoothing technique for easier optimization.

4.1. Lattice Mask Experts as Convolutional Networks

Attention modules in neural networks usually include an attention mechanism with learnable linear transformations of the inputs2 followed by a feed-forward network (FFN), as in the Transformer encoder layer (Vaswani et al., 2017).

To infuse core geometry priors in the attention module, we propose to modulate the attention weights with a mask generated by an additional layer, as shown in Figure 3a. We refer to this layer as Lattice mask expert, as it specializes towards specific transformations of the lattice. To understand the purpose of this layer, it is useful to remember that, by the analysis conducted in Section 3, even if the attention and FFN layers are initialized to the identity function, the mask expert can generate attention masks that produce precise geometric transformations of the input.

By Corollary 3.3, we know that each group action on the lattice can be represented by a mask that is a convolution of the identity and we have an analytical expression to calculate the kernels of the convolution. We can leverage this notion to design CNNs that produce attention masks corresponding to specific group actions by following the general formulation:

4.2. Mask Smoothing for Easier Training

The framework described so far parameterizes discrete transformations of a lattice in a differentiable manner. Nevertheless, to improve the training of LATFORMER, we found it beneficial to also apply a smoothing operation on the attention masks. Our approach entails defining an adjacency relation between group elements and applying graph convolution with a heat kernel on the corresponding graph. This encourages the optimizer to favor weight updates that change the masks in a smooth manner w.r.t. the geodesic distance implied by the graph. Concretely, we define the neighbors of each element gi on the lattice as those gj = e ◦ gi reachable by an application of a primitive action e, such as translation by a single pixel in one dimension, rotation by 90◦ , and vertical/horizontal reflection. The notion of neighborhood gives rise to a graph whose vertex set is the lattice group and that contains one edge for every pair of neighboring actions.

As before, it helps to consider different kinds of transformations separately. For instance, as shown in Figure 4, for 2D rotations the underlying graph is a cycle with 4 elements due to the underlying point group for 4-fold rotations being the cyclic group C4. Performing heat diffusion can be achieved by repeated neighborhood averaging over the cycle and yields a smoothed rotation mask that performs all rotations at the same time (rightmost image in Figure 4). We can extend the same approach to all lattice transformations: for instance, in the case of translation, the underlying graph is a grid and the smoothing operation is akin to convolution with a Gaussian kernel.

To train LATFORMER with smoothed masks, we compute two predictions: one with the non-smooth mask predicted by the model and one with a smoothed version of the same mask. The final loss is the sum of two cross-entropy losses calculated separately for the two predictions.

Figure 3: A LATFORMER layer (a) and an architecture for a Lattice translation expert (b). The LATFORMER layer (a)is a standard Transformer encoder layer augmented with a Lattice mask expert constrained to generate attention masks

Figure 4: Rotational smoothing can be obtained by heat diffusion over the cyclic graph of rotation masks.