The Abstraction and Reasoning Corpus: Related Work

cover
11 Mar 2024

This paper is available on arxiv under CC 4.0 license.

Authors:

(1) Mattia Atzeni, EPFL, Switzerland and [email protected];

(2) Mrinmaya Sachan, ETH Zurich, Switzerland;

(3) Andreas Loukas, Prescient Design, Switzerland.

Our work was inspired by a previous investigation of selfattention layers which identified sufficient conditions such that they can perform convolution when equipped with relative positional encodings (Cordonnier et al., 2020; Andreoli, 2019). Rather than relying on relative encodings, we here show how soft-masking can be used to learn sample efficiently more general input transformations, such as rotation, reflection, and scaling.

To the extent of our knowledge, the group-action learning problem has not been explicitly and generally formulated in previous work. That being said, many previous works have focused on specific instances, such as learning to sort (Graves et al., 2014; Reed & De Freitas, 2015; Li et al., 2020) by selecting an element of the permutation group Sn, docking/folding by roto-translating objects according to an action in the special Euclidean group SE(3) (Sverrisson et al., 2022; Stark et al. ¨ , 2022; Jumper et al., 2021), and graph spectrum generation where the learned actions belong to the Stiefel manifold (Martinkus et al., 2022).

Our work is similar in spirit to recent efforts in neurosymbolic visual reasoning (Johnson et al., 2017b;a; Goyal et al., 2017; Mao et al., 2019; Higgins et al., 2018) and in the area of enhancing machine-learning models with the ability to reason over structured data (Atzeni et al., 2021; Murugesan et al., 2021a; Atzeni & Atzori, 2018; Murugesan et al., 2021b). Many approaches based on attention mechanisms have been proposed in the past few years (Hudson & Manning, 2018; 2019). Our work differentiates from previous lines of research in that we aim to learn basic geometric reasoning in a sample-efficient way, rather than modeling relationships between high-level concepts.

Finally, some recent works came to our same conclusion on the advantages of using attention masks to incorporate prior knowledge in neural networks. As an example, Yan et al. (2020) focus on the task of learning subroutines (e.g., sorting algorithms) and use a CNN to generate an attention mask for a Transformer encoder. Similarly, Sartran et al. (2022) used precomputed attention masks to incorporate syntactic biases in language models.