Phase-Associative Memory
Phase-Associative Memory (PAM) is a new sequence model using complex-valued representations. It achieves 30.0 validation perplexity on WikiText-103, close to a matched transformer's 27.1.
Researchers have introduced Phase-Associative Memory (PAM), a recurrent sequence model that utilizes complex-valued representations. In PAM, associations are accumulated in a matrix state via outer products, and retrieval is operated through the conjugate inner product. This approach has been tested on WikiText-103, where it reached a validation perplexity of 30.0 with approximately 100M parameters.
The significance of PAM lies in its ability to perform competitively with transformer models, despite the additional arithmetic overhead from complex computation. Notably, PAM's performance is within 10% of a matched transformer trained under identical conditions, which achieved a validation perplexity of 27.1. This suggests that PAM could offer a viable alternative for sequence modeling tasks, particularly where complex-valued representations are beneficial.
As PAM is a new development in the field of sequence modeling, its future applications and potential improvements are yet to be fully explored. Further research could focus on optimizing PAM's performance, exploring its suitability for various tasks, and understanding the implications of using complex-valued representations in sequence modeling.