Reformer: The Efficient Transformer
E899034
Reformer: The Efficient Transformer is a research paper introducing a more memory- and computation-efficient Transformer architecture using techniques like locality-sensitive hashing attention and reversible layers.
All labels observed (1)
| Label | Occurrences |
|---|---|
| Reformer: The Efficient Transformer canonical | 1 |
How this entity was disambiguated
This entity first appeared as the object of triple T11003397 — resolving that mention is where its identity was fixed. The disambiguator weighed these candidate entities and picked the highlighted one (or “None”, minting a new entity). This is how homonymy is resolved: the same surface form can point to different entities.
NED1
Entity disambiguation (via context triple)
gpt-5-mini-2025-08-07
Target entity: Reformer: The Efficient Transformer Context triple: [Łukasz Kaiser, coAuthorOf, Reformer: The Efficient Transformer]
-
A.
Attention Is All You Need
"Attention Is All You Need" is the landmark 2017 research paper that introduced the Transformer architecture and revolutionized modern natural language processing and sequence modeling.
-
B.
Transformer-XL
Transformer-XL is a neural network architecture for language modeling that extends the Transformer with segment-level recurrence and relative positional encodings to better capture long-range dependencies.
-
C.
Language Models are Unsupervised Multitask Learners
"Language Models are Unsupervised Multitask Learners" is a 2019 OpenAI research paper that demonstrated how large-scale unsupervised language models like GPT-2 can perform a wide range of tasks without task-specific training.
-
D.
Swin Transformer
Swin Transformer is a hierarchical vision transformer architecture that uses shifted windows for efficient and scalable image recognition and related computer vision tasks.
-
E.
Longformer
Longformer is a transformer-based neural network architecture designed for efficient processing of very long sequences using sparse attention mechanisms.
- F. None of above. chosen
- G. Unsure - the case is ambiguous/there is not enough information to decide.
NED2
Entity disambiguation (via description)
gpt-5-mini-2025-08-07
Target entity: Reformer: The Efficient Transformer Target entity description: Reformer: The Efficient Transformer is a research paper introducing a more memory- and computation-efficient Transformer architecture using techniques like locality-sensitive hashing attention and reversible layers.
-
A.
Attention Is All You Need
"Attention Is All You Need" is the landmark 2017 research paper that introduced the Transformer architecture and revolutionized modern natural language processing and sequence modeling.
-
B.
Transformer-XL
Transformer-XL is a neural network architecture for language modeling that extends the Transformer with segment-level recurrence and relative positional encodings to better capture long-range dependencies.
-
C.
Language Models are Unsupervised Multitask Learners
"Language Models are Unsupervised Multitask Learners" is a 2019 OpenAI research paper that demonstrated how large-scale unsupervised language models like GPT-2 can perform a wide range of tasks without task-specific training.
-
D.
Swin Transformer
Swin Transformer is a hierarchical vision transformer architecture that uses shifted windows for efficient and scalable image recognition and related computer vision tasks.
-
E.
Longformer
Longformer is a transformer-based neural network architecture designed for efficient processing of very long sequences using sparse attention mechanisms.
- F. None of above. chosen
Statements (48)
| Predicate | Object |
|---|---|
| instanceOf |
machine learning paper
ⓘ
neural network architecture ⓘ research paper ⓘ scientific publication ⓘ |
| addresses |
quadratic memory complexity of self-attention
ⓘ
quadratic time complexity of self-attention ⓘ |
| aimsTo |
enable training on very long sequences
ⓘ
reduce computational cost of Transformers ⓘ reduce memory usage of Transformers ⓘ |
| applicationDomain |
language modeling
ⓘ
long-context tasks ⓘ sequence modeling ⓘ |
| assumes | similar tokens attend mostly to each other ⓘ |
| basedOn | Transformer architecture ⓘ |
| category | efficient Transformer variant ⓘ |
| complexityClaim | reduces attention complexity from O(L^2) to approximately O(L log L) ⓘ |
| contribution |
demonstrates training on sequences with tens of thousands of tokens
ⓘ
shows LSH attention can approximate full attention with lower cost ⓘ shows reversible layers can significantly reduce activation memory ⓘ |
| field |
deep learning
ⓘ
machine learning ⓘ natural language processing ⓘ neural networks ⓘ |
| goal | scale Transformers to longer sequences without prohibitive resource usage ⓘ |
| improvesOn | standard Transformer ⓘ |
| influencedBy |
Reversible residual networks
NERFINISHED
ⓘ
locality-sensitive hashing ⓘ |
| introducesTechnique |
chunked feed-forward layers
ⓘ
locality-sensitive hashing attention ⓘ reversible residual layers ⓘ shared query-key projections for attention ⓘ |
| LSHAttentionProperty |
computes attention only within buckets
ⓘ
groups similar queries into buckets ⓘ |
| optimizationTarget |
memory efficiency
ⓘ
time efficiency ⓘ |
| proposes | Reformer architecture ⓘ |
| relatedTo |
Linformer
NERFINISHED
ⓘ
Longformer NERFINISHED ⓘ Performer ⓘ Transformer NERFINISHED ⓘ sparse attention models ⓘ |
| reversibleLayerProperty |
reconstructs intermediate activations during backpropagation
ⓘ
stores only activations at boundaries between layers ⓘ |
| title | Reformer: The Efficient Transformer NERFINISHED ⓘ |
| uses |
approximate nearest neighbor search via LSH
ⓘ
position-wise feed-forward networks ⓘ reversible layers to recompute activations ⓘ self-attention mechanism ⓘ |
How these facts were elicited
The pipeline generated the facts above by prompting gpt-5.1 with this entity's name + description and the instruction below.
Instruction
You are a knowledge base construction expert. Given a subject entity and a description of it, return factual statements that you know for the subject as a JSON list of dictionaries(triples), where keys must be "subject", "predicate" and "object". The number of facts may be very high, between 25 to 50 or more, for very popular subjects. For less popular subjects, the number of facts can be very low, like 5 or 10. # Requirements - If you don't know the subject at all, return an empty list. - If the subject is not a named entity, return an empty list. - Include at least one triple where predicate is "instanceOf". - Do not get too wordy. - Separate several objects into multiple triples with one object.
Input
Subject: Reformer: The Efficient Transformer Description of subject: Reformer: The Efficient Transformer is a research paper introducing a more memory- and computation-efficient Transformer architecture using techniques like locality-sensitive hashing attention and reversible layers.
Referenced by (1)
Full triples — surface form annotated when it differs from this entity's canonical label.
subject surface form:
Łukasz Kaiser