Embeddings from Language Models
E771680
deep contextual word representation
natural language processing method
neural network model
word embedding technique
Embeddings from Language Models (ELMo) is a deep contextual word representation technique that uses bidirectional language models to capture rich, context-dependent meanings of words for natural language processing tasks.
Statements (52)
| Predicate | Object |
|---|---|
| instanceOf |
deep contextual word representation
ⓘ
natural language processing method ⓘ neural network model ⓘ word embedding technique ⓘ |
| basedOn | bidirectional language models ⓘ |
| captures |
context-dependent word meaning
ⓘ
semantic information ⓘ syntactic information ⓘ |
| combinationMethod | learned weighted sum of internal layers ⓘ |
| combines |
backward language model representations
ⓘ
forward language model representations ⓘ |
| comparedTo |
GloVe
NERFINISHED
ⓘ
word2vec NERFINISHED ⓘ |
| developedAt |
Allen Institute for Artificial Intelligence
NERFINISHED
ⓘ
University of Washington NERFINISHED ⓘ |
| developedBy |
Christopher Clark
NERFINISHED
ⓘ
Kenton Lee NERFINISHED ⓘ Luke Zettlemoyer NERFINISHED ⓘ Mark Neumann NERFINISHED ⓘ Matt Gardner NERFINISHED ⓘ Matthew E. Peters NERFINISHED ⓘ Mohit Iyyer NERFINISHED ⓘ |
| differenceFromStaticEmbeddings | context-dependent representations ⓘ |
| embeddingDimension | 1024 ⓘ |
| hasAbbreviation | ELMo NERFINISHED ⓘ |
| implementedIn | AllenNLP NERFINISHED ⓘ |
| improves |
coreference resolution performance
ⓘ
named entity recognition performance ⓘ question answering performance ⓘ semantic role labeling performance ⓘ textual entailment performance ⓘ |
| influenced |
BERT
NERFINISHED
ⓘ
GPT-style contextual embeddings ⓘ |
| inputRepresentation | character-based ⓘ |
| inputUnit | word ⓘ |
| layerTypes |
character CNN layer
ⓘ
first BiLSTM layer ⓘ second BiLSTM layer ⓘ |
| license | Apache License 2.0 ⓘ |
| numLayers | 3 ⓘ |
| pretrainedOn | 1 Billion Word Benchmark NERFINISHED ⓘ |
| produces | contextualized word embeddings ⓘ |
| publicationTitle | Deep contextualized word representations NERFINISHED ⓘ |
| publicationYear | 2018 ⓘ |
| publishedIn | NAACL 2018 NERFINISHED ⓘ |
| representationLevel | token-level ⓘ |
| trainingDirection |
backward
ⓘ
forward ⓘ |
| trainingObjective | language modeling ⓘ |
| usage | feature-based transfer learning ⓘ |
| usesArchitecture |
bidirectional LSTM
ⓘ
character-level convolutional neural network ⓘ |
Referenced by (1)
Full triples — surface form annotated when it differs from this entity's canonical label.