COSAEMB: Contrastive Section-aware Aspect Embeddings for Scientific Articles

Research papers are long documents that contain information about various aspects such as background, prior work, methodology, and results. Existing works on scientific document representation learning only leverage the title and abstract of the paper. We present COSAEMB, a model that learns representations from the full-text of 97402 scientific papers from the S2ORC dataset. We present a novel supervised contrastive training framework for long documents using triplet loss and margin gradation. Our framework can be used to learn representations of long documents with any existing encoder-only transformer model without retraining it from scratch. COSAEMB shows improved performance on information retrieval from the paper’s full-text in comparison to models trained only on paper titles and abstracts. We also evaluate COSAEMB on SCIREPEVAL and CSFCube benchmarks, showing comparable performance with existing state-of-the-art models.