Extractive summarization of documents with images based on multi-modal RNN

Jingqiang Chen, Hai Zhuge*

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


Rapid growth of multi-modal documents containing images on the Internet expresses strong demand on multi-modal summarization. The challenge is to create a computing method that can uniformly process text and image. Deep learning provides basic models for meeting this challenge. This paper treats extractive multi-modal summarization as a classification problem and proposes a sentence–image classification method based on the multi-modal RNN model. Our method encodes words and sentences with the hierarchical RNN models and encodes the ordered image set with the CNN model and the RNN model, and then calculates the selection probability of sentences and the sentence–image alignment probability through a logistic classifier taking text coverage, text redundancy, image set coverage, and image set redundancy as features. Two methods are proposed to compute the image set redundancy feature by combining the important scores of sentences and the hidden sentence–image alignment. Experiments on the extended DailyMail corpora constructed by collecting images and captions from the Web show that our method outperforms 11 baseline text summarization methods and that adopting the two image-related features in the classification method can improve text summarization. Our method is able to mine the hidden sentence–image alignments and to create informative well-aligned multi-modal summaries.

Original languageEnglish
Pages (from-to)186-196
Number of pages11
JournalFuture Generation Computer Systems
Early online date25 Apr 2019
Publication statusPublished - 1 Oct 2019

Bibliographical note

© 2019, Elsevier. Licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International http://creativecommons.org/licenses/by-nc-nd/4.0/

Funding: National Natural Science Foundation of China (No. 61806101, No. 61876048, No. 61602256, No. 61876091), and the Open Foundation of Key Laboratory of Intelligent Information Processing, ICT, CAS, China (IIP2019-2).


  • Document summarization
  • Extractive summarization
  • Multi-modal summarization
  • RNN
  • Summarization


Dive into the research topics of 'Extractive summarization of documents with images based on multi-modal RNN'. Together they form a unique fingerprint.

Cite this