恭喜盈瑜 (Yola) 以優異成績通過碩士論文口試。論文題目為「文本意圖的多模態分析:以 Instagram 為例 (An Analysis of Multimodal Document Intent in Instagram Posts)」。

摘要如下：

時至今日,社群媒體(如 Instagram)趨向結合圖片以及文字表徵,建構出一種
新的「多模態」溝通方式。利用計算方法分析多模態關係已成為一個熱門的主題,然而,尚未有研究針對台灣的百大網紅發文中的多模態圖文配對(Image-caption Pair)來分析文本意圖和圖文關係。利用文字和圖片的多模態表徵,本研究沿用 Kruk et al. (2019) 的圖文關係分類方法(contextual relationship/semiotic relationship/authors intent), 對此三種分類提出新的圖文表徵方式(Sentence-
BERT 及 image embedding), 並利用計算模型(Random Forest, Decision Tree Classifier)精準分類三種圖文關係,研究結果顯示正確率高達 86.23%。

Present-day, a majority of representation style on social media (i.e., Instagram) tends to combine visual and textual content in the same message as a consequence of building up a modern way of communication. Effective computational approaches for understanding documents with multiple modalities are needed to identify the relationship between them. This study extends recent advances in authors’ intent classification by putting forward an approach using Image-caption Pair(ICPs). Several Maching Learning algorithms like Decision Tree Classifier (DTC), Random Forest (RF), and encoders like Sentence-BERT and picture embedding are undertaken in the tasks in order to classify the relationships between multiple modalities, which are 1) contextual relationship 2) semiotic relationship and 3) authors intent. This study points to two possible results. First, despite the prior studies consider incorporating the two synergistic modalities in a combined model will improve the accuracy in the relationship classification task, this study does not prove the preconceived notion. The results suggest that the incorporating of text and image needs more effort to complement each other. Second, we show that these text-image relationships can be classified with high accuracy (86.23%) by using only text modality. To conclude, this study may be of essential in demonstrating a computational approach to access multimodal documents as well as providing a better understanding of classifying the relationships between modalities.

keywords: Instagram, multimodal documents understanding, contexual relationship, semiotic relationship, authors intent, Natural Language Processing, Decision Tree Classifier, Random Forest, Sentence-BERT, image embedding