Vol.67, No.2, 2021, pp.1963-1977, doi:10.32604/cmc.2021.014931
OPEN ACCESS
ARTICLE
Image-Based Lifelogging: User Emotion Perspective
  • Junghyun Bum1, Hyunseung Choo1, Joyce Jiyoung Whang2,*
1 College of Computing, Sungkyunkwan University, Suwon, 16417, Korea
2 School of Computing, KAIST, Daejeon, 34141, Korea
* Corresponding Author: Joyce Jiyoung Whang. Email:
Received 28 October 2020; Accepted 13 December 2020; Issue published 05 February 2021
Abstract
Lifelog is a digital record of an individual’s daily life. It collects, records, and archives a large amount of unstructured data; therefore, techniques are required to organize and summarize those data for easy retrieval. Lifelogging has been utilized for diverse applications including healthcare, self-tracking, and entertainment, among others. With regard to the image-based lifelogging, even though most users prefer to present photos with facial expressions that allow us to infer their emotions, there have been few studies on lifelogging techniques that focus upon users’ emotions. In this paper, we develop a system that extracts users’ own photos from their smartphones and configures their lifelogs with a focus on their emotions. We design an emotion classifier based on convolutional neural networks (CNN) to predict the users’ emotions. To train the model, we create a new dataset by collecting facial images from the CelebFaces Attributes (CelebA) dataset and labeling their facial emotion expressions, and by integrating parts of the Radboud Faces Database (RaFD). Our dataset consists of 4,715 high-resolution images. We propose Representative Emotional Data Extraction Scheme (REDES) to select representative photos based on inferring users’ emotions from their facial expressions. In addition, we develop a system that allows users to easily configure diaries for a special day and summaize their lifelogs. Our experimental results show that our method is able to effectively incorporate emotions into lifelog, allowing an enriched experience.
Keywords
Lifelog; facial expression; emotion; emotion classifier; transfer learning
Cite This Article
J. Bum, H. Choo and J. J. Whang, "Image-based lifelogging: user emotion perspective," Computers, Materials & Continua, vol. 67, no.2, pp. 1963–1977, 2021.
This work is licensed under a Creative Commons Attribution 4.0 International License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.