Photo Filter Recommendation by Category-Aware Aesthetic Learning

Wei-Tse Sun, Ting-Hsuan Chao, Yin-Hsi Kuo, Winston Hsu

Abstract

Nowadays, social media have become popular platforms for the public to share photos. To make photos more visually appealing, most social media provide filters by which users can change the appearance of their photos without domain knowledge. However, due to the growing number of filter types, it becomes a major issue for users to choose the best filter type. For this purpose, filter recommendation for photo aesthetics takes an important role in image quality ranking problems. In these years, several works have declared that Convolutional Neural Networks (CNNs) outperform traditional methods in image aesthetic categorization, which classifies images into high or low quality. However, most of them do not consider the effect on filtered images. In this paper, we propose a novel method for image aesthetic learning for filter recommendation. Instead of binarizing image quality, we adjust the state-of-the-art CNN architectures and design a pairwise ranking loss function to learn the aesthetic response for filtered images. By utilizing pairwise image comparison, the models embed aesthetic responses in the hidden layers. Moreover, based on our pilot study, we observe that users usually prefer different filters for various image categories (e.g., portrait, landscape, food). To improve the filter aesthetic ranking, we further integrate the automatically detected image category into our proposed aesthetic-oriented models. Meanwhile, to the best of our knowledge, there is no public dataset for aesthetics judgement with filtered images. We create a new dataset called Filter Aesthetic Comparison Dataset (FACD). The dataset contains around 28,000 filtered images based on the AVA dataset and 42,240 reliable image pairs with aesthetic comparison annotations using Amazon Mechanical Turk. It is the first dataset containing filtered images and the user preference labels. Based on the collected FACD, we conduct experiments on our proposed methods for photo filter recommendation. The experimental results show that our method which learns aesthetic ranking by category-aware pairwise learning outperforms the traditional aesthetic classification methods (e.g., 12\% relative improvement).

Publication

  • Wei-Tse Sun, Ting-Hsuan Chao, Yin-Hsi Kuo, Winston H. Hsu. "Photo Filter Recommendation by Category-Aware Aesthetic Learning," IEEE Transactions on Multimedia, 2017. [pdf]

Dataset

Please notice that this dataset is made available for academic research purpose only. The copyright of all images belongs to the photographers/dpchallenge.

Filter Aesthetic Comparison Dataset (FACD)

Filter Aesthetic Comparison Dataset (FACD) contains 1,280 unfiltered images collected from the AVA dataset within 8 categories. It also includes 28,160 filtered images and 42,240 image pairs. The filtered images are generated by applying 22 different filter types provided by GIMP toolkit on the unfiltered images. The image pairs are sampled from the filtered images and annotated by users on Amazon Mechanical Turk. The pairwise comparison results are used for both training and evaluation in this paper.
The entire dataset includes two zip files:
  • FACD_image - contains directories of the original and filtered images
    • original images - Origin
    • filtered images - 1977, Amaro, Apollp, Brannan, Earlybird, Gotham, Hefe, Hudson, Inkwell, Lofi, LordKevin, Mayfair, Nashville, Poprocket, Rise, Sierra, Sutro, Toaster, Valencia, Walden, Willow, and XProII
  • FACD_metadata - contains two pkl files about the information of the images and the pairs
    • pairwise_comparison.pkl
      • category - image category
      • imgId - image ID
      • f1 - the first filter
      • f2 - the second filter
      • ans - user preference between two filtered images
      • score - the aesthetic score computed from comparison results
      • class - high or low quality
    • image_score.pkl
      • filterName - filter name
      • imgId - image ID
      • score - the aesthetic score computed from comparison results
      • class - high or low quality
  • The zip file containing the images can be downloaded [here] (2.76G)
  • The zip file containing the metadata can be downloaded [here] (1.15M)

Notes
*The details of data collection and annotation are described in this paper.
*The image quality label and the aesthetic score are generated from the comparison results rather than directly annotated by human. Thus we do not suggest to apply the labels to other works about binary quality classification.

Please contact Wilson Sun for any questions/problems/bug reports/etc.