publications

2024

1. Double Reverse Diffusion for Realistic Garment Reconstruction from Images

   Jeonghaeng Lee, Duc Nguyen, Jongyoo Kim, Jiwoo Kang, and Sanghoon Lee

   Engineering Applications of Artificial Intelligence, Jan 2024

   Abs

   Creating realistic digital 3D avatars has been getting more attention thanks to the introduction of new multimedia formats such as augmented and virtual reality. An important factor making avatars realistic is clothes. In this paper, we investigate a new method to reconstruct realistic garments from a set of images and body information. Early methods working on realistic images struggle to faithfully reconstruct the garment details. As deep learning is increasingly applied to geometric data which can conveniently represent garments, we devise a novel deep learning-based solution to the garment reconstruction problem. We offer a new perspective on the reconstruction problem and treat it as a reversion of the smoothing diffusion process. To achieve this goal, we propose to deform the smoothed human mesh into a clothed human via a Double Reverse Diffusion (DReD) process. For the first reverse diffusion, we introduce a novel operator called Graph Long Short-Term Memory (GraphLSTM) which recursively diffuses features to produce a deformed mesh by modeling the relationships between vertices. Then, the output mesh can be repeatedly upsampled and deformed by the above pipeline to obtain finer garment details, which can be seen as another reverse diffusion process. To obtain features for the reverse diffusion, we extract pixel-aligned features transferred from images and explore to incorporate the visibility of garments from the image viewpoints. Through detailed experiments on two public datasets, we demonstrate that DReD synthesizes more realistic wrinkled garments with lower errors and offers faster inference than previous methods.

2. Single-Image 3-D Reconstruction: Rethinking Point Cloud Deformation

   Anh-Duc Nguyen, Seonghwa Choi, Woojae Kim, Jongyoo Kim, Heeseok Oh, Jiwoo Kang, and Sanghoon Lee

   IEEE Transactions on Neural Networks and Learning Systems, May 2024

   Abs

   Single-image 3-D reconstruction has long been a challenging problem. Recent deep learning approaches have been introduced to this 3-D area, but the ability to generate point clouds still remains limited due to inefficient and expensive 3-D representations, the dependency between the output and the number of model parameters, or the lack of a suitable computing operation. In this article, we present a novel deep-learning-based method to reconstruct a point cloud of an object from a single still image. The proposed method can be decomposed into two steps: feature fusion and deformation. The first step extracts both global and point-specific shape features from a 2-D object image, and then injects them into a randomly generated point cloud. In the second step, which is deformation, we introduce a new layer termed as GraphX that considers the interrelationship between points like common graph convolutions but operates on unordered sets. The framework can be applicable to realistic image data with background as we optionally learn a mask branch to segment objects from input images. To complement the quality of point clouds, we further propose an objective function to control the point uniformity. In addition, we introduce different variants of GraphX that cover from best performance to best memory budget. Moreover, the proposed model can generate an arbitrary-sized point cloud, which is the first deep method to do so. Extensive experiments demonstrate that we outperform the existing models and set a new height for different performance metrics in single-image 3-D reconstruction.

2023

1. FreeEnricher: Enriching Face Landmarks without Additional Cost

   Yangyu Huang, Xi Chen, Jongyoo Kim, Hao Yang, Chong Li, Jiaolong Yang, and Dong Chen

   In AAAI Conference on Artificial Intelligence , May 2023

   Abs

   Recent years have witnessed significant growth of face alignment. Though dense facial landmark is highly demanded in various scenarios, e.g., cosmetic medicine and facial beautification, most works only consider sparse face alignment. To address this problem, we present a framework that can enrich landmark density by existing sparse landmark datasets, e.g., 300W with 68 points and WFLW with 98 points. Firstly, we observe that the local patches along each semantic contour are highly similar in appearance. Then, we propose a weakly-supervised idea of learning the refinement ability on original sparse landmarks and adapting this ability to enriched dense landmarks. Meanwhile, several operators are devised and organized together to implement the idea. Finally, the trained model is applied as a plug-and-play module to the existing face alignment networks. To evaluate our method, we manually label the dense landmarks on 300W testset. Our method yields state-of-the-art accuracy not only in newly-constructed dense 300W testset but also in the original sparse 300W and WFLW testsets without additional cost.

2. MNET++: Music-Driven Pluralistic Dancing Toward Multiple Dance Genre Synthesis

   Jinwoo Kim, Beom Kwon, Jongyoo Kim, and Sanghoon Lee

   IEEE Transactions on Pattern Analysis and Machine Intelligence, Dec 2023

   Abs

   Numerous task-specific variants of autoregressive networks have been developed for dance generation. Nonetheless, a severe limitation remains in that all existing algorithms can return repeated patterns for a given initial pose, which may be inferior. We examine and analyze several key challenges of previous works, and propose variations in both model architecture (namely MNET++) and training methods to address these. In particular, we devise the beat synchronizer and dance synthesizer. First, generated dance should be locally and globally consistent with given music beats, circumvent repetitive patterns, and look realistic. To achieve this, the beat synchronizer implicitly catches the rhythm enabling it to stay in sync with the music as it dances. Then, the dance synthesizer infers the dance motions in a seamless patch-by-patch manner conditioned by music. Second, to generate diverse dance lines, adversarial learning is performed by leveraging the transformer architecture. Furthermore, MNET++ learns a dance genre-aware latent representation that is scalable for multiple domains to provide fine-grained user control according to the dance genre. Compared with the state-of-the-art methods, our method synthesizes plausible and diverse outputs according to multiple dance genres as well as generates remarkable dance sequences qualitatively and quantitatively.

2022

1. MPS-NeRF: Generalizable 3D Human Rendering From Multiview Images

   Xiangjun Gao, Jiaolong Yang, Jongyoo Kim, Sida Peng, Zicheng Liu, and Xin Tong

   IEEE Transactions on Pattern Analysis and Machine Intelligence, Dec 2022

   Abs

   There has been rapid progress recently on 3D human rendering, including novel view synthesis and pose animation, based on the advances of neural radiance fields (NeRF). However, most existing methods focus on person-specific training and their training typically requires multi-view videos. This paper deals with a new challenging task – rendering novel views and novel poses for a person unseen in training, using only multiview still images as input without videos. For this task, we propose a simple yet surprisingly effective method to train a generalizable NeRF with multiview images as conditional input. The key ingredient is a dedicated representation combining a canonical NeRF and a volume deformation scheme. Using a canonical space enables our method to learn shared properties of human and easily generalize to different people. Volume deformation is used to connect the canonical space with input and target images and query image features for radiance and density prediction. We leverage the parametric 3D human model fitted on the input images to derive the deformation, which works quite well in practice when combined with our canonical NeRF. The experiments on both real and synthetic data with the novel view synthesis and pose animation tasks collectively demonstrate the efficacy of our method.

2021

1. ADNet: Leveraging Error-Bias Towards Normal Direction in Face Alignment

   Yangyu Huang, Hao Yang, Chong Li, Jongyoo Kim, and Fangyun Wei

   In IEEE/CVF International Conference on Computer Vision (ICCV) , Oct 2021

   Abs

   The recent progress of CNN has dramatically improved face alignment performance. However, few works have paid attention to the error-bias with respect to error distribution of facial landmarks. In this paper, we investigate the error-bias issue in face alignment, where the distributions of landmark errors tend to spread along the tangent line to landmark curves. This error-bias is not trivial since it is closely connected to the ambiguous landmark labeling task. Inspired by this observation, we seek a way to leverage the error-bias property for better convergence of CNN model. To this end, we propose anisotropic direction loss (ADL) and anisotropic attention module (AAM) for coordinate and heatmap regression, respectively. ADL imposes strong binding force in normal direction for each landmark point on facial boundaries. On the other hand, AAM is an attention module which can get anisotropic attention mask focusing on the region of point and its local edge connected by adjacent points, it has a stronger response in tangent than in normal, which means relaxed constraints in the tangent. These two methods work in a complementary manner to learn both facial structures and texture details. Finally, we integrate them into an optimized end-to-end training pipeline named ADNet. Our ADNet achieves state-of-the-art results on 300W, WFLW and COFW datasets, which demonstrates the effectiveness and robustness.

2. Diverse and Adjustable Versatile Image Enhancer

   Woojae Kim, Anh-Duc Nguyen, Jinwoo Kim, Jongyoo Kim, Heeseok Oh, and Sanghoon Lee

   IEEE Access, Oct 2021
3. Learning High-Fidelity Face Texture Completion without Complete Face Texture

   Jongyoo Kim, Jiaolong Yang, and Xin Tong

   In IEEE/CVF International Conference on Computer Vision (ICCV) , Oct 2021

   Abs

   For face texture completion, previous methods typically use some complete textures captured by multiview imaging systems or 3D scanners for supervised learning. This paper deals with a new challenging problem - learning to complete invisible texture in a single face image without using any complete texture. We simply leverage a large corpus of face images of different subjects (e. g., FFHQ) to train a texture completion model in an unsupervised manner. To achieve this, we propose DSD-GAN, a novel deep neural network based method that applies two discriminators in UV map space and image space. These two discriminators work in a complementary manner to learn both facial structures and texture details. We show that their combination is essential to obtain high-fidelity results. Despite the network never sees any complete facial appearance, it is able to generate compelling full textures from single images.

2019

1. Deep CNN-Based Blind Image Quality Predictor

   Jongyoo Kim, Anh-Duc Nguyen, and Sanghoon Lee

   IEEE Transactions on Neural Networks and Learning Systems, Jan 2019

   Abs

   Image recognition based on convolutional neural networks (CNNs) has recently been shown to deliver the state-of-the-art performance in various areas of computer vision and image processing. Nevertheless, applying a deep CNN to no-reference image quality assessment (NR-IQA) remains a challenging task due to critical obstacles, i.e., the lack of a training database. In this paper, we propose a CNN-based NR-IQA framework that can effectively solve this problem. The proposed method—deep image quality assessor (DIQA)—separates the training of NR-IQA into two stages: 1) an objective distortion part and 2) a human visual system-related part. In the first stage, the CNN learns to predict the objective error map, and then the model learns to predict subjective score in the second stage. To complement the inaccuracy of the objective error map prediction on the homogeneous region, we also propose a reliability map. Two simple handcrafted features were additionally employed to further enhance the accuracy. In addition, we propose a way to visualize perceptual error maps to analyze what was learned by the deep CNN model. In the experiments, the DIQA yielded the state-of-the-art accuracy on the various databases.

2. Video Frame Synthesis Via Plug-and-Play Deep Locally Temporal Embedding

   Anh-Duc Nguyen, Woojae Kim, Jongyoo Kim, Weisi Lin, and Sanghoon Lee

   IEEE Access, Jan 2019

2018

1. Deep Blind Image Quality Assessment by Learning Sensitivity Map

   Jongyoo Kim, Woojae Kim, and Sanghoon Lee

   In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , Jan 2018
2. Deep Video Quality Assessor: From Spatio-Temporal Visual Sensitivity to a Convolutional Neural Aggregation Network

   Woojae Kim, Jongyoo Kim, Sewoong Ahn, Jinwoo Kim, and Sanghoon Lee

   In European Conference on Computer Vision , Jan 2018
3. Multiple Level Feature-Based Universal Blind Image Quality Assessment Model

   Jongyoo Kim, Anh-Duc Nguyen, Sewoong Ahn, Chong Luo, and Sanghoon Lee

   In IEEE International Conference on Image Processing (ICIP) , Jan 2018
4. Deep Visual Saliency on Stereoscopic Images

   Anh-Duc Nguyen, Jongyoo Kim, Heeseok Oh, Haksub Kim, Weisi Lin, and Sanghoon Lee

   IEEE Transactions on Image Processing, Jan 2018
5. Video Frame Interpolation by Plug-and-Play Deep Locally Linear Embedding

   Anh-Duc Nguyen, Woojae Kim, Jongyoo Kim, and Sanghoon Lee

   arXiv preprint arXiv:1807.01462, Jan 2018

2017

1. Deep Blind Image Quality Assessment by Employing FR-IQA

   Jongyoo Kim, and Sanghoon Lee

   In IEEE International Conference on Image Processing (ICIP) , Jan 2017
2. Deep Convolutional Neural Models for Picture-Quality Prediction: Challenges and Solutions to Data-Driven Image Quality Assessment

   Jongyoo Kim, Hui Zeng, Deepti Ghadiyaram, Sanghoon Lee, Lei Zhang, and Alan C. Bovik

   IEEE Signal Processing Magazine, Nov 2017

   Abs

   Convolutional neural networks (CNNs) have been shown to deliver standout performance on a wide variety of visual information processing applications. However, this rapidly developing technology has only recently been applied with systematic energy to the problem of picture-quality prediction, primarily because of limitations imposed by a lack of adequate ground-truth human subjective data. This situation has begun to change with the development of promising data-gathering methods that are driving new approaches to deep-learning-based perceptual picture-quality prediction. Here, we assay progress in this rapidly evolving field, focusing, in particular, on new ways to collect large quantities of ground-truth data and on recent CNN-based picture-quality prediction models that deliver excellent results in a large, real-world, picture-quality database.

3. Deep Learning of Human Visual Sensitivity in Image Quality Assessment Framework

   Jongyoo Kim, and Sanghoon Lee

   In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , Nov 2017
4. Fully Deep Blind Image Quality Predictor

   Jongyoo Kim, and Sanghoon Lee

   IEEE Journal of Selected Topics in Signal Processing, Feb 2017

   Abs

   In general, owing to the benefits obtained from original information, full-reference image quality assessment (FR-IQA) achieves relatively higher prediction accuracy than no-reference image quality assessment (NR-IQA). By fully utilizing reference images, conventional FR-IQA methods have been investigated to produce objective scores that are close to subjective scores. In contrast, NR-IQA does not consider reference images; thus, its performance is inferior to that of FR-IQA. To alleviate this accuracy discrepancy between FR-IQA and NR-IQA methods, we propose a blind image evaluator based on a convolutional neural network (BIECON). To imitate FR-IQA behavior, we adopt the strong representation power of a deep convolutional neural network to generate a local quality map, similar to FR-IQA. To obtain the best results from the deep neural network, replacing hand-crafted features with automatically learned features is necessary. To apply the deep model to the NR-IQA framework, three critical problems must be resolved: 1) lack of training data; 2) absence of local ground truth targets; and 3) different purposes of feature learning. BIECON follows the FR-IQA behavior using the local quality maps as intermediate targets for conventional neural networks, which leads to NR-IQA prediction accuracy that is comparable with that of state-of-the-art FR-IQA methods.

5. Quality Assessment of Perceptual Crosstalk on Two-View Auto-Stereoscopic Displays

   Jongyoo Kim, Taewan Kim, Sanghoon Lee, and Alan Conrad Bovik

   IEEE Transactions on Image Processing, Feb 2017
6. An Identification Framework for Print-Scan Books in a Large Database

   Sang-Hoon Lee, Jongyoo Kim, and Sanghoon Lee

   Information sciences, Feb 2017
7. Blind Deep S3D Image Quality Evaluation via Local to Global Feature Aggregation

   Heeseok Oh, Sewoong Ahn, Jongyoo Kim, and Sanghoon Lee

   IEEE Transactions on Image Processing, Feb 2017
8. Enhancement of Visual Comfort and Sense of Presence on Stereoscopic 3D Images

   Heeseok Oh, Jongyoo Kim, Jinwoo Kim, Taewan Kim, Sanghoon Lee, and Alan Conrad Bovik

   IEEE Transactions on Image Processing, Feb 2017

2016

1. Blind Sharpness Prediction for Ultrahigh-Definition Video Based on Human Visual Resolution

   Haksub Kim, Jongyoo Kim, Taegeun Oh, and Sanghoon Lee

   IEEE Transactions on Circuits and Systems for Video Technology, Feb 2016
2. No-Reference Perceptual Sharpness Assessment for Ultra-High-Definition Images

   Woojae Kim, Haksub Kim, Heeseok Oh, Jongyoo Kim, and Sanghoon Lee

   In IEEE International Conference on Image Processing (ICIP) , Feb 2016
3. Perceptual Crosstalk Prediction on Autostereoscopic 3D Display

   Taewan Kim, Jongyoo Kim, SeongYong Kim, Sungho Cho, and Sanghoon Lee

   IEEE Transactions on Circuits and Systems for Video Technology, Feb 2016
4. Framework Implementation of Image-Based Indoor Localization System Using Parallel Distributed Computing

   Beom Kwon, Donghyun Jeon, Jongyoo Kim, Junghwan Kim, Doyoung Kim, Hyewon Song, and Sanghoon Lee

   The Journal of Korean Institute of Communications and Information Sciences, Feb 2016

2015

1. Implementation of an Omnidirectional Human Motion Capture System Using Multiple Kinect Sensors

   Junghwan Kim, Inwoong Lee, Jongyoo Kim, and Sanghoon Lee

   IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, Feb 2015
2. Video Sharpness Prediction Based on Motion Blur Analysis

   Jongyoo Kim, Junghwan Kim, Woojae Kim, Jisoo Lee, and Sanghoon Lee

   In IEEE International Conference on Multimedia and Expo (ICME) , Feb 2015
3. Implementation of Human Action Recognition System Using Multiple Kinect Sensors

   Beom Kwon, Doyoung Kim, Junghwan Kim, Inwoong Lee, Jongyoo Kim, Heeseok Oh, Haksub Kim, and Sanghoon Lee

   In Advances in Multimedia Information Processing–PCM 2015 , Feb 2015
4. 3D Visual Discomfort Predictor Based on Neural Activity Statistics

   Heeseok Oh, Jongyoo Kim, Sanghoon Lee, and Alan C Bovik

   In IEEE International Conference on Image Processing (ICIP) , Feb 2015

2014

1. Quality Assessment of Perceptual Crosstalk in Autostereoscopic Display

   Jongyoo Kim, Taewan Kim, and Sanghoon Lee

   In IEEE International Conference on Image Processing (ICIP) , Feb 2014