CloTH-VTON+: Clothing Three-dimensional reconstruction for Hybrid image-based Virtual Try-ON


Matiur Rahman Minar
Thai Thanh Tuan
Heejune Ahn

Seoul National University of Science and Technology

IEEE Access 2021

[paper]
[bibtex]


We proposed a novel hybrid and fully automatic method for 3D clothing reconstruction from single image and applying it to image-based virtual try-on (VTON) for fashion clothing, which generates realistically deformed try-on results with superior quality.

Image-based virtual try-on (VTON) systems based on deep learning have attracted research and commercialinterests. Although they show their strengths in blending the person and try-on clothing image andsynthesizing the dis-occluded regions, their results for complex-posed persons are often unsatisfactorydue to the limitations in their geometry deformation and texture-preserving capacity. To address thesechallenges, we propose CloTH-VTON+ for seamlessly integrating the image-based deep learning methodsand the strength of the 3D model in shape deformation. Specifically, a fully automatic pipeline is developedfor 3D clothing model reconstruction and deformation using a reference human model: first, the try-onclothing is matched to the target clothing regions in the simple shaped reference human model, and thenthe 3D clothing model is reconstructed. The reconstructed 3D clothing model can generate a very naturalpose and shape transfer, retaining the textures of clothes. A clothing refinement network further refines thealignment, eliminating the misalignment due to the errors in human pose estimation and 3D deformation.The deformed clothing images are combined utilizing conditional generative networks to in-paint the dis-occluded areas and blend them all. Experiments on an existing benchmark dataset demonstrate that CloTH-VTON+ generates higher quality results in comparison to the state-of-the-art VTON systems and CloTH-VTON. CloTH-VTON+ can be incorporated into extended applications such as multi-pose guided and Video VTON.







Pipeline










Citation

M. R. Minar, T. T. Tuan and H. Ahn, "CloTH-VTON+: Clothing Three-Dimensional Reconstruction for Hybrid Image-Based Virtual Try-ON," in IEEE Access, vol. 9, pp. 30960-30978, 2021, doi: 10.1109/ACCESS.2021.3059701.

@article{{Minar_2021_Access,,
    title={CloTH-VTON+: Clothing Three-dimensional reconstruction for Hybrid image-based Virtual Try-ON},
    author={M. R. {Minar} and T. T. {Tuan} and H. {Ahn}},
    journal={IEEE Access},
    year={2021},
    volume={9},
    number={},
    pages={30960-30978},
    doi={10.1109/ACCESS.2021.3059701},
}




Acknowledgements

This webpage template was borrowed from Articulation-aware CSM, who borrowed it from some colorful folks.