Being natural to everyone, language-based inputs have demonstrated effective for various tasks such as object detection and image generation. This paper for the first time presents a language-based system for interactive colorization of scene sketches, based on their semantic comprehension. Compared with prior scribble-based interfaces, which require a minimum level of professional skills, our language-based interface is more natural for novice users. The proposed system is built upon deep neural networks trained on a large-scale repository of scene sketches and cartoon-style color images with text descriptions. Given a scene sketch, our system allows users, via language-based instructions, to interactively localize and colorize specific object instances to meet various colorization requirements in a progressive way. We demonstrate the effectiveness of our approach via comprehensive experimental results including alternative studies, comparison with the state-of-the-art, and generalization user studies.
VIDEO
Publications
ACM Transactions on Graphics (SIGGRAPH Asia), 2019.
Keywords: deep neural networks; image segmentation; language-based editing; scene sketch; sketch colorization, interactive graphics, interactive perception, augmented communication
European Conference on Computer Vision (ECCV), 2018.
Keywords: sketch dataset, scene sketch, sketch segmentation, interactive graphics
Cited By
Sketch-Based Creativity Support Tools Using Deep Learning Artificial Intelligence for Human Computer Interaction: A Modern Approach.
Forrest Huang , Eldon Schoop , David Ha , Jeffrey Nichols , and John Canny .
source | cite | search
Emergent Graphical Conventions in a Visual Communication Game https://arxiv.org/pdf/2111.14210.pdf.
Shuwen Qiu , Sirui Xie , Lifeng Fan , Tao Gao , Song-Chun Zhu , and Yixin Zhu .
source | cite | search
Write-an-Animation: High-level Text-based Animation Editing with Character-Scene Interaction Computer Graphics Forum.
Jia-Qi Zhang , Xiang Xu , Zhi-Meng Shen , Ze-Huan Huang , Yang Zhao , Yan-Pei Cao , Pengfei Wan , and Miao Wang .
source | cite | search
One Sketch for All: One-Shot Personalized Sketch Segmentation https://arxiv.org/pdf/2112.10838.pdf.
Anran Qi , Yulia Gryaditskaya , Tao Xiang , and Yi-Zhe Song .
source | cite | search
FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context https://arxiv.org/pdf/2203.02113.pdf.
Pinaki Nath Chowdhury , Aneeshan Sain , Yulia Gryaditskaya , Ayan Kumar Bhunia , Tao Xiang , and Yi-Zhe Song .
source | cite | search
Delaunay Painting: Perceptual Image Colouring From Raster Contours with Gaps Computer Graphics Forum.
Amal Dev Parakkat , Pooran Memari , and Marie-Paule Cani .
source | cite | search
SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval Using Adaptive GCNs IEEE Transactions on Image Processing.
Fang Liu , Xiaoming Deng , Changqing Zou , Yu-Kun Lai , Keqi Chen , Ran Zuo , Cuixia Ma , Yong-Jin Liu , and Hongan Wang .
source | cite | search
A Review of Image and Video Colorization: From Analogies to Deep Learning Visual Informatics.
Shu-Yu Chen , Jia-Qi Zhang , You-You Zhao , Paul L. Rosin , Yu-Kun Lai , and Lin Gao .
source | cite | search
Exemplar-Based Sketch Colorization with Cross-Domain Dense Semantic Correspondence Mathematics.
Jinrong Cui , Haowei Zhong , Hailong Liu , and Yulu Fu .
source | cite | search
Neural Image Recolorization for Creative Domains 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).
Boyi Li , Serge Belongie , Ser-nam Lim , and Abe Davis .
source | cite | search
SketchMaker: Sketch Extraction and Reuse for Interactive Scene Sketch Composition ACM Transactions on Interactive Intelligent Systems.
Fang Liu , Xiaoming Deng , Jiancheng Song , Yu-Kun Lai , Yong-Jin Liu , Hao Wang , Cuixia Ma , Shengfeng Qin , and Hongan Wang .
source | cite | search
Modulating Bottom-Up and Top-Down Visual Processing Via Language-Conditional Filters 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).
Ilker Kesen , Ozan Arkan Can , Erkut Erdem , Aykut Erdem , and Deniz Yuret .
source | cite | search
Unsupervised Scene Sketch to Photo Synthesis arXiv.2209.02834.
Jiayun Wang , Sangryul Jeon , Stella X. Yu , Xi Zhang , Himanshu Arora , and Yu Lou .
source | cite | search
Cartoon Image Processing: A Survey International Journal of Computer Vision.
Yang Zhao , Diya Ren , Yuan Chen , Wei Jia , Ronggang Wang , and Xiaoping Liu .
source | cite | search
DrawMon: A Distributed System for Detection of Atypical Sketch Content in Concurrent Pictionary Games Proceedings of the 30th ACM International Conference on Multimedia.
Nikhil Bansal , Kartik Gupta , Kiruthika Kannan , Sivani Pentapati , and Ravi Kiran Sarvadevabhatla .
source | cite | search
Generating Compositional Color Representations From Text https://arxiv.org/abs/2109.10477.
Paridhi Maheshwari , Nihal Jain , Praneetha Vaddamanu , Dhananjay Raut , Shraiysh Vaishay , and Vishwa Vinay .
source | cite | search
Painting Style-Aware Manga Colorization Based on Generative Adversarial Networks 2021 IEEE International Conference on Image Processing (ICIP).
Yugo Shimizu , Ryosuke Furuta , Delong Ouyang , Yukinobu Taniguchi , Ryota Hinami , and Shonosuke Ishiwatari .
source | cite | search
Adversarial Segmentation Loss for Sketch Colorization 2021 IEEE International Conference on Image Processing (ICIP).
Samet Hicsonmez , Nermin Samet , Emre Akbas , and Pinar Duygulu .
source | cite | search
Focusing on Persons 4.
Xin Jin , Zhonglan Li , Ke Liu , Dongqing Zou , Xiaodong Li , Xingfan Zhu , Ziyin Zhou , Qilong Sun , and Qingyu Liu .
source | cite | search
Sketchy Scene Captioning: Learning Multi-level Semantic Information From Sparse Visual Scene Cues Lecture Notes in Computer Science.
Lian Zhou , Yangdong Chen , and Yuejie Zhang .
source | cite | search
Multi-style Chinese Art Painting Generation of Flowers IET Image Processing.
Feifei Fu , Jiancheng Lv , Chenwei Tang , and Mao Li .
source | cite | search
TediGAN: Text-Guided Diverse Face Image Generation and Manipulation Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Weihao Xia , Yujiu Yang , Jing-Hao Xue , and Baoyuan Wu .
source | cite | search
DanbooRegion: An Illustration Region Dataset Computer Vision –ECCV 2020.
Lvmin Zhang , Yi Ji , and Chunping Liu .
source | cite | search
SketchyDepth: From Scene Sketches to RGB-D Images ICCV 2021.
Gianluca Berardi , Samuele Salti , and Luigi Di Stefano .
source | cite | search
Generative Adversarial Networks\textendashEnabled Human\textendashArtificial Intelligence Collaborative Applications for Creative and Design Industries: A Systematic Review of Current Approaches and Trends Frontiers in Artificial Intelligence.
Rowan T. Hughes , Liming Zhu , and Tomasz Bednarz .
source | cite | search
XCI-Sketch: Extraction of Color Information From Images for Generation of Colored Outlines and Sketche https://arxiv.org/abs/2108.11554.
Harsh Rathod , Manisimha Varma , Parna Chowdhury , Sameer Saxena , V. Manushree , Ankita Ghosh , and Sahil Khose .
source | cite | search
Generating Compositional Color Representations From Text https://arxiv.org/abs/2109.10477.
Paridhi Maheshwari , Nihal Jain , Praneetha Vaddamanu , Dhananjay Raut , Shraiysh Vaishay , and Vishwa Vinay .
source | cite | search
Text As Neural Operator:Image Manipulation by Text Instruction MM '21: Proceedings of the 29th ACM International Conference on Multimedia.
Tianhao Zhang , Hung-Yu Tseng , Lu Jiang , Weilong Yang , Honglak Lee , and Irfan Essa .
source | cite | search
DLA-Net for FG-SBIR 5.
Jiaqing Xu , Haifeng Sun , Qi Qi , Jingyu Wang , Ce Ge , Lejian Zhang , and Jianxin Liao .
source | cite | search
Grayscale Image Colorization Using a Convolutional Neural Network Journal of the Korean Society for Industrial and Applied Mathematics.
Minje Jwa and Myungjoo Kang .
source | cite | search
Exploring Local Detail Perception for Scene Sketch Semantic Segmentation IEEE Transactions on Image Processing.
Ce Ge , Haifeng Sun , Yi-Zhe Song , Zhanyu Ma , and Jianxin Liao .
source | cite | search
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input arXiv.2203.14804.
Pinaki Nath Chowdhury , Ayan Kumar Bhunia , Viswanatha Reddy Gajjala , Aneeshan Sain , Tao Xiang , and Yi-Zhe Song .
source | cite | search
ColorizeDiffusion: Adjustable Sketch Colorization with Reference Image and Text arXiv.2401.01456.
Dingkun Yan , Liang Yuan , Yuma Nishioka , Issei Fujishiro , and Suguru Saito .
source | cite | search
OpenSketch: A Richly-annotated Dataset of Product Design Sketches ACM Transactions on Graphics.
Yulia Gryaditskaya , Mark Sypesteyn , Jan Willem Hoftijzer , Sylvia Pont , Frédo Dur , and Adrien Bousseau .
source | cite | search
Language-based Photo Color Adjustment for Graphic Designs ACM Transactions on Graphics.
Zhenwei Wang , Nanxuan Zhao , Gerhard Hancke , and Rynson W.H. Lau .
source | cite | search
Semi-supervised Reference-based Sketch Extraction Using a Contrastive Learning Framework ACM Transactions on Graphics.
Chang Wook Seo , Amirsaman Ashtari , and Junyong Noh .
website , source | cite | search
Stroke-based Semantic Segmentation for Scene-level Free-hand Sketches The Visual Computer.
Zhengming Zhang , Xiaoming Deng , Jinyao Li , Yukun Lai , Cuixia Ma , Yongjin Liu , and Hongan Wang .
source | cite | search
Deep Learning for Studying Drawing Behavior: A Review Frontiers in Psychology.
Benjamin Beltzung , Marie Pel{\'{e}} , Julien P. Renoult , and C{\'{e}}dric Sueur .
source | cite | search
Human\textendashmachine Hybrid Intelligence for the Generation of Car Frontal Forms Advanced Engineering Informatics.
Yu Wu , Lisha Ma , Xiaofang Yuan , and Qingnan Li .
source | cite | search
SVCNet: Scribble-based Video Colorization Network with Temporal Aggregation arXiv.2303.11591.
Yuzhi Zhao , Lai-Man Po , Kangcheng Liu , Xuehui Wang , Wing-Yin Yu , Pengfei Xian , Yujia Zhang , and Mengyang Liu .
source | cite | search
Adversarial Interactive Cartoon Sketch Colourization with Texture Constraint and Auxiliary Auto-Encoder Computer Graphics Forum.
Xiaoyu Liu , Shaoqiang Zhu , Yao Zeng , and Junsong Zhang .
source | cite | search
Fine-Grained Video Retrieval with Scene Sketches IEEE Transactions on Image Processing.
Ran Zuo , Xiaoming Deng , Keqi Chen , Zhengming Zhang , Yu-Kun Lai , Fang Liu , Cuixia Ma , Hao Wang , Yong-Jin Liu , and Hongan Wang .
source | cite | search
A State-of-Art Review on~Intelligent Systems for~Drawing Assisting Lecture Notes in Computer Science.
Juexiao Qin , Xiaohua Sun , and Weijian Xu .
source | cite | search
Improving Sketch Colorization Using Adversarial Segmentation Consistency arXiv.2301.08590.
Samet Hicsonmez , Nermin Samet , Emre Akbas , and Pinar Duygulu .
source | cite | search
CustomSketching: Sketch Concept Extraction for Sketch-based Image Synthesis and Editing arXiv.2402.17624.
Chufeng Xiao and Hongbo Fu .
source | cite | search
Learning Inclusion Matching for Animation Paint Bucket Colorization arXiv.2403.18342.
Yuekun Dai , Shangchen Zhou , Qinyue Li , Chongyi Li , and Chen Change Loy .
source | cite | search
CreativeSeg: Semantic Segmentation of Creative Sketches IEEE Transactions on Image Processing.
Yixiao Zheng , Kaiyue Pang , Ayan Das , Dongliang Chang , Yi-Zhe Song , and Zhanyu Ma .
source | cite | search
