Author(s)

Hao Chen

Affiliation(s)

Queen Mary School Hainan, Beijing University of Posts and Telecommunications, Beijing 100876, China.

Corresponding Author

Hao Chen

ABSTRACT

Current controllable image generation methods predominantly rely on external architectural modifications, such as auxiliary control networks, which require substantial computational overhead and struggle to unify diverse control modalities including text, pose, depth, and sketches. These approaches fundamentally limit scalability and real-time applicability due to their additive nature and complex multi-condition integration challenges. We introduce HyperEdit, a novel hypernetwork-driven framework that achieves multi-modal controllable generation through dynamic parameter perturbation of pre-trained diffusion models, moving beyond external control paradigms toward intrinsic model adaptation. Our approach employs a unified hypernetwork that learns to map diverse control conditions—ranging from textual descriptions and pose skeletons to depth maps and edge sketches—into targeted parameter perturbations, enabling seamless integration of multiple modalities without architectural modifications to the base model. Through systematic perturbation discovery on carefully constructed condition-image pairs and progressive parameter injection strategies, HyperEdit demonstrates remarkable efficiency gains, achieving up to 6× faster inference compared to existing methods while requiring significantly fewer parameters. Extensive experiments across diverse control scenarios show that our unified framework not only maintains generation quality comparable to specialized control methods but also enables novel capabilities such as real-time condition mixing, dynamic editing strength adjustment, and reversible modifications. This work establishes a new paradigm for controllable generation that bridges the gap between research innovation and practical deployment requirements.

KEYWORDS

Model editing; Image generation; Hypernetwork

CITE THIS PAPER

Hao Chen. Beyond external control: hypernetwork-driven parameter editing for multi-modal image generation. Journal of Computer Science and Electrical Engineering. 2025, 7(5): 1-12. DOI: https://doi.org/10.61784/jcsee3072.

REFERENCES

[1] Goodfellow I J, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks. arXiv preprint arXiv:1406.2661, 2014.

[2] Tang Jian, Guo Haitao, Xia Heng, et al. A survey on image generation for industrial processes and its applications. Acta Automatica Sinica, 2024, 50(2): 211-240. DOI: 10.16383/j.aas.c230126.

[3] Liu Zerun, Yin Yufei, Xue Wenhao, et al. A survey on conditional guided image generation based on diffusion models. Journal of Zhejiang University (Science Edition), 2023, 50(6): 651-667. DOI: 10.3785/j.issn.1008-9497.2023.06.001.

[4] Jiang Rui, Zheng Guangcong, Li Teng, et al. Survey on multimodal controllable diffusion models. Journal of Computer Science and Technology, 2024. DOI: 10.1007/s11390-024-3814-0.

[5] Zhang L, Rao A, Agrawala M. Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023: 3836-3847.

[6] Mou C, Wang X, Xie L, et al. T2I-Adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. Proceedings of the AAAI Conference on Artificial Intelligence, 2024.

[7] Ye H, Zhang J, Liu S, Han X, et al. IP-Adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721, 2023.

[8] Li M, Yang T, Kuang H, et al. ControlNet++: Improving conditional controls with efficient consistency feedback. European Conference on Computer Vision (ECCV), 2024.

[9] Zavadski D, Feiden J F, Rother C. ControlNet-XS: Rethinking the control of text-to-image diffusion models as feedback-control systems. European Conference on Computer Vision (ECCV), arXiv preprint arXiv:2312.06573, 2024.

[10] Yang H, Han W, Zhou Y, et al. DC-ControlNet: Decoupling inter- and intra-element conditions in image generation with diffusion models. arXiv preprint arXiv:2502.14779, 2025.

[11] Li Ming, Wang Jianhua, Chen Siyuan. Research status of diffusion models in computer vision. CAAI Transactions on Intelligent Systems, 2024, 19(2): 234-248.

[12] Cao Yin, Qin Junping, Ma Qianli, et al. A survey on text-to-image generation. Journal of Zhejiang University (Engineering Science), 2024, 58(2): 219-238. DOI: 10.3785/j.issn.1008-973X.2024.02.001.

[13] Ha David, Dai Andrew, Le Quoc V. HyperNetworks. arXiv preprint arXiv:1609.09106, 2016.

[14] Deutsch Lior. Generating Neural Networks with Neural Networks. arXiv preprint arXiv:1801.01952, 2018.

[15] Li Zherui, Jiang Houcheng, Chen Hao, et al. Reinforced Lifelong Editing for Language Models. arXiv preprint arXiv:2502.05759, 2025.

[16] Hu E J, Shen Y, Wallis P, et al. LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations (ICLR), arXiv preprint arXiv:2106.09685, 2022.

[17] Wang Ruipeng, Fang Junfeng, Li Jiaqi, et al. ACE: Concept Editing in Diffusion Models without Performance Degradation. arXiv preprint arXiv:2503.08116, 2025.

[18] Tang Yuying, Zhang Ningning, Ciancia Mariana, et al. Exploring the Impact of AI-generated Image Tools on Professional and Non-professional Users in the Art and Design Fields. arXiv preprint arXiv:2406.10640, 2024.

[19] Yuan Jinghui, Chen Hao, Luo Renwei, et al. A Margin-Maximizing Fine-Grained Ensemble Method. arXiv preprint arXiv:2409.12849, 2024.