Recently, an open-source project named Ultralight-Digital-Human has garnered significant attention in the developer community. This project has successfully addressed the challenges of deploying digital human technology on mobile devices, enabling ordinary smartphones to run digital human applications in real-time, thereby opening new possibilities for the popularization of related technologies.

This ultra-lightweight digital human model employs innovative deep learning techniques, optimizing algorithms and compressing models to effectively "slim down" the massive digital human system to a level where it can run smoothly on mobile devices. The system supports real-time processing of video and audio inputs and can quickly synthesize digital human images with prompt and smooth performance.

image.png

Technically, the project integrates Wenet and Hubert audio feature extraction solutions, allowing developers to choose flexibly based on specific application scenarios. Additionally, by introducing the syncnet technology, the lip-sync effect of digital humans has been significantly improved. To ensure smooth operation on mobile devices, the development team employed parameter pruning techniques during the training and deployment process, effectively reducing the demand for computational resources.

Another highlight of the project is the provision of comprehensive training process documentation. Developers only need to prepare a high-quality facial video of 3-5 minutes to start training their digital human model according to the guide. The system has clear requirements for videos; Wenet mode requires a frame rate of 20fps, while Hubert mode requires 25fps.

To ensure training effectiveness, the project team specifically reminds developers to pay attention to the following key aspects: using a pre-trained model as a base; ensuring the quality of training data; regularly monitoring the training process; and adjusting training parameters as needed. These details will directly impact the final digital human effect.

Currently, this open-source project has shown great potential in areas such as social applications, mobile games, and virtual reality. Compared to traditional digital human technology, it not only lowers the hardware threshold but also achieves cross-platform compatibility, allowing stable operation on various smartphones.

Project link: https://github.com/anliyuan/Ultralight-Digital-Human