4M

Multi-modal and Multi-task Model Training Framework

InternationalSelectionImageMulti-modal learningTransformer model

4M is a framework for training multi-modal and multi-task models capable of handling various visual tasks and performing multi-modal conditional generation. The model demonstrates its generalizability and scalability through experimental analysis, laying the foundation for further exploration of multi-modal learning in vision and other domains.

Multi-modal and multi-task training paradigm
capable of predicting or generating any modality.
Transforms modalities into discrete token sequences for training on a unified Transformer encoder-decoder.
Supports prediction from partial inputs
enabling chained multi-modal generation.
Can generate any modality based on other modalities in any subset
achieving self-consistent prediction.
Supports fine-grained multi-modal generation and editing tasks
such as semantic segmentation or depth map generation.
Performs controllable multi-modal generation by weighting different conditions to control the output.

The 4M model is targeted towards researchers and developers in the computer vision and machine learning domains
especially those interested in multi-modal data processing and generative models. This technology has applications in image and video analysis
content creation
data augmentation
and multi-modal interaction scenarios.

Use the 4M model to generate a depth map and surface normal from an RGB image.
Use 4M for image editing
such as reconstructing a complete RGB image based on partial input.
In multi-modal retrieval
use the 4M model to retrieve corresponding images based on text descriptions.

Visit the 4M GitHub repository to access the code and pre-trained models.
Install the required dependencies and environment according to the documentation.
Download and load the pre-trained 4M model.
Prepare input data
which can be text