MotionGPT
MotionGPT: Human Motion as Foreign Language
Last updated
MotionGPT: Human Motion as Foreign Language
Last updated
MotionGPT works by first converting 3D motion into motion tokens, similar to the generation process of word tokens. This allows MotionGPT to learn the semantic coupling of motion and language. Once the motion is converted into tokens, MotionGPT can then be used to generate text descriptions of the motion, or to generate new motions based on a text description.
Though the advancement of pre-trained large language models unfolds, the exploration of building a unified model for language and other multi-modal data, such as motion, remains challenging and untouched so far.
Fortunately, human motion displays a semantic coupling akin to human language, often perceived as a form of body language. By fusing language data with large-scale motion models, motion-language pre-training that can enhance the performance of motion-related tasks becomes feasible.
MotionGPT is driven by this insight. It is a unified, versatile, and user-friendly motion-language model to handle multiple motion-relevant tasks. Specifically, they employ the discrete vector quantization for human motion and transfer 3D motion into motion tokens, similar to the generation process of word tokens. Building upon this “motion vocabulary”, they perform language modeling on both motion and text in a unified manner, treating human motion as a specific language.
Moreover, inspired by prompt learning, they pre-train MotionGPT with a mixture of motion-language data and fine-tune it on prompt-based question-and-answer tasks.
Extensive experiments demonstrate that MotionGPT achieves state-of-the-art performances on multiple motion tasks including text-driven motion generation, motion captioning, motion prediction, and motion in-between.
To involve large language data and models in the motion generation tasks, we propose a unified motion-language framework named MotionGPT. MotionGPT consists of a motion tokenizer responsible for converting raw motion data into discrete motion tokens, as well as a motion-aware language model that learns to understand the motion tokens from large language pre-training models by corresponding textual descriptions.
MotionGPT has been shown to be effective on a variety of motion-related tasks, including:
Motion captioning: Generating text descriptions of human motion.
Motion generation: Generating new human motions based on a text description.
Motion retrieval: Finding similar motions to a given motion.
Motion editing: Editing a motion to make it more realistic or to fit a specific purpose.
MotionGPT is a powerful tool that can be used to generate and manipulate human motion. It has the potential to be used in a variety of applications, such as:
Virtual reality: Creating realistic and immersive virtual environments.
Animation: Creating realistic and expressive animations.
Sports: Training athletes and analyzing their performance.
Medicine: Rehabilitation and therapy.
MotionGPT is still under development, but it has the potential to revolutionize the way we interact with human motion.
