As the developer of Llama, a large-scale open-source foundational language model, Meta believes that the computational power required for future model training will significantly increase.
On Tuesday, during Meta's second-quarter earnings call, Mark Zuckerberg revealed that the computational power needed to train Llama4 will be ten times that required for Llama3. He emphasized that despite this, Meta must build the capability to train models to avoid falling behind competitors.
Zuckerberg stated, "The computational demand for training Llama4 might be nearly ten times that of Llama3, and the computational requirements for future models will continue to increase." He also noted that it is difficult to predict the trends of future generations of models, but it is better to build the necessary capabilities in advance rather than too late, given the long preparation time required for new inference projects.
In April this year, Meta released Llama3 with 80 billion parameters. Last week, the company launched its upgraded version, Llama3.1405B, with 405 billion parameters, becoming Meta's largest open-source model.
Susan Li, Meta's Chief Financial Officer, also mentioned that the company is considering various data center projects and building capabilities for training future AI models. She stated that Meta expects this investment to increase capital expenditures in 2025.
It's worth noting that training large-scale language models is a capital-intensive business. In the second quarter of 2024, Meta's capital expenditures, driven by investments in servers, data centers, and network infrastructure, increased from $6.4 billion a year ago to $8.5 billion, a growth of nearly 33%.
Key Points:
🎯 Meta requires approximately ten times the computational power to train Llama4 compared to Llama3.
🎯 Meta anticipates that investments in building capabilities will increase capital expenditures in 2025.
🎯 Training large-scale language models is costly, with Meta's capital expenditures significantly rising in the second quarter.