Llama 3

What is Llama 3?

Llama 3 is Meta’s latest open-source next-generation large language model (LLM), available in 8-byte and 70-byte parameter sizes, marking another significant advancement in the open-source AI field. As the third generation of the Llama series, Llama 3 not only inherits the powerful capabilities of its predecessors but also provides more efficient and reliable AI solutions through a series of innovations and improvements. It aims to support a wide range of applications, including but not limited to programming, problem-solving, translation, and dialogue generation, through advanced natural language processing technology.

Llama 3 series models

Llama 3 is currently available in two models: the 8B (8 billion parameters) and the 70B (70 billion parameters) versions. These two models are designed to meet different levels of application needs, providing users with flexibility and freedom of choice.

Llama-3-8B : An 8B parameter model, this is a relatively small but efficient model with 8 billion parameters. It is designed for applications requiring fast inference and limited computing resources, while maintaining high performance standards.
Llama-3-70B : The 70B parameter model, a larger-scale model with 70 billion parameters. It can handle more complex tasks, providing deeper language understanding and generation capabilities, and is suitable for applications with higher performance requirements.

Later, Llama 3 will also release a model with 400B parameters, which is currently under training. Meta also stated that a detailed research paper will be published once the training of Llama 3 is complete.

Llama 3 official website entrance

Official project homepage : https://llama.meta.com/llama3/
GitHub model weights and code : https://github.com/meta-llama/llama3/
Hugging Face model : https://huggingface.co/collections/meta-llama/meta-llama-3-66214712577ca38149ebb2b6

Improvements in Llama 3

Parameter size : Llama 3 offers models with two parameter sizes: 8B and 70B. Compared to Llama 2, the increased number of parameters allows the model to capture and learn more complex language patterns.
Training dataset : Llama 3’s training dataset is 7 times larger than Llama 2’s, containing over 15 trillion tokens, including 4 times the amount of code data, which makes Llama 3 superior in understanding and generating code.
Model architecture : Llama 3 employs a more efficient tokenizer and Grouped Query Attention (GQA) technology, which improves the model’s inference efficiency and ability to handle long texts.
Performance improvements : Llama 3 achieves advancements in reducing false rejection rates, improving response alignment, and increasing model response diversity through improved pre-training and post-training processes.
Security : New trust and security tools such as Llama Guard 2, as well as Code Shield and CyberSec Eval 2, have been introduced to enhance the security and reliability of the model.
Multilingual support : Llama 3 incorporates high-quality non-English data in more than 30 languages into its pre-training data, laying the foundation for future multilingual capabilities.
Reasoning and code generation : Llama 3 demonstrates significantly improved capabilities in reasoning, code generation, and instruction following, making it more accurate and efficient in handling complex tasks.

Llama 3 Performance Evaluation

According to Meta’s official blog, the finely tuned Llama 3 8B model outperforms models of similar parameter size (Gemma 7B, Mistral 7B) in benchmark tests on datasets such as MMLU, GPQA, HumanEval, GSM-8K, and MATH. The finely tuned Llama 3 70B also outperforms models of similar size, such as Gemini Pro 1.5 and Claude 3 Sonnet, in benchmark tests on MLLU, HumanEval, and GSM-8K.

In addition, Meta developed a new high-quality human evaluation set containing 1800 prompts covering 12 key use cases: seeking advice, brainstorming, classification, closed-ended question answering, coding, creative writing, extraction, character/role creation, open-ended question answering, reasoning, rewriting, and summarizing. Human evaluators ranked Llama 3 based on this evaluation set, comparing it with competing models such as Claude Sonnet, Mistral Medium, and GPT-3.5. The results showed that Llama 3 performed exceptionally well in real-world scenarios, achieving a minimum win rate of 52.9%.

Llama 3 technical architecture

Decoder architecture : Llama 3 adopts a decoder-only architecture, which is a standard Transformer model architecture mainly used for natural language generation tasks.
Tokenizer and vocabulary : Llama 3 uses a tokenizer with 128K tokens, which enables the model to encode language more efficiently, thus significantly improving performance.
Grouped Query Attention (GQA) : To improve inference efficiency, Llama 3 employs GQA in both the 8B and 70B models. This technique reduces computation by grouping queries within the attention mechanism while maintaining model performance.
Long sequence processing : Llama 3 supports sequences of up to 8,192 tokens and uses masking to ensure that self-attention does not cross document boundaries, which is especially important for processing long text.
Pre-training dataset : Llama 3 was pre-trained on over 15TB of tokens. This dataset is not only massive in scale but also of high quality, providing rich linguistic information for the model.
Multilingual Data : To support multilingual capabilities, the Llama 3 pre-trained dataset contains more than 5% high-quality non-English data, covering more than 30 languages.
Data filtering and quality control : The Llama 3 development team developed a series of data filtering pipelines, including heuristic filters, NSFW (Not Suitable for the Workplace) filters, semantic deduplication methods, and text classifiers, to ensure high-quality training data.
Scalability and Parallelism : Llama 3 employs data parallelism, model parallelism, and pipeline parallelism during training, enabling the model to be trained efficiently on a large number of GPUs.
Instruction Fine-Tuning : Llama 3 further improves the model’s performance on specific tasks, such as dialogue and programming, by fine-tuning instructions on the basis of the pre-trained model.

How to use Llama 3

Developers

Meta has open-sourced its Llama 3 model on GitHub, Hugging Face, and Replicate. Developers can use tools such as Torchtune to customize and fine-tune Llama 3 to suit specific use cases and requirements. Interested developers can check out the official getting started guide and download and deploy it.

Official model download : https://llama.meta.com/llama-downloads
GitHub address : https://github.com/meta-llama/llama3/
Hugging Face website : https://huggingface.co/meta-llama
Replicate URL : https://replicate.com/meta

Regular users

For non-technical users who want to experience Llama 3, the following methods can be used:

Experience Meta’s newly launched Meta AI chat assistant (Note: Meta.AI is region-locked and only available in certain countries).
Experience it at Chat with Llama provided by Replicate : https://llama3.replicate.dev/
You can manually switch your model to Llama 3 using Hugging Chat ( https://huggingface.co/chat/ ).