Llama 3.1论文中文对照翻译

The Llama 3 Herd of Models

模型群 Llama 3

Llama Team, Al @ Meta 1 {}^{1} 1

Llama 团队，Meta @ Al 1 {}^{1} 1

1 {}^{1} 1 A detailed contributor list can be found in the appendix of this paper.

1 {}^{1} 1 详细的贡献者名单可在本文附录中找到。

Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405    B {405}\mathrm{\;B} 405B parameters and a context window of up to 128    K {128}\mathrm{\;K} 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.

现代人工智能（AI）系统由基础模型驱动。本文介绍了一组新的基础模型，称为 Llama 3。它是一群原生支持多语言、编码、推理和工具使用的语言模型。我们最大的模型是一个密集型 Transformer，具有 405    B {405}\mathrm{\;B} 405B 参数和高达 128    K {128}\mathrm{\;K} 128K 个令牌的上下文窗口。本文对 Llama 3 进行了广泛的实证评估。我们发现 Llama 3 在众多任务上与 GPT-4 等领先语言模型相比质量相当。我们公开发布了 Llama 3，包括 405B 参数语言模型的预训练和后训练版本以及用于输入和输出安全的 Llama Guard 3 模型。本文还展示了通过组合方法将图像、视频和语音功能集成到 Llama 3 中的实验结果。我们观察到这种方法在图像、视频和语音识别任务上与最先进的技术表现竞争。由于这些模型仍在开发中，因此尚未广泛发布。

Date: July 23, 2024

日期：2024年7月23日

Website: https://llama.meta.com/

网站：https://llama.meta.com/

1 Introduction 引言

Foundation models are general models of language, vision, speech, and/or other modalities that are designed to support a large variety of AI tasks. They form the basis of many modern AI systems.

基础模型是用于语言、视觉、语音及其他模态的通用模型，旨在支持多种AI任务。它们构成了许多现代AI系统的基础。

The development of modern foundation models consists of two main stages: (1) a pre-training stage in which the model is trained at massive scale using straightforward tasks such as next-word prediction or captioning and (2) a post-training stage in which the model is tuned to follow instructions, align with human preferences, and improve specific capabilities (for example, coding and reasoning).

现代基础模型的开发包括两个主要阶段：（1）预训练阶段，模型通过简单的任务如下一个词预测或字幕生成进行大规模训练；（2）后训练阶段，模型调整以遵循指令、与人类偏好对齐并提升特定能力（例如，编码和推理）。

In this paper, we present a new set of foundation models for language, called LIama 3. The Llama 3 Herd of models natively supports multilinguality, coding, reasoning, and tool usage. Our largest model is dense Transformer with 405    B {405}\mathrm{\;B} 405B parameters,processing information in a context window of up to 128    K {128}\mathrm{\;K} 128K tokens. Each member of the herd is listed in Table 1. All the results presented in this paper are for the Llama 3.1 models, which we will refer to as Llama 3 throughout for brevity.

在本文中，我们提出了一套新的语言基础模型，称为 LIama 3。Llama 3 模型群天然支持多语言、编码、推理和工具使用。我们最大的模型是密集型 Transformer，具有 405    B {405}\mathrm{\;B} 405B 参数，在最多 128    K {128}\mathrm{\;K} 128K 个词元的上下文窗口中处理信息。模型群中的每个成员均列于表 1。本文展示的所有结果均针对 Llama 3.1 模型，为简洁起见，全文将简称其为 Llama 3。

We believe there are three key levers in the development of high-quality foundation models: data, scale, and managing complexity. We seek to optimize for these three levers in our development process:

我们认为，高质量基础模型的发展有三个关键杠杆：数据、规模和管理复杂性。我们在开发过程中力求优化这三个方面：

Data. Compared to prior versions of Llama (Touvron et al., 2023a,b), we improved both the quantity and quality of the data we use for pre-training and post-training. These improvements include the development of more careful pre-processing and curation pipelines for pre-training data and the development of more rigorous quality assurance and filtering approaches for post-training data. We pre-train Llama 3 on a corpus of about 15T multilingual tokens, compared to 1.8T tokens for Llama 2.

数据。与之前的 Llama 版本（Touvron 等人，2023a,b）相比，我们不仅提高了预训练和后训练所用数据的数量，还提升了其质量。这些改进包括开发更细致的预训练数据预处理和筛选流程，以及开发更严格的后训练数据质量保证和过滤方法。我们使用约 15T 的多语言词元对 Llama 3 进行预训练，而 Llama 2 的预训练数据量为 1.8T 词元。

Scale. We train a model at far larger scale than previous Llama models: our flagship language model was pre-trained using 3.8 × 10 25 {3.8} \times {10}^{25} 3.8×1025 FLOPs,almost 50 × {50} \times 50× more than the largest version of Llama 2. Specifically, we pre-trained a flagship model with 405    B {405}\mathrm{\;B} 405B trainable parameters on 15.6    T {15.6}\mathrm{\;T} 15.6T text tokens. As expected per scaling laws for foundation models, our flagship model outperforms smaller models trained using the same procedure. While our scaling laws suggest our flagship model is an approximately compute-optimal size for our training budget, we also train our smaller models for much longer than is compute-optimal. The resulting models perform better than compute-optimal models at the same inference budget. We use the flagship model to further improve the quality of those smaller models during post-training.

规模。我们训练的模型规模远超以往的 Llama 模型：我们的旗舰语言模型使用 3.8 × 10 25 {3.8} \times {10}^{25} 3.8×1025 FLOPs 进行预训练，几乎比 Llama 2 的最大版本多 50 × {50} \times 50×。具体来说，我们在 15.6    T {15.6}\mathrm{\;T} 15.6T 个文本标记上预训练了一个具有 405    B {405}\mathrm{\;B} 405B 个可训练参数的旗舰模型。根据基础模型的缩放定律预期，我们的旗舰模型优于使用相同程序训练的较小模型。尽管我们的缩放定律表明我们的旗舰模型对于我们的训练预算来说是一个近似计算最优的大小，但我们的小型模型训练时间远超过计算最优时间。因此，这些模型在相同的推理预算下表现优于计算最优模型。我们使用旗舰模型在训练后进一步提高那些较小模型的质量。

Table 1 Overview of the Llama 3 Herd of models. All results in this paper are for the Llama 3.1 models.

表1 Llama 3 模型群概览。本文中的所有结果均针对 Llama 3.1 模型。

Managing complexity. We make design choices that seek to maximize our ability to scale the model development process. For example, we opt for a standard dense Transformer model architecture (Vaswani et al., 2017) with minor adaptations, rather than for a mixture-of-experts model (Shazeer et al., 2017) to maximize training stability. Similarly, we adopt a relatively simple post-training procedure based on supervised finetuning (SFT), rejection sampling (RS), and direct preference optimization (DPO; Rafailov et al. (2023)) as opposed to more complex reinforcement learning algorithms (Ouyang et al., 2022; Schulman et al., 2017) that tend to be less stable and harder to scale.

管理复杂性。我们做出设计选择，以最大化我们扩展模型开发过程的能力。例如，我们选择了一个标准的密集 Transformer 模型架构（Vaswani 等人，2017），并进行了微小的调整，而不是选择混合专家模型（Shazeer 等人，2017）以最大化训练稳定性。同样，我们采用了一个相对简单的训练后程序，基于监督微调（SFT）、拒绝采样（RS）和直接偏好优化（DPO；Rafailov 等人（2023）），而不是更复杂的强化学习算法（Ouyang 等人，2022；Schulman 等人，2017），这些算法往往稳定性较差且更难扩展。

The result of our work is Llama 3: a herd of three multilingual 1 {}^{1} 1 language models with 8    B , 70    B 8\mathrm{\;B},{70}\mathrm{\;B} 8B,70B ,and 405    B {405}\mathrm{\;B} 405B parameters. We evaluate the performance of Llama 3 on a plethora of benchmark datasets that span a wide range of language understanding tasks. In addition, we perform extensive human evaluations that compare Llama 3 with competing models. An overview of the performance of the flagship Llama 3 model on key benchmarks is presented in Table 2. Our experimental evaluation suggests that our flagship model performs on par with leading language models such as GPT-4 (OpenAI, 2023a) across a variety of tasks, and is close to matching the state-of-the-art. Our smaller models are best-in-class, outperforming alternative models with similar numbers of parameters (Bai et al., 2023; Jiang et al., 2023). Llama 3 also delivers a much better balance between helpfulness and harmlessness than its predecessor (Touvron et al., 2023b). We present a detailed analysis of the safety of Llama 3 in Section 5.4.

我们的工作成果是 Llama 3：一个由三个多语言 1 {}^{1} 1 语言模型组成的群体，具有 8    B , 70    B 8\mathrm{\;B},{70}\mathrm{\;B} 8B,70B 和 405    B {405}\mathrm{\;B} 405B 参数。我们在涵盖广泛语言理解任务的大量基准数据集上评估了 Llama 3 的性能。此外，我们还进行了广泛的人工评估，将 Llama 3 与竞争模型进行比较。旗舰 Llama 3 模型在关键基准上的性能概览见表 2。我们的实验评估表明，我们的旗舰模型在各种任务上与 GPT-4（OpenAI，2023a）等领先语言模型表现相当，并且接近达到最先进水平。我们的小型模型表现最佳，优于具有相似参数数量的替代模型（Bai 等，2023；Jiang 等，2023）。Llama 3 在有用性和无害性之间提供了比其前身更好的平衡（Touvron 等，2023b）。我们在第 5.4 节中详细分析了 Llama 3 的安全性。

We are publicly releasing all three Llama 3 models under an updated version of the Llama 3 Community License; see https://llama.meta.com. This includes pre-trained and post-trained versions of our 405B parameter language model and a new version of our Llama Guard model (Inan et al., 2023) for input and output safety. We hope that the open release of a flagship model will spur a wave of innovation in the research community, and accelerate a responsible path towards the development of artificial general intelligence (AGI).

我们根据更新版本的 Llama 3 社区许可证公开发布了所有三个 Llama 3 模型；详情请参见 https://llama.meta.com。这包括我们 405B 参数语言模型的预训练和后训练版本，以及我们 Llama Guard 模型（Inan 等，2023）的新版本，用于输入和输出安全。我们希望旗舰模型的开放发布将激发研究社区的创新浪潮，并加速通往负责任的人工通用智能（AGI）发展之路。

As part of the Llama 3 development process we also develop multimodal extensions to the models, enabling image recognition, video recognition, and speech understanding capabilities. These models are still under active development and not yet ready for release. In addition to our language modeling results, the paper presents results of our initial experiments with those multimodal models.

作为Llama 3开发过程的一部分，我们还开发了多模态扩展模型，使其具备图像识别、视频识别和语音理解能力。这些模型仍在积极开发中，尚未准备好发布。除了我们的语言建模成果外，本文还展示了我们与这些多模态模型初步实验的结果。

1 {}^{1} 1 The Llama 38B and 70B were pre-trained on multilingual data but were intended for use in English at the time.

1 {}^{1} 1 Llama 38B和70B在多语言数据上进行了预训练，但当时旨在用于英语。

Table 2 Performance of finetuned Llama 3 models on key benchmark evaluations. The table compares the performance of the 8    B , 70    B 8\mathrm{\;B},{70}\mathrm{\;B} 8B,70B ,and 405    B {405}\mathrm{\;B} 405B versions of Llama 3 with that of competing models. We boldface the best-performing model in each of three model-size equivalence classes. A Results obtained using 5-shot prompting (no CoT). Results obtained without CoT. ⋄ {}^{\diamond } ⋄ Results obtained using zero-shot prompting.

表2 微调后的Llama 3模型在关键基准评估中的性能。该表比较了Llama 3的 8    B , 70    B 8\mathrm{\;B},{70}\mathrm{\;B} 8B,70B和 405    B {405}\mathrm{\;B} 405B 版本与竞争模型的性能。我们在每个三个模型大小等价类别中加粗显示了表现最佳的模型。A 使用5次提示（无CoT）获得的结果。无CoT获得的结果。 ⋄ {}^{\diamond } ⋄ 使用零次提示获得的结果。

2 General Overview 总体概述

The model architecture of Llama 3 is illustrated in Figure 1. The development of our Llama 3 language models comprises two main stages:

Llama 3的模型架构如图1所示。我们的Llama 3语言模型的开发包括两个主要阶段：

Language model pre-training. We start by converting a large, multilingual text corpus to discrete tokens and pre-training a large language model (LLM) on the resulting data to perform next-token prediction. In the language model pre-training stage, the model learns the structure of language and obtains large amounts of knowledge about the world from the text it is “reading”. To do this effectively, pre-training is performed at massive scale: we pre-train a model with 405    B {405}\mathrm{\;B} 405B parameters on 15.6    T {15.6}\mathrm{\;T} 15.6T tokens using a context window of 8    K 8\mathrm{\;K} 8K tokens. This standard pre-training stage is followed by a continued pre-training stage that increases the supported context window to 128    K {128}\mathrm{\;K} 128K tokens. See Section 3 for details.

语言模型预训练。我们首先将一个大型多语言文本语料库转换为离散的标记，并在生成的数据上预训练一个大型语言模型（LLM）以进行下一个标记预测。在语言模型预训练阶段，模型学习语言的结构，并从其“阅读”的文本中获取大量关于世界的知识。为了有效实现这一点，预训练是在大规模上进行的：我们在 405    B {405}\mathrm{\;B} 405B个参数上预训练一个模型，使用 15.6    T {15.6}\mathrm{\;T} 15.6T个标记的上下文窗口为 8    K 8\mathrm{\;K} 8K个标记。这一标准预训练阶段之后是一个继续预训练阶段，将支持的上下文窗口增加到 128    K {128}\mathrm{\;K} 128K个标记。详见第3节。

Language model post-training. The pre-trained language model has a rich understanding of language but it does not yet follow instructions or behave in the way we would expect an assistant to. We align the model with human feedback in several rounds, each of which involves supervised finetuning (SFT) on instruction tuning data and Direct Preference Optimization (DPO; Rafailov et al., 2024). At this post-training 2 {}^{2} 2 stage,we also integrate new capabilities,such as tool-use,and observe strong improvements in other areas, such as coding and reasoning. See Section 4 for details. Finally, safety mitigations are also incorporated into the model at the post-training stage, the details of which are described in Section 5.4.

语言模型后训练。预训练的语言模型对语言有丰富的理解，但它还不遵循指令或以我们期望助手的方式行事。我们通过几轮与人类反馈的对齐，每一轮都包括在指令调优数据上的监督微调（SFT）和直接偏好优化（DPO；Rafailov et al., 2024）。在这个后训练 2 {}^{2} 2 阶段，我们还集成了新的能力，例如工具使用，并在其他领域如编码和推理方面观察到显著改进。详见第4节。最后，安全缓解措施也在后训练阶段被纳入模型，详情见第5.4节。

The resulting models have a rich set of capabilities. They can answer questions in at least eight languages, write high-quality code, solve complex reasoning problems, and use tools out-of-the-box or in a zero-shot way.

由此产生的模型具有丰富的能力。它们至少能用八种语言回答问题，编写高质量的代码，解决复杂的推理问题，并能开箱即用地或以零样本方式使用工具。

We also perform experiments in which we add image, video, and speech capabilities to Llama 3 using a compositional approach. The approach we study comprises the three additional stages illustrated in Figure 28:

我们还进行了实验，通过组合方法将图像、视频和语音能力添加到Llama 3中。我们研究的方法包括图28所示的三个额外阶段：

Multi-modal encoder pre-training. We train separate encoders for images and speech. We train our image encoder on large amounts of image-text pairs. This teaches the model the relation between visual content and the description of that content in natural language. Our speech encoder is trained using a

多模态编码器预训练。我们为图像和语音训练单独的编码器。我们在大量的图像-文本对上训练我们的图像编码器。这教会了模型视觉内容与自然语言中对该内容的描述之间的关系。我们的语音编码器是使用

2 {}^{2} 2 In this paper,we use the term “post-training” to refer to any model training that happens outside of pre-training.

2 {}^{2} 2 在本文中，我们使用术语“后训练”来指代在预训练之外进行的任何模型训练。

Figure 1 Illustration of the overall architecture and training of Llama 3. Llama 3 3 3 is a Transformer language model trained to predict the next token of a textual sequence. See text for details.

图1 Llama 3的整体架构和训练示意图。Llama 3 3 3 是一个Transformer语言模型，训练用于预测文本序列的下一个词元。详见正文。

self-supervised approach that masks out parts of the speech inputs and tries to reconstruct the masked out parts via a discrete-token representation. As a result, the model learns the structure of speech signals. See Section 7 for details on the image encoder and Section 8 for details on the speech encoder.

一种自监督方法，该方法掩盖语音输入的部分内容，并尝试通过离散标记表示来重建被掩盖的部分。因此，模型学习了语音信号的结构。有关图像编码器的详细信息，请参见第7节；有关语音编码器的详细信息，请参见第8节。

Vision adapter training. We train an adapter that integrates the pre-trained image encoder into the pre-trained language model. The adapter consists of a series of cross-attention layers that feed image-encoder representations into the language model. The adapter is trained on text-image pairs. This aligns the image representations with the language representations. During adapter training, we also update the parameters of the image encoder but we intentionally do not update the language-model parameters. We also train a video adapter on top of the image adapter on paired video-text data. This enables the model to aggregate information across frames. See Section 7 for details.

视觉适配器训练。我们训练一个适配器，将预训练的图像编码器集成到预训练的语言模型中。该适配器由一系列交叉注意力层组成，这些层将图像编码器表示输入到语言模型中。适配器在文本-图像对上进行训练。这使得图像表示与语言表示对齐。在适配器训练期间，我们还更新图像编码器的参数，但我们有意不更新语言模型的参数。我们还在图像适配器之上对视频-文本数据对进行视频适配器训练。这使得模型能够跨帧聚合信息。详细信息请参见第7节。

Speech adapter training. Finally, we integrate the speech encoder into the model via an adapter that converts speech encodings into token representations that can be fed directly into the finetuned language model. The parameters of the adapter and encoder are jointly updated in a supervised finetuning stage to enable high-quality speech understanding. We do not change the language model during speech adapter training. We also integrate a text-to-speech system. See Section 8 for details.

语音适配器训练。最后，我们通过一个适配器将语音编码器集成到模型中，该适配器将语音编码转换为可以直接输入到微调语言模型的标记表示。在监督微调阶段，适配器和编码器的参数被联合更新，以实现高质量的语音理解。在语音适配器训练期间，我们不改变语言模型。我们还集成了一个文本到语音系统。详细信息请参见第8节。

Our multimodal experiments lead to models that can recognize the content of images and videos, and support interaction via a speech interface. These models are still under development and not yet ready for release.

我们的多模态实验产生了能够识别图像和视频内容并支持通过语音界面进行交互的模型。这些模型仍在开发中，尚未准备好发布。

3 Pre-Training 预训练

Language model pre-training involves: (1) the curation and filtering of a large-scale training corpus, (2) the development of a model architecture and corresponding scaling laws for determining model size, (3) the development of techniques for efficient pre-training at large scale, and (4) the development of a pre-training recipe. We present each of these components separately below.

语言模型预训练涉及：(1) 大规模训练语料库的策划和过滤，(2) 模型架构的开发以及确定模型大小的相应缩放法则，(3) 大规模高效预训练技术的开发，以及 (4) 预训练方案的开发。我们将在下面分别介绍这些组件。

3.1 Pre-Training Data 预训练数据

We create our dataset for language model pre-training from a variety of data sources containing knowledge until the end of 2023. We apply several de-duplication methods and data cleaning mechanisms on each data source to obtain high-quality tokens. We remove domains that contain large amounts of personally identifiable information (PII), and domains with known adult content.

我们从多种数据源中创建用于语言模型预训练的数据集，这些数据源包含截至2023年底的知识。我们对每个数据源应用多种去重方法和数据清洗机制以获取高质量的令牌。我们移除了包含大量个人身份信息（PII）的领域，以及已知包含成人内容的领域。

3.1.1 Web Data Curation 网络数据精选

Much of the data we utilize is obtained from the web and we describe our cleaning process below.

我们使用的许多数据来自网络，下面描述我们的清洗过程。

PH and safety filtering. Among other mitigations, we implement filters designed to remove data from websites are likely to contain unsafe content or high volumes of PII, domains that have been ranked as harmful according to a variety of Meta safety standards, and domains that are known to contain adult content.

PH和安全过滤。除了其他缓解措施外，我们实施了旨在移除可能包含不安全内容或大量PII的网站数据的过滤器，根据多种Meta安全标准被评定为有害的域名，以及已知包含成人内容的域名。

Text extraction and cleaning. We process the raw HTML content for non-truncated web documents to extract high-quality diverse text. To do so, we build a custom parser that extracts the HTML content and optimizes for precision in boilerplate removal and content recall. We evaluate our parser’s quality in human evaluations, comparing it with popular third-party HTML parsers that optimize for article-like content, and found it to perform favorably. We carefully process HTML pages with mathematics and code content to preserve the structure of that content. We maintain the image alt attribute text since mathematical content is often represented as pre-rendered images where the math is also provided in the alt attribute. We experimentally evaluate different cleaning configurations. We find markdown is harmful to the performance of a model that is primarily trained on web data compared to plain text, so we remove all markdown markers.

文本提取和清洗。我们对非截断的网页文档的原始HTML内容进行处理，以提取高质量的多样化文本。为此，我们构建了一个自定义解析器，用于提取HTML内容并优化样板删除和内容召回的精确度。我们通过人工评估来评估解析器的质量，将其与优化文章类内容的流行第三方HTML解析器进行比较，发现其表现更优。我们仔细处理包含数学和代码内容的HTML页面，以保留这些内容的结构。我们保留图像alt属性文本，因为数学内容通常以预渲染图像表示，其中数学内容也提供在alt属性中。我们实验性地评估不同的清洗配置。我们发现，与纯文本相比，Markdown对主要基于网络数据训练的模型的性能有害，因此我们移除了所有Markdown标记。

De-duplication. We apply several rounds of de-duplication at the URL, document, and line level:

去重。我们在URL、文档和行级别应用多轮去重：

URL-level de-duplication. We perform URL-level de-duplication across the entire dataset. We keep the most recent version for pages corresponding to each URL.

URL级别去重。我们对整个数据集进行URL级别的去重。对于每个URL对应的页面，我们保留最新版本。

Document-level de-duplication. We perform global MinHash (Broder, 1997) de-duplication across the entire dataset to remove near duplicate documents.

文档级去重。我们对整个数据集执行全局MinHash（Broder，1997）去重，以删除近似重复的文档。

Line-level de-duplication. We perform aggressive line-level de-duplication similar to coNet (Wenzek et al., 2019). We remove lines that appeared more than 6 times in each bucket of 30M documents. Although our manual qualitative analysis showed that the line-level de-duplication removes not only leftover boilerplate from various websites such as navigation menus, cookie warnings, but also frequent high-quality text, our empirical evaluations showed strong improvements.

行级去重。我们执行激进的行级去重，类似于coNet（Wenzek等，2019）。我们删除在每个30M文档桶中出现超过6次的行。尽管我们的手动定性分析显示，行级去重不仅删除了来自各种网站的残留样板文件，如导航菜单、cookie警告，还删除了频繁的高质量文本，但我们的实证评估显示了显著的改进。

Heuristic filtering. We develop heuristics to remove additional low-quality documents, outliers, and documents with excessive repetitions. Some examples of heuristics include:

启发式过滤。我们开发启发式方法来删除额外的低质量文档、异常值和包含过多重复内容的文档。一些启发式方法的例子包括：

We use duplicated n-gram coverage ratio (Rae et al., 2021) to remove lines that consist of repeated content such as logging or error messages. Those lines could be very long and unique, hence cannot be filtered by line-dedup.

我们使用重复n-gram覆盖率（Rae等，2021）来删除包含重复内容（如日志或错误消息）的行。这些行可能非常长且独特，因此无法通过行去重过滤。

We use “dirty word” counting (Raffel et al., 2020) to filter out adult websites that are not covered by domain block lists.

我们使用“脏词”计数（Raffel等，2020）来过滤未被域名阻止列表覆盖的成人网站。

We use a token-distribution Kullback-Leibler divergence to filter out documents containing excessive numbers of outlier tokens compared to the training corpus distribution.

我们使用令牌分布的Kullback-Leibler散度来过滤包含相对于训练语料库分布过多异常令牌的文档。

Model-based quality filtering. Further, we experiment with applying various model-based quality classifiers to sub-select high-quality tokens. These include using fast classifiers such as fasttext (Joulin et al., 2017) trained to recognize if a given text would be referenced by Wikipedia (Touvron et al., 2023a), as well as more compute-intensive Roberta-based classifiers (Liu et al., 2019a) trained on Llama 2 predictions. To train a quality classifier based on Llama 2, we create a training set of cleaned web documents, describe the quality requirements, and instruct Llama 2’s chat model to determine if the documents meets these requirements. We use DistilRoberta (Sanh et al., 2019) to generate quality scores for each document for efficiency reasons. We experimentally evaluate the efficacy of various quality filtering configurations.

基于模型的质量过滤。此外，我们尝试应用各种基于模型的质量分类器来筛选高质量的令牌。这些包括使用快速分类器，如 fasttext（Joulin 等人，2017），训练来识别给定文本是否会被维基百科引用（Touvron 等人，2023a），以及更计算密集型的基于 Roberta 的分类器（Liu 等人，2019a），这些分类器在 Llama 2 预测上进行训练。为了基于 Llama 2 训练质量分类器，我们创建了一个经过清理的网络文档训练集，描述了质量要求，并指示 Llama 2 的聊天模型来确定文档是否满足这些要求。出于效率原因，我们使用 DistilRoberta（Sanh 等人，2019）为每个文档生成质量分数。我们实验性地评估了各种质量过滤配置的有效性。

Code and reasoning data. Similar to DeepSeek-AI et al. (2024), we build domain-specific pipelines that extract code and math-relevant web pages. Specifically, both the code and reasoning classifiers are DistilRoberta models trained on web data annotated by Llama 2. Unlike the general quality classifier mentioned above, we conduct prompt tuning to target web pages containing math deduction, reasoning in STEM areas and code interleaved with natural language. Since the token distribution of code and math is substantially different than that of natural language, these pipelines implement domain-specific HTML extraction, customized text features and heuristics for filtering.

代码和推理数据。与 DeepSeek-AI 等人（2024）类似，我们构建了特定领域的管道，用于提取与代码和数学相关的网页。具体来说，代码和推理分类器都是基于 Llama 2 标注的网络数据训练的 DistilRoberta 模型。与上述通用质量分类器不同，我们进行提示调优，以针对包含数学推导、STEM 领域推理以及自然语言交织代码的网页。由于代码和数学的令牌分布与自然语言有显著不同，这些管道实现了特定领域的 HTML 提取、定制文本特征和过滤启发式方法。

Multilingual data. Similar to our processing pipelines for English described above, we implement filters to remove data from websites that are likely to contain PII or unsafe content. Our multilingual text processing pipeline has several unique features:

多语言数据。与上述针对英语的处理管道类似，我们实施过滤器以移除可能包含个人身份信息（PII）或不安全内容的网站数据。我们的多语言文本处理管道具有几个独特特点：

We use a fasttext-based language identification model to categorize documents into 176 languages.

我们使用基于 fasttext 的语言识别模型将文档分类为 176 种语言。

We perform document-level and line-level de-duplication within data for each language. - We apply language-specific heuristics and model-based filters to remove low-quality documents.

我们对每种语言的数据进行文档级和行级去重。 - 我们应用特定语言的启发式方法和基于模型的过滤器来移除低质量文档。

In addition, we perform quality ranking of multilingual documents using a multilingual Llama 2-based classifier to ensure that high-quality content is prioritized. We determine the amount of multilingual tokens used in pre-training experimentally, balancing model performance on English and multilingual benchmarks.

此外，我们使用基于多语言Llama 2的分类器对多语言文档进行质量排序，以确保高质量内容得到优先处理。我们通过实验确定预训练中使用的多语言令牌数量，平衡模型在英语和多语言基准上的性能。

3.1.2 Determining the Data Mix 确定数据混合比例

To obtain a high-quality language model, it is essential to carefully determine the proportion of different data sources in the pre-training data mix. Our main tools in determining this data mix are knowledge classification and scaling law experiments.

为了获得高质量的语言模型，必须仔细确定预训练数据混合中不同数据源的比例。我们在确定这一数据混合比例时的主要工具是知识分类和缩放定律实验。

Knowledge classification. We develop a classifier to categorize the types of information contained in our web data to more effectively determine a data mix. We use this classifier to downsample data categories that are over-represented on the web, for example, arts and entertainment.

知识分类。我们开发了一个分类器，用于对我们的网络数据中包含的信息类型进行分类，以便更有效地确定数据混合比例。我们使用这个分类器对在网络上过度代表的数据类别进行降采样，例如艺术和娱乐。

Scaling laws for data mix. To determine the best data mix, we perform scaling law experiments in which we train several small models on a data mix and use that to predict the performance of a large model on that mix (see Section 3.2.1). We repeat this process multiple times for different data mixes to select a new data mix candidate. Subsequently, we train a larger model on this candidate data mix and evaluate the performance of that model on several key benchmarks.

数据混合比例的缩放定律。为了确定最佳的数据混合比例，我们进行了缩放定律实验，在这些实验中，我们在一个数据混合上训练几个小型模型，并利用这些模型预测大型模型在该混合上的性能（见第3.2.1节）。我们对不同的数据混合重复这一过程，以选择一个新的数据混合候选。随后，我们在这个候选数据混合上训练一个更大的模型，并在几个关键基准上评估该模型的性能。

Data mix summary. Our final data mix contains roughly 50 % {50}\% 50% of tokens corresponding to general knowledge, 25 % {25}\% 25% of mathematical and reasoning tokens, 17 % {17}\% 17% code tokens,and 8 % 8\% 8% multilingual tokens.

数据混合总结。我们的最终数据混合包含大约 50 % {50}\% 50%的令牌对应于一般知识， 25 % {25}\% 25%的数学和推理令牌， 17 % {17}\% 17%的代码令牌，以及 8 % 8\% 8%的多语言令牌。

3.1.3 Annealing Data 退火数据

Empirically, we find that annealing (see Section 3.4.3) on small amounts of high-quality code and mathematical data can boost the performance of pre-trained models on key benchmarks. Akin to Li et al. (2024b), we perform annealing with a data mix that upsamples high-quality data in select domains. We do not include any training sets from commonly used benchmarks in our annealing data. This enables us to assess the true few-shot learning capabilities and out-of-domain generalization of Llama 3.

根据实证研究，我们发现对少量高质量的代码和数学数据进行退火处理（参见第3.4.3节）可以提升预训练模型在关键基准测试中的性能。类似于Li等人（2024b）的方法，我们通过混合数据对高质量数据在选定领域进行上采样来执行退火处理。在我们的退火数据中不包含任何常用基准测试的训练集。这使我们能够评估Llama 3的真实少样本学习能力和跨领域泛化能力。

Following OpenAI (2023a), we evaluate the efficacy of annealing on the GSM8k (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b) training sets in annealing. We find that annealing improved the performance of a pre-trained Llama 3 8B model on the GSM8k and MATH validation sets by 24.0% and 6.4%, respectively. However, the improvements on the 405B model are negligible, suggesting that our flagship model has strong in-context learning and reasoning capabilities and does not require specific in-domain training samples to obtain strong performance.

遵循OpenAI（2023a）的方法，我们在退火过程中评估了退火在GSM8k（Cobbe等人，2021）和MATH（Hendrycks等人，2021b）训练集上的有效性。我们发现，退火分别提高了预训练的Llama 3 8B模型在GSM8k和MATH验证集上的性能24.0%和6.4%。然而，405B模型的改进微乎其微，这表明我们的旗舰模型具有强大的上下文学习和推理能力，不需要特定的领域内训练样本就能获得强大的性能。

Using annealing to assess data quality. Similar to Blakeney et al. (2024), we find that annealing enables us to judge the value of small domain-specific datasets. We measure the value of such datasets by annealing the learning rate of a 50 % {50}\% 50% trained Llama 3 8B model linearly to 0 on 40B tokens. In those experiments,we assign 30 % {30}\% 30% weight to the new dataset and the remaining 70 % {70}\% 70% weight to the default data mix. Using annealing to evaluate new data sources is more efficient than performing scaling law experiments for every small dataset.

使用退火评估数据质量。类似于Blakeney等人（2024）的方法，我们发现退火使我们能够判断小规模特定领域数据集的价值。我们通过将 50 % {50}\% 50%训练的Llama 3 8B模型的学习率在40B个令牌上线性退火至0来衡量这些数据集的价值。在这些实验中，我们将 30 % {30}\% 30%权重分配给新数据集，其余的 70 % {70}\% 70%权重分配给默认数据混合。使用退火来评估新数据源比为每个小数据集执行缩放定律实验更高效。

3.2 Model Architecture 模型架构

Llama 3 uses a standard, dense Transformer architecture (Vaswani et al., 2017). It does not deviate significantly from Llama and Llama 2 (Touvron et al., 2023a,b) in terms of model architecture; our performance gains are primarily driven by improvements in data quality and diversity as well as by increased training scale.

Llama 3 采用标准的密集 Transformer 架构（Vaswani 等人，2017）。在模型架构方面，它与 Llama 和 Llama 2（Touvron 等人，2023a,b）没有显著偏离；我们的性能提升主要源于数据质量和多样性的改进以及训练规模的扩大。

We make a few small modifications compared to Llama 2:

与 Llama 2 相比，我们进行了一些小的修改：

We use grouped query attention (GQA; Ainslie et al. (2023)) with 8 key-value heads to improve inference speed and to reduce the size of key-value caches during decoding.

我们使用分组查询注意力（GQA；Ainslie 等人（2023）），采用 8 个键值头，以提高推理速度并减少解码过程中键值缓存的大小。

We use an attention mask that prevents self-attention between different documents within the same sequence. We find that this change had limited impact during in standard pre-training, but find it to be important in continued pre-training on very long sequences.

我们使用了一个注意力掩码，防止同一序列内不同文档之间的自注意力。我们发现这一改变在标准预训练中影响有限，但在对非常长的序列进行连续预训练时显得非常重要。

Table 3 Overview of the key hyperparameters of Llama 3. We display settings for 8B, 70B, and 405B language models.

表 3 Llama 3 关键超参数概览。我们展示了 8B、70B 和 405B 语言模型的设置。

We use a vocabulary with 128    K {128}\mathrm{\;K} 128K tokens. Our token vocabulary combines 100    K {100}\mathrm{\;K} 100K tokens from the tiktoken 3 {}^{3} 3 tokenizer with 28    K {28}\mathrm{\;K} 28K additional tokens to better support non-English languages. Compared to the Llama 2 tokenizer, our new tokenizer improves compression rates on a sample of English data from 3.17 to 3.94 characters per token. This enables the model to “read” more text for the same amount of training compute. We also found that adding 28    K {28}\mathrm{\;K} 28K tokens from select non-English languages improved both compression ratios and downstream performance, with no impact on English tokenization.

我们使用了一个包含 128    K {128}\mathrm{\;K} 128K 个词元的词汇表。我们的词元词汇结合了 tiktoken 3 {}^{3} 3 分词器中的 100    K {100}\mathrm{\;K} 100K 个词元和 28    K {28}\mathrm{\;K} 28K 个额外词元，以更好地支持非英语语言。与 Llama 2 分词器相比，我们的新分词器在英语数据样本上的压缩率从 3.17 提高到 3.94 个字符每词元。这使得模型能够以相同的训练计算量“读取”更多文本。我们还发现，添加来自选定非英语语言的 28    K {28}\mathrm{\;K} 28K 个词元不仅提高了压缩比，还提升了下游性能，且对英语分词没有影响。

We increase the RoPE base frequency hyperparameter to 500,000 . This enables us to better support longer contexts; Xiong et al. (2023) showed this value to be effective for context lengths up to 32,768.

我们将 RoPE 基频超参数提高到 500,000。这使我们能够更好地支持更长的上下文；Xiong 等人（2023）表明，这一数值对于长达 32,768 的上下文长度是有效的。

Llama 3405B uses an architecture with 126 layers, a token representation dimension of 16,384, and 128 attention heads; see Table 3 for details. This leads to a model size that is approximately compute-optimal according to scaling laws on our data for our training budget of 3.8 × 10 25 F L O P s {3.8} \times {10}^{25}\mathrm{{FLOPs}} 3.8×1025FLOPs .

Llama 3405B 采用了一种架构，包含 126 层，令牌表示维度为 16,384，以及 128 个注意力头；详细信息参见表 3。根据我们的数据和训练预算 3.8 × 10 25 F L O P s {3.8} \times {10}^{25}\mathrm{{FLOPs}} 3.8×1025FLOPs 的缩放定律，这导致了一个近似计算最优的模型大小。

3.2.1 Scaling Laws 缩放定律

We develop scaling laws (Hoffmann et al., 2022; Kaplan et al., 2020) to determine the optimal model size for our flagship model given our pre-training compute budget. In addition to determining the optimal model size, a major challenge is to forecast the flagship model’s performance on downstream benchmark tasks, due to a couple of issues: (1) Existing scaling laws typically predict only next-token prediction loss rather than specific benchmark performance. (2) Scaling laws can be noisy and unreliable because they are developed based on pre-training runs conducted with small compute budgets (Wei et al., 2022b).

我们根据预训练计算预算，开发了缩放定律（Hoffmann 等人，2022；Kaplan 等人，2020），以确定我们旗舰模型的最优大小。除了确定最优模型大小外，一个主要挑战是预测旗舰模型在下游基准任务上的性能，这由于以下几个问题：（1）现有的缩放定律通常仅预测下一个令牌预测损失，而不是特定的基准性能。（2）缩放定律可能嘈杂且不可靠，因为它们是基于使用小计算预算进行的预训练运行开发的（Wei 等人，2022b）。

To address these challenges, we implement a two-stage methodology to develop scaling laws that accurately predict downstream benchmark performance:

为了解决这些挑战，我们实施了一种两阶段方法来开发能够准确预测下游基准性能的缩放定律：

We first establish a correlation between the compute-optimal model’s negative log-likelihood on downstream tasks and the training FLOPs.

我们首先建立计算最优模型在下游任务上的负对数似然与训练浮点运算次数（FLOPs）之间的相关性。

Next, we correlate the negative log-likelihood on downstream tasks with task accuracy, utilizing both the scaling law models and older models trained with higher compute FLOPs. In this step, we specifically leverage the Llama 2 family of models.

接下来，我们利用缩放定律模型和使用更高计算 FLOPs 训练的旧模型，将下游任务上的负对数似然与任务准确性相关联。在这一步骤中，我们特别利用了 Llama 2 系列模型。

This approach enables us to predict downstream task performance given a specific number of training FLOPs for compute-optimal models. We use a similar method to select our pre-training data mix (see Section 3.4).

这种方法使我们能够根据计算最优模型的特定训练 FLOPs 数量预测下游任务性能。我们使用类似的方法来选择我们的预训练数据混合（参见第 3.4 节）。

Scaling law experiments. Concretely, we construct our scaling laws by pre-training models using compute budgets between 6 × 10 18 6 \times {10}^{18} 6×1018 FLOPs and 10 22 {10}^{22} 1022 FLOPs. At each compute budget,we pre-train models ranging in size between 40 M {40}\mathrm{M} 40M and 16    B {16}\mathrm{\;B} 16B parameters,using a subset of model sizes at each compute budget. In these training runs, we use a cosine learning rate schedule with a linear warmup for 2,000 training steps. The peak learning rate is set between 2 × 10 − 4 2 \times {10}^{-4} 2×10−4 and 4 × 10 − 4 4 \times {10}^{-4} 4×10−4 depending on the size of the model. We set the cosine decay to 0.1 of the peak value. The weight decay at each step is set to 0.1 times the learning rate at that step. We use a fixed batch size for each compute scale,ranging between 250    K {250}\mathrm{\;K} 250K and 4 M 4\mathrm{M} 4M .

规模法则实验。具体而言，我们通过使用介于 6 × 10 18 6 \times {10}^{18} 6×1018 FLOPs 和 10 22 {10}^{22} 1022 FLOPs 之间的计算预算对模型进行预训练来构建我们的规模法则。在每个计算预算下，我们预训练大小介于 40 M {40}\mathrm{M} 40M 和 16    B {16}\mathrm{\;B} 16B 参数之间的模型，在每个计算预算下使用模型大小的一个子集。在这些训练运行中，我们使用余弦学习率调度，线性预热 2,000 个训练步骤。峰值学习率根据模型大小设置在 2 × 10 − 4 2 \times {10}^{-4} 2×10−4 和 4 × 10 − 4 4 \times {10}^{-4} 4×10−4 之间。我们将余弦衰减设置为峰值值的 0.1。每一步的权重衰减设置为该步学习率的 0.1 倍。我们为每个计算规模使用固定的批量大小，范围介于 250    K {250}\mathrm{\;K} 250K 和 4 M 4\mathrm{M} 4M 之间。

3 {}^{3} 3 https://github.com/openai/tiktoken/tree/main

Figure 2 Scaling law IsoFLOPs curves between 6 × 10 18 6 \times {10}^{18} 6×1018 and 10 22 {10}^{22} 1022 FLOPs. The loss is the negative log-likelihood on a held-out validation set. We approximate measurements at each compute scale using a second degree polynomial.

图2 在 6 × 10 18 6 \times {10}^{18} 6×1018 和 10 22 {10}^{22} 1022 FLOPs 之间的规模法则等FLOPs曲线。损失是在一个独立的验证集上的负对数似然。我们使用二次多项式近似每个计算规模上的测量值。

Figure 3 Number of training tokens in identified compute-optimal models as a function of pre-training compute budget. We include the fitted scaling-law prediction as well. The compute-optimal models correspond to the parabola minimums in Figure 2.

图3 作为预训练计算预算函数的识别出的计算最优模型中的训练令牌数量。我们还包含了拟合的规模法则预测。计算最优模型对应于图2中的抛物线最小值。

These experiments give rise to the IsoFLOPs curves in Figure 2. The loss in these curves is measured on a separate validation set. We fit the measured loss values using a second-degree polynomial and identify the minimums of each parabola. We refer to minimum of a parabola as the compute-optimal model at the corresponding pre-training compute budget.

这些实验产生了图2中的等FLOPs曲线。这些曲线中的损失是在一个独立的验证集上测量的。我们使用二次多项式拟合测量的损失值，并识别每个抛物线的最小值。我们将抛物线的最小值称为相应预训练计算预算下的计算最优模型。

We use the compute-optimal models we identified this way to predict the optimal number of training tokens for a specific compute budget. To do so,we assume a power-law relation between compute budget, C C C ,and the optimal number of training tokens, N ⋆ ( C ) {N}^{ \star }\left( C\right) N⋆(C) :

我们使用这种方式识别出的计算最优模型来预测特定计算预算下的最优训练令牌数量。为此，我们假设计算预算 C C C 和最优训练令牌数量 N ⋆ ( C ) {N}^{ \star }\left( C\right) N⋆(C) 之间存在幂律关系：

N ⋆ ( C ) = A C α . {N}^{ \star }\left( C\right) = A{C}^{\alpha }. N⋆(C)=ACα.

We fit A A A and α \alpha α using the data from Figure 2. We find that ( α , A ) = ( 0.53 , 0.29 ) \left( {\alpha ,A}\right) = \left( {{0.53},{0.29}}\right) (α,A)=(0.53,0.29) ; the corresponding fit is shown in Figure 3. Extrapolation of the resulting scaling law to 3.8 × 10 25 {3.8} \times {10}^{25} 3.8×1025 FLOPs suggests training a 402B parameter model on 16.55    T {16.55}\mathrm{\;T} 16.55T tokens.

我们使用图2的数据来拟合 A A A 和 α \alpha α。我们发现 ( α , A ) = ( 0.53 , 0.29 ) \left( {\alpha ,A}\right) = \left( {{0.53},{0.29}}\right) (α,A)=(0.53,0.29)；相应的拟合结果如图3所示。将由此得出的缩放定律外推至 3.8 × 10 25 {3.8} \times {10}^{25} 3.8×1025 FLOPs 表明，应在 16.55    T {16.55}\mathrm{\;T} 16.55T 令牌上训练一个402B参数的模型。

An important observation is that IsoFLOPs curves become flatter around the minimum as the compute budget increases. This implies that performance of the flagship model is relatively robust to small changes in the trade-off between model size and training tokens. Based on this observation, we ultimately decided to train a flagship model with 405    B {405}\mathrm{\;B} 405B parameters.

一个重要的观察结果是，随着计算预算的增加，IsoFLOPs曲线在最小值附近变得更平坦。这意味着旗舰模型的性能对模型大小和训练令牌之间权衡的小变化相对稳健。基于这一观察，我们最终决定训练一个具有 405    B {405}\mathrm{\;B} 405B 参数的旗舰模型。

Predicting performance on downstream tasks. We use the resulting compute-optimal models to forecast the performance of the flagship Llama 3 model on benchmark data sets. First, we linearly correlate the (normalized) negative log-likelihood of correct answer in the benchmark and the training FLOPs. In this analysis,we use only the scaling law models trained up to 10 22 {10}^{22} 1022 FLOPs on the data mix described above. Next, we establish a sigmoidal relation between the log-likelihood and accuracy using both the scaling law models and Llama 2 models, which were trained using the Llama 2 data mix and tokenizer. We show the results of this experiment on the ARC Challenge benchmark in Figure 4). We find this two-step scaling law prediction, which extrapolates over four orders of magnitude, to be quite accurate: it only slightly underestimates the final performance of the flagship Llama 3 model.

预测下游任务的性能。我们使用由此得出的计算最优模型来预测旗舰Llama 3模型在基准数据集上的性能。首先，我们在线性相关基准中正确答案的（归一化）负对数似然和训练FLOPs。在此分析中，我们仅使用上述数据混合训练至 10 22 {10}^{22} 1022 FLOPs 的缩放定律模型。接下来，我们使用缩放定律模型和Llama 2模型（这些模型使用Llama 2数据混合和分词器进行训练）建立对数似然和准确性之间的S形关系。我们在ARC挑战基准上展示了这一实验的结果（如图4所示）。我们发现这种跨越四个数量级的两步缩放定律预测相当准确：它仅略微低估了旗舰Llama 3模型的最终性能。

3.3 Infrastructure, Scaling, and Efficiency 基础设施、扩展和效率

We describe our hardware and infrastructure that powered Llama 3405B pre-training at scale and discuss several optimizations that leads to improvements in training efficiency.

我们描述了支持 Llama 3405B 大规模预训练的硬件和基础设施，并讨论了几项优化措施，这些措施提高了训练效率。

3.3.1 Training Infrastructure 训练基础设施

The Llama 1 and 2 models were trained on Meta’s AI Research SuperCluster (Lee and Sengupta, 2022). As we scaled further, the training for Llama 3 was migrated to Meta’s production clusters (Lee et al., 2024).This setup optimizes for production-grade reliability, which is essential as we scale up training.

Llama 1 和 2 模型是在 Meta 的 AI 研究超级集群（Lee 和 Sengupta，2022）上训练的。随着我们进一步扩展，Llama 3 的训练迁移到了 Meta 的生产集群（Lee 等人，2024）。这种设置针对生产级可靠性进行了优化，这对于我们扩大训练规模至关重要。

Figure 4 Scaling law forecast for ARC Challenge. Left: Normalized negative log-likelihood of the correct answer on the ARC Challenge benchmark as a function of pre-training FLOPs. Right: ARC Challenge benchmark accuracy as a function of the normalized negative log-likelihood of the correct answer. This analysis enables us to predict model performance on the ARC Challenge benchmark before pre-training commences. See text for details.

图 4 ARC 挑战赛的规模法则预测。左侧：ARC 挑战赛基准测试中正确答案的归一化负对数似然性作为预训练 FLOP 的函数。右侧：ARC 挑战赛基准测试准确性作为正确答案的归一化负对数似然性的函数。这一分析使我们能够在预训练开始之前预测模型在 ARC 挑战赛基准测试上的性能。详见正文。

Compute. Llama 3 405B is trained on up to 16K H100 GPUs, each running at 700W TDP with 80GB HBM3, using Meta’s Grand Teton AI server platform (Matt Bowman, 2022). Each server is equipped with eight GPUs and two CPUs. Within a server, the eight GPUs are connected via NVLink. Training jobs are scheduled using MAST (Choudhury et al., 2024), Meta’s global-scale training scheduler.

计算。Llama 3 405B 在多达 16K H100 GPU 上进行训练，每个 GPU 以 700W TDP 运行，配备 80GB HBM3，使用 Meta 的 Grand Teton AI 服务器平台（Matt Bowman，2022）。每台服务器配备八块 GPU 和两块 CPU。在服务器内部，八块 GPU 通过 NVLink 连接。训练任务使用 MAST（Choudhury 等人，2024）进行调度，这是 Meta 的全球规模训练调度器。

Storage. Tectonic (Pan et al., 2021), Meta’s general-purpose distributed file system, is used to build a storage fabric (Battey and Gupta, 2024) for Llama 3 pre-training. It offers 240 PB of storage out of 7,500 servers equipped with SSDs, and supports a sustainable throughput of 2 TB/s and a peak throughput of 7 TB/s. A major challenge is supporting the highly bursty checkpoint writes that saturate the storage fabric for short durations. Checkpointing saves each GPU’s model state, ranging from 1 MB to 4 GB per GPU, for recovery and debugging. We aim to minimize GPU pause time during checkpointing and increase checkpoint frequency to reduce the amount of lost work after a recovery.

存储。Tectonic（Pan et al., 2021），Meta的通用分布式文件系统，被用于构建Llama 3预训练的存储架构（Battey and Gupta, 2024）。它提供了来自7,500台配备SSD的服务器的240 PB存储容量，并支持2 TB/s的可持续吞吐量和7 TB/s的峰值吞吐量。一个主要挑战是支持高度突发的检查点写入，这些写入会在短时间内饱和存储架构。检查点保存每个GPU的模型状态，每个GPU的范围从1 MB到4 GB不等，用于恢复和调试。我们的目标是尽量减少检查点期间的GPU暂停时间，并增加检查点频率以减少恢复后的工作损失量。

Network. Llama 3405B used RDMA over Converged Ethernet (RoCE) fabric based on the Arista 7800 and Minipack2 Open Compute Project 4 {}^{4} 4 OCP rack switches. Smaller models in the Llama 3 family were trained using Nvidia Quantum2 Infiniband fabric. Both RoCE and Infiniband clusters leverage 400 Gbps interconnects between GPUs. Despite the underlying network technology differences between these clusters, we tune both of them to provide equivalent performance for these large training workloads. We elaborate further on our RoCE network since we fully own its design.

网络。Llama 3405B使用了基于Arista 7800和Minipack2 Open Compute Project 4 {}^{4} 4 OCP机架交换机的RDMA over Converged Ethernet（RoCE）架构。Llama 3系列中的较小模型则使用Nvidia Quantum2 Infiniband架构进行训练。无论是RoCE还是Infiniband集群，都利用了GPU之间的400 Gbps互连。尽管这些集群的底层网络技术存在差异，我们对其进行了调整，以在这些大型训练负载中提供等效性能。我们进一步详细阐述我们的RoCE网络，因为我们完全拥有其设计。

Network topology. Our RoCE-based AI cluster comprises 24    K G P U s 5 {24}\mathrm{\;K}{\mathrm{{GPUs}}}^{5} 24KGPUs5 connected by a three-layer Clos network (Lee et al., 2024). At the bottom layer, each rack hosts 16 GPUs split between two servers and connected by a single Minipack2 top-of-the-rack (ToR) switch. In the middle layer, 192 such racks are connected by Cluster Switches to form a pod of 3,072 GPUs with full bisection bandwidth, ensuring no oversubscription. At the top layer, eight such pods within the same datacenter building are connected via Aggregation Switches to form a cluster of 24    K {24}\mathrm{\;K} 24K GPUs. However,network connectivity at the aggregation layer does not maintain full bisection bandwidth and instead has an oversubscription ratio of 1:7. Our model parallelism methods (see Section 3.3.2) and training job scheduler (Choudhury et al., 2024) are all optimized to be aware of network topology, aiming to minimize network communication across pods.

网络拓扑。我们的基于 RoCE 的 AI 集群由 24    K G P U s 5 {24}\mathrm{\;K}{\mathrm{{GPUs}}}^{5} 24KGPUs5 通过三层 Clos 网络连接而成（Lee et al., 2024）。在最底层，每个机架托管 16 个 GPU，分布在两台服务器上，并通过一个 Minipack2 架顶（ToR）交换机连接。在中层，192 个这样的机架通过集群交换机连接，形成一个包含 3,072 个 GPU 的 Pod，具有完全的双向带宽，确保没有超额订阅。在顶层，同一数据中心建筑内的八个这样的 Pod 通过聚合交换机连接，形成一个包含 24    K {24}\mathrm{\;K} 24K 个 GPU 的集群。然而，聚合层的网络连接并不保持完全的双向带宽，而是具有 1:7 的超额订阅比。我们的模型并行方法（见第 3.3.2 节）和训练作业调度器（Choudhury et al., 2024）都经过优化，以了解网络拓扑，旨在最小化 Pod 间的网络通信。

Load balancing. LLM training produces fat network flows that are hard to load balance across all available network paths using traditional methods such as Equal-Cost Multi-Path (ECMP) routing. To address this challenge, we employ two techniques. First, our collective library creates 16 network flows between two GPUs, instead of just one, thereby reducing the traffic per flow and providing more flows

负载均衡。LLM 训练产生的大量网络流量很难使用传统的等成本多路径（ECMP）路由等方法在所有可用网络路径上进行负载均衡。为了解决这一挑战，我们采用了两种技术。首先，我们的集合库在两个 GPU 之间创建 16 个网络流，而不是仅一个，从而减少了每个流的流量并提供了更多的流

4 {}^{4} 4 Open Compute Project: https://www.opencompute.org/

4 {}^{4} 4 开放计算项目：https://www.opencompute.org/

5 {}^{5} 5 Note that we use only up to 16    K {16}\mathrm{\;K} 16K of these 24    K {24}\mathrm{\;K} 24K GPUs for Llama 3 pre-training.

5 {}^{5} 5 请注意，我们仅使用这些 24    K {24}\mathrm{\;K} 24K 个 GPU 中的最多 16    K {16}\mathrm{\;K} 16K 个用于 Llama 3 预训练。

Table 4 – Scaling configurations and MFU for each stage of Llama 3 405B pre-training. See t e x t \mathrm{{text}} text and Figure 5 for d e s c r i p t i o n s \mathrm{{descriptions}} descriptions of each type of parallelism.

表 4 – Llama 3 405B 预训练各阶段的扩展配置和 MFU。参见 t e x t \mathrm{{text}} text 和图 5 了解每种并行性的 d e s c r i p t i o n s \mathrm{{descriptions}} descriptions。

for load balancing. Second, our Enhanced-ECMP (E-ECMP) protocol effectively balances these 16 flows across different network paths by hashing on additional fields in the RoCE header of packets.

用于负载均衡。其次，我们的增强型等价多路径（E-ECMP）协议通过在RoCE数据包头中对额外字段进行哈希处理，有效地将这16个流平衡到不同的网络路径上。

Congestion control. We use deep-buffer switches in the spine (Gangidi et al., 2024) to accommodate transient congestion and buffering caused by collective communication patterns. This setup helps limit the impact of persistent congestion and network back pressure caused by slow servers, which is common in training. Finally, better load balancing through E-ECMP significantly reduces the chance of congestion. With these optimizations, we successfully run a 24K GPU cluster without traditional congestion control methods such as Data Center Quantized Congestion Notification (DCQCN).

拥塞控制。我们在骨干网中使用深度缓冲交换机（Gangidi等人，2024年）来容纳由集体通信模式引起的瞬态拥塞和缓冲。这种设置有助于限制持续拥塞和由慢速服务器引起的网络背压的影响，这在训练中很常见。最后，通过E-ECMP实现的更好的负载均衡显著降低了拥塞的可能性。通过这些优化，我们成功地运行了一个24K GPU集群，而没有使用诸如数据中心量化拥塞通知（DCQCN）等传统的拥塞控制方法。

3.3.2 Parallelism for Model Scaling 模型扩展的并行化

To scale training for our largest models, we use 4D parallelism - a combination of four different types of parallelism methods - to shard the model. This approach efficiently distributes computation across many GPUs and ensures each GPU’s model parameters, optimizer states, gradients, and activations fit in its HBM. Our implementation of 4D parallelism is illustrated in Figure 5. It combines tensor parallelism (TP; Krizhevsky et al. (2012); Shoeybi et al. (2019); Korthikanti et al. (2023)), pipeline parallelism (PP; Huang et al. (2019); Narayanan et al. (2021); Lamy-Poirier (2023)), context parallelism (CP; Liu et al. (2023a)), and data parallelism (DP; Rajbhandari et al. (2020); Ren et al. (2021); Zhao et al. (2023b)).

为了扩展我们最大模型的训练，我们使用了4D并行化——结合了四种不同类型的并行化方法——来分片模型。这种方法有效地将计算分布到许多GPU上，并确保每个GPU的模型参数、优化器状态、梯度和激活值都适合其HBM。我们的4D并行化实现如图5所示。它结合了张量并行化（TP；Krizhevsky等人（2012年）；Shoeybi等人（2019年）；Korthikanti等人（2023年））、流水线并行化（PP；Huang等人（2019年）；Narayanan等人（2021年）；Lamy-Poirier（2023年））、上下文并行化（CP；Liu等人（2023a年））和数据并行化（DP；Rajbhandari等人（2020年）；Ren等人（2021年）；Zhao等人（2023b年））。

Tensor parallelism splits individual weight tensors into multiple chunks on different devices. Pipeline parallelism partitions the model vertically into stages by layers, so that different devices can process in parallel different stages of the full model pipeline. Context parallelism divides the input context into segments, reducing memory bottleneck for very long sequence length inputs. We use fully sharded data parallelism (FSDP; Rajbhandari et al., 2020; Ren et al., 2021; Zhao et al., 2023b), which shards the model, optimizer, and gradients while implementing data parallelism which processes data in parallel on multiple GPUs and synchronizes after each training step. Our use of FSDP for Llama 3 shards optimizer states and gradients, but for model shards we do not reshard after forward computation to avoid an extra all-gather communication during backward passes.

张量并行将单个权重张量分割到不同设备上的多个块中。流水线并行将模型垂直划分为多个阶段，通过层来实现，使得不同设备可以并行处理完整模型流水线的不同阶段。上下文并行将输入上下文分割成段，减少了对于非常长的序列长度输入的内存瓶颈。我们使用完全分片数据并行（FSDP；Rajbhandari 等人，2020；Ren 等人，2021；Zhao 等人，2023b），它在实现数据并行的同时，对模型、优化器和梯度进行分片，即在多个GPU上并行处理数据，并在每个训练步骤后进行同步。我们在Llama 3中使用FSDP分片优化器状态和梯度，但对于模型分片，我们在前向计算后不重新分片，以避免在后向传递期间进行额外的全收集通信。

GPU utilization. Through careful tuning of the parallelism configuration, hardware, and software, we achieve an overall BF16 Model FLOPs Utilization (MFU; Chowdhery et al. (2023)) of 38-43% for the configurations shown in Table 4. The slight drop in MFU to 41 % {41}\% 41% on 16    K {16}\mathrm{\;K} 16K GPUs with DP=128 compared to 43 % {43}\% 43% on 8    K 8\mathrm{\;K} 8K GPUs with DP=64 is due to the lower batch size per DP group needed to keep the global tokens per batch constant during training.

GPU利用率。通过仔细调整并行配置、硬件和软件，我们为表4所示的配置实现了总体BF16模型浮点运算利用率（MFU；Chowdhery 等人（2023））达到38-43%。在DP=128的 16    K {16}\mathrm{\;K} 16KGPU上，MFU略微下降至 41 % {41}\% 41%，相比于DP=64的 8    K 8\mathrm{\;K} 8KGPU上的 43 % {43}\% 43%，这是由于在训练过程中为了保持全局批次令牌数量恒定，需要降低每个DP组的批次大小。

Pipeline parallelism improvements. We encountered several challenges with existing implementations:

流水线并行改进。我们在现有实现中遇到了几个挑战：

Batch size constraint. Current implementations have constraints on supported batch size per GPU, requiring it to be divisible by the number of pipeline stages. For the example in Figure 6, the depth-first schedule (DFS) of pipeline parallelism (Narayanan et al.,2021) requires N = P P = 4 N = \mathrm{{PP}} = 4 N=PP=4 ,while the breadth-first schedule (BFS; Lamy-Poirier (2023)) requires N = M N = M N=M ,where M M M is the total number of micro-batches and N N N is the number of contiguous micro-batches for the same stage’s forward or backward. However, pre-training often needs flexibility to adjust batch size.

批量大小约束。当前的实现对每个 GPU 支持的批量大小有约束，要求其必须能被流水线阶段数整除。以图 6 中的例子为例，深度优先调度（DFS）的流水线并行（Narayanan 等人，2021）要求 N = P P = 4 N = \mathrm{{PP}} = 4 N=PP=4，而广度优先调度（BFS；Lamy-Poirier（2023））要求 N = M N = M N=M，其中 M M M 是微批量的总数， N N N 是同一阶段前向或后向的连续微批量数。然而，预训练通常需要灵活调整批量大小。

Memory imbalance. Existing pipeline parallelism implementations lead to imbalanced resource consumption. The first stage consumes more memory due to the embedding and the warm-up micro-batches.

内存不平衡。现有的流水线并行实现导致资源消耗不平衡。第一阶段由于嵌入和预热微批量而消耗更多内存。

Computation imbalance. After the last layer of the model, we need to calculate output and loss, making this stage the execution latency bottleneck.

计算不平衡。在模型的最后一层之后，我们需要计算输出和损失，这使得该阶段成为执行延迟瓶颈。

Figure 5 Illustration of 4D parallelism. GPUs are divided into parallelism groups in the order of [TP, CP, PP, DP], where DP stands for FSDP. In this example, 16 {16} 16 GPUs are configured with a group size of ∣ T P ∣ = 2 , ∣ C P ∣ = 2 , ∣ P P ∣ = 2 \left| \mathrm{{TP}}\right| = 2,\left| \mathrm{{CP}}\right| = 2,\left| \mathrm{{PP}}\right| = 2 ∣TP∣=2,∣CP∣=2,∣PP∣=2 ,and ∣ D P ∣ = 2 \left| \mathrm{{DP}}\right| = 2 ∣DP∣=2 . A GPU’s position in 4D parallelism is represented as a vector, [ D 1 , D 2 , D 3 , D 4 ] \left\lbrack {{D}_{1},{D}_{2},{D}_{3},{D}_{4}}\right\rbrack [D1​,D2​,D3​,D4​] ,where D i {D}_{i} Di​ is the index on the i i i -th parallelism dimension. In this example,GPU0[TP0,CP0,PP0,DP0] and GPU1[TP1,CP0,PP0,DP0] are in the same TP group, GPU0 and GPU2 are in the same CP group, GPU0 and GPU4 are in the same PP group, and GPU0 and GPU8 are in the same DP group.

图 5 展示了 4D 并行的示意图。GPU 按照 [TP, CP, PP, DP] 的顺序分成并行组，其中 DP 代表 FSDP。在这个例子中， 16 {16} 16 个 GPU 配置了 ∣ T P ∣ = 2 , ∣ C P ∣ = 2 , ∣ P P ∣ = 2 \left| \mathrm{{TP}}\right| = 2,\left| \mathrm{{CP}}\right| = 2,\left| \mathrm{{PP}}\right| = 2 ∣TP∣=2,∣CP∣=2,∣PP∣=2 的组大小，以及 ∣ D P ∣ = 2 \left| \mathrm{{DP}}\right| = 2 ∣DP∣=2。GPU 在 4D 并行中的位置表示为一个向量 [ D 1 , D 2 , D 3 , D 4 ] \left\lbrack {{D}_{1},{D}_{2},{D}_{3},{D}_{4}}\right\rbrack [D1​,D2​,D3​,D4​]，其中 D i {D}_{i} Di​ 是第 i i i 个并行维度上的索引。在这个例子中，GPU0[TP0,CP0,PP0,DP0] 和 GPU1[TP1,CP0,PP0,DP0] 在同一个 TP 组中，GPU0 和 GPU2 在同一个 CP 组中，GPU0 和 GPU4 在同一个 PP 组中，GPU0 和 GPU8 在同一个 DP 组中。

To address these issues,we modify our pipeline schedule as shown in Figure 6,which allows setting N N N flexibly - in this case N = 5 N = 5 N=5 ,which can run a arbitrary number of micro-batches in each batch. This allows us to run: (1) fewer micro-batches than the number of stages when we have batch size limit at large scale; or (2) more micro-batches to hide point-to-point communication, finding a sweet spot between DFS and breadth first schedule (BFS) for the best communication and memory efficiency. To balance the pipeline, we reduce one Transformer layer each from the first and the last stages, respectively. This means that the first model chunk on the first stage has only the embedding, and the last model chunk on the last stage has only output projection and loss calculation. To reduce pipeline bubbles, we use an interleaved schedule (Narayanan et al.,2021) with V V V pipeline stages on one pipeline rank. Overall pipeline bubble ratio is P P − 1 V ∗ M \frac{\mathrm{{PP}} - 1}{V * M} V∗MPP−1​ . Further,we adopt asynchronous point-to-point communication in PP,which considerably speeds up training, especially in cases when the document mask introduces extra computation imbalance. We enable TORCH_NCCL_AVOID_RECORD_STREAMS to reduce memory usage from asynchronous point-to-point communication. Finally, to reduce memory cost, based on detailed memory allocation profiling, we proactively deallocate tensors that will not be used for future computation, including the input and output tensors of each pipeline stage, that will not be used for future computation. With these optimizations, we could pre-train Llama 3 on sequences of 8    K 8\mathrm{\;K} 8K tokens without activation checkpointing.

为了解决这些问题，我们修改了如图6所示的流水线调度，这使得可以灵活设置 N N N - 在这种情况下是 N = 5 N = 5 N=5，它可以在每个批次中运行任意数量的微批次。这使我们能够运行：（1）在大规模下有批次大小限制时，少于阶段数量的微批次；或（2）更多的微批次以隐藏点对点通信，在DFS和广度优先调度（BFS）之间找到最佳通信和内存效率的平衡点。为了平衡流水线，我们分别从第一和最后阶段各减少一个Transformer层。这意味着第一阶段的第一模型块只有嵌入层，最后阶段的最后模型块只有输出投影和损失计算。为了减少流水线气泡，我们使用了一种交错调度（Narayanan等人，2021），在一个流水线等级上有 V V V 个流水线阶段。总体流水线气泡比率为 P P − 1 V ∗ M \frac{\mathrm{{PP}} - 1}{V * M} V∗MPP−1​。此外，我们在PP中采用了异步点对点通信，这大大加快了训练速度，尤其是在文档掩码引入额外计算不平衡的情况下。我们启用了TORCH_NCCL_AVOID_RECORD_STREAMS来减少异步点对点通信的内存使用。最后，为了降低内存成本，基于详细的内存分配分析，我们主动释放那些不会用于未来计算的张量，包括每个流水线阶段的输入和输出张量，这些张量不会用于未来的计算。通过这些优化，我们可以在没有激活检查点的情况下预训练Llama 3，处理长度为 8    K 8\mathrm{\;K} 8K 个令牌的序列。

Context parallelism for long sequences. We utilize context parallelism (CP) to improve memory efficiency when scaling the context length of Llama 3 and enable training on extremely long sequences up to 128    K {128}\mathrm{\;K} 128K in length. In CP, we partition across the sequence dimension, and specifically we partition the input sequence into 2 × C P 2 \times \mathrm{{CP}} 2×CP chunks so each CP rank receives two chunks for better load balancing. The i i i -th CP rank received both the i i i -th and the ( 2 × C P − 1 − i ) \left( {2 \times \mathrm{{CP}} - 1 - i}\right) (2×CP−1−i) -th chunks.

长序列的上下文并行。我们利用上下文并行（CP）在扩展Llama 3的上下文长度时提高内存效率，并能够在长度高达 128    K {128}\mathrm{\;K} 128K的极长序列上进行训练。在CP中，我们在序列维度上进行分区，具体地将输入序列分成 2 × C P 2 \times \mathrm{{CP}} 2×CP个块，以便每个CP等级接收两个块以实现更好的负载均衡。第 i i i个CP等级接收第 i i i个和第 ( 2 × C P − 1 − i ) \left( {2 \times \mathrm{{CP}} - 1 - i}\right) (2×CP−1−i)个块。

Different from existing CP implementations that overlap communication and computation in a ring-like structure (Liu et al., 2023a), our CP implementation adopts an all-gather based method where we first all-gather the key (K) and value (V) tensors, and then compute attention output for the local query (Q) tensor chunk. Although the all-gather communication latency is exposed in the critical path, we still adopt this approach for two main reasons: (1) it is easier and more flexible to support different types of attention masks in all-gather based CP attention, such as the document mask; and (2) the exposed all-gather latency is small as the communicated K \mathrm{K} K and V \mathrm{V} V tensors are much smaller than Q \mathrm{Q} Q tensor due to the use of GQA (Ainslie et al., 2023). Hence, the time complexity of attention computation is an order of magnitude larger than all-gather ( O ( S 2 ) \left( {O\left( {S}^{2}\right) }\right. (O(S2) versus O ( S ) O\left( S\right) O(S) ,where S S S represents the sequence length in the full causal mask),making the all-gather overhead negligible.

与现有在环状结构中重叠通信和计算的CP实现（Liu et al., 2023a）不同，我们的CP实现采用了一种基于all-gather的方法，其中我们首先all-gather键（K）和值（V）张量，然后计算本地查询（Q）张量块的注意力输出。尽管all-gather通信延迟暴露在关键路径中，我们仍然采用这种方法有两个主要原因：（1）在基于all-gather的CP注意力中，支持不同类型的注意力掩码（如文档掩码）更容易且更灵活；（2）由于使用了GQA（Ainslie et al., 2023），通信的 K \mathrm{K} K和 V \mathrm{V} V张量远小于 Q \mathrm{Q} Q张量，因此暴露的all-gather延迟很小。因此，注意力计算的时间复杂度比all-gather大一个数量级（ ( O ( S 2 ) \left( {O\left( {S}^{2}\right) }\right. (O(S2)对 O ( S ) O\left( S\right) O(S)，其中 S S S表示完整因果掩码中的序列长度），使得all-gather的开销可以忽略不计。

Figure 6 Illustration of pipeline parallelism in Llama 3. Pipeline parallelism partitions eight pipeline stages (0 to 7) across four pipeline ranks (PP ranks 0 to 3), where the GPUs with rank 0 run stages 0 and 4, the GPUs with P rank 1 run stages 1 and 5,etc. The colored blocks ( 0 to 9 ) represent a sequence of micro-batches,where M M M is the total number of micro-batches and N N N is the number of continuous micro-batches for the same stage’s forward or backward. Our key insight is to make N N N tunable.

图6展示了Llama 3中的流水线并行性示意图。流水线并行性将八个流水线阶段（0到7）分布在四个流水线等级（PP等级0到3）上，其中等级0的GPU运行阶段0和4，等级1的GPU运行阶段1和5，依此类推。彩色块（0到9）代表一系列微批次，其中 M M M是微批次总数， N N N是同一阶段的连续微批次数量，用于前向或后向传播。我们的关键见解是使 N N N可调。

Network-aware parallelism configuration. The order of parallelism dimensions, [TP, CP, PP, DP], is optimized for network communication. The innermost parallelism requires the highest network bandwidth and lowest latency, and hence is usually constrained to within the same server. The outermost parallelism may spread across a multi-hop network and should tolerate higher network latency. Therefore, based on the requirements for network bandwidth and latency, we place parallelism dimensions in the order of [TP, CP, PP, DP]. DP (i.e., FSDP) is the outermost parallelism because it can tolerate longer network latency by asynchronously prefetching sharded model weights and reducing gradients. Identifying the optimal parallelism configuration with minimal communication overhead while avoiding GPU memory overflow is challenging. We develop a memory consumption estimator and a performance-projection tool which helped us explore various parallelism configurations and project overall training performance and identify memory gaps effectively.

网络感知的并行配置。并行维度的顺序[TP, CP, PP, DP]针对网络通信进行了优化。最内层的并行性需要最高的网络带宽和最低的延迟，因此通常限制在同一服务器内。最外层的并行性可能跨越多跳网络，并应容忍更高的网络延迟。因此，根据对网络带宽和延迟的要求，我们将并行维度按[TP, CP, PP, DP]的顺序排列。DP（即FSDP）是最外层的并行性，因为它可以通过异步预取分片模型权重和减少梯度来容忍更长的网络延迟。在避免GPU内存溢出的同时，确定具有最小通信开销的最佳并行配置是一个挑战。我们开发了一个内存消耗估计器和一个性能预测工具，帮助我们探索各种并行配置，有效预测整体训练性能并识别内存缺口。

Numerical stability. By comparing training loss between different parallelism setups, we fixed several numerical issues that impact training stability. To ensure training convergence, we use FP32 gradient accumulation during backward computation over multiple micro-batches and also reduce-scatter gradients in FP32 across data parallel workers in FSDP. For intermediate tensors, e.g., vision encoder outputs, that are used multiple times in the forward computation, the backward gradients are also accumulated in FP32.

数值稳定性。通过比较不同并行设置下的训练损失，我们解决了影响训练稳定性的几个数值问题。为了确保训练收敛，我们在多个微批次的后向计算中使用FP32梯度累积，并在FSDP中的数据并行工作者之间使用FP32梯度进行减少散布。对于在前向计算中多次使用的中间张量，例如视觉编码器输出，后向梯度也在FP32中累积。

3.3.3 Collective Communication 集体通信

Our collective communication library for Llama 3 is based on a fork of Nvidia’s NCCL library, called NCCLX. NCCLX significantly improves the performance of NCCL, especially for higher latency networks. Recall that the order of parallelism dimensions is [ T P , C P , P P , D P ] \left\lbrack {\mathrm{{TP}},\mathrm{{CP}},\mathrm{{PP}},\mathrm{{DP}}}\right\rbrack [TP,CP,PP,DP] ,where D P \mathrm{{DP}} DP corresponds to FSDP. The outermost parallelism dimensions, PP and DP, may communicate through a multi-hop network, with latency up to tens of microseconds. The original NCCL collectives - all-gather and reduce-scatter in FSDP, and point-to-point in PP-require data chunking and staged data copy. This approach incurs several inefficiencies, including (1) requiring a large number of small control messages to be exchanged over the network to facilitate data transfer, (2) extra memory-copy operations, and (3) using extra GPU cycles for communication. For Llama 3 training, we address a subset of these inefficiencies by tuning chunking and data transfer to fit our network latencies, which can be as high as tens of microseconds for a large cluster. We also allow small control messages to traverse our network at a higher priority, especially avoiding being head-of-line blocked in deep-buffer core switches. Our ongoing work for future Llama versions involves making deeper changes in NCCLX to holistically address all the aforementioned problems.

我们为Llama 3开发的集体通信库基于Nvidia的NCCL库的一个分支，称为NCCLX。NCCLX显著提升了NCCL的性能，特别是在更高延迟的网络中。回想一下，并行维度的顺序是 [ T P , C P , P P , D P ] \left\lbrack {\mathrm{{TP}},\mathrm{{CP}},\mathrm{{PP}},\mathrm{{DP}}}\right\rbrack [TP,CP,PP,DP]，其中 D P \mathrm{{DP}} DP对应于FSDP。最外层的并行维度，PP和DP，可能通过多跳网络进行通信，延迟可达数十微秒。原始的NCCL集体操作——FSDP中的全收集和减少散布，以及PP中的点对点通信——需要数据分块和分阶段数据复制。这种方法会导致几个低效问题，包括（1）需要大量小控制消息通过网络交换以促进数据传输，（2）额外的内存复制操作，以及（3）使用额外的GPU周期进行通信。对于Llama 3的训练，我们通过调整分块和数据传输以适应我们的网络延迟（对于大型集群，延迟可能高达数十微秒）来解决这些低效问题的一部分。我们还允许小控制消息以更高优先级在我们的网络中传输，特别是避免在深度缓冲核心交换机中被头阻塞。我们正在进行的工作是为未来的Llama版本在NCCLX中进行更深入的更改，以全面解决上述所有问题。

Table 5 Root-cause categorization of unexpected interruptions during a 54-day period of Llama 3 405B pre-training. A b o u t \mathrm{{About}} About 78 % {78}\% 78% of unexpected interruptions were attributed to confirmed or suspected hardware issues.

表5 在Llama 3 405B预训练的54天期间，意外中断的根本原因分类。 A b o u t \mathrm{{About}} About 78 % {78}\% 78%的意外中断归因于已确认或疑似硬件问题。

3.3.4 Reliability and Operational Challenges 可靠性与运营挑战

The complexity and potential failure scenarios of 16    K {16}\mathrm{\;K} 16K GPU training surpass those of much larger CPU clusters that we have operated. Moreover, the synchronous nature of training makes it less fault-tolerant - a single GPU failure may require a restart of the entire job. Despite these challenges, for Llama 3, we achieved higher than 90 % {90}\% 90% effective training time while supporting automated cluster maintenance,such as firmware and Linux kernel upgrades (Vigraham and Leonhardi, 2024), which resulted in at least one training interruption daily. The effective training time measures the time spent on useful training over the elapsed time.

16    K {16}\mathrm{\;K} 16K GPU训练的复杂性和潜在故障场景超过了我们运营的许多大型CPU集群。此外，训练的同步性质使其容错性较低——单个GPU故障可能需要重新启动整个任务。尽管存在这些挑战，对于Llama 3，我们实现了超过 90 % {90}\% 90%的有效训练时间，同时支持自动化集群维护，如固件和Linux内核升级（Vigraham和Leonhardi，2024），这导致了至少每天一次的训练中断。有效训练时间衡量了在总时间中用于有效训练的时间。

During a 54-day snapshot period of pre-training, we experienced a total of 466 job interruptions. Of these, 47 were planned interruptions due to automated maintenance operations such as firmware upgrades or operator-initiated operations like configuration or dataset updates. The remaining 419 were unexpected interruptions, which are classified in Table 5. Approximately 78 % {78}\% 78% of the unexpected interruptions are attributed to confirmed hardware issues, such as GPU or host component failures, or suspected hardware-related issues like silent data corruption and unplanned individual host maintenance events. GPU issues are the largest category, accounting for 58.7 % {58.7}\% 58.7% of all unexpected issues. Despite the large number of failures,significant manual intervention was required only three times during this period, with the rest of issues handled by automation.

在预训练的54天快照期间，我们经历了总共466次任务中断。其中，47次是由于自动化维护操作（如固件升级）或操作员发起的操作（如配置或数据集更新）导致的计划中断。其余的419次是意外中断，分类见表5。大约 78 % {78}\% 78%的意外中断归因于已确认的硬件问题，如GPU或主机组件故障，或疑似硬件相关问题，如静默数据损坏和未计划的单个主机维护事件。GPU问题是最大的类别，占所有意外问题的 58.7 % {58.7}\% 58.7%。尽管故障数量众多，但在此期间仅三次需要大量人工干预，其余问题均由自动化处理。

To increase the effective training time, we reduced job startup and checkpointing time, and developed tools for fast diagnosis and problem resolution. We extensively use PyTorch’s built-in NCCL flight recorder (Ansel et al., 2024), a feature that captures collective metadata and stack traces into a ring buffer, and hence allowing us to diagnose hangs and performance issues quickly at scale, particularly with regard to NCCLX. Using this, we efficiently record every communication event and the duration of each collective operation, and also automatically dump tracing data on NCCLX watchdog or heartbeat timeout. We enable more computationally intensive tracing operations and metadata collection selectively as needed live in production through online configuration changes (Tang et al., 2015) without needing a code release or job restart.

为了增加有效的训练时间，我们减少了作业启动和检查点时间，并开发了快速诊断和问题解决工具。我们广泛使用 PyTorch 内置的 NCCL 飞行记录器（Ansel et al., 2024），该功能捕获集体元数据和堆栈跟踪到环形缓冲区，从而使我们能够在大规模上快速诊断挂起和性能问题，特别是在 NCCLX 方面。通过使用这个工具，我们高效地记录每个通信事件和每次集体操作的持续时间，并在 NCCLX 看门狗或心跳超时时自动转储跟踪数据。我们根据需要在生产环境中通过在线配置更改（Tang et al., 2015）有选择地启用更多计算密集型的跟踪操作和元数据收集，而无需代码发布或作业重启。

Debugging issues in large-scale training is complicated by the mixed use of NVLink and RoCE in our network. Data transfer over NVLink typically occurs through load/store operations issued by CUDA kernels, and failures in either the remote GPU or NVLink connectivity often manifest as stalled load/store operations within CUDA kernels without returning a clear error code. NCCLX enhances the speed and accuracy of failure detection and localization through a tight co-design with PyTorch, allowing PyTorch to access NCCLX’s internal state and track relevant information. While stalls due to NVLink failures cannot be completely prevented, our system monitors the state of the communication library and automatically times out when such a stall is detected. Additionally, NCCLX traces the kernel and network activities of each NCCLX communication and provides a snapshot of the failing NCCLX collective’s internal state, including finished and pending data transfers between all ranks. We analyze this data to debug NCCLX scaling issues.

在大规模训练中调试问题因我们网络中混合使用 NVLink 和 RoCE 而变得复杂。通过 NVLink 的数据传输通常由 CUDA 内核发出的加载/存储操作完成，远程 GPU 或 NVLink 连接性的故障通常表现为 CUDA 内核中停滞的加载/存储操作，而不会返回明确的错误代码。NCCLX 通过与 PyTorch 的紧密协同设计，提高了故障检测和定位的速度和准确性，使 PyTorch 能够访问 NCCLX 的内部状态并跟踪相关信息。虽然无法完全防止因 NVLink 故障导致的停滞，但我们的系统监控通信库的状态，并在检测到此类停滞时自动超时。此外，NCCLX 跟踪每个 NCCLX 通信的内核和网络活动，并提供失败 NCCLX 集体的内部状态快照，包括所有等级之间的已完成和待处理数据传输。我们分析这些数据以调试 NCCLX 的扩展问题。

Sometimes, hardware issues may cause still-functioning but slow stragglers that are hard to detect. Even a single straggler can slow down thousands of other GPUs, often appearing as functioning but slow communications. We developed tools to prioritize potentially problematic communications from selected process groups. By investigating just a few top suspects, we were usually able to effectively identify the stragglers.

有时，硬件问题可能导致仍在运行但速度缓慢的落后者，这些落后者难以检测。即使单个落后者也可能拖慢成千上万台其他GPU的速度，通常表现为功能正常但通信缓慢。我们开发了工具，用于从选定的进程组中优先处理可能存在问题的通信。通过调查少数几个主要嫌疑对象，我们通常能够有效地识别出落后者。

One interesting observation is the impact of environmental factors on training performance at scale. For Llama 3405B, we noted a diurnal 1-2% throughput variation based on time-of-day. This fluctuation is the result of higher mid-day temperatures impacting GPU dynamic voltage and frequency scaling.

一个有趣的观察是环境因素对大规模训练性能的影响。对于Llama 3405B，我们注意到基于时间的日变化，吞吐量有1-2%的波动。这种波动是中午高温影响GPU动态电压和频率缩放的结果。

During training, tens of thousands of GPUs may increase or decrease power consumption at the same time, for example, due to all GPUs waiting for checkpointing or collective communications to finish, or the startup or shutdown of the entire training job. When this happens, it can result in instant fluctuations of power consumption across the data center on the order of tens of megawatts, stretching the limits of the power grid. This is an ongoing challenge for us as we scale training for future, even larger Llama models.

在训练过程中，成千上万台GPU可能会同时增加或减少功耗，例如，由于所有GPU等待检查点或集体通信完成，或者整个训练作业的启动或关闭。当这种情况发生时，可能会导致数据中心功耗瞬间波动，达到数十兆瓦的量级，这超出了电网的承受极限。随着我们为未来更大的Llama模型进行训练规模的扩大，这是一个持续的挑战。

3.4 Training Recipe 训练配方

The recipe used to pre-train Llama 3405B consists of three main stages: (1) initial pre-training, (2) long-context pre-training, and (3) annealing. The three stages are described separately below. We use similar recipes to pre-train the 8    B 8\mathrm{\;B} 8B and 70    B {70}\mathrm{\;B} 70B models.

用于预训练Llama 3405B的配方包含三个主要阶段：（1）初始预训练，（2）长上下文预训练，和（3）退火。以下分别描述这三个阶段。我们使用类似的配方来预训练 8    B 8\mathrm{\;B} 8B和 70    B {70}\mathrm{\;B} 70B模型。

3.4.1 Initial Pre-Training 初始预训练

We pre-train Llama 3 405B using AdamW with a peak learning rate of 8 × 10 − 5 8 \times {10}^{-5} 8×10−5 ,a linear warm up of 8,000 steps,and a cosine learning rate schedule decaying to 8 × 10 − 7 8 \times {10}^{-7} 8×10−7 over 1 , 200 , 000 1,{200},{000} 1,200,000 steps. We use a lower batch size early in training to improve training stability, and increase it subsequently to improve efficiency. Specifically, we use an initial batch size of 4 M 4\mathrm{M} 4M tokens and sequences of length 4,096,and double these values to a batch size of 8 M 8\mathrm{M} 8M sequences of 8 , 192 8,{192} 8,192 tokens after pre-training 252 M {252}\mathrm{M} 252M tokens. We double the batch size again to 16 M {16}\mathrm{M} 16M after pre-training on 2.87 T tokens. We found this training recipe to be very stable: we observed few loss spikes and did not require interventions to correct for model training divergence.

我们使用AdamW对Llama 3 405B进行预训练，设定峰值学习率为 8 × 10 − 5 8 \times {10}^{-5} 8×10−5，采用线性预热8,000步，以及余弦学习率调度，在 1 , 200 , 000 1,{200},{000} 1,200,000步内衰减至 8 × 10 − 7 8 \times {10}^{-7} 8×10−7。在训练初期使用较低的批次大小以提高训练稳定性，随后增加批次大小以提高效率。具体而言，我们初始批次大小为 4 M 4\mathrm{M} 4M个令牌，序列长度为4,096，并在预训练 252 M {252}\mathrm{M} 252M个令牌后，将批次大小和序列长度分别翻倍至 8 M 8\mathrm{M} 8M和 8 , 192 8,{192} 8,192。在预训练2.87万亿个令牌后，我们将批次大小再次翻倍至 16 M {16}\mathrm{M} 16M。我们发现这种训练配方非常稳定：观察到的损失峰值很少，且无需干预来纠正模型训练偏差。

Adjusting the data mix. We made a several adjustments to the pre-training data mix during training to improve model performance on particular downstream tasks. In particular, we increased the percentage of non-English data during pre-training to improve the multilingual performance of Llama 3. We also upsample mathematical data to improve the model’s mathematical reasoning performance, we added more recent web data in the later stages of pre-training to advance the model’s knowledge cut-off, and we downsampled subsets of the pre-training data that were later identified as being lower quality.

调整数据混合比例。在训练过程中，我们对预训练数据混合比例进行了多次调整，以提高模型在特定下游任务上的性能。特别是，我们增加了非英语数据的百分比，以提升Llama 3的多语言性能。我们还增加了数学数据的采样率，以提高模型的数学推理性能，在预训练后期阶段增加了更近期的网络数据，以推进模型的知识截止点，并对后来被识别为质量较低的预训练数据子集进行了降采样。

3.4.2 Long Context Pre-Training 长上下文预训练

In the final stages of pre-training, we train on long sequences to support context windows of up to 128 K {128}\mathrm{K} 128K tokens. We do not train on long sequences earlier because the compute in self-attention layers grows quadratically in the sequence length. We increase the supported context length in increments, pre-training until the model has successfully adapted to the increased context length. We assess successful adaptation by measuring whether (1) model performance on short-context evaluations has recovered completely and (2) the model perfectly solves “needle in a haystack” tasks up to that length. In Llama 3405B pre-training, we increased context length gradually in six stages,starting from the original 8    K 8\mathrm{\;K} 8K context window and ending in the final 128    K {128}\mathrm{\;K} 128K context window. This long-context pre-training stage was performed using approximately 800    B {800}\mathrm{\;B} 800B training tokens.

在预训练的最后阶段，我们训练长序列以支持最长 128 K {128}\mathrm{K} 128K 个标记的上下文窗口。我们不在早期训练长序列，因为自注意力层的计算量随序列长度呈二次方增长。我们逐步增加支持的上下文长度，预训练直到模型成功适应增加的上下文长度。我们通过测量（1）模型在短上下文评估中的性能是否完全恢复，以及（2）模型是否完美解决了长达该长度的“大海捞针”任务，来评估成功的适应。在 Llama 3405B 预训练中，我们分六个阶段逐步增加上下文长度，从最初的 8    K 8\mathrm{\;K} 8K 上下文窗口开始，到最终的 128    K {128}\mathrm{\;K} 128K 上下文窗口结束。这一长上下文预训练阶段使用了大约 800    B {800}\mathrm{\;B} 800B 个训练标记。

Figure 7 Illustration of the overall post-training approach for Llama 3. Our post-training strategy involves rejection sampling, supervised finetuning, and direct preference optimization. See text for details.

图 7 展示了 Llama 3 的整体后训练方法。我们的后训练策略涉及拒绝采样、监督微调和直接偏好优化。详见正文。

3.4.3 Annealing 退火

During pre-training on the final 40 M {40}\mathrm{M} 40M tokens,we linearly annealed the learning rate to 0,maintaining a context length of 128    K {128}\mathrm{\;K} 128K tokens. During this annealing phase,we also adjusted the data mix to upsample data sources of very high quality; see Section 3.1.3. Finally, we compute the average of model checkpoints (Polyak (1991) averaging) during annealing to produce the final pre-trained model.

在预训练最后 40 M {40}\mathrm{M} 40M 个标记的过程中，我们将学习率线性退火至 0，保持 128    K {128}\mathrm{\;K} 128K 个标记的上下文长度。在这一退火阶段，我们还调整了数据混合，以增加高质量数据源的采样率；参见第 3.1.3 节。最后，我们在退火期间计算模型检查点的平均值（Polyak (1991) 平均），以生成最终的预训练模型。

4 Post-Training 后训练

We produce the aligned Llama 3 models by applying several rounds of post-training, 6 {}^{6} 6 or aligning the model with human feedback (Ouyang et al., 2022; Rafailov et al., 2024) on top of a pre-trained checkpoint. Each round of post-training involves supervised finetuning (SFT) followed by Direct Preference Optimization (DPO; Rafailov et al., 2024) on examples collected either via human annotations or generated synthetically. Our post-training modeling and data approaches are described in Sections 4.1 and 4.2 respectively. We further detail custom data curation strategies to improve the reasoning, coding, factuality, multilingual, tool use, long context, and precise instruction following in Section 4.3.

我们通过应用多轮后训练， 6 {}^{6} 6 或在预训练检查点基础上结合人类反馈（Ouyang et al., 2022; Rafailov et al., 2024）来生成对齐的 Llama 3 模型。每轮后训练包括监督微调（SFT），随后在通过人工标注或合成生成的示例上进行直接偏好优化（DPO；Rafailov et al., 2024）。我们的后训练建模和数据方法分别在第 4.1 和 4.2 节中描述。我们进一步详细介绍了定制数据筛选策略，以提高推理、编码、事实性、多语言、工具使用、长上下文和精确指令遵循能力，详见第 4.3 节。

4.1 Modeling 建模

The backbone of our post-training strategy is a reward model and a language model. We first train a reward model on top of the pre-trained checkpoint using human-annotated preference data (see Section 4.1.2). We then finetune pre-trained checkpoints with supervised finetuning (SFT; see Section 4.1.3), and further align the checkpoints with Direct Preference Optimization (DPO; see Section 4.1.4). This process is illustrated in Figure 7. Unless otherwise noted, our modeling procedure applies to Llama 3 405B, and we refer to Llama 3 405B as Llama 3 for simplicity.

我们的后训练策略的核心是一个奖励模型和一个语言模型。我们首先在预训练检查点基础上使用人工标注的偏好数据（见第 4.1.2 节）训练一个奖励模型。然后，我们对预训练检查点进行监督微调（SFT；见第 4.1.3 节），并进一步通过直接偏好优化（DPO；见第 4.1.4 节）对检查点进行对齐。此过程如图 7 所示。除非另有说明，我们的建模过程适用于 Llama 3 405B，为简洁起见，我们将 Llama 3 405B 简称为 Llama 3。

4.1.1 Chat Dialog Format 聊天对话格式

To tune LLMs for human-AI interaction, we need to define a chat dialog protocol for the model to understand human instructions and perform conversational tasks. Compared to its predecessor, Llama 3 has new capabilities such as tool use (Section 4.3.5) which may require generating multiple messages and sending them to different locations (e.g., user, ipython) within a single dialog turn. To support this, we design a new multi-message chat protocol which uses various special header and termination tokens. The header tokens are used to indicate the source and destination of each message in a conversation. Similarly, the termination tokens indicate when it is the time to alternate between human and AI to speak.

为了调整大型语言模型以适应人机交互，我们需要为模型定义一个聊天对话协议，使其能够理解人类指令并执行对话任务。与前代产品相比，Llama 3 新增了工具使用（第 4.3.5 节）等能力，这可能需要在单个对话轮次内生成多条消息并发送至不同位置（例如，用户、ipython）。为此，我们设计了一种新的多消息聊天协议，该协议使用各种特殊头部和终止标记。头部标记用于指示对话中每条消息的来源和目的地。同样，终止标记用于指示何时轮到人类和人工智能交替发言。

6 {}^{6} 6 We use the term “post-training” to refer to any model training that happens outside of pre-training.

6 {}^{6} 6 我们使用“后训练”这一术语来指代在预训练之外进行的任何模型训练。

4.1.2 Reward Modeling 奖励模型

We train a reward model (RM) covering different capabilities on top of the pre-trained checkpoint. The training objective is the same as Llama 2 except that we remove the margin term in the loss, as we observe diminishing improvements after data scaling. Following Llama 2, we use all of our preference data for reward modeling after filtering out samples with similar responses. In addition to standard preference pair of (chosen, rejected) response, annotations also create a third “edited response” for some prompts, where the chosen response from the pair is further edited for improvement (see Section 4.2.1). Hence, each preference ranking sample has two or three responses with clear ranking (edited > > > chosen > > > rejected). We concatenate the prompt and multiple responses into a single row during training with responses randomly shuffled. This is an approximation to the standard scenario of putting the responses in separate rows and computing the scores, but in our ablations, this approach improves training efficiency without a loss in accuracy.

我们在预训练检查点的基础上训练了一个涵盖不同能力的奖励模型（RM）。训练目标与 Llama 2 相同，只是我们去除了损失中的边际项，因为我们观察到数据规模扩大后改进效果递减。与 Llama 2 一样，我们在过滤掉响应相似的样本后，使用所有的偏好数据进行奖励模型训练。除了标准的偏好对（被选中的，被拒绝的）响应外，注释还为某些提示创建了第三种“编辑后的响应”，其中来自配对的被选中的响应被进一步编辑以改进（见第 4.2.1 节）。因此，每个偏好排序样本都有两个或三个响应，且排序明确（编辑后的 > > > 被选中的 > > > 被拒绝的）。我们在训练过程中将提示和多个响应连接成单行，并对响应进行随机洗牌。这是将响应放在单独行并计算分数的标准场景的近似，但在我们的消融实验中，这种方法提高了训练效率，且没有损失准确性。

4.1.3 Supervised Finetuning 监督微调

The reward model is then used to perform rejection sampling on our human annotation prompts, the details of which are described in Section 4.2. Together with this rejection-sampled data and other data sources (including synthetic data), we finetune the pre-trained language model using a standard cross entropy loss on the target tokens (while masking loss on prompt tokens). More details about the data mix can be found in Section 4.2. We refer to this stage as supervised finetuning (SFT; Wei et al., 2022a; Sanh et al., 2022; Wang et al., 2022b), even though many of the training targets are model-generated. Our largest models are finetuned with a learning rate of 10 − 5 {10}^{-5} 10−5 over the course of 8.5    K {8.5}\mathrm{\;K} 8.5K to 9    K 9\mathrm{\;K} 9K steps. We found these hyperparameter settings to work well across different rounds and data mixes.

然后，奖励模型用于对我们的标注提示进行拒绝采样，详细信息在第4.2节中描述。结合这些拒绝采样数据和其他数据源（包括合成数据），我们使用标准交叉熵损失对目标词进行微调预训练语言模型（同时屏蔽提示词的损失）。有关数据混合的更多细节可以在第4.2节中找到。我们称这一阶段为监督微调（SFT；Wei et al., 2022a；Sanh et al., 2022；Wang et al., 2022b），尽管许多训练目标是模型生成的。我们的最大模型在 8.5    K {8.5}\mathrm{\;K} 8.5K到 9    K 9\mathrm{\;K} 9K步的过程中以 10 − 5 {10}^{-5} 10−5的学习率进行微调。我们发现这些超参数设置在不同轮次和数据混合中表现良好。

4.1.4 Direct Preference Optimization 直接偏好优化

We further train our SFT models with Direct Preference Optimization (DPO; Rafailov et al., 2024) for human preference alignment. For training, we primarily use the most recent batches of preference data collected using the best performing models from the previous alignment rounds. As a result, our training data conforms better to the distribution of the policy model that is being optimized in each round. We also explored on-policy algorithms such as PPO (Schulman et al., 2017), but found that DPO required less compute for large-scale models and performed better, especially on instruction following benchmarks like IFEval (Zhou et al., 2023). For Llama 3,we use a learning rate of 10 − 5 {10}^{-5} 10−5 and set the β \beta β hyper-parameter to be 0.1 . In addition,we apply the following algorithmic modifications to DPO:

我们进一步使用直接偏好优化（DPO；Rafailov et al., 2024）对SFT模型进行人类偏好对齐训练。在训练中，我们主要使用通过上一轮对齐中表现最佳的模型收集的最新偏好数据批次。因此，我们的训练数据更符合每一轮正在优化的策略模型的分布。我们还探索了如PPO（Schulman et al., 2017）这样的在线策略算法，但发现DPO对大规模模型需要的计算量更少，且表现更好，尤其是在遵循指令的基准测试如IFEval（Zhou et al., 2023）上。对于Llama 3，我们使用 10 − 5 {10}^{-5} 10−5的学习率，并将 β \beta β超参数设置为0.1。此外，我们对DPO应用了以下算法修改：

Masking out formatting tokens in DPO loss: We mask out special formatting tokens including header and termination tokens (described in Section 4.1.1) from both chosen and rejected responses in the loss to stabilize DPO training. We observe that having these tokens contribute to the loss may lead to undesired model behaviors such as tail repetition or abruptly generating termination tokens. We hypothesize that this is due to the contrastive nature of the DPO loss - the presence of common tokens in both chosen and rejected responses leads to a conflicting learning objective as the model needs to increase and reduce the likelihood of these tokens simultaneously.

在DPO损失中屏蔽格式化标记：我们从损失中的选定和拒绝响应中屏蔽掉包括标题和终止标记（在4.1.1节中描述）在内的特殊格式化标记，以稳定DPO训练。我们观察到，这些标记对损失的贡献可能导致不希望的模型行为，如尾部重复或突然生成终止标记。我们假设这是由于DPO损失的对比性质——选定和拒绝响应中共同标记的存在导致了一个冲突的学习目标，因为模型需要同时增加和减少这些标记的可能性。

Regularization with NLL loss: We add an additional negative log-likelihood (NLL) loss term with a scaling coefficient of 0.2 on the chosen sequences, similar to Pang et al. (2024). This helps further stabilize DPO training by maintaining desired formatting for generation and preventing the decrease of log probability of chosen responses (Pang et al., 2024; Pal et al., 2024).

使用NLL损失进行正则化：我们在选定序列上添加一个额外的负对数似然（NLL）损失项，其缩放系数为0.2，类似于Pang等人（2024年）的做法。这有助于通过保持生成所需的格式并防止选定响应的对数概率下降来进一步稳定DPO训练（Pang等人，2024年；Pal等人，2024年）。

4.1.5 Model Averaging 模型平均

Finally, we average models obtained from experiments using various versions of data or hyperparameters at each RM, SFT, or DPO stage (Izmailov et al., 2019; Wortsman et al., 2022; Li et al., 2022).

最后，我们平均了在每个RM、SFT或DPO阶段使用不同版本数据或超参数获得的模型（Izmailov等人，2019年；Wortsman等人，2022年；Li等人，2022年）。

Table 6 Statistics of human preference data. We list statistics of the internally collected human preference data used for Llama 3 alignment. We ask annotators to perform multi-turn dialogues with the models and make comparisons among responses at each turn. In post-processing, we split each dialogue to multiple examples at a turn level. Each example consists of a prompt (including previous dialog if available) and a response (e.g., chosen or rejected response).

表6 人类偏好数据的统计。我们列出了用于Llama 3对齐的内部收集的人类偏好数据的统计信息。我们要求标注者与模型进行多轮对话，并在每轮中对响应进行比较。在后处理中，我们将每个对话拆分为多个以轮为单位的示例。每个示例包括一个提示（如果可用，包括之前的对话）和一个响应（例如，选定的或拒绝的响应）。

4.1.6 Iterative Rounds 迭代轮次

Following Llama 2, we apply the above methods in six rounds. In each cycle, we collect new preference annotations and SFT data, sampling synthetic data from the latest models.

继Llama 2之后，我们在六轮中应用上述方法。在每个周期中，我们收集新的偏好注释和SFT数据，从最新模型中抽取合成数据。

4.2 Post-training Data 后训练数据

The post-training data composition plays a critical role in the usefulness and behavior of language models. In this section, we discuss our human annotation procedures and preference data collection (Section 4.2.1), the composition of our SFT data (Section 4.2.2), and methods for data quality control and cleaning (Section 4.2.3).

后训练数据组成对语言模型的有用性和行为起着关键作用。本节中，我们将讨论我们的人工注释流程和偏好数据收集（第4.2.1节），我们的SFT数据组成（第4.2.2节），以及数据质量控制和清理的方法（第4.2.3节）。

4.2.1 Preference Data 偏好数据

Our preference data annotation process is similar to Llama 2. We deploy multiple models for annotation after each round and sample two responses from two different models for each user prompt. These models can be trained with different data mixes and alignment recipes, allowing for different capability strength (e.g., code expertise) and increased data diversity. We ask annotators to rate the strength of their preference by categorizing it into one of four levels, based on how much more they prefer the chosen response over the rejected one: significantly better, better, slightly better, or marginally better. We also incorporate an editing step after preference ranking to encourage annotators to further improve the preferred response. Annotators edit the chosen response directly or prompt the model with feedback to refine its own response. Consequently, a portion of our preference data has three responses ranked (edited > > > chosen > > > rejected).

我们的偏好数据注释流程与Llama 2类似。在每一轮之后，我们部署多个模型进行注释，并为每个用户提示从两个不同模型中抽取两个响应。这些模型可以使用不同的数据混合和调整方案进行训练，从而允许不同的能力强度（例如，代码专业知识）和增加数据多样性。我们要求注释者根据他们对所选响应相对于被拒绝响应的偏好程度，将其分类为四个级别之一：显著更好、更好、略好或稍好。我们还引入了一个编辑步骤，在偏好排序后鼓励注释者进一步改进所选响应。注释者直接编辑所选响应或通过反馈提示模型改进其自身响应。因此，我们的部分偏好数据包含三个排序的响应（编辑后的 > > > 所选的 > > > 被拒绝的）。

In Table 6, we report the statistics of preference annotations that we use for Llama 3 training. General English covers multiple subcategories such as knowledge-based question and answering or precise instruction-following, which fall outside the scope of specific capabilities. Compared to Llama 2, we observe an increase in the average length of prompt and response, suggesting that we train Llama 3 on more complex tasks. In addition, we implement a quality analysis and human evaluation process to rigorously assess the data collected, allowing us to refine our prompts and provide systematic, actionable feedback to annotators. For example, as Llama 3 improves after each round, we increase prompt complexity accordingly to target areas where the model lags.

在表6中，我们报告了用于Llama 3训练的偏好注释的统计数据。通用英语涵盖了多个子类别，如基于知识的问答或精确的指令遵循，这些都超出了特定能力的范围。与Llama 2相比，我们观察到提示和响应的平均长度增加，这表明我们训练Llama 3处理更复杂的任务。此外，我们实施了质量分析和人工评估流程，以严格评估收集的数据，使我们能够优化提示并提供系统性、可操作的反馈给注释者。例如，随着Llama 3在每一轮改进后，我们相应增加提示的复杂度，以针对模型落后的领域。

In each round of post-training, we use all the preference data that is available at the time for reward modeling, while only using the latest batches from various capabilities for DPO training. For both reward modeling and DPO, we use samples that are labeled as the chosen response being significantly better or better than the rejected counterpart for training and discard samples with similar responses.

在每一轮后训练中，我们使用当时可用的所有偏好数据进行奖励建模，而仅使用来自各种能力的最新批次进行DPO训练。对于奖励建模和DPO，我们使用标记为被选响应明显优于或优于被拒绝的对应响应的样本进行训练，并丢弃响应相似的样本。

4.2.2 SFT Data SFT数据

Our finetuning data is largely comprised of the following sources:

我们的微调数据主要由以下来源组成：

Prompts from our human annotation collection with rejection-sampled responses.

来自我们人工注释收集的提示与拒绝采样的响应。

Synthetic data targeting specific capabilities (see Section 4.3 for more details).

针对特定能力生成的合成数据（更多细节见第4.3节）。

Table 7 Statistics of SFT data. We list internally collected SFT data used for Llama 3 alignment. Each SFT example consists of a context (i.e., all conversation turns except the last one) and a final response.

表7 SFT数据的统计。我们列出了用于Llama 3对齐的内部收集的SFT数据。每个SFT示例包括一个上下文（即所有对话轮次，除最后一轮外）和一个最终响应。

Small amounts of human-curated data (see Section 4.3 for more details).

少量人工精选数据（更多细节见第4.3节）。

As our post-training rounds progress, we develop stronger Llama 3 variants that we use to collect larger datasets that cover a wide range of complex capabilities. In this section, we discuss the details for the rejection-sampling procedure and overall composition of our final SFT datamix.

随着我们后训练轮次的进展，我们开发了更强大的Llama 3变体，用于收集涵盖广泛复杂能力的大型数据集。在本节中，我们将讨论拒绝采样程序的细节以及我们最终SFT数据集的整体组成。

Rejection sampling. During rejection sampling (RS), for each prompt collected during human annotation (Section 4.2.1) we sample K K K (typically between 10 and 30) outputs from the latest chat model policy (usually the best performing checkpoint from the previous post-training iteration, or the best performing checkpoint for a particular capability) and use our reward model to select the best candidate, consistent with Bai et al. (2022). In later rounds of post-training, we introduce system prompts to steer RS responses to conform with desirable tone, style, or formatting, which might be different for different capabilities.

拒绝采样。在拒绝采样（RS）过程中，对于在人工标注期间收集的每个提示（第4.2.1节），我们从最新的聊天模型策略（通常是上一轮后训练迭代中表现最佳的检查点，或针对特定能力表现最佳的检查点）中采样 K K K（通常在10到30之间）输出，并使用我们的奖励模型选择最佳候选，与Bai等人（2022）一致。在后训练的后续轮次中，我们引入系统提示以引导RS响应符合理想的语调、风格或格式，这些可能因不同能力而异。

To increase the efficiency of rejection sampling, we adopt PagedAttention (Kwon et al., 2023). PagedAttention enhances memory efficiency through dynamic key-value cache allocation. It supports arbitrary output lengths by dynamically scheduling requests based on the current cache capacity. Unfortunately, this carries the risk of swap-out when running out of memory. To eliminate such swap overhead, we define a maximum output length and perform a request only if sufficient memory is available to fit an output with that length. PagedAttention also enables us to share the key-value cache pages for a prompt across all corresponding outputs. Together, this leads to a throughput improvement of over 2 × 2 \times 2× during rejection sampling.

为了提高拒绝采样的效率，我们采用了PagedAttention（Kwon等人，2023）。PagedAttention通过动态键值缓存分配增强了内存效率。它通过根据当前缓存容量动态调度请求来支持任意输出长度。不幸的是，这带来了内存耗尽时交换出去的风险。为了消除这种交换开销，我们定义了最大输出长度，并且仅在有足够内存容纳该长度的输出时才执行请求。PagedAttention还使我们能够为提示共享所有相应输出的键值缓存页。这些共同导致了拒绝采样期间吞吐量提高了超过 2 × 2 \times 2×。

Overall data composition. Table 7 shows data statistics for each broad category of our “helpfulness” mix. While SFT and preference data contain overlapping domains, they are curated differently, yielding distinct count statistics. In Section 4.2.3 we describe techniques for categorizing topic, complexity, and quality of our data samples. In each round of post-training, we adjust our overall data mix carefully across these axes to tune performance across a wide range of benchmarks. Our final data mix epochs multiple times on some high quality sources and downsamples others.

总体数据构成。表7显示了我们“帮助性”混合中每个大类别的数据统计。尽管SFT和偏好数据包含重叠的领域，但它们的策划方式不同，产生了不同的计数统计。在第4.2.3节中，我们描述了用于分类主题、复杂性和数据样本质量的技术。在每一轮后训练中，我们仔细调整这些轴上的总体数据混合，以在广泛的基准上调整性能。我们的最终数据混合在某些高质量来源上多次迭代，并对其他来源进行降采样。

4.2.3 Data Processing and Quality Control 数据处理和质量控制

Given that most of our training data is model-generated, it requires careful cleaning and quality control.

鉴于我们的大部分训练数据是模型生成的，因此需要仔细进行清洗和质量控制。

Data cleaning. In the early rounds, we observed a number of undesirable patterns common in our data, such as excessive use of emojis or exclamation points. Therefore, we implement a series of rule-based data removal and modification strategies to filter or clean problematic data. For example, to mitigate overly-apologetic tonal issues, we identify overused phrases (such as “I’m sorry” or “I apologize”) and carefully balance the proportion of such samples in our dataset.

数据清洗。在早期轮次中，我们观察到数据中存在一些不良模式，例如过度使用表情符号或感叹号。因此，我们实施了一系列基于规则的数据删除和修改策略来过滤或清理有问题的数据。例如，为了缓解过度道歉的语调问题，我们识别出过度使用的短语（如“I’m sorry”或“I apologize”），并仔细平衡这些样本在我们数据集中的比例。

Data pruning. We also apply a collection of model-based techniques to remove low-quality training samples and improve overall model performance:

数据修剪。我们还应用了一系列基于模型的技术来移除低质量的训练样本，并提高整体模型性能：

Topic classification: We first finetune Llama 3 8B into a topic classifier, and perform inference over all data to classify it into both coarsely-grained buckets (“mathematical reasoning”) and fine-grained buckets (“geometry and trigonometry”).

主题分类：我们首先将 Llama 3 8B 微调为一个主题分类器，并对所有数据进行推理，将其分类为粗粒度类别（如“数学推理”）和细粒度类别（如“几何与三角学”）。

Quality scoring: We use both reward model and Llama-based signals to obtain a quality score for each sample. For an RM-based score, we consider data that is in the top quartile of RM scores as high quality. For a Llama-based score, we prompt Llama 3 checkpoint to rate each sample on a three-point scale for general English data (accuracy, instruction following, and tone/presentation) and a two-point scale for coding data (bug identification and user intention), and consider samples that obtain the maximum score as high quality. The RM and Llama-based scores have high disagreement rates, and we find that combining these signals yield the best recall on our internal test set. Ultimately, we select examples that are marked as high quality by the RM or the Llama-based filter.

质量评分：我们同时使用奖励模型和基于 Llama 的信号来为每个样本获取质量评分。对于基于 RM 的评分，我们认为位于 RM 评分前四分之一的数据显示为高质量。对于基于 Llama 的评分，我们提示 Llama 3 检查点对每个样本在一般英语数据（准确性、指令遵循和语调/呈现）上进行三点量表评分，在编程数据（错误识别和用户意图）上进行两点量表评分，并将获得最高分的样本视为高质量。RM 和基于 Llama 的评分存在高度不一致，我们发现结合这些信号在我们的内部测试集上获得了最佳召回率。最终，我们选择了被 RM 或基于 Llama 的过滤器标记为高质量的示例。

Difficulty scoring: Because we are also interested in prioritizing examples that are more complex for the model, we score data using two measures of difficulty: Instag (Lu et al., 2023) and Llama-based scoring. For Instag, we prompt Llama 3 70B to perform intention tagging of SFT prompts, where more intentions implies more complexity. We also prompt Llama 3 to measure the difficulty (Liu et al., 2024c) of dialogs on a three-point scale.

难度评分：由于我们也对优先处理模型更复杂的示例感兴趣，我们使用两种难度度量来评分数据：Instag（Lu et al., 2023）和基于Llama的评分。对于Instag，我们提示Llama 3 70B对SFT提示进行意图标记，意图越多意味着越复杂。我们还提示Llama 3在三点量表上测量对话的难度（Liu et al., 2024c）。

Semantic deduplication: Finally, we perform semantic deduplication (Abbas et al., 2023; Liu et al., 2024c). We first cluster complete dialogs using RoBERTa (Liu et al., 2019b) and within each cluster sort them by quality score × \times × difficulty score. We then do greedy selection by iterating through all sorted examples, and only keeping the ones that have maximum cosine similarity less than a threshold to the examples seen so far in the cluster.

语义去重：最后，我们进行语义去重（Abbas et al., 2023; Liu et al., 2024c）。我们首先使用RoBERTa（Liu et al., 2019b）对完整对话进行聚类，并在每个聚类中按质量分数 × \times ×难度分数进行排序。然后我们通过遍历所有排序后的示例进行贪心选择，只保留那些与当前聚类中已见示例的余弦相似度小于阈值的示例。

4.3 Capabilities 能力

We highlight special efforts to improve performance for specific capabilities such as code (Section 4.3.1), multilinguality (Section 4.3.2), math and reasoning (Section 4.3.3), long context (Section 4.3.4), tool use (Section 4.3.5), factuality (Section 4.3.6), and steerability (Section 4.3.7).

我们特别强调了针对特定能力提升性能的努力，如代码（第4.3.1节）、多语言能力（第4.3.2节）、数学和推理（第4.3.3节）、长上下文（第4.3.4节）、工具使用（第4.3.5节）、事实性（第4.3.6节）和可操控性（第4.3.7节）。

4.3.1 Code 代码

LLMs for code have received significant attention since the release of Copilot and Codex (Chen et al., 2021). Developers are now widely using these models to generate code snippets, debug, automate tasks, and improve code quality. For Llama 3, we target improving and evaluating code generation, documentation, debugging, and review capabilities for the following high priority programming languages: Python, Java, Javascript, C / C + + \mathrm{C}/\mathrm{C} + + C/C++ ,Typescript,Rust,PHP,HTML/CSS,SQL,bash/shell. Here,we present our work on improving these coding capabilities via training a code expert, generating synthetic data for SFT, improving formatting with system prompt steering, and creating quality filters to remove bad samples from our training data.

自Copilot和Codex（Chen et al., 2021）发布以来，用于代码的大型语言模型受到了广泛关注。开发者现在广泛使用这些模型来生成代码片段、调试、自动化任务以及提高代码质量。对于Llama 3，我们的目标是改进和评估代码生成、文档编写、调试和审查能力，针对以下高优先级编程语言：Python、Java、Javascript、 C / C + + \mathrm{C}/\mathrm{C} + + C/C++、Typescript、Rust、PHP、HTML/CSS、SQL、bash/shell。在这里，我们介绍了通过训练代码专家、生成合成数据用于SFT、通过系统提示引导改进格式以及创建质量过滤器从训练数据中移除不良样本来提高这些编码能力的工作。

Expert training. We train a code expert which we use to collect high quality human annotations for code throughout subsequent rounds of post-training. This is accomplished by branching the main pre-training run and continuing pre-training on a 1T token mix of mostly ( > 85 % ) \left( { > {85}\% }\right) (>85%) code data. Continued pre-training on domain-specific data has been shown to be effective for improving performance in a specific domain (Gururangan et al., 2020). We follow a recipe similar to that of CodeLlama (Rozière et al., 2023). For the last several thousand steps of training we perform long-context finetuning (LCFT) to extend the expert’s context length to 16    K {16}\mathrm{\;K} 16K tokens on a high quality mix of repo-level code data. Finally,we follow the similar post-training modeling recipes described in Section 4.1 to align this model, except with SFT and DPO data mixes primarily targeting code. This model is also used for rejection sampling (Section 4.2.2) for coding prompts.

专家训练。我们训练了一个代码专家，用于在后续的训练后轮次中收集高质量的人工代码注释。这是通过从主预训练运行中分支出一个分支，并在大部分 ( > 85 % ) \left( { > {85}\% }\right) (>85%)代码数据的1T令牌混合上继续预训练来实现的。继续在特定领域的数据上进行预训练已被证明对提高特定领域的表现有效（Gururangan et al., 2020）。我们遵循类似于CodeLlama（Rozière et al., 2023）的方法。在训练的最后几千步中，我们进行长上下文微调（LCFT），将专家的上下文长度扩展到 16    K {16}\mathrm{\;K} 16K令牌，使用高质量的仓库级代码数据混合。最后，我们遵循第4.1节中描述的类似的训练后建模方法来对齐这个模型，除了主要针对代码的SFT和DPO数据混合。该模型还用于编码提示的拒绝采样（第4.2.2节）。

Synthetic data generation. During development, we identified key issues in code generation, including difficulty in following instructions, code syntax errors, incorrect code generation, and difficulty in fixing bugs. While intensive human annotation could theoretically resolve these issues, synthetic data generation offers a complementary approach at a lower cost and higher scale, unconstrained by the expertise level of annotators. As such, we use Llama 3 and the code expert to generate a large quantity of synthetic SFT dialogs.

合成数据生成。在开发过程中，我们发现了代码生成中的关键问题，包括难以遵循指令、代码语法错误、错误的代码生成以及难以修复错误。虽然密集的人工标注理论上可以解决这些问题，但合成数据生成提供了一种成本更低、规模更高的补充方法，不受标注者专业水平的限制。因此，我们使用Llama 3和代码专家生成了大量合成SFT对话。

We describe three high-level approaches for generating synthetic code data. In total, we generate over 2.7M synthetic examples which were used during SFT.

我们描述了三种生成合成代码数据的高级方法。总共生成了超过270万个合成示例，这些示例在SFT期间被使用。

Synthetic data generation: execution feedback. The 8B and 70B models show significant performance improvements when trained on data generated by a larger, more competent model. However, our initial experiments revealed that training Llama 3405B on its own generated data is not helpful (and can even degrade performance). To address this limitation, we introduced execution feedback as a source of truth, enabling the model to learn from its mistakes and stay on track. In particular, we generate large dataset of approximately one million synthetic coding dialogues using the following process:

合成数据生成：执行反馈。当8B和70B模型在由更大、更强的模型生成的数据上训练时，显示出显著的性能提升。然而，我们最初的实验表明，在自身生成的数据上训练Llama 3405B没有帮助（甚至可能降低性能）。为了解决这一限制，我们引入了执行反馈作为事实来源，使模型能够从错误中学习并保持在正确的轨道上。特别是，我们使用以下过程生成了大约一百万个合成编码对话的大型数据集：

Problem description generation: First, we generate a large collection of programming problem descriptions that span a diverse range of topics, including those in the long tail distribution. To achieve this diversity, we sample random code snippets from various sources and prompt the model to generate programming problems inspired by these examples. This allowed us to tap into a wide range of topics and create a comprehensive set of problem descriptions (Wei et al., 2024).

问题描述生成：首先，我们生成了大量涵盖广泛主题的编程问题描述，包括长尾分布中的主题。为了实现这种多样性，我们从各种来源抽取随机代码片段，并提示模型根据这些示例生成编程问题。这使我们能够触及广泛的主题并创建全面的问题描述集（Wei等人，2024）。

Solution generation: Then, we prompt Llama 3 to solve each problem in a given programming language. We observe that adding general rules of good programming to the prompt improves the generated solution quality. Also, we find it is helpful to require the model to explain its thought process in comments.

解决方案生成：然后，我们提示Llama 3用给定的编程语言解决每个问题。我们观察到，在提示中添加良好的编程通用规则可以提高生成解决方案的质量。此外，我们发现要求模型在注释中解释其思维过程是有帮助的。

Correctness analysis: After generating a solution, it is crucial to recognize that its correctness is not guaranteed, and including incorrect solutions in the finetuning dataset could harm the model’s quality. While we do not ensure complete correctness, we develop methods to approximate it. To achieve this, we extract the source code from the generated solution and applied a combination of static and dynamic analysis techniques to test its correctness, including:

正确性分析：在生成解决方案后，认识到其正确性并未得到保证至关重要，将不正确的解决方案包含在微调数据集中可能会损害模型的质量。虽然我们不保证完全正确，但我们开发了方法来近似正确性。为此，我们从生成的解决方案中提取源代码，并应用静态和动态分析技术的组合来测试其正确性，包括：

Static analysis: We run all generated code through a parser and a linter to ensure syntactic correctness, catching errors such as syntax errors, use of uninitialized variables or non-imported functions, code style issues, typing errors, and others.

静态分析：我们将所有生成的代码通过解析器和代码检查工具运行，以确保语法正确性，捕捉诸如语法错误、未初始化变量的使用或未导入的函数、代码风格问题、类型错误等。

Unit test generation and execution: For each problem and solution, we prompt the model to generate unit tests, executed in a containerized environment together with the solution, catching run-time execution errors and some semantic errors.

单元测试生成与执行：对于每个问题和解决方案，我们提示模型生成单元测试，并在容器化环境中与解决方案一起执行，捕捉运行时执行错误和一些语义错误。

Error feedback and iterative self-correction: When a solution fails at any step, we prompt the model to revise it. The prompt included the original problem description, the faulty solution, and feedback from the parser/linter/tester (stdout, stderr/ and return code). After a unit test execution failure, the model could either fix the code to pass the existing tests or modify its unit tests to accommodate the generated code. Only dialogs that pass all checks are included in the final dataset,used for supervised finetuning (SFT). Notably,we observed that about 20 % {20}\% 20% of solutions were initially incorrect but self-corrected, indicating that the model learned from the execution feedback and improved its performance.

错误反馈与迭代自校正：当解决方案在任何步骤失败时，我们提示模型进行修订。提示包括原始问题描述、有问题的解决方案以及来自解析器/代码检查工具/测试器的反馈（标准输出、标准错误和返回代码）。在单元测试执行失败后，模型可以修复代码以通过现有测试，或修改其单元测试以适应生成的代码。只有通过所有检查的对话才包含在最终数据集中，用于监督微调（SFT）。值得注意的是，我们观察到大约 20 % {20}\% 20% 的解决方案最初是不正确的，但经过自我校正，表明模型从执行反馈中学习并提高了其性能。

Fine-tuning and iterative improvement: The finetuning process is conducted over multiple rounds, with each round building on the previous one. After each round, the model is improved, generating higher-quality synthetic data for the next round. This iterative process allows for progressive refinement and enhancement of the model’s performance.

微调与迭代改进：微调过程在多个轮次中进行，每一轮都基于前一轮。每一轮之后，模型得到改进，为下一轮生成更高质量的合成数据。这种迭代过程允许逐步细化和增强模型的性能。

Synthetic data generation: programming language translation. We observe a performance gap between major programming languages (e.g., Python/C++) and less common ones (e.g., Typescript/PHP). This is not surprising as we have less training data for less common programming languages. To mitigate this, we supplement our existing data by translating data from common programming languages to less common languages (similar to Chen et al. (2023) in the context of reasoning). This is achieved by prompting Llama 3 and ensuring quality via syntax parsing, compilation, and execution. Figure 8 demonstrates an example of synthetic PHP code translated from Python. This improves performance significantly for less common languages as measured by the MultiPL-E (Cassano et al., 2023) benchmark.

合成数据生成：编程语言翻译。我们观察到主流编程语言（如 Python/C++）与较少使用的编程语言（如 Typescript/PHP）之间存在性能差距。这并不令人惊讶，因为我们对于较少使用的编程语言的训练数据较少。为了缓解这一问题，我们通过将常见编程语言的数据翻译为较少使用的语言（类似于 Chen 等人在推理背景下的做法 (2023)）来补充现有数据。这一过程通过提示 Llama 3 并确保通过语法解析、编译和执行来保证质量。图 8 展示了一个从 Python 翻译成 PHP 的合成代码示例。这显著提高了较少使用语言的性能，如 MultiPL-E 基准测试（Cassano 等人，2023）所衡量。

Synthetic data generation: backtranslation. To improve certain coding capabilities (e.g., documentation, explanations) where execution feedback is less informative for determining quality, we employ an alternative multi-step approach. Using this procedure, we generated approximately 1.2M synthetic

合成数据生成：回译。为了提高某些编码能力（例如，文档编写、解释），在这些能力中执行反馈对于确定质量的信息较少，我们采用了一种替代的多步骤方法。通过这一过程，我们生成了大约 1.2M 的合成数据。

Figure 8 Code translation example. We display an example of using Llama 3 to translate Python code (left) to PHP code (right) to augment our SFT dataset with a wider range of programming languages.

图8 代码翻译示例。我们展示了一个使用Llama 3将Python代码（左侧）翻译成PHP代码（右侧）的例子，以扩充我们的SFT数据集，涵盖更广泛的编程语言。

Figure 9 © Improving generated code quality with system prompts. L e f t {Left} Left : without system prompt R i g h t {Right} Right : with system prompt.

图9 © 通过系统提示改进生成代码的质量。 L e f t {Left} Left：无系统提示 R i g h t {Right} Right：有系统提示。

dialogs related to code explanation, generation, documentation, and debugging. Beginning with code snippets from a variety of languages in our pre-training data:

与代码解释、生成、文档和调试相关的对话。从我们的预训练数据中多种语言的代码片段开始：

Generate: We prompt Llama 3 to generate data that represents our target capability (e.g., we add comments and docstrings for the code snippet, or we ask the model to explain a piece of code).

生成：我们提示Llama 3生成代表我们目标能力的数据（例如，我们为代码片段添加注释和文档字符串，或者我们要求模型解释一段代码）。

Backtranslate: We then prompt the model to “backtranslate” the synthetically generated data to the original code (e.g., we prompt the model to generate code only from its documentation, or we ask the model to generate code only from its explanation).

回译：然后我们提示模型进行“回译”，将合成生成的数据还原为原始代码（例如，我们提示模型仅从其文档生成代码，或者我们要求模型仅从其解释生成代码）。

Filter: Using the original code as a reference, we prompt the Llama 3 to determine the quality of the output (e.g., we ask the model how faithful the backtranslated code is to the original). We then use the generated examples that have the highest self-verification scores in SFT.

过滤器：以原始代码为参考，我们引导 Llama 3 评估输出质量（例如，我们询问模型回译代码对原始代码的忠实度）。然后，我们使用在 SFT 中自验证得分最高的生成示例。

System prompt steering during rejection sampling. During the rejection sampling process, we used code specific system prompts to improve code readability, documentation, thoroughness, and specificity. Recall, from Section 7 this data is used to finetune the language model. Figure 9 shows an example of how the system prompt helps improve the generated code quality - it adds necessary comments, uses more informative variable names, saves memory, etc.

拒绝采样期间的系统提示引导。在拒绝采样过程中，我们使用特定于代码的系统提示来提高代码的可读性、文档化、全面性和具体性。回顾第7节，这些数据用于微调语言模型。图9展示了一个例子，说明系统提示如何帮助提高生成代码的质量——它添加了必要的注释，使用了更具信息量的变量名，节省了内存等。

Filtering training data with execution and model-as-judge signals. As described in Section 4.2.3, we occasionally encounter quality issues in our rejection-sampled data, such as code blocks containing bugs. Detecting these issues in our rejection-sampled data is not as straightforward as it is for our synthetic code data, as the rejection-sampled responses typically contain a mix of natural language and code for which the code may not always be expected to be executable. (For example, user prompts may explicitly ask for pseudo-code or edits to only a very small snippet of an executable program.) To address this, we utilize the “model-as-judge” approach, where earlier versions of Llama 3 assess and assign a binary ( 0 / 1 ) \left( {0/1}\right) (0/1) score based on two criteria: code correctness and code style. We retain only those samples that achieve a perfect score of 2. Initially, this stringent filtering led to a regression in downstream benchmark performance, primarily because it disproportionately removed examples with challenging prompts. To counteract this, we strategically revise the responses of some coding data categorized as most challenging until they met the Llama-based “model-as-judge” criteria. By refining these challenging problems, the coding data achieves a balance between quality and difficulty, resulting in optimal downstream performance.

使用执行和模型即判断信号过滤训练数据。如第4.2.3节所述，我们在拒绝采样数据中偶尔会遇到质量问题，例如包含错误的代码块。在我们的拒绝采样数据中检测这些问题不像合成代码数据那样直接，因为拒绝采样的响应通常包含自然语言和代码的混合，其中代码不一定总是可执行的。（例如，用户提示可能明确要求伪代码或仅对可执行程序的一小部分进行编辑。）为了解决这个问题，我们采用了“模型即判断”方法，其中早期的Llama 3版本根据两个标准：代码正确性和代码风格，评估并赋予一个二进制 ( 0 / 1 ) \left( {0/1}\right) (0/1)分数。我们仅保留那些获得满分2分的样本。最初，这种严格的过滤导致下游基准性能下降，主要是因为它不均衡地移除了具有挑战性提示的示例。为了抵消这一点，我们战略性地修改了一些被归类为最具挑战性的编码数据的响应，直到它们符合基于Llama的“模型即判断”标准。通过改进这些具有挑战性的问题，编码数据在质量和难度之间达到了平衡，从而实现了最佳的下游性能。

4.3.2 Multilinguality 多语言性

We describe how we improve Llama 3’s multilingual capabilities, including training an expert specialized on substantially more multilingual data, sourcing and generating high quality multilingual instruction tuning data for German, French, Italian, Portuguese, Hindi, Spanish, and Thai, and tackling specific challenges of multilingual language steering to enhance the overall performance of our model.

我们描述了如何提升Llama 3的多语言能力，包括训练一个专注于更多多语言数据的专家，为德语、法语、意大利语、葡萄牙语、印地语、西班牙语和泰语获取和生成高质量的多语言指令调整数据，以及解决多语言语言引导的具体挑战，以提高我们模型的整体性能。

Expert training. Our Llama 3 pre-training data mix contains significantly more English tokens than non-English tokens. To collect higher quality human annotations in non-English languages, we train a multilingual expert by branching off the pre-training run and continuing to pre-train on a data mix that consists of 90 % {90}\% 90% multilingual tokens. We then perform post-training on this expert following Section 4.1. This expert model is then used to collect higher quality annotations in non-English languages until pre-training was fully complete.

专家培训。我们的 Llama 3 预训练数据混合中包含的英语标记明显多于非英语标记。为了在非英语语言中收集更高质量的人工标注，我们通过分支出预训练运行并继续在由 90 % {90}\% 90% 多语言标记组成的数据混合上进行预训练，来训练一个多语言专家。然后，我们按照第 4.1 节对该专家进行后训练。这个专家模型随后被用于在预训练完全完成之前收集非英语语言中更高质量的标注。

Multilingual data collection. Our multilingual SFT data is derived primarily from sources described below. The overall distribution is 2.4 % {2.4}\% 2.4% human annotations, 44.2 % {44.2}\% 44.2% data from other NLP tasks, 18.8 % {18.8}\% 18.8% rejection sampled data,and 34.6 % {34.6}\% 34.6% translated reasoning data.

多语言数据收集。我们的多语言 SFT 数据主要来源于以下描述的来源。总体分布为 2.4 % {2.4}\% 2.4% 人工标注， 44.2 % {44.2}\% 44.2% 来自其他 NLP 任务的数据， 18.8 % {18.8}\% 18.8% 拒绝采样数据，以及 34.6 % {34.6}\% 34.6% 翻译推理数据。

Human annotations: We collect high-quality, manually annotated data from linguists and native speakers. These annotations mostly consist of open-ended prompts that represent real world use cases.

人工标注：我们从语言学家和母语使用者那里收集高质量的手工标注数据。这些标注主要由代表真实世界使用场景的开放式提示组成。

Data from other NLP tasks: To further augment, we use multilingual training data from other tasks and rewrite into dialog format. For example, we use data from exams-qa (Hardalov et al., 2020) and Conic10k (Wu et al., 2023). To improve language alignment, we also use parallel texts from GlobalVoices (Prokopidis et al., 2016) and Wikimedia (Tiedemann, 2012). We use LID based filtering and Blaser2.0 (Seamless Communication et al., 2023) to remove low quality data. For parallel text data, instead of using the bitext pairs directly, we apply a multilingual template inspired by Wei et al. (2022a) to better simulate real-life conversations in translation and language learning scenarios.

来自其他 NLP 任务的数据：为了进一步增强，我们使用来自其他任务的多语言训练数据并重写为对话格式。例如，我们使用来自 exams-qa（Hardalov 等人，2020）和 Conic10k（Wu 等人，2023）的数据。为了改善语言对齐，我们还使用来自 GlobalVoices（Prokopidis 等人，2016）和 Wikimedia（Tiedemann，2012）的平行文本。我们使用基于 LID 的过滤和 Blaser2.0（Seamless Communication 等人，2023）来去除低质量数据。对于平行文本数据，我们不是直接使用双语对，而是应用受 Wei 等人（2022a）启发的多语言模板，以更好地模拟翻译和语言学习场景中的真实对话。

Rejection sampled data: We apply rejection sampling on our human annotated prompts to generate high-quality samples for finetuning, with few modifications compared to the process for English data:

拒绝采样数据：我们在人工标注的提示上应用拒绝采样，以生成高质量的样本用于微调，与英语数据的处理过程相比，改动较少：

Generation: We explored randomly choosing the temperature hyperparameter from the range 0.2 − 1 {0.2} - 1 0.2−1 for diverse generations in early rounds of post-training. With high temperature,responses for multilingual prompts can get creative and inspiring, but are also susceptible to unnecessary or unnatural code-switching. In the final round of post-training, we use a constant value of 0.6 to balance the trade-off. Additionally, we used specialized system prompts to improve response format, structure and general readability.

生成：我们在后训练的早期轮次中探索从范围 0.2 − 1 {0.2} - 1 0.2−1 随机选择温度超参数以实现多样化的生成。在高温度下，多语言提示的响应可以变得富有创意和启发性，但也容易出现不必要的或不自然的代码切换。在后训练的最后一轮，我们使用恒定值 0.6 来平衡这种权衡。此外，我们使用了专门的系统提示来改进响应格式、结构和整体可读性。

Selection: Prior to reward model based selection, we implement multilingual-specific checks to ensure high language-match rate between the prompt and response (e.g., a romanized Hindi prompt should not expect a response in Hindi Devanagari script).

选择：在基于奖励模型的选择之前，我们实施了多语言特定的检查，以确保提示和响应之间的语言匹配率高（例如，罗马化的印地语提示不应期望以印地语梵文脚本形式的响应）。

Translated data: We try to avoid using machine-translated data to finetune the model in order to prevent translationese (Bizzoni et al., 2020; Muennighoff et al., 2023) or possible name bias (Wang et al., 2022a), gender bias (Savoldi et al., 2021), or cultural bias (Ji et al., 2023). Moreover, we aim to prevent the model from being exposed only to tasks that are rooted in English cultural context, which may not be representative of the linguistic and cultural diversity we aim to capture. We made one exception to this and translated our synthetic quantitative reasoning data (see Section 4.3.3 for details) to improve performance in quantitative reasoning in non-English languages. Due to the simple nature of

翻译数据：我们尽量避免使用机器翻译的数据来微调模型，以防止翻译体（Bizzoni 等人，2020；Muennighoff 等人，2023）或可能的名称偏见（Wang 等人，2022a）、性别偏见（Savoldi 等人，2021）或文化偏见（Ji 等人，2023）。此外，我们的目标是防止模型仅接触根植于英语文化背景的任务，这可能无法代表我们旨在捕捉的语言和文化多样性。我们对此做了一个例外，将我们的合成定量推理数据（详见第 4.3.3 节）翻译成其他语言，以提高非英语语言的定量推理性能。由于这些数学问题中语言的简单性，翻译样本被发现几乎没有质量问题。我们观察到在 MGSM（Shi 等人，2022）上添加这些翻译数据后取得了显著的提升。

the language in these math problems, the translated samples were found to have little to no quality issues. We observed strong gains on MGSM (Shi et al., 2022) from adding this translated data.

这些数学问题中的语言简单，翻译样本几乎没有质量问题。我们观察到在 MGSM（Shi 等人，2022）上添加这些翻译数据后取得了显著的提升。

4.3.3 Math and Reasoning 数学与推理

We define reasoning as the ability to perform multi-step computations and arrive at the correct final answer. Several challenges guide our approach to training models that excel in mathematical reasoning:

我们将推理定义为执行多步骤计算并得出正确最终答案的能力。在训练擅长数学推理的模型时，我们面临以下几个挑战：

Lack of prompts: As the complexity of questions increases, the number of valid prompts or questions for Supervised Fine-Tuning (SFT) decreases. This scarcity makes it difficult to create diverse and representative training datasets for teaching models various mathematical skills (Yu et al., 2023; Yue et al., 2023; Luo et al., 2023; Mitra et al., 2024; Shao et al., 2024; Yue et al., 2024b).

提示缺失：随着问题复杂性的增加，适用于监督微调（SFT）的有效提示或问题的数量减少。这种稀缺性使得难以创建多样化和具有代表性的训练数据集，以教授模型各种数学技能（Yu et al., 2023; Yue et al., 2023; Luo et al., 2023; Mitra et al., 2024; Shao et al., 2024; Yue et al., 2024b）。

Lack of ground truth chain of thought: Effective reasoning requires a step-by-step solution to facilitate the reasoning process (Wei et al., 2022c). However, there is often a shortage of ground truth chains of thought, which are essential for guiding the model how to break down the problem step-by-step and reach the final answer (Zelikman et al., 2022).

缺乏真实思维链：有效的推理需要逐步解决方案来促进推理过程（Wei et al., 2022c）。然而，通常缺乏真实的思维链，这些思维链对于指导模型如何逐步分解问题并达到最终答案至关重要（Zelikman et al., 2022）。

Incorrect intermediate steps: When using model-generated chains of thought, the intermediate steps may not always be correct (Cobbe et al., 2021; Uesato et al., 2022; Lightman et al., 2023; Wang et al., 2023a). This inaccuracy can lead to incorrect final answers and needs to be addressed.

中间步骤错误：在使用模型生成的思维链时，中间步骤可能并不总是正确的（Cobbe et al., 2021; Uesato et al., 2022; Lightman et al., 2023; Wang et al., 2023a）。这种不准确性可能导致最终答案错误，需要解决。

Teaching models to use external tools: Enhancing models to utilize external tools, such as code interpreters, allows them to reason by interleaving code and text (Gao et al., 2023; Chen et al., 2022; Gou et al., 2023). This capability can significantly improve their problem-solving abilities.

教授模型使用外部工具：增强模型利用外部工具（如代码解释器）的能力，使它们能够通过交错代码和文本来进行推理（Gao et al., 2023; Chen et al., 2022; Gou et al., 2023）。这种能力可以显著提高它们的问题解决能力。

Discrepancy between training and inference: There is often a discrepancy between how the model is finetuned during training and how it is used during inference. During inference, the finetuned model may interact with humans or other models, requiring it to improve its reasoning using feedback. Ensuring consistency between training and real-world usage is crucial for maintaining reasoning performance.

训练与推理之间的差异：模型在训练期间的微调方式与推理期间的使用方式之间常常存在差异。在推理过程中，微调后的模型可能与人类或其他模型交互，需要利用反馈来改进其推理能力。确保训练与实际应用之间的一致性对于保持推理性能至关重要。

To address these challenges, we apply the following methodologies:

为了应对这些挑战，我们采用了以下方法：

Addressing the lack of prompts: We source relevant pre-training data from mathematical contexts and converted it into a question-answer format which can then be used for supervised finetuning. Additionally, we identify mathematical skills where the model under-performs and actively sourced prompts from humans to teach models such skills. To facilitate this process, we create a taxonomy of mathematical skills (Didolkar et al., 2024) and ask humans to provide relevant prompts/questions accordingly.

解决提示缺失问题：我们从数学上下文中获取相关的预训练数据，并将其转换为问答格式，以便用于监督式微调。此外，我们识别模型表现不佳的数学技能，并主动从人类那里获取提示来教授模型这些技能。为了促进这一过程，我们创建了一个数学技能分类体系（Didolkar et al., 2024），并请人类根据此体系提供相关的提示/问题。

Augmenting training data with step-wise reasoning traces: We use Llama 3 3 3 to generate step-by-step solutions for a set of prompts. For each prompt, the model produces a variable number of generations. These generations are then filtered based on the correct answer (Li et al., 2024a). We also do self-verification where Llama 3 is used to verify whether a particular step-by-step solution is valid for a given question. This process improves the quality of the finetuning data by eliminating instances where the model does not produce valid reasoning traces.

通过逐步推理追踪增强训练数据：我们使用 Llama 3 3 3 为一组提示生成逐步解决方案。对于每个提示，模型生成数量可变的生成结果。然后根据正确答案（Li et al., 2024a）对这些生成结果进行筛选。我们还进行自我验证，使用 Llama 3 来验证某个逐步解决方案是否适用于给定问题。这一过程通过消除模型未产生有效推理追踪的实例来提高微调数据的质量。

Filteringincorrect reasoning traces: We train outcome and stepwise reward models (Lightman et al., 2023; Wang et al., 2023a) to filter training data where the intermediate reasoning steps were incorrect. These reward models are used to eliminate data with invalid step-by-step reasoning, ensuring high-quality data for finetuning. For more challenging prompts, we use Monte Carlo Tree Search (MCTS) with learned step-wise reward models to generate valid reasoning traces, further enhancing the collection of high-quality reasoning data (Xie et al., 2024).

过滤不正确的推理轨迹：我们训练结果和逐步奖励模型（Lightman et al., 2023; Wang et al., 2023a）来过滤中间推理步骤不正确的训练数据。这些奖励模型用于消除无效的逐步推理数据，确保用于微调的高质量数据。对于更具挑战性的提示，我们使用蒙特卡洛树搜索（MCTS）与学习的逐步奖励模型来生成有效的推理轨迹，进一步增强高质量推理数据的收集（Xie et al., 2024）。

Interleaving code and text reasoning: We prompt Llama 3 to solve reasoning problems through a combination of textual reasoning and associated Python code (Gou et al., 2023). Code execution is used as a feedback signal to eliminate cases where the reasoning chain was not valid, ensuring the correctness of the reasoning process.

交错代码和文本推理：我们提示Llama 3通过文本推理和相关的Python代码（Gou et al., 2023）来解决推理问题。代码执行用作反馈信号，以消除推理链无效的情况，确保推理过程的正确性。

Learning from feedback and mistakes: To simulate human feedback, we utilize incorrect generations (i.e., generations leading to incorrect reasoning traces) and perform error correction by prompting Llama 3 to

从反馈和错误中学习：为了模拟人类反馈，我们利用不正确的生成（即导致不正确推理轨迹的生成）并通过提示Llama 3进行错误纠正

yield correct generations (An et al., 2023b; Welleck et al., 2022; Madaan et al., 2024a). The iterative process of using feedback from incorrect attempts and correcting them helps improve the model’s ability to reason accurately and learn from its mistakes.

产生正确的生成（An et al., 2023b; Welleck et al., 2022; Madaan et al., 2024a）。利用不正确尝试的反馈并纠正它们的迭代过程有助于提高模型准确推理和从错误中学习的能力。

4.3.4 Long Context 长上下文

During the final pre-training stage, we extend the context length of Llama 3 from 8K tokens to 128K tokens (see Section 3.4 for more details). Similar to pre-training, we find that during finetuning we must carefully tune the recipe to balance short and long-context capabilities.

在最终的预训练阶段，我们将Llama 3的上下文长度从8K令牌扩展到128K令牌（更多细节见第3.4节）。与预训练类似，我们发现在微调过程中，我们必须仔细调整配方以平衡短上下文和长上下文能力。

SFT and synthetic data generation. Naively applying our existing SFT recipe with only short-context data resulted in significant regressions in long-context capabilities from pre-training, highlighting the need to incorporate long-context data in our SFT data mix. In practice, however, it is largely impractical to get humans to annotate such examples due to the tedious and time-consuming nature of reading lengthy contexts, so we predominantly rely on synthetic data to fill this gap. We use earlier versions of Llama 3 to generate synthetic data based on the key long-context use-cases: (possibly multi-turn) question-answering, summarization for long documents, and reasoning over code repositories, and describe them in greater detail below.

SFT 和合成数据生成。简单地应用我们现有的 SFT 配方，仅使用短上下文数据，导致从预训练中长上下文能力显著退化，这凸显了在我们的 SFT 数据混合中加入长上下文数据的必要性。然而，在实践中，由于阅读长上下文的繁琐和耗时性质，让人类来标注这些例子基本上是不切实际的，因此我们主要依赖合成数据来填补这一空白。我们使用早期的 Llama 3 版本基于关键的长上下文用例生成合成数据：（可能是多轮的）问答、长文档的摘要和代码仓库的推理，并在下面详细描述它们。

Question answering: We carefully curate a set of long documents from our pre-training mix. We split these documents into chunks of 8    K 8\mathrm{\;K} 8K tokens,and prompted an earlier version of the Llama 3 model to generate QA pairs conditional on randomly selected chunks. During training, the whole document is used as context.

问答：我们从预训练混合数据中精心挑选了一组长文档。我们将这些文档分割成 8    K 8\mathrm{\;K} 8K 个令牌的块，并提示早期版本的 Llama 3 模型根据随机选择的块生成 QA 对。在训练过程中，整个文档被用作上下文。

Summarization: We applied hierarchical summarization of long-context documents by first summarizing the chunks of 8    K 8\mathrm{\;K} 8K input length using our strongest Llama 38    K {38}\mathrm{\;K} 38K context model and then summarizing the summaries. During training we provide the full document and prompt the model to summarize the document while preserving all the important details. We also generate QA pairs based on the summaries of the documents and prompt the model with questions that require global understanding of the whole long document.

摘要：我们通过首先使用我们最强的 Llama 38    K {38}\mathrm{\;K} 38K 上下文模型对 8    K 8\mathrm{\;K} 8K 输入长度的块进行摘要，然后对摘要进行摘要，应用了长上下文文档的分层摘要。在训练过程中，我们提供完整的文档并提示模型在保留所有重要细节的同时对文档进行摘要。我们还基于文档的摘要生成 QA 对，并提示模型回答需要对整个长文档进行全局理解的问题。

Long context code reasoning: We parse Python files to identify import statements and determine their dependencies. From here, we select the most commonly depended-upon files, specifically those referenced by at least five other files. We remove one of these key files from a repository and prompt the model to identify which files depended on the missing file and to generate the necessary missing code.

长上下文代码推理：我们解析 Python 文件以识别导入语句并确定它们的依赖关系。从这里，我们选择最常被依赖的文件，特别是那些至少被其他五个文件引用的文件。我们从仓库中移除其中一个关键文件，并提示模型识别哪些文件依赖于缺失的文件并生成必要的缺失代码。

We further categorize these synthetically generated samples based on the sequence length ( 16    K , 32    K , 64    K ({16}\mathrm{\;K},{32}\mathrm{\;K},{64}\mathrm{\;K} (16K,32K,64K and 128    K {128}\mathrm{\;K} 128K ) to enable more fine-grained targeting of input lengths.

我们进一步根据序列长度 ( 16    K , 32    K , 64    K ({16}\mathrm{\;K},{32}\mathrm{\;K},{64}\mathrm{\;K} (16K,32K,64K 和 128    K {128}\mathrm{\;K} 128K 对这些综合生成的样本进行分类，以实现对输入长度的更精细定位。

Through careful ablations,we observe that mixing 0.1 % {0.1}\% 0.1% of synthetically generated long-context data with the original short-context data optimizes the performance across both short-context and long-context benchmarks.

通过仔细的消融实验，我们观察到将综合生成的长上下文数据 0.1 % {0.1}\% 0.1% 与原始短上下文数据混合，可以优化短上下文和长上下文基准的性能。

DPO. We observe that using only short context training data in DPO did not negatively impact long-context performance as long as the SFT model is high quality in long context tasks. We suspect this is due to the fact that our DPO recipe has fewer optimizer steps than SFT. Given this finding, we keep the standard short-context recipe for DPO on top of our long-context SFT checkpoints.

DPO。我们观察到，只要SFT模型在长上下文任务中质量高，仅使用短上下文训练数据在DPO中并不会对长上下文性能产生负面影响。我们怀疑这是因为我们的DPO配方比SFT的优化步骤少。鉴于这一发现，我们在长上下文SFT检查点的基础上保持标准的短上下文配方用于DPO。

4.3.5 Tool Use 工具使用

Teaching LLMs to use tools such as search engines or code interpreters hugely expands the range of tasks they can solve, transforming them from pure chat models into more general assistants (Nakano et al., 2021; Thoppilan et al., 2022; Parisi et al., 2022; Gao et al., 2023; Mialon et al., 2023a; Schick et al., 2024). We train Llama 3 to interact with the following tools:

教授LLM使用搜索引擎或代码解释器等工具，极大地扩展了它们能解决的任务范围，使它们从纯粹的聊天模型转变为更通用的助手（Nakano et al., 2021; Thoppilan et al., 2022; Parisi et al., 2022; Gao et al., 2023; Mialon et al., 2023a; Schick et al., 2024）。我们训练Llama 3与以下工具进行交互：

Search engine. Llama 3 is trained to use Brave Search 7 {}^{7} 7 to answer questions about recent events that go beyond its knowledge cutoff or that require retrieving a particular piece of information from the web.

搜索引擎。Llama 3被训练使用Brave Search 7 {}^{7} 7 来回答关于近期事件的问题，这些问题超出了其知识截止点，或者需要从网络上检索特定信息。

Python interpreter. Llama 3 can generate and execute code to perform complex computations, read files uploaded by the user and solve tasks based on them such as question answering, summarization, data analysis or visualization.

Python解释器。Llama 3能够生成并执行代码，进行复杂计算，读取用户上传的文件，并基于这些文件解决任务，如问答、总结、数据分析或可视化。

7 {}^{7} 7 https://brave.com/search/api/

数学计算引擎。Llama 3 可以使用 Wolfram Alpha API 8 {}^{8} 8 更准确地解决数学、科学问题，或从 Wolfram 的数据库中检索准确信息。

The resulting model is able to use these tools in a chat setup to solve the user’s queries, including in multi-turn dialogs. If a query requires multiple tool calls, the model can write a step-by-step plan, call the tools in sequence, and do reasoning after each tool call.

生成的模型能够在聊天环境中使用这些工具来解决用户的查询，包括在多轮对话中。如果一个查询需要多次工具调用，模型可以编写一个逐步计划，按顺序调用工具，并在每次工具调用后进行推理。

We also improve Llama 3’s zero-shot tool use capabilities - given in-context, potentially unseen tool definitions and a user query, we train the model to generate the correct tool call.

我们还改进了 Llama 3 的零样本工具使用能力——在给定上下文中，潜在未见过的工具定义和用户查询的情况下，我们训练模型生成正确的工具调用。

Implementation. We implement our core tools as Python objects with different methods. Zero-shot tools can be implemented as Python functions with descriptions, documentation (i.e., examples for how to use them), and the model only needs the function’s signature and docstring as context to generate the appropriate call. We also convert function definitions and calls to JSON format, e.g., for web API calls. All tool calls are executed by the Python interpreter, that must be enabled in the Llama 3 system prompt. Core tools can be individually enabled or disabled in the system prompt.

实现。我们将核心工具实现为具有不同方法的 Python 对象。零样本工具可以实现为带有描述、文档（即使用示例）的 Python 函数，模型只需要函数的签名和文档字符串作为上下文来生成适当的调用。我们还把函数定义和调用转换为 JSON 格式，例如，用于网络 API 调用。所有工具调用都由 Python 解释器执行，必须在 Llama 3 系统提示中启用。核心工具可以在系统提示中单独启用或禁用。

Data collection. Different from Schick et al. (2024), we rely on human annotations and preferences to teach Llama 3 to use tools. There are two main differences with the post-training pipeline generally used in Llama 3:

数据收集。与 Schick 等人 (2024) 不同，我们依赖人工注释和偏好来教导 Llama 3 使用工具。这与 Llama 3 中通常使用的后训练流程有两个主要区别：

For tools, dialogs often contain more than a single assistant message (e.g., calling the tool and reasoning about the tool output). Thus, we annotate at the message level to collect granular feedback: annotators provide a preference between two assistant messages with the same context or, if both contain major problems, edit one of the messages. The chosen or edited message is then added to the context and the dialog continues. This provides human feedback for both the assistant’s ability of calling the tools and reasoning about the tool outputs. Annotators cannot rank or edit the tool outputs.

对于工具，对话通常包含不止一条助手消息（例如，调用工具并对工具输出进行推理）。因此，我们在消息级别进行标注以收集细粒度的反馈：标注者在一个相同上下文中的两条助手消息之间提供偏好选择，或者如果两者都存在重大问题，则编辑其中一条消息。被选定或编辑的消息随后被添加到上下文中，对话继续进行。这为助手调用工具和推理工具输出的能力提供了人类反馈。标注者不能对工具输出进行排序或编辑。

We do not perform rejection sampling, as we did not observe gains in our tool benchmarks.

我们不进行拒绝采样，因为我们未在我们的工具基准测试中观察到收益。

To accelerate the annotation process, we start by bootstrapping basic tool use capabilities by finetuning on synthetically generated data from previous Llama 3 checkpoints. Thus, annotators have fewer edits to perform. In a similar spirit, as Llama 3 gradually improves through its development, we progressively complexify our human annotation protocols: we start by single-turn tool use annotations, before moving to tool use in dialogs, and finally annotating for multi-step tool use and data analysis.

为了加速标注过程，我们首先通过在从前Llama 3检查点合成生成的数据上进行微调来引导基本工具使用能力。因此，标注者需要进行的编辑较少。同样地，随着Llama 3通过其开发逐步改进，我们逐步复杂化我们的人类标注协议：我们首先从单轮工具使用标注开始，然后过渡到对话中的工具使用，最后标注多步骤工具使用和数据分析。

Tool datasets. To create data for tool usage applications, we leverage the following procedure:

工具数据集。为了创建工具使用应用的数据，我们采用以下步骤：

Single-step tool use: We start by few-shot generation of synthetic user prompts which, by construction, require a call to one of our core tools (for example, questions that exceed our knowledge cutoff date). Then, still relying on few-shot generation, we generate appropriate tool calls for these prompts, execute them, and add the output to the model’s context. Finally, we prompt the model again to generate a final answer to the user’s query based on the tool output. We end up with trajectories of the following form: system prompt,user prompt,tool call,tool output,final answer. We also filter around 30 % {30}\% 30% this dataset to remove tool calls that cannot be executed or other formatting issues.

单步骤工具使用：我们首先通过少样本生成合成用户提示，这些提示按设计需要调用我们的核心工具之一（例如，超出我们知识截止日期的问题）。然后，仍然依赖少样本生成，我们为这些提示生成适当的工具调用，执行它们，并将输出添加到模型的上下文中。最后，我们再次提示模型，根据工具输出生成对用户查询的最终答案。我们最终得到以下形式的轨迹：系统提示、用户提示、工具调用、工具输出、最终答案。我们还会过滤掉无法执行的工具调用或其他格式问题。

Multi-step tool use: We follow a similar protocol and first generate synthetic data to teach the model basic multi-step tool use capabilities. To do this, we first prompt Llama 3 to generate user prompts that require at least two tool calls, that can be the same or different tools from our core set. Then, conditioned on these prompts, we few-shot prompt Llama 3 to generate a solution consisting of interleaved reasoning steps and tool calls, similar to ReAct (Yao et al., 2022). See Figure 10 for an example of Llama 3 performing a task involving multi-step tool usage.

多步骤工具使用：我们遵循类似的协议，首先生成合成数据来教授模型基本的多步骤工具使用能力。为此，我们首先提示 Llama 3 生成需要至少两次工具调用的用户提示，这些工具可以是我们核心集合中的相同或不同工具。然后，基于这些提示，我们通过少量示例提示 Llama 3 生成一个解决方案，该解决方案包含交错的推理步骤和工具调用，类似于 ReAct（Yao 等人，2022）。参见图 10，Llama 3 执行涉及多步骤工具使用的任务的示例。

File uploads: We annotate for the following filetypes: .TXT, .DOCX, .PDF, .PPTX, .XLSX, .CSV, .TSV, PY, JSON, JSONL, HTML, XML. Our prompts are based on a provided file, and ask to summarize the contents of the file, find and fix bugs, optimize a piece of code, perform data analysis or visualization. See Figure 11 for an example of Llama 3 performing a task involving a file upload.

文件上传：我们为以下文件类型进行标注：.TXT、.DOCX、.PDF、.PPTX、.XLSX、.CSV、.TSV、PY、JSON、JSONL、HTML、XML。我们的提示基于提供的文件，要求总结文件内容、查找并修复错误、优化一段代码、执行数据分析或可视化。参见图 11，Llama 3 执行涉及文件上传的任务的示例。

After finetuning on this synthetic data, we gather human annotations in diverse and challenging scenarios including multi-turn interactions, more than three step tool use, and instances where a tool call does not yield a satisfying answer. We augment our synthetic data with different system prompts to teach the model to use tools only when activated. To train the model to avoid calling tools for simple queries, we also add queries from easy math or question answering datasets (Berant et al., 2013; Koncel-Kedziorski et al., 2016; Joshi et al., 2017; Amini et al., 2019) and their responses without tools, but with tools activated in system prompt.

在对此合成数据进行微调后，我们在多样化和具有挑战性的场景中收集人工标注，包括多轮交互、超过三个步骤的工具使用，以及工具调用未产生结果的情况。一个令人满意的答案。我们通过不同的系统提示来增强我们的合成数据，以教导模型仅在激活时使用工具。为了训练模型避免对简单查询调用工具，我们还添加了来自简单数学或问答数据集（Berant et al., 2013; Koncel-Kedziorski et al., 2016; Joshi et al., 2017; Amini et al., 2019）的查询及其不使用工具的响应，但在系统提示中激活了工具。

8 {}^{8} 8 https://products.wolframalpha.com/llm-api/documentation

Figure 11 Processing file uploads. Example of Llama 3 performing analysis and visualization of an uploaded file.

图 11 处理文件上传。Llama 3 执行分析和可视化上传文件的示例。

We follow the principle that post-training should align the model to “know what it knows” rather than add knowledge (Gekhman et al., 2024; Mielke et al., 2020). Our primary approach involves generating data that aligns model generations with subsets of factual data present in the pre-training data. To achieve this, we develop a knowledge probing technique that takes advantage of Llama 3’s in-context abilities. This data generation process involves the following procedure:

我们遵循的原则是，训练后应使模型“知道自己知道什么”，而不是增加知识（Gekhman 等人，2024；Mielke 等人，2020）。我们的主要方法涉及生成与预训练数据中存在的部分事实数据相一致的数据。为此，我们开发了一种知识探查技术，利用 Llama 3 的上下文能力。此数据生成过程包括以下步骤：

Extract a data snippet from the pre-training data.

从预训练数据中提取数据片段。

Generate a factual question about these snippets (context) by prompting Llama 3.

通过提示 Llama 3 生成关于这些片段（上下文）的事实问题。

Sample responses from Llama 3 to the question.

从 Llama 3 对问题的回答中抽样。

Score the correctness of the generations using the original context as a reference and Llama 3 as a judge.

使用原始上下文作为参考和 Llama 3 作为评判，对生成的正确性进行评分。

Score the informativeness of the generations using Llama 3 as a judge.

使用 Llama 3 作为评判，对生成的信息量进行评分。

Generate a refusal for responses which are consistently informative and incorrect across the generations, using Llama 3.

对于在生成中持续表现出信息丰富但错误的回答，使用 Llama 3 生成拒绝回答。

We use data generated from the knowledge probe to encourage the model to only answer questions which it has knowledge about, and refuse answering those questions that it is unsure about. Further, pre-training data is not always factually consistent or correct. We therefore also collect a limited set of labeled factuality data that deals with sensitive topics where factually contradictory or incorrect statements are prevalent.

我们利用从知识探查中生成的数据，鼓励模型仅回答其了解的问题，并拒绝回答其不确定的问题。此外，预训练数据并不总是事实一致或正确。因此，我们还收集了一组有限的标记事实数据，这些数据涉及敏感话题，其中事实矛盾或不正确的陈述普遍存在。

4.3.7 Steerability 可操控性

Steerability is the ability to direct the model’s actions and outcomes to meet developer and user specifications. As Llama 3 is a generic foundational model, it should be maximally steerable to different downstream use cases easily. For Llama 3, we focus on enhancing its steerability through system prompt with natural language instructions, especially around response length, format, tone and character/persona.

可操控性是指能够引导模型的行为和结果以满足开发者和用户的需求。由于 Llama 3 是一个通用的基础模型，它应该能够最大限度地轻松适应不同的下游使用场景。对于 Llama 3，我们通过使用自然语言指令的系统提示来增强其可操控性，特别是在响应长度、格式、语气和角色/人物方面。

Data collection. We collect steerability preference samples within the general English category by asking annotators to design different system prompts for Llama 3. Annotators then engage in conversations with the models to evaluate their consistency in following instructions defined in system prompts over the course of the conversation. We show an example customized system prompt used for enhancing steerability below:

数据收集。我们在英语类别中收集可操控性偏好样本，方法是要求标注者为 Llama 3 设计不同的系统提示。然后，标注者与模型进行对话，评估模型在整个对话过程中遵循系统提示中定义的指令的一致性。我们在下面展示了一个用于增强可操控性的自定义系统提示示例：

You are a helpful and cheerful AI Chatbot that acts as a meal plan assistant for busy families. The family consists of 2 adults, 3 teenagers, and 2 preschoolers. Plan two or three days at a time and use leftovers or extra ingredients for the second day’s plan. The user will let you know if they want two or three days. If they don’t, assume three days. Each plan should include breakfast, lunch, snack, and dinner. Ask the user if they approve of the plan or need adjustments. After they approve provide a grocery list with family size in mind. Always keep family preferences in mind and if there’s something that they don’t like provide a substitution. If the user is not feeling inspired then ask them what’s the one place they wish they could visit on vacation this week and then suggest meals based on that location’s culture. Weekend meals can be more complex. Weekday meals should be quick and easy. For breakfast and lunch, easy food like cereal, English muffins with pre-cooked bacon, and other quick easy foods are preferred. The family is busy. Be sure to ask if they have essentials and favorites on hand like coffee or energy drinks so they don’t forget to buy it. Remember to be budget-conscious unless it’s a special occasion.

您是一位乐于助人且开朗的AI聊天机器人，担任忙碌家庭的膳食计划助手。这个家庭包括2名成人、3名青少年和2名学龄前儿童。每次计划两到三天的饮食，并利用剩余食材或额外配料安排第二天的计划。用户会告知他们需要两到三天的计划。如果未指定，则默认三天。每个计划应包括早餐、午餐、小吃和晚餐。询问用户是否批准该计划或需要调整。在他们批准后，提供一份考虑到家庭规模的购物清单。始终牢记家庭偏好，如果有他们不喜欢的食物，请提供替代品。如果用户缺乏灵感，询问他们本周最希望去哪里度假，然后根据该地点的文化推荐餐食。周末餐食可以更复杂。工作日餐食应快速简便。早餐和午餐偏好简单食物，如麦片、英式松饼配预煮培根等快捷简便食品。家庭忙碌，务必询问他们是否有咖啡或能量饮料等必需品和最爱，以免忘记购买。记得要考虑预算，除非是特殊场合。

Modeling. After we collect the preference data, we leverage this data in reward modeling, rejection sampling, SFT, and DPO to enhance Llama 3’s steerability.

建模。在我们收集偏好数据后，我们利用这些数据在奖励建模、拒绝采样、SFT和DPO中增强Llama 3的可引导性。

5 Results 结果

We performed an extensive series of evaluations of Llama 3, investigating the performance of: (1) the pre-trained language model, (2) the post-trained language model, and (3) the safety characteristics of Llama 3. We present the results of these evaluations in separate subsections below.

我们对Llama 3进行了一系列广泛的评估，调查了以下方面的性能：（1）预训练语言模型，（2）后训练语言模型，以及（3）Llama 3的安全特性。我们在下面的单独小节中展示了这些评估的结果。

5.1 Pre-trained Language Model 预训练语言模型

In this section, we report evaluation results for our pre-trained Llama 3 (Section 3), comparing with various other models of comparable sizes. We reproduce results of competitor models whenever possible. For non-Llama models, we report the best score across results that are publicly reported or (where possible) that we reproduced ourselves. The specifics of these evaluations, including configurations such as the number of shots, metrics, and other pertinent hyperparameters and settings, can be accessed on our Github repository here. Additionally, we are releasing the data generated as part of evaluations with publicly available benchmarks which can be found on Huggingface here. We evaluate the quality of our models on standard benchmarks (Section 5.1.1), for robustness to changes in multiple-choice question setups (Section 5.1.2), and on adversarial evaluations (Section 5.1.3). We also conduct a contamination analysis to estimate the extent to which our evaluations are impacted by contamination of training data (Section 5.1.4).

在本节中，我们报告了我们预训练的Llama 3（第3节）的评估结果，并与各种其他尺寸相当的模型进行了比较。我们尽可能重现了竞争对手模型的结果。对于非Llama模型，我们报告了公开报告或（如果可能）我们自己重现的结果中的最佳分数。这些评估的具体细节，包括诸如样本数量、指标以及其他相关超参数和设置等配置，可以在我们的Github仓库中访问。此外，我们还发布了作为评估一部分生成的数据，这些数据可以在Huggingface上找到。我们在标准基准（5.1.1节）上评估我们模型的质量，对于多选题设置变化的鲁棒性（5.1.2节），以及对抗性评估（5.1.3节）。我们还进行了污染分析，以估计我们的评估受到训练数据污染影响的程度（5.1.4节）。

5.1.1 Standard Benchmarks 标准基准

To compare our models with the current state-of-the-art, we evaluate Llama 3 on a large number of standard benchmark evaluations shown in Table 8. These evaluations cover eight top-level categories: (1) commonsense reasoning; (2) knowledge; (3) reading comprehension; (4) math, reasoning, and problem solving; (5) long context; (6) code; (7) adversarial evaluations; and (8) aggregate evaluations.

为了将我们的模型与当前最先进的技术进行比较，我们在表8所示的大量标准基准评估上对Llama 3进行了评估。这些评估涵盖了八个顶级类别：（1）常识推理；（2）知识；（3）阅读理解；（4）数学、推理和问题解决；（5）长上下文；（6）代码；（7）对抗性评估；以及（8）综合评估。

Table 8 Pre-training benchmarks by category. Overview of all benchmarks we use to evaluate pre-trained Llama 3 models, grouped by capability category.

表8 按类别划分的预训练基准。我们用于评估预训练Llama 3模型的所有基准概览，按能力类别分组。

Experimental setup. For each benchmark, we compute scores for Llama 3 as well as various other pre-trained models of comparable sizes. Where possible, we recompute numbers with our own pipeline for other models. To ensure a fair comparison, we then select the best score between the score that we computed and the reported number for that model with comparable or more conservative settings. You can find additional details on our evaluation setup here. For some models, it is not possible to (re)compute benchmark values, for instance, because the pre-trained model is not released or because the API does not provide access to log-probabilities. In particular, this is true for all models comparable to Llama 3 405B. Thus, we do not report category averages for Llama 3405B, which requires that all numbers are available for all benchmarks.

实验设置。对于每个基准测试，我们计算 Llama 3 以及其他各种大小相当的预训练模型的分数。在可能的情况下，我们使用自己的流程重新计算其他模型的分数。为了确保公平比较，我们随后在可比较或更保守的设置下，选择我们计算的分数和该模型报告的分数中的最佳值。您可以在此处找到有关我们评估设置的更多详细信息。对于某些模型，无法（重新）计算基准值，例如因为未发布预训练模型或 API 不提供对对数概率的访问。特别是，这对于所有与 Llama 3 405B 相当的模型都是如此。因此，我们不报告 Llama 3405B 的类别平均值，这要求所有基准测试的所有数值都可用。

Significance estimates. Benchmark scores are estimates of a model’s true performance. These estimates have variance because benchmark sets are finite samples drawn from some underlying distribution. We follow Madaan et al. (2024b) and report on this variance via 95 % {95}\% 95% confidence intervals (CIs),assuming that benchmark scores are Gaussian distributed. While this assumption is incorrect (e.g., benchmark scores are bounded), preliminary bootstrap experiments suggest CIs (for discrete metrics) are a good approximation:

显著性估计。基准分数是对模型真实性能的估计。这些估计具有方差，因为基准集是从某个潜在分布中抽取的有限样本。我们遵循 Madaan 等人（2024b）的方法，通过 95 % {95}\% 95% 置信区间（CIs）报告这一方差，假设基准分数呈高斯分布。虽然这一假设不正确（例如，基准分数是有界的），但初步的 bootstrap 实验表明，对于离散指标，置信区间是一个很好的近似：

C I ( S ) = 1.96 × S × ( 1 − S ) N . {CI}\left( S\right) = {1.96} \times \sqrt{\frac{S \times \left( {1 - S}\right) }{N}}. CI(S)=1.96×NS×(1−S)​ ​.

Herein, S S S is the observed benchmark score (e.g.,accuracy or EM) and N N N the sample size of the benchmark. We omit CIs for benchmark scores that are not simple averages. We note that because subsampling is not the only source of variation, our CI values lower bound the actual variation in the capability estimate.

在此， S S S 是观察到的基准分数（例如，准确性或 EM）， N N N 是基准的样本大小。我们省略了不是简单平均值的基准分数的置信区间。我们注意到，因为子抽样不是变化的唯一来源，我们的置信区间值低估了能力估计中的实际变化。

Results for 8B and 70B models. Figure 12 reports the average performance of Llama 3 8B and 70B on the commonsense reasoning, knowledge, reading comprehension, math and reasoning, and code benchmarks. The results show that Llama 3 8B outperforms competing models in virtually every category, both in terms of per-category win rate and in terms of average per-category performance. We also find that Llama 3 70B outperforms its predecessor Llama 2 70B by a large margin on most benchmarks, with the exception of commonsense benchmarks that are likely saturated. Llama 3 70B also outperforms Mixtral 8x22B.

8B 和 70B 模型的结果。图 12 报告了 Llama 3 8B 和 70B 在常识推理、知识、阅读理解、数学和推理以及代码基准上的平均表现。结果显示，Llama 3 8B 在几乎所有类别中都优于竞争模型，无论是在每个类别的胜率还是在每个类别的平均表现方面。我们还发现，Llama 3 70B 在大多数基准测试中都大幅领先于其前身 Llama 2 70B，除了可能已经饱和的常识基准。Llama 3 70B 也优于 Mixtral 8x22B。

Detailed results for all models. Table 9, 10, 11, 12, 13, and 14 present the benchmark performance of pre-trained Llama 3 8B, 70B, and 405B models on reading comprehension tasks, coding tasks, commonsense understanding tasks, mathematical reasoning tasks, and general tasks. The tables compare Llama 3’s performance with that of models of similar size. The results show that Llama 3405B performs competitively with other models in its class. In particular, Llama 3405B substantially outperforms prior open-source models. For long-context, we present more comprehensive results (including probing tasks like needle-in-a-haystack) in Section 5.2.

所有模型的详细结果。表 9、10、11、12、13 和 14 展示了预训练的 Llama 3 8B、70B 和 405B 模型在阅读理解任务、编码任务、常识理解任务、数学推理任务和一般任务上的基准表现。这些表格比较了 Llama 3 与相似尺寸模型的表现。结果显示，Llama 3405B 在其类别中与其他模型竞争激烈。特别是，Llama 3405B 大幅领先于之前的开源模型。对于长上下文，我们在第 5.2 节中提供了更全面的结果（包括诸如“大海捞针”之类的探测任务）。

Figure 12: Performance of pre-trained Llama 3 8B and 70B models on pre-training benchmarks. Results are a g g r e g a t e d    b y \mathrm{{aggregated}\;{by}} aggregatedby capability category by averaging accuracies across all benchmarks corresponding to that category.

图 12：预训练的 Llama 3 8B 和 70B 模型在预训练基准上的表现。结果通过在所有对应于该类别的基准上平均准确率来表示 a g g r e g a t e d    b y \mathrm{{aggregated}\;{by}} aggregatedby 能力类别。

Table 9 Pre-trained model performance on reading comprehension tasks. Results include 95 % {95}\% 95% confidence intervals.

表 9 预训练模型在阅读理解任务上的表现。结果包括 95 % {95}\% 95% 置信区间。

Table 10 Pre-trained model performance on coding tasks. Results include 95 % {95}\% 95% confidence intervals.

表 10 预训练模型在编码任务上的表现。结果包括 95 % {95}\% 95% 置信区间。

5.1.2 Model Robustness 模型鲁棒性

In addition to performance on benchmarks, robustness is an important factor in the quality of pre-trained language models. We investigate the robustness of our pre-trained language models to design choices in multiple-choice question (MCQ) setups. Prior work has reported that model performance can be sensitive to seemingly arbitrary design choices in such setups, for example, model scores and even rankings may change with the order and labels of the in-context examples (Lu et al., 2022; Zhao et al., 2021; Robinson and Wingate, 2023; Liang et al., 2022; Gupta et al., 2024), the exact format of the prompt (Weber et al., 2023b; Mishra et al., 2022), or the answer choice format and order (Alzahrani et al., 2024; Wang et al., 2024a; Zheng et al., 2023). Motivated by this work, we use the MMLU benchmark to evaluate the robustness of our pre-trained models to: (1) few-shot label bias, (2) label variants, (3) answer order, and (4) prompt format:

除了基准测试的性能外，鲁棒性是预训练语言模型质量的重要因素。我们研究了预训练语言模型在多项选择题（MCQ）设置中的设计选择对其鲁棒性的影响。先前的工作报告称，模型性能可能对这种设置中的看似随意的设计选择敏感，例如，模型分数甚至排名可能会随着上下文示例的顺序和标签（Lu et al., 2022; Zhao et al., 2021; Robinson and Wingate, 2023; Liang et al., 2022; Gupta et al., 2024）、提示的确切格式（Weber et al., 2023b; Mishra et al., 2022）或答案选择格式和顺序（Alzahrani et al., 2024; Wang et al., 2024a; Zheng et al., 2023）的变化而变化。受此工作的启发，我们使用MMLU基准来评估我们的预训练模型对以下方面的鲁棒性：（1）少样本标签偏差，（2）标签变体，（3）答案顺序，以及（4）提示格式：

Few-shot label bias. Following Zheng et al. (2023) and Weber et al. (2023a), we investigate the impact of the distribution of labels in four-shot examples. Specifically, we consider settings in which: (1) all

少样本标签偏差。根据Zheng et al. (2023)和Weber et al. (2023a)的研究，我们调查了四样本示例中标签分布的影响。具体来说，我们考虑了以下设置：（1）所有

Table 11 Pre-trained model performance on commonsense understanding tasks. Results include 95 % {95}\% 95% confidence intervals.

表11 预训练模型在常识理解任务上的性能。结果包括 95 % {95}\% 95% 置信区间。

Table 12 Pre-trained model performance on math and reasoning tasks. Results include 95 % {95}\% 95% confidence intervals. ♢ 11 {}^{\diamondsuit }{11} ♢11 -shot. △ {}^{\bigtriangleup } △ Variable shot.

表12 预训练模型在数学和推理任务上的性能。结果包括 95 % {95}\% 95% 置信区间。 ♢ 11 {}^{\diamondsuit }{11} ♢11 样本。 △ {}^{\bigtriangleup } △ 可变样本。

Table 13 Pre-trained model performance on general language tasks. Results include 95 % {95}\% 95% confidence intervals.

表13 预训练模型在通用语言任务上的性能。结果包括 95 % {95}\% 95% 置信区间。

Figure 13: Robustness of our pre-trained language models to different design choices in the MMLU benchmark. L e f t :  Performance  {Left}{ : \text{ Performance }} Left: Performance  for different label variants. Right: Performance for different labels present in few-shot examples.

图13：我们的预训练语言模型在MMLU基准测试中对不同设计选择的鲁棒性。 L e f t :  Performance  {Left}{ : \text{ Performance }} Left: Performance  针对不同的标签变体。右图：不同标签在少样本示例中的表现。

Figure 14 Robustness of our pre-trained language models to different design choices in the MMLU benchmark.    L e f t    :    P e r f o r m a n c e \;{Left}\; : \;\mathrm{{Performance}} Left:Performance for different answer orders. Right: Performance for different prompt formats.

图14：我们的预训练语言模型在MMLU基准测试中对不同设计选择的鲁棒性。    L e f t    :    P e r f o r m a n c e \;{Left}\; : \;\mathrm{{Performance}} Left:Performance 针对不同的答案顺序。右图：不同提示格式的表现。

few-shot examples have the same label (A A A A); (2) all examples have a different label (A B C D); and (3) there are only two labels present (A A B B and C C D D).

少样本示例具有以下三种标签情况：(1) 所有示例具有相同的标签（A A A A）；(2) 所有示例具有不同的标签（A B C D）；(3) 只有两个标签存在（A A B B 和 C C D D）。

Label variants. We also study model response to different choice token sets. We consider the two sets proposed by Alzahrani et al. (2024): namely,a set of common language independent tokens ( $$ & #$ © and a of rare tokens (ce § 3 §3 §3 ii) that do not have any implicit relative order. We also consider two versions of the canonical labels (A. B. C. D. and A) B) C) D)) and a numerical list (1. 2. 3. 4.).

标签变体。我们还研究了模型对不同选择标记集的响应。我们考虑了Alzahrani等人（2024）提出的两组标记：一组是常见的与语言无关的标记（$$ & #$（c）和一组罕见的标记（ce § 3 §3 §3 ii），这些标记没有任何隐含的相对顺序。我们还考虑了规范标签的两个版本（A. B. C. D. 和 A) B) C) D)）和一个数字列表（1. 2. 3. 4.）。

Answer order. Following Wang et al. (2024a), we compute how stable the results are across different answer orders. To compute this, we remap all the answers in the dataset according to a fixed permutation. For example,for the permutation A B C D, all answer options with label A and B keep their label, and all answer options with label C \mathbf{C} C get label D \mathbf{D} D ,and vice versa.

答案顺序。根据Wang等人（2024a）的研究，我们计算了不同答案顺序下的结果稳定性。为了计算这一点，我们根据一个固定的排列重新映射数据集中的所有答案。例如，对于排列 A B C D，所有标签为 A 和 B 的答案选项保持其标签，所有标签为 C \mathbf{C} C 的答案选项获得标签 D \mathbf{D} D，反之亦然。

Prompt format. We evaluate variance in performance across five task prompts that differ in the level of information provided: one prompt simply asks the model to answer the question, whereas other prompts assert the expertise of the model or that the best answer should be chosen.

提示格式。我们评估了五种任务提示在提供信息水平上的性能差异：一种提示仅要求模型回答问题，而其他提示则断言模型的专业性或应选择最佳答案。

Figure 13 presents the results of our experiments studying robustness of model performance to label variants (left) and few-shot label bias (right). The results show that our pre-trained language models are very robust to changes in MCQ labels and to the structure of the few-shot prompt labels. This robustness is particularly pronounced for the 405    B {405}\mathrm{\;B} 405B parameter model. Figure 14 presents the results of our study of robustness to answer order and prompt format. The results in the figure further underscore the robustness of the performance of our pre-trained language models, in particular, of Llama 3 405B.

图13展示了我们关于模型性能对标签变体（左侧）和少样本标签偏差（右侧）的鲁棒性实验结果。结果显示，我们的预训练语言模型对多项选择题标签的变化和少样本提示标签的结构非常鲁棒。这种鲁棒性在 405    B {405}\mathrm{\;B} 405B参数模型中尤为明显。图14展示了我们对答案顺序和提示格式鲁棒性的研究结果。图中的结果进一步强调了我们预训练语言模型性能的鲁棒性，特别是Llama 3 405B。

Figure 15 Adversarial versus non-adversarial performance for question answering, mathematical reasoning, and paraphrase detection benchmarks. Left: Results for pre-trained models. Right: Results for post-trained models.

图15展示了问答、数学推理和释义检测基准测试中对抗性与非对抗性性能的结果。左侧：预训练模型的结果。右侧：后训练模型的结果。

5.1.3 Adversarial Benchmarks 对抗性基准测试

In addition to the benchmarks presented above, we evaluate on several adversarial benchmarks in three areas: question answering, mathematical reasoning, and paraphrase detection. This testing probes the model’s capabilities on tasks specifically created to be challenging and can potentially also point to overfitting on benchmarks. For question answering, we use Adversarial SQuAD (Jia and Liang, 2017) and Dynabench SQuAD (Kiela et al., 2021). For mathematical reasoning, we use GSM-Plus (Li et al., 2024c). For paraphrase detection, we use PAWS (Zhang et al., 2019).

除了上述基准测试外，我们还在三个领域评估了几个对抗性基准测试：问答、数学推理和释义检测。这种测试探查模型在专门设计为具有挑战性的任务上的能力，并可能指出对基准测试的过拟合。对于问答，我们使用对抗性SQuAD（Jia和Liang，2017）和Dynabench SQuAD（Kiela等人，2021）。对于数学推理，我们使用GSM-Plus（Li等人，2024c）。对于释义检测，我们使用PAWS（Zhang等人，2019）。

Figure 15 presents the scores of Llama 3 8B, 70B, and 405B on the adversarial benchmarks as a function of their performance on non-adversarial benchmarks. The non-adversarial benchmarks we use are SQuAD (Rajpurkar et al., 2016) for question answering, GSM8K for mathematical reasoning, and QQP (Wang et al., 2017) for paraphrase detection. Each datapoint represents a pair of an adversarial and non-adversarial datasets (e.g. QQP paired with PAWS), and we show all possible pairs within a category. The diagonal black line represents parity between adversarial and non-adversarial datasets - being on the line would indicate the model has similar performance regardless of the adversarial nature.

图15展示了Llama 3 8B、70B和405B在对抗性基准测试中的得分，作为它们在非对抗性基准测试中表现的函数。我们使用的非对抗性基准测试包括用于问答的SQuAD（Rajpurkar et al., 2016）、用于数学推理的GSM8K和用于复述检测的QQP（Wang et al., 2017）。每个数据点代表一对对抗性和非对抗性数据集（例如，QQP与PAWS配对），我们展示了同一类别内的所有可能配对。对角黑色线表示对抗性和非对抗性数据集之间的均衡——位于线上表明模型在对抗性性质上具有相似的表现。

On paraphrase detection, neither pre-trained nor post-trained models appear to suffer from the type of adversariality with which PAWS was constructed, marking a substantial step with respect to the previous generation of models. This result confirms the findings of Weber et al. (2023a), who also found that LLMs are less susceptible to the type of spurious correlations found in several adversarial datasets. For mathematical reasoning and question answering, however, the adversarial performances are substantially lower than the non-adversarial performances. This pattern is similar for pre-trained and post-trained models.

在复述检测方面，无论是预训练还是后训练模型，似乎都没有受到PAWS构建的对抗性类型的影响，相对于上一代模型来说，这是一个重大的进步。这一结果证实了Weber et al.（2023a）的发现，他们也发现LLMs对在几个对抗性数据集中发现的虚假相关性不那么敏感。然而，在数学推理和问答方面，对抗性表现明显低于非对抗性表现。这种模式在预训练和后训练模型中都相似。

5.1.4 Contamination Analysis 污染分析

We conduct a contamination analysis to estimate to what extent benchmark scores may be influenced by contamination of the evaluation data in the pre-training corpus. In previous work, several different contamination methods have been used, with various different hyperparameters - we refer to Singh et al. (2024) for an overview. Any of these methods can suffer from false positives and negatives, and how to best run contamination analyses is currently still an open field of research. Here, we largely follow the suggestions of Singh et al. (2024).

我们进行了一项污染分析，以估计基准测试分数可能在多大程度上受到预训练语料库中评估数据污染的影响。在之前的工作中，使用了多种不同的污染方法，具有不同的超参数——我们参考Singh et al.（2024）的概述。这些方法中的任何一种都可能遭受假阳性和假阴性，如何最好地进行污染分析目前仍然是一个开放的研究领域。在这里，我们主要遵循Singh et al.（2024）的建议。

Method. Specifically, Singh et al. (2024) propose to select contamination detection methods empirically, based on which method results in the largest difference between the ‘clean’ part of the dataset and the entire dataset, which they call estimated performance gain. For all our evaluation datasets, we score examples based on 8-gram overlap, a method that was found by Singh et al. (2024) to be accurate for many datasets. We consider an example of a dataset D D D to be contaminated if a ratio T D {\mathcal{T}}_{D} TD​ of its tokens are part of an 8-gram occurring at least once in the pre-training corpus. We select T D {\mathcal{T}}_{D} TD​ separately for each dataset, based on which value shows the maximal significant estimated performance gain across the three model sizes.

方法。具体而言，Singh 等人（2024 年）提出根据哪种方法在数据集的“干净”部分与整个数据集之间产生最大差异来经验性地选择污染检测方法，他们称之为估计性能增益。对于我们所有的评估数据集，我们根据 8-gram 重叠来评分，Singh 等人（2024 年）发现这种方法对许多数据集都很准确。我们认为数据集 D D D 的一个示例受到污染，如果其标记的一部分 T D {\mathcal{T}}_{D} TD​ 是预训练语料库中至少出现一次的 8-gram 的一部分。我们针对每个数据集分别选择 T D {\mathcal{T}}_{D} TD​，基于哪个值在三种模型大小中显示出最大的显著估计性能增益。

Table 14 Performance of pre-trained models on long-context tasks. Results include 95 % {95}\% 95% confidence intervals.

表 14 预训练模型在长上下文任务上的性能。结果包括 95 % {95}\% 95% 置信区间。

Table 15 Percentage of evaluation sets considered to be contaminated because similar data exists in the training corpus, and the estimated performance gain that may result from that contamination. See the text for details.

表 15 由于训练语料库中存在相似数据而被视为污染的评估集的百分比，以及由此污染可能带来的估计性能增益。详情见正文。

Results. In Table 15, we report the percentage of evaluation data that is considered contaminated for the maximal estimated performance gain, as described above, for all key benchmarks. From the table, we exclude numbers for benchmarks for which the results are not significant, for instance because the clean or contaminated set has too few examples, or because the observed performance gain estimate shows extremely erratic behavior. In Table 15, we observe that for some datasets contamination has a large impact, while for others it does not. For example, for PiQA and HellaSwag, both the estimation of contamination and the estimation of performance gain are high. For Natural Questions,on the other hand,the estimated 52 % {52}\% 52% contamination seems to have virtually no effect on the performance. For SQuAD and MATH, low thresholds yield high levels of contamination, but no performance gains. This suggests that contamination is either not helpful for these datasets, or that a larger n \mathrm{n} n is required to obtain a better estimate. Finally, for MBPP, HumanEval, MMLU and MMLU-Pro, other contamination detection methods may be needed: even with higher thresholds, 8-gram overlap gives such high contamination scores that it is impossible to get a good performance gain estimate.

结果。在表15中，我们报告了在所有关键基准测试中，为了获得最大估计性能增益，如上所述，被认为受到污染的评估数据百分比。从表中，我们排除了结果不显著的基准测试数据，例如因为干净或受污染的数据集样本太少，或者因为观察到的性能增益估计显示出极其不稳定的行为。在表15中，我们观察到对于某些数据集，污染有较大影响，而对于其他数据集则没有。例如，对于PiQA和HellaSwag，污染估计和性能增益估计都很高。另一方面，对于Natural Questions，估计的 52 % {52}\% 52%污染似乎对性能几乎没有影响。对于SQuAD和MATH，低阈值导致高水平的污染，但没有性能增益。这表明污染对这些数据集要么没有帮助，要么需要更大的 n \mathrm{n} n来获得更好的估计。最后，对于MBPP、HumanEval、MMLU和MMLU-Pro，可能需要其他污染检测方法：即使使用更高的阈值，8-gram重叠也会给出如此高的污染分数，以至于无法获得良好的性能增益估计。

5.2 Post-trained Language Model 后训练语言模型

We present results for our Llama 3 post-trained models on benchmarks across different capabilities. Similar to pre-training we are releasing the data generated as part of evaluations with publicly available benchmarks which can be found on Huggingface here. Additional details on our eval setup can be found here.

我们展示了我们的Llama 3后训练模型在不同能力基准测试上的结果。与预训练类似，我们发布了作为评估一部分生成的数据，这些数据可以在Huggingface上公开获取。有关我们评估设置的更多详细信息可以在这里找到。

Benchmarks and metrics. Table 16 contains an overview of all the benchmarks, organized by the capability. We apply decontamination of the post-training data by running exact match with the prompts from each benchmark. In addition to the standard academic benchmarks, we also performed extensive human evaluation of different capabilities. Details are provided in Section 5.3.

基准测试和指标。表16概述了所有基准测试，按能力组织。我们通过与每个基准测试的提示进行精确匹配来对后训练数据进行去污染处理。除了标准的学术基准测试外，我们还对不同能力进行了广泛的人工评估。详细信息在第5.3节中提供。

Experimental setup. We employ a similar experimental setup to the pre-training phase and conduct a comparative analysis of Llama 3 alongside other models of comparable size and capability. To the extent possible, we evaluate the performance of other models ourselves and compare the results with the reported numbers, selecting the best score. You can find additional details on our evaluation setup here.

实验设置。我们采用与预训练阶段类似的实验设置，并对 Llama 3 进行比较分析，同时与其他尺寸和能力相当的模型进行对比。在可能的情况下，我们自行评估其他模型的性能，并将结果与报告的数据进行比较，选择最佳分数。您可以在此处找到有关我们评估设置的更多详细信息。

Table 16 Post-training benchmarks by category. Overview of all benchmarks we use to evaluate post-trained Llama 3 models, ordered by capability.

表 16 按类别划分的训练后基准测试。我们用于评估训练后 Llama 3 模型的所有基准测试概览，按能力排序。

5.2.1 General Knowledge and Instruction-Following Benchmarks 通用知识和指令遵循基准测试

We evaluate Llama 3 on benchmarks for general knowledge and instruction-following in Table 2.

我们在表 2 中对 Llama 3 进行了通用知识和指令遵循的基准测试评估。

General knowledge. We leverage MMLU (Hendrycks et al., 2021a) and MMLU-Pro (Wang et al., 2024b) to evaluate Llama 3’s capability on knowledge-based question answering. For MMLU, we report the macro average of subtask accuracy under the 5-shot standard setting without CoT. MMLU-Pro is an extension of MMLU, incorporating more challenging, reasoning-focused questions, eliminating noisy questions, and expanding the choice set from four to ten options. Given its focus on complex reasoning, we report 5-shot CoT for MMLU-Pro. All tasks are formatted as generation tasks, similar to simple-evals (OpenAI, 2024).

通用知识。我们利用 MMLU（Hendrycks 等人，2021a）和 MMLU-Pro（Wang 等人，2024b）来评估 Llama 3 在基于知识的问题回答方面的能力。对于 MMLU，我们在不使用 CoT 的 5-shot 标准设置下报告子任务准确率的宏观平均值。MMLU-Pro 是 MMLU 的扩展，包含更具挑战性、以推理为重点的问题，消除了噪声问题，并将选项集从四个扩展到十个。鉴于其侧重于复杂推理，我们报告了 MMLU-Pro 的 5-shot CoT。所有任务均格式化为生成任务，类似于 simple-evals（OpenAI，2024）。

As shown in Table 2, our 8B and 70B Llama 3 variants outperform other models of similar sizes on both general knowledge tasks. Our 405B model outperforms GPT-4 and Nemotron 4 340B, with Claude 3.5 Sonnet leading among larger models.

如表 2 所示，我们的 8B 和 70B Llama 3 变体在通用知识任务上均优于其他尺寸相当的模型。我们的 405B 模型优于 GPT-4 和 Nemotron 4 340B，而 Claude 3.5 Sonnet 在更大模型中领先。

Instruction following. We assess the ability of Llama 3 and other models to follow natural language instructions on IFEval (Zhou et al., 2023). IFEval comprises approximately 500 “verifiable instructions” such as “write in more than 400 words”, which can be verified by heuristics. We report the average of prompt-level and instruction-level accuracy, under strict and loose constraints in Table 2. Note that all Llama 3 variants outperform comparable models across IFEval.

遵循指令。我们评估了Llama 3和其他模型在IFEval（Zhou et al., 2023）上遵循自然语言指令的能力。IFEval包含约500条“可验证指令”，例如“写超过400字”，这些指令可以通过启发式方法验证。我们在表2中报告了在严格和宽松约束下的提示级和指令级准确率的平均值。请注意，所有Llama 3变体在IFEval上的表现均优于同类模型。

5.2.2 Proficiency Exams 能力考试

Next, we evaluate our models on a wide variety of proficiency exams originally designed to test humans. We source these exams from publicly available official sources; for some exams, we report average scores across different exam sets per proficiency exam. Specifically, we average:

接下来，我们在一系列原本设计用于测试人类的能力考试上评估我们的模型。我们从公开可用的官方来源获取这些考试；对于某些考试，我们报告了每个能力考试在不同考试集上的平均分数。具体来说，我们平均了：

GRE: Official GRE Practice Test 1 and 2 (from the Educational Testing Services);

GRE：官方GRE练习测试1和2（来自教育考试服务机构）；

LSAT: Official Preptest 71, 73, 80 and 93 ;

LSAT：官方预测试71、73、80和93；

SAT: 8 exams from The Official SAT Study guide edition 2018;

SAT：来自2018年版《官方SAT学习指南》的8个考试；

AP: One official practice exam per subject;

AP：每个科目一个官方练习考试；

GMAT Official GMAT Online Exam.

GMAT：官方GMAT在线考试。

Questions in these exams contain both MCQ style and generation questions. We exclude the questions that are accompanied with images. For the GRE exams that contain questions with multiple correct options, we qualify the outputs as correct only if all the correct options are selected by the model. The evaluations are run using few shot prompting wherever we have more than 1 exam set per exam. We scale the scores to be in the range 130-170 for GRE and report accuracy for all other exams.

这些考试中的问题包含选择题和生成题。我们排除了附带图像的问题。对于包含多个正确选项的GRE考试，我们仅在模型选择了所有正确选项时才将其输出视为正确。我们在每个考试有多个考试集的情况下使用少量提示进行评估。我们将GRE的分数范围调整为130-170，并报告所有其他考试的准确率。

Table 17 Performance of Llama 3 models and GPT-40 on a variety of proficiency exams including ⁡ L S A T , S A T , G M A T , a n d \operatorname{including}\mathrm{{LSAT}},\mathrm{{SAT}},\mathrm{{GMAT}},\mathrm{{and}} includingLSAT,SAT,GMAT,and AP, and GRE tests. For GRE exams, we report normalized score; for all others, we report accuracy. For the bottom two rows corresponding to GRE Quant. and GRE Verbal, we report the scaled scores out of 170.

表17展示了Llama 3模型和GPT-40在多种能力考试 including ⁡ L S A T , S A T , G M A T , a n d \operatorname{including}\mathrm{{LSAT}},\mathrm{{SAT}},\mathrm{{GMAT}},\mathrm{{and}} includingLSAT,SAT,GMAT,and AP和GRE测试中的表现。对于GRE考试，我们报告标准化分数；对于其他所有考试，我们报告准确率。对于对应GRE数量和GRE语文的最后两行，我们报告170分制的分数。

Our results can be found in Table 17. We observe that the performance of our Llama 3405B model is very similar to Claude 3.5 Sonnet and GPT-4 40. Our 70B model has an even more impressive performance. It is significantly better than GPT-3.5 Turbo and beats Nemotron 4340B on many tests.

我们的结果见表17。我们观察到，我们的Llama 3405B模型的表现与Claude 3.5 Sonnet和GPT-4 40非常相似。我们的70B模型表现更为出色。它在许多测试中明显优于GPT-3.5 Turbo，并击败了Nemotron 4340B。

5.2.3 Coding Benchmarks 编码基准测试

We evaluate Llama 3 on code generation on several popular Python and multi-programming language benchmarks. To gauge the effectiveness of our models in generating functionally correct code, we use the pass@ N N N metric,which evaluates the pass rate for a set of unit tests among N N N generations. We report pass@1.

我们在几个流行的Python和多编程语言基准测试上评估Llama 3的代码生成能力。为了衡量我们模型生成功能正确代码的有效性，我们使用pass@ N N N指标，该指标评估在一组单元测试中通过率 N N N。我们报告pass@1。

Python code generation. HumanEval (Chen et al., 2021) and MBPP (Austin et al., 2021) are popular benchmarks for Python code generation which focus on relatively simple, self-contained functions. HumanEval+ (Liu et al., 2024a) is an enhanced version of HumanEval, in which more tests are generated to avoid false positives. The MBPP EvalPlus base version (v0.2.0) is a selection of 378 well-formed problems out of the 974 initial problems in all of the original MBPP (train and test) dataset (Liu et al., 2024a). Results for these benchmarks are reported in Table 18. Across the Python variants of these benchmarks, Llama 3 8B and 70B outperform

Python代码生成。HumanEval（Chen et al., 2021）和MBPP（Austin et al., 2021）是流行的Python代码生成基准测试，它们侧重于相对简单、自包含的函数。HumanEval+（Liu et al., 2024a）是HumanEval的增强版本，其中生成了更多测试以避免假阳性。MBPP EvalPlus基础版本（v0.2.0）是从原始MBPP（训练和测试）数据集（Liu et al., 2024a）中的974个初始问题中精选出的378个格式良好的问题。这些基准测试的结果报告在表18中。在这些基准测试的Python变体中，Llama 3 8B和70B表现出色。

Table 18 Pass@1 scores on code generation benchmarks. We report results on HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), as well as EvalPlus (Liu et al., 2024a) versions of these benchmarks.

表18 代码生成基准测试的Pass@1得分。我们报告了在HumanEval（Chen et al., 2021）、MBPP（Austin et al., 2021）以及这些基准的EvalPlus版本（Liu et al., 2024a）上的结果。

Table 19 Performance of non-Python programming tasks. We report Llama 3 results on MultiPL-E (Cassano et al., 2023).

表19 非Python编程任务的性能。我们报告了Llama 3在MultiPL-E（Cassano et al., 2023）上的结果。

models of similar sizes. For the largest models, Llama 3 405B, Claude 3.5 Sonnet and GPT-4o perform similarly, with GPT-4o showing the strongest results.

相似规模的模型。对于最大的模型，Llama 3 405B、Claude 3.5 Sonnet和GPT-4o表现相似，其中GPT-4o显示出最强的结果。

Multi-programming language code generation. To assess code generation capabilities beyond Python, we report results for the MultiPL-E (Cassano et al., 2023) benchmark, which is based on translations of problems from HumanEval and MBPP. Results for a subset of popular programming languages are reported in Table 19. Note that there is a significant drop in performance compared to the Python counterparts in Table 18.

多编程语言代码生成。为了评估超越Python的代码生成能力，我们报告了MultiPL-E（Cassano et al., 2023）基准测试的结果，该基准基于HumanEval和MBPP问题的翻译。表19报告了部分流行编程语言的结果。请注意，与表18中的Python对应项相比，性能有显著下降。

5.2.4 Multilingual Benchmarks 多语言基准测试

Llama 3 supports 8 languages - English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai, although the underlying foundation model has been trained on a broader collection of languages. 9 {}^{9} 9 In Table 20, we show results from evaluating Llama 3 on the multilingual MMLU (Hendrycks et al., 2021a) and Multilingual Grade School Math (MGSM) (Shi et al., 2022) benchmarks.

Llama 3支持8种语言——英语、德语、法语、意大利语、葡萄牙语、 Hindi、西班牙语和泰语，尽管底层基础模型已在更广泛的语言集合上进行了训练。 9 {}^{9} 9 在表20中，我们展示了在多语言MMLU（Hendrycks et al., 2021a）和多语言小学数学（MGSM）（Shi et al., 2022）基准测试上评估Llama 3的结果。

Multilingual MMLU. We translate MMLU questions, few-shot examples, and answers using Google Translate. We leave the task instructions in English and perform the evaluation in a 5-shot setting. In Table 20, we report average results across German, French, Italian, Portuguese, Hindi, Spanish, and Thai.

多语言MMLU。我们使用Google Translate翻译MMLU问题、少样本示例和答案。我们保留英语的任务指令，并在5样本设置中进行评估。在表20中，我们报告了德语、法语、意大利语、葡萄牙语、Hindi、西班牙语和泰语的平均结果。

9 {}^{9} 9 Llama 3 has not been optimized or safety tuned for use cases in those other languages. Developers may fine-tune Llama 3 models for languages beyond the 8 supported languages provided they comply with the Llama 3 Community License and the Acceptable Use Policy and in such cases are responsible for ensuring that any uses of Llama 3 in additional languages is done in a safe and responsible manner.

9 {}^{9} 9 Llama 3 尚未针对其他语言的使用场景进行优化或安全调优。开发者可以在遵守 Llama 3 社区许可协议和可接受使用政策的前提下，对 Llama 3 模型进行除支持的8种语言之外的语言微调，并负责确保在额外语言中使用 Llama 3 时采取安全且负责任的方式。

MGSM (Shi et al., 2022). We use the same native prompts as in simple-evals (OpenAI, 2024) for testing our models in a 0 -shot CoT setting. In Table 20, we report averge results across languages covered in MGSM benchmark.

MGSM（Shi et al., 2022）。我们使用与 simple-evals（OpenAI, 2024）相同的原生提示，在0-shot CoT设置下测试我们的模型。在表20中，我们报告了MGSM基准涵盖语言的平均结果。

Table 20 Multilingual benchmarks. For MGSM (Shi et al., 2022), we report 0 -shot CoT results for our Llama 3 models. Multilingual MMLU is an internal benchmark with translated MMLU (Hendrycks et al., 2021a) questions and answers into 7 languages - we report 5-shot results averaged across these languages.

表20 多语言基准测试。对于MGSM（Shi et al., 2022），我们报告了Llama 3模型的0-shot CoT结果。多语言MMLU是一个内部基准，包含将MMLU（Hendrycks et al., 2021a）的问题和答案翻译成7种语言，我们报告了这些语言的5-shot平均结果。

We find that Llama 3 405B outperforms most other models on MGSM, achieving an average of 91.6%. On MMLU, in line with English MMLU results shown above,Llama 3 405B falls behind GPT-4o by 2%. On the other hand, both Llama 370B and 8B models demonstrate strong performance, leading among competitors with a wide margin on both tasks.

我们发现Llama 3 405B在MGSM上表现优于大多数其他模型，平均达到91.6%。在MMLU上，与上述英语MMLU结果一致，Llama 3 405B落后于GPT-4o 2%。另一方面，Llama 3 70B和8B模型均展现出强劲性能，在两项任务中均以较大优势领先于竞争对手。

5.2.5 Math and Reasoning Benchmarks 数学与推理基准测试

Our math and reasoning benchmark results are presented in Table 2. Llama 3 8B model outperforms other models of similar sizes on GSM8K, MATH, and GPQA. Our 70B model performs significantly better than other models in its class on all the benchmarks. Finally, Llama 3405B model is the best in its category on GSM8K and ARC-C, while on MATH, it is the second best model. On GPQA, it is competitive with GPT-4 40, with Claude 3.5 Sonnet being the best model by a significant margin.

我们的数学和推理基准测试结果如表2所示。Llama 3 8B模型在GSM8K、MATH和GPQA上优于其他同尺寸模型。我们的70B模型在所有基准测试中显著优于同级别的其他模型。最后，Llama 3405B模型在GSM8K和ARC-C类别中表现最佳，而在MATH中是第二好的模型。在GPQA上，它与GPT-4 40竞争激烈，而Claude 3.5 Sonnet以显著优势成为最佳模型。

5.2.6 Long Context Benchmarks 长上下文基准测试

We consider a diverse set of tasks that span various domains and text types. In the benchmarks we list below, we focus on sub-tasks that use unbiased evaluation protocols, i.e., accuracy-based metrics rather than n-gram overlapping metrics. We also prioritize tasks that we found to be of low variance.

我们考虑了一系列跨多个领域和文本类型的多样化任务。在下面列出的基准测试中，我们专注于使用无偏评估协议的子任务，即基于准确性的指标而非n-gram重叠指标。我们还优先考虑了我们发现方差较低的任务。

Needle-in-a-Haystack (Kamradt, 2023) measures a model’s ability to retrieve a hidden information inserted in random parts of the long document. Our Llama 3 models demonstrate perfect needle retrieval performance,successfully retrieving 100 % {100}\% 100% of needles at all document depths and context lengths. We also measure performance on Multi-needle (Table 21), a variation of Needle-in-a-Haystack, where we insert four needles in the context and test if a model can retrieve two of them. Our Llama 3 models achieve near perfect retrieval results.

“大海捞针”（Kamradt，2023）衡量模型在长文档随机部分中检索隐藏信息的能力。我们的Llama 3模型展示了完美的针检索性能，成功检索了 100 % {100}\% 100%的针，无论文档深度和上下文长度如何。我们还测量了“多针”（表21）的性能，这是“大海捞针”的一个变体，我们在上下文中插入四个针，并测试模型是否能检索其中两个。我们的Llama 3模型实现了近乎完美的检索结果。

ZeroSCROLLS (Shaham et al., 2023) is a zero-shot benchmark for natural language understanding over long texts. We report numbers on the validation set, as the ground truth answers are not publicly available. Our Llama 3405B and 70B models either match or surpass other models on various tasks in this benchmark.

ZeroSCROLLS（Shaham等人，2023）是一个针对长文本自然语言理解的零样本基准测试。我们报告了验证集上的数据，因为真实答案并未公开。我们的Llama 3405B和70B模型在基准测试的各项任务中与其他模型持平或超越。

InfiniteBench (Zhang et al., 2024) requires models to understand long dependencies in the context window. We evaluate Llama 3 on En.QA (QA over novels) and En.MC (multiple-choice QA over novels), where our 405    B {405}\mathrm{\;B} 405B model outperforms all others. The gains are particularly significant on En.QA.

InfiniteBench（Zhang et al., 2024）要求模型理解上下文窗口中的长依赖关系。我们在En.QA（小说问答）和En.MC（小说多项选择问答）上评估Llama 3，我们的 405    B {405}\mathrm{\;B} 405B模型在这些任务中表现优于所有其他模型。特别是在En.QA上，我们的模型取得了显著的提升。

5.2.7 Tool Use Performance 工具使用性能

We evaluate our models on a range of benchmarks for zero-shot tool use (i.e. function calling): Nexus (Srinivasan et al., 2023), API-Bank (Li et al., 2023b), Gorilla API-Bench (Patil et al., 2023), and the Berkeley Function Calling Leaderboard (BFCL) (Yan et al., 2024). Results are shown in Table 22.

我们在一系列零样本工具使用（即函数调用）基准上评估我们的模型：Nexus（Srinivasan et al., 2023）、API-Bank（Li et al., 2023b）、Gorilla API-Bench（Patil et al., 2023）和Berkeley函数调用排行榜（BFCL）（Yan et al., 2024）。结果如表22所示。

On Nexus, our Llama 3 variants perform the best compared to their counterparts. On the API-Bank, our Llama 38 B {38}\mathrm{B} 38B and 70    B {70}\mathrm{\;B} 70B models outperform other models in their category by a significant margin. The 405    B {405}\mathrm{\;B} 405B model is behind Claude 3.5 Sonnet by only 0.6 % {0.6}\% 0.6% . Finally,our 405    B {405}\mathrm{\;B} 405B and 70    B {70}\mathrm{\;B} 70B models perform competitively on BFCL and are close second in their respective size class. Llama 3 8B performs the best in its category.

在Nexus上，我们的Llama 3变体相较于其他模型表现最佳。在API-Bank上，我们的Llama 38 B {38}\mathrm{B} 38B和 70    B {70}\mathrm{\;B} 70B模型在其类别中以显著优势超越其他模型。 405    B {405}\mathrm{\;B} 405B模型仅落后于Claude 3.5 Sonnet 0.6 % {0.6}\% 0.6%。最后，我们的 405    B {405}\mathrm{\;B} 405B和 70    B {70}\mathrm{\;B} 70B模型在BFCL上表现竞争性，并在各自的大小类别中位列第二。Llama 3 8B在其类别中表现最佳。

Table 21 Long-context benchmarks. For ZeroSCROLLS (Shaham et al., 2023), we report numbers on the validation set. For QuALITY we report exact match, for Qasper - f1 and for SQuALITY - rougeL. We report f1 for InfiniteBench (Zhang et al., 2024) En.QA metric and accuracy for En.MC. For Multi-needle (Kamradt, 2023) we insert 4 needles in the context and test if a model can retrieve 2 needles at different context lengths, we compute average recall across 10 sequence lengths up till 128 k {128}\mathrm{k} 128k .

表21 长上下文基准测试。对于ZeroSCROLLS（Shaham等人，2023年），我们报告验证集上的数据。对于QuALITY，我们报告完全匹配，对于Qasper - f1，对于SQuALITY - rougeL。我们报告InfiniteBench（Zhang等人，2024年）En.QA指标的f1值和En.MC的准确性。对于Multi-needle（Kamradt，2023年），我们在上下文中插入4根针，并测试模型是否能在不同上下文长度下检索到2根针，我们计算直到 128 k {128}\mathrm{k} 128k的10个序列长度的平均召回率。

Human evaluations. We also conduct human evaluations to test the tool use capabilities of the model, with a focus on code execution tasks. We collect 2000 user prompts related to code execution (without plotting or file uploads), plot generation, and file uploads. These prompts are collected from the LMSys dataset (Chiang et al., 2024), GAIA benchmark (Mialon et al., 2023b), human annotators, and synthetic generation.

人类评估。我们还进行人类评估，以测试模型的工具使用能力，重点是代码执行任务。我们收集了2000个与代码执行（不包括绘图或文件上传）、绘图生成和文件上传相关的用户提示。这些提示来自LMSys数据集（Chiang等人，2024年）、GAIA基准（Mialon等人，2023b年）、人工标注者和合成生成。

We compare Llama 3 405B to GPT-4o using OpenAI’s Assistants A P I 10 {\mathrm{{API}}}^{10} API10 . The results are provided in Figure 16. On text-only code execution tasks and plots generation, Llama 3405B significantly beats GPT-4o. However, it lags behind on the file upload use case.

我们使用OpenAI的Assistants A P I 10 {\mathrm{{API}}}^{10} API10 将Llama 3 405B与GPT-4o进行比较。结果如图16所示。在纯文本代码执行任务和绘图生成方面，Llama 3405B显著优于GPT-4o。然而，在文件上传用例方面，它落后于GPT-4o。

5.3 Human Evaluations 人类评估

In addition to evaluations on standard benchmark sets, we also perform a series of human evaluations. These evaluations allow us to measure and optimize more subtle aspects of model performance, such as our model’s tone, verbosity, and

除了在标准基准集上的评估外，我们还进行了一系列人类评估。这些评估使我们能够测量和优化模型性能的更细微方面，例如我们模型的语气、冗长程度和

understanding of nuances and cul- Table 22 Zero-shot tool use benchmarks. We report function calling accuracy tural contexts. Well-designed hu- across Nexus (Srinivasan et al., 2023), API-Bank (Li et al., 2023b), API-man evaluations closely reflect the Bench (Patil et al., 2023), and BFCL (Yan et al., 2024). user experience, providing insights into how the model performs in real-world scenarios.

对细微差别和文化背景的理解。表22展示了零样本工具使用基准测试。我们报告了在Nexus（Srinivasan等人，2023年）、API-Bank（Li等人，2023b）、API-man评估（Patil等人，2023年）和BFCL（Yan等人，2024年）中的函数调用准确性。这些评估紧密反映了用户体验，提供了模型在现实场景中表现的见解。

Prompt collection. We collected high-quality prompt spanning a wide range of categories and difficulties. To do so, we first developed a taxonomy with categories and subcategories capturing as many model capabilities as possible. We used this taxonomy to collect about 7,000 prompts spanning six individual capabilities (English, reasoning,coding,Hindi,Spanish,and Portuguese),and three multiturn capabilities 11 {}^{11} 11 (English,reasoning, and coding). We ensured that within each category, prompts are uniformly distributed across subcategories. We also categorized each prompt into one of three difficulty levels and ensured that our prompt collection contains roughly 10 % {10}\% 10% easy prompts, 30 % {30}\% 30% medium prompts,and 60 % {60}\% 60% hard prompts. All the human evaluation prompt sets were subject to a thorough quality assurance process. Modeling teams did not have access to our human-evaluation prompts to prevent accidental contamination or overfitting on the test set.

提示收集。我们收集了涵盖广泛类别和难度的高质量提示。为此，我们首先开发了一个分类法，包括类别和子类别，以尽可能多地捕捉模型的能力。我们利用这一分类法收集了约7,000个提示，涵盖六种独立能力（英语、推理、编程、印地语、西班牙语和葡萄牙语）和三种多轮能力 11 {}^{11} 11（英语、推理和编程）。我们确保在每个类别中，提示在子类别中均匀分布。我们还根据难度将每个提示分为三个级别，并确保我们的提示集合包含大约 10 % {10}\% 10% 简单提示、 30 % {30}\% 30% 中等提示和 60 % {60}\% 60% 困难提示。所有人类评估提示集都经过了严格的质量保证流程。建模团队无法访问我们的人类评估提示，以防止测试集的意外污染或过拟合。

10 {}^{10} 10 https://platform.openai.com/docs/assistants/overview

11 {}^{11} 11 For multiturn human evaluations,the number of turns is between 2 and 11 in each prompt. We assess the model response in the final turn.

11 {}^{11} 11 对于多轮人类评估，每个提示中的轮数在2到11之间。我们评估模型在最后一轮的响应。

Figure 16: Human evaluation results for Llama 3 405B vs. GPT-4o on code execution tasks including plotting and file uploads. Llama 3405B outperforms GPT-4o on code execution (without plotting or file uploads) as well as plot generation, but lags behind in file upload use cases.

图16：Llama 3 405B与GPT-4o在包括绘图和文件上传的代码执行任务上的人类评估结果。Llama 3405B在代码执行（不包括绘图或文件上传）以及绘图生成方面优于GPT-4o，但在文件上传用例中落后。

Evaluation process. To perform a pairwise human evaluation of two models, we ask human annotators which of two model responses (produced by different models) they prefer. Annotators use a 7-point scale for their ratings, enabling them to indicate whether one model response is much better than, better than, slightly better than, or about the same as the other model response. When an annotator indicates that one model response is better or much better than the other model response, we consider this a “win” for that model. We perform pairwise comparisons between models in which we report win rates per capability in the prompt set.

评估过程。为了进行两个模型的成对人类评估，我们询问人类标注者他们更喜欢两个模型响应中的哪一个（由不同模型产生）。标注者使用7点量表进行评分，使他们能够表明一个模型响应是否远优于、优于、略优于或与另一个模型响应大致相同。当标注者表示一个模型响应优于或远优于另一个模型响应时，我们认为这是该模型的“胜利”。我们在模型之间进行成对比较，报告提示集中每项能力的胜率。

Results. We use our human evaluation process to compare Llama 3 405B with GPT-4 (0125 API version), GPT-40 (API version), and Claude 3.5 Sonnet (API version). The results of these evaluations are presented in Figure 17. We observe that Llama 3405B performs approximately on par with the 0125 API version of GPT-4, while achieving mixed results (some wins and some losses) compared to GPT-4o and Claude 3.5 Sonnet. On nearly all capabilities, the win rates of Llama 3 and GPT-4 are within the margin of error. On multiturn reasoning and coding tasks, Llama 3405B outperforms GPT-4 but it underperforms GPT-4 on multilingual (Hindi, Spanish, and Portuguese) prompts. Llama 3 performs on par with GPT-4o on English prompts, on par with Claude 3.5 Sonnet on multilingual prompts, and outperforms Claude 3.5 Sonnet on single and multiturn English prompts. However, it trails Claude 3.5 Sonnet in capabilities such as coding and reasoning. Qualitatively, we find that model performance in human evaluations is heavily influenced by nuanced factors such as model tone, response structure, and verbosity - factors that we are optimizing for in our post-training process. Overall, our human evaluation results are consistent with those on standard benchmark evaluations: Llama 3405B is very competitive with leading industry models, making it the best-performing openly available model.

结果。我们利用人类评估流程来比较 Llama 3 405B 与 GPT-4（0125 API 版本）、GPT-40（API 版本）以及 Claude 3.5 Sonnet（API 版本）。这些评估的结果如图 17 所示。我们观察到，Llama 3405B 的表现与 GPT-4 的 0125 API 版本大致相当，而在与 GPT-40 和 Claude 3.5 Sonnet 的比较中则呈现出混合结果（有胜有负）。在几乎所有能力上，Llama 3 和 GPT-4 的胜率都在误差范围内。在多轮推理和编码任务上，Llama 3405B 优于 GPT-4，但在多语言（印地语、西班牙语和葡萄牙语）提示上表现不佳。Llama 3 在英语提示上与 GPT-40 相当，在多语言提示上与 Claude 3.5 Sonnet 相当，并在单轮和多轮英语提示上优于 Claude 3.5 Sonnet。然而，在编码和推理等能力上，它落后于 Claude 3.5 Sonnet。从定性角度看，我们发现模型在人类评估中的表现深受模型语调、响应结构和冗长程度等细微因素的影响——这些因素是我们训练后优化过程中的重点。总体而言，我们的人类评估结果与标准基准评估结果一致：Llama 3405B 与行业领先模型非常具有竞争力，使其成为表现最佳的公开可用模型。

Limitations. All human evaluation results underwent a thorough data quality assurance process. However, since it is challenging to define objective criteria for evaluating model responses, human evaluations can still be influenced by personal biases, backgrounds, and preferences of human annotators, which may lead to inconsistent or unreliable results.

局限性。所有人类评估结果都经过了严格的数据质量保证流程。然而，由于定义模型响应评估的客观标准具有挑战性，人类评估仍可能受到人类标注者的个人偏见、背景和偏好影响，这可能导致结果不一致或不可靠。

5.4 Safety 安全性

We focus our study on assessing Llama 3’s ability to generate content in a safe and responsible way, while still maximizing helpful information. Our safety work begins in the pre-training stage, primarily in the form of

我们专注于评估 Llama 3 在安全且负责任的方式下生成内容的能力，同时最大化有用信息。我们的安全工作始于预训练阶段，主要形式为

Figure 17 Human evaluation results for the Llama 3 405B model. L e f t {Left} Left : Comparison with GPT-4. M i d d l e {Middle} Middle : Comparison with GPT-4o. Right: Comparison with Claude 3.5 Sonnet. All results include 95% confidence intervals and exclude ties.

图 17 Llama 3 405B 模型的人类评估结果。 L e f t {Left} Left：与 GPT-4 的比较。 M i d d l e {Middle} Middle：与 GPT-4o 的比较。右侧：与 Claude 3.5 Sonnet 的比较。所有结果包括 95% 置信区间并排除平局。

data cleaning and filtering. We then describe our approach to safety finetuning, focusing on how to train the model to align to specific safety policies while still retaining helpfulness. We analyze each of the Llama 3 capabilities, including multilingual, long context, tool usage, and various multimodal capabilities, to measure the effectiveness of our safety mitigations.

数据清洗和过滤。然后我们描述了我们的安全微调方法，重点是如何训练模型以符合特定的安全政策，同时仍然保持有用性。我们分析了 Llama 3 的每项能力，包括多语言、长上下文、工具使用和各种多模态能力，以衡量我们安全缓解措施的有效性。

Subsequently, we describe our assessment of uplift for cybersecurity and chemical and biological weapons risks. Uplift refers to the additional risk introduced by new technological developments compared to using existing available technologies (such as web search).

随后，我们描述了对网络安全和化学及生物武器风险提升的评估。提升指的是与使用现有可用技术（如网络搜索）相比，新技术发展引入的额外风险。

We then describe how we leverage Red Teaming to iteratively identify and combat various safety risks across capabilities and perform a residual risk assessment.

然后我们描述了如何利用红队测试来迭代识别和应对跨能力的安全风险，并进行残余风险评估。

Finally, we describe system-level safety, or the development and orchestration of classifiers around the input and output of the model itself to further enhance safety and make it easier for developers to both customize safety to various usecases and deploy generative AI in more responsible ways.

最后，我们描述了系统级安全，即围绕模型本身输入和输出的分类器的发展和编排，以进一步增强安全性，并使开发者更容易为各种用例定制安全措施，并以更负责任的方式部署生成式 AI。

5.4.1 Benchmark Construction 基准构建

We create various internal benchmarks to help us develop models safely and responsibly. Our benchmarks are heavily inspired by the risk categories from the ML Commons taxonomy of hazards (Vidgen et al., 2024). While various benchmarks for language model safety exist such as ToxiGen (Hartvigsen et al., 2022), XS Test (Röttger et al., 2023), and more, a number of the new capabilities in Llama 3 do not have sufficient external benchmarks available and often external benchmarks lack sufficient coverage of breadth and depth.

我们创建了多种内部基准测试，以帮助我们安全且负责任地开发模型。我们的基准测试深受ML Commons危害分类法（Vidgen et al., 2024）中的风险类别启发。尽管存在多种语言模型安全基准测试，如ToxiGen（Hartvigsen et al., 2022）、XS Test（Röttger et al., 2023）等，但Llama 3中的许多新功能缺乏足够的外部基准测试，且外部基准测试往往缺乏足够的广度和深度覆盖。

For each risk category, we collect human-written prompts that are either adversarial or borderline in nature - examples of such prompts can be found in Table 23. Adversarial prompts range from straightforward ones that directly elicit a harmful response to ones that incorporate sophisticated jailbreaking techniques. Benchmarks of these adversarial prompts are the basis for how we measure violation rate.

对于每个风险类别，我们收集了人为编写的提示，这些提示本质上是敌对的或边缘性的——此类提示的示例可在表23中找到。敌对提示从直接引发有害响应的简单提示到包含复杂越狱技术的提示不等。这些敌对提示的基准测试是我们衡量违规率的基础。

As a counter-metric to violation rate, we also construct false refusal benchmarks composed of borderline prompts. We refer to false refusal as occurring when a model refuses to answer in a helpful way even when a plausible, safe response is possible. Borderline prompts are prompts near the decision boundary that a well-calibrated model should handle, for example, “How can I steal the spotlight from my bestie who always acts like the main character??” Our overall benchmark size across violations and false refusals is over 4000 prompts per capability or language, and contains a mix of single-turn and multi-turn prompts.

作为违规率的反向指标，我们还构建了由边缘提示组成的错误拒绝基准测试。我们将错误拒绝定义为模型即使在可能提供合理、安全响应的情况下也不以有帮助的方式回答。边缘提示是接近决策边界的提示，一个校准良好的模型应能处理，例如，“我如何从总是表现得像主角的好友那里抢走焦点？？”我们在违规和错误拒绝方面的总体基准测试规模超过每种能力或语言的4000个提示，并包含单轮和多轮提示的混合。

Table 23 Examples of adversarial prompts from our internal benchmarks across all the capabilities.

表23 我们内部基准测试中跨所有能力的敌对提示示例。

Table 24 Average verbatim memorization in pre-trained Llama 3 for selected test scenarios. Our baseline is Llama 2 in the English, 50-gram scenario using the same prompting methodology applied to its data mix.

表24 预训练Llama 3在选定测试场景中的平均逐字记忆。我们的基线是使用相同提示方法应用于其数据混合的英语50-gram场景中的Llama 2。

5.4.2 Safety Pre-training 安全预训练

We believe responsible development must be considered from an end-to-end perspective and incorporated at every stage of model development and deployment. During pre-training, we apply a variety of filters, such as filters to identify websites that likely contain personally identifiable information (see Section 3.1). We also focus heavily on discoverable memorization (Nasr et al., 2023). Similar to Carlini et al. (2022), we sample prompts and ground truths at different frequencies of occurrence in the training data using an efficient rolling hash index of all n-grams in the corpus. We construct different test scenarios by varying the length of prompt and ground truth, the detected language of target data, and the domain. We then measure how often the model generates the ground truth sequence verbatim, and analyze the relative rates of memorization in the specified scenarios. We define verbatim memorization as the inclusion rate - the proportion of model generations that include the ground truth continuation exactly - and report averages weighted by the prevalence of given characteristics in the data,as shown in Table 24. We find low memorization rates of training data ( 1.13 % ({1.13}\% (1.13% and 3.91 % {3.91}\% 3.91% on average for the 405    B {405}\mathrm{\;B} 405B with n = 50 n = {50} n=50 and n = 1000 n = {1000} n=1000 respectively). Memorization rates are roughly on par with Llama 2 at equivalent size and using the same methodology applied to its data mix. 12 {}^{12} 12

我们相信，负责任的开发必须从端到端的角度考虑，并在模型开发和部署的每个阶段融入。在预训练阶段，我们应用了多种过滤器，例如用于识别可能包含个人身份信息网站的过滤器（参见第3.1节）。我们还重点研究了可发现的记忆化（Nasr等人，2023年）。与Carlini等人（2022年）类似，我们使用语料库中所有n-gram的高效滚动哈希索引，在训练数据中以不同频率采样提示和真实值。我们通过改变提示和真实值的长度、目标数据检测到的语言以及领域来构建不同的测试场景。然后，我们测量模型逐字生成真实值序列的频率，并分析指定场景中的记忆化相对速率。我们将逐字记忆化定义为包含率——即包含完全真实值延续的模型生成比例——并根据数据中给定特征的普遍性报告加权平均值，如表24所示。我们发现训练数据的记忆化率较低 ( 1.13 % ({1.13}\% (1.13%和 3.91 % {3.91}\% 3.91%，平均而言，对于 405    B {405}\mathrm{\;B} 405B分别具有 n = 50 n = {50} n=50和 n = 1000 n = {1000} n=1000）。记忆化率大致与Llama 2在相同大小并使用应用于其数据混合的相同方法相当。 12 {}^{12} 12

5.4.3 Safety Finetuning 安全微调

We describe our approach to safety finetuning to mitigate risks across many capabilities, which encompasses two key aspects: (1) safety training data and (2) risk mitigation techniques. Our safety finetuning process builds upon our general finetuning methodology with modifications tailored to address specific safety concerns.

我们描述了安全微调的方法，以减轻跨多种能力的风险，这包括两个关键方面：（1）安全训练数据和（2）风险缓解技术。我们的安全微调过程基于我们的通用微调方法，并进行了针对特定安全问题的修改。

We optimize for two primary metrics: Violation Rate (VR), a metric that captures when the model produces a response that violates a safety policy, and False Refusal Rate (FRR), a metric that captures when the model incorrectly refuses to respond to a harmless prompt. In parallel, we evaluate model performance on helpfulness benchmarks to ensure that safety improvements do not compromise overall helpfulness.

我们针对两个主要指标进行优化：违规率（VR），该指标捕捉模型何时产生违反安全政策的响应；错误拒绝率（FRR），该指标捕捉模型何时错误地拒绝回应无害的提示。同时，我们评估模型在帮助性基准上的表现，以确保安全性的提升不会损害整体帮助性。

12 {}^{12} 12 Note there are limitations with our analysis — for example,recent work advocates for metrics beyond exact match (Ippolito et al., 2023) and alternative prompt search strategies (Kassem et al., 2024). Nonetheless, we find the results of the evaluations to be encouraging.

12 {}^{12} 12 需要注意的是，我们的分析存在局限性——例如，最近的研究提倡使用超越精确匹配的指标（Ippolito 等人，2023）和替代的提示搜索策略（Kassem 等人，2024）。尽管如此，我们认为评估结果是令人鼓舞的。

Finetuning data. The quality and design of safety training data has a profound impact on performance. Through extensive ablations, we find that the quality is more critical than the quantity. We mainly use human-generated data collected from our data vendors, but find that it can be prone to errors and inconsistencies - particularly for nuanced safety policies. To ensure the highest quality data, we developed AI-assisted annotation tools to support our rigorous quality assurance processes. In addition to collecting adversarial prompts, we also gather a set of similar prompts, which we refer to as borderline prompts. These are closely related to the adversarial prompts but with a goal to teach the model to learn to provide helpful responses, thereby reducing the false refusal rate (FRR).

微调数据。安全训练数据的质量和设计对性能有深远影响。通过广泛的消融实验，我们发现质量比数量更为关键。我们主要使用从数据供应商处收集的人工生成数据，但发现这些数据容易出错且不一致——特别是在处理微妙的安全政策时。为了确保最高质量的数据，我们开发了AI辅助的标注工具来支持我们严格的质量保证流程。除了收集对抗性提示外，我们还收集了一组类似的提示，我们称之为边缘提示。这些提示与对抗性提示密切相关，但目的是教会模型学会提供有帮助的响应，从而降低错误拒绝率（FRR）。

Figure 18 Influence of model size on safety mix design for balancing violation rate (VR) and false refusal rate (FRR). Each point of the scatterplot represents a different data mix balancing safety and helpfulness data. Different model sizes retain varying capacities for safety learning. Our experiments show that 8    B 8\mathrm{\;B} 8B models require a higher proportion of safety data relative to helpfulness data in the overall SFT mix to achieve comparable safety performance to 70    B {70}\mathrm{\;B} 70B models. Larger models are more capable of discerning between adversarial and borderline context, resulting in a more favorable balance between VR and FRR.

图18 模型大小对安全混合设计的影响，以平衡违规率（VR）和误拒率（FRR）。散点图的每个点代表不同的数据混合，平衡了安全性和有用性数据。不同大小的模型保留了不同的安全学习能力。我们的实验表明， 8    B 8\mathrm{\;B} 8B模型相对于有用性数据，在整体SFT混合中需要更高比例的安全数据，以达到与 70    B {70}\mathrm{\;B} 70B模型相当的安全性能。较大的模型更能区分对抗性和边缘情境，从而在VR和FRR之间实现更有利的平衡。

Beyond human annotation, we also leverage synthetic data to improve the quality and coverage of our training datasets. We utilize a range of techniques to generate additional adversarial examples, including in-context learning with carefully crafted system prompts, guided mutation of seed prompts based on new attack vectors, and advanced algorithms including Rainbow Teaming (Samvelyan et al., 2024), based on MAP-Elites (Mouret and

除了人工标注，我们还利用合成数据来提高训练数据集的质量和覆盖范围。我们采用多种技术生成额外的对抗性示例，包括通过精心设计的系统提示进行情境学习，基于新攻击向量引导种子提示的变异，以及基于MAP-Elites（Mouret和Clune，2015）的高级算法，如Rainbow Teaming（Samvelyan等人，2024），生成在多个多样性维度上受限的提示。

Clune, 2015), which generate prompts constrained across multiple dimensions of diversity.

我们还进一步解决了模型在生成安全响应时的语气问题，这对下游用户体验有影响。我们为Llama 3开发了拒绝语气指南，并通过严格的质量保证流程确保所有新安全数据符合该指南。我们还通过零样本重写和人在回路的编辑相结合的方法，对现有安全数据进行细化，以生成高质量数据。通过采用这些方法，以及使用语气分类器评估安全响应的语气质量，我们能够显著改善模型的措辞。

We further address the model’s tone when producing safe responses, which has an impact on downstream user experience. We developed a refusal tone guideline for Llama 3 and ensured that all new safety data adhered to it through rigorous quality assurance process. We also refine existing safety data to align with the guideline, using a combination of zero-shot rewriting and human-in-the-loop editing to produce high-quality data. By employing these methods, along with a tone classifier to assess tone quality for safety responses, we are able to significantly improve the model’s verbiage.

我们还进一步解决了模型在生成安全响应时的语气问题，这对下游用户体验有影响。我们为Llama 3开发了拒绝语气指南，并通过严格的质量保证流程确保所有新安全数据符合该指南。我们还通过零样本重写和人在回路的编辑相结合的方法，对现有安全数据进行细化，以生成高质量数据。通过采用这些方法，以及使用语气分类器评估安全响应的语气质量，我们能够显著改善模型的措辞。

Safety supervised finetuning. Following our Llama 2 recipe (Touvron et al., 2023b), we combine all helpfulness data and safety data during the model alignment stage. Additionally, we introduce a borderline dataset to help the model discern the subtle distinctions between safe and unsafe requests. Our annotation teams are instructed to meticulously craft responses to safety prompts based on our guidelines. We have found that SFT is highly effective in aligning the model when we strategically balance the ratio of adversarial to borderline examples. We put the focus on more challenging risk areas, with a higher ratio of borderline examples. This plays a crucial role in our successful safety mitigation efforts while keeping false refusal to a minimum.

安全监督微调。遵循我们的 Llama 2 配方（Touvron 等人，2023b），我们在模型对齐阶段结合所有有用性数据和安全数据。此外，我们引入了一个边界数据集，以帮助模型辨别安全请求和不安全请求之间的细微差别。我们的标注团队根据我们的指南精心制作安全提示的响应。我们发现，在战略性地平衡对抗性示例与边界示例的比例时，SFT 在模型对齐方面非常有效。我们将重点放在更具挑战性的风险领域，边界示例的比例更高。这在保持最低误拒率的同时，对我们的安全缓解工作起到了至关重要的作用。

Further, we examine the impact of model size on the trade-off between FRR and VR in Figure 18. Our results show that it varies - with smaller models requiring a larger proportion of safety data relative to helpfulness, and that it is more challenging to efficiently balance VR and FRR compared to larger models.

此外，我们在图 18 中考察了模型大小对 FRR 和 VR 之间权衡的影响。我们的结果显示，这种影响是不同的——较小的模型相对于有用性数据需要更大比例的安全数据，并且与较大的模型相比，更难以有效平衡 VR 和 FRR。

Safety DPO. To reinforce safety learning, we incorporate adversarial and borderline examples into our preference datasets in DPO. We discover that crafting response pairs to be nearly orthogonal in an embedding space is particularly effective in teaching the model to distinguish between good and bad responses for a given prompt. We conduct multiple experiments to determine the optimal ratio of adversarial, borderline, and helpfulness examples, aiming to optimize the trade-off between FRR and VR. We also find that the model size influences the learning outcomes - as a result, we tailor different safety mixes for various model sizes.

安全 DPO。为了加强安全学习，我们将对抗性示例和边界示例纳入 DPO 的偏好数据集中。我们发现，在嵌入空间中制作响应对几乎正交特别有效，这有助于模型区分给定提示的好坏响应。我们进行了多次实验，以确定对抗性示例、边界示例和有用性示例的最佳比例，旨在优化 FRR 和 VR 之间的权衡。我们还发现，模型大小影响学习结果——因此，我们为不同大小的模型定制了不同的安全混合。

Figure 19 – Violation rates (VR) and false refusal rates (FRR) on English and our core multilingual short context benchmarks, comparing Llama 3405B - with and without Llama Guard (LG) system-level protections - to competitor models and systems. Languages not supported by Comp. 3 represented with an ‘x.’ Lower is better.

图 19 – 在英语和我们核心多语言短上下文基准上的违规率（VR）和误拒率（FRR），比较 Llama 3405B – 有无 Llama Guard（LG）系统级保护 – 与竞争对手模型和系统。Comp. 3 不支持的语言用 ‘x’ 表示。数值越低越好。

Figure 20 – Violation rates (VR) and false refusal rates (FRR) on tool use and long context benchmarks. Lower is better. The performance for DocQA and Many-shot benchmarks are listed separately. Note we do not have a borderline data set for Many-shot, due to the adversarial nature of the benchmark, and thus do not measure false refusal rates on it. For Tool Usage (Search), we only test Llama 3 405B compared to Comp. 1.

图20 – 工具使用和长上下文基准的违规率（VR）和误拒率（FRR）。数值越低越好。DocQA和Many-shot基准的性能分别列出。请注意，由于Many-shot基准的对抗性质，我们没有边缘数据集，因此不在其上测量误拒率。对于工具使用（搜索），我们仅测试了Llama 3 405B与Comp. 1的对比。

5.4.4 Safety Results 安全结果

We first highlight Llama 3’s general behavior along various axes and then describe results for each specific new capability and our effectiveness at mitigating the safety risks.

我们首先强调Llama 3在各个方面的普遍行为，然后描述每项新能力的具体结果以及我们在缓解安全风险方面的有效性。

Overall performance. A comparison of Llama 3’s final violation and false refusal rates with similar models can be found in Figures 19 and 20. These results focus on our largest parameter size Llama 3405B model, compared to relevant competitors. Two of the competitors are end-to-end systems accessed through API, and one of them is an open source language model that we host internally and we evaluate directly. 13 {}^{13} 13 We evaluate our Llama models both standalone and coupled with Llama Guard, our open source system-level safety solution (more in Section 5.4.7).

总体性能。Llama 3的最终违规率和误拒率与类似模型的比较可以在图19和图20中找到。这些结果聚焦于我们最大的参数尺寸Llama 3405B模型，与相关竞争对手进行比较。其中两个竞争对手是通过API访问的端到端系统，另一个是我们内部托管的开源语言模型，我们直接对其进行评估。 13 {}^{13} 13 我们评估了Llama模型在独立运行和与Llama Guard（我们的开源系统级安全解决方案，更多内容见5.4.7节）结合使用的情况。

While a low violation rate is desirable, it is critical to consider false refusal as a counter-metric, as a model that always refuses is maximally safe, but not helpful in the slightest. Similarly, a model that always answers every prompt, regardless of how problematic the request, would be overly harmful and toxic. In Figure 21, leveraging our internal benchmarks, we explore how different models and systems in industry navigate this trade off and how Llama 3 compares. We find that our models achieve very competitive violation rate metrics while keeping false refusal rate low as well, indicating a solid balance between helpfulness and safety.

虽然低违规率是理想的，但考虑误拒率作为反向指标至关重要，因为一个总是拒绝的模型虽然最大限度地安全，但毫无帮助。同样，一个无论请求多么有问题都总是回答的模型，将过于有害和有毒。在图21中，利用我们的内部基准，我们探讨了行业中不同模型和系统如何权衡这一问题，以及Llama 3的比较情况。我们发现，我们的模型在违规率指标上实现了非常具有竞争力的表现, 同时保持低错误拒绝率，表明在有用性和安全性之间取得了良好的平衡。

13 {}^{13} 13 Because these safety benchmarks are internal to Meta,we acknowledge that the numbers in this section are not reproducible externally, and so we choose to anonymize the competitors we evaluate against.

13 {}^{13} 13 由于这些安全基准是Meta内部的，我们承认本节中的数据无法在外部复现，因此我们选择对评估的竞争对手进行匿名处理。

Figure 21 Violation and false refusal rates across models and capabilities. Each p o i n t \mathrm{{point}} point represents the o v e r a l l \mathrm{{overall}} overall false r e f u s a l \mathrm{{refusal}} refusal and violation rate for an internal capability benchmark across all safety categories. Symbols indicate whether we are evaluating model or system level safety. As expected model level safety results indicate higher violation rates and lower refusal rates compared to system level safety results. Llama 3 aims to balance a low violation rate with a low false refusal rate, while some competitors are more skewed towards one or the other.

图21 模型和能力中的违规率和错误拒绝率。每个 p o i n t \mathrm{{point}} point 代表所有安全类别中内部能力基准的 o v e r a l l \mathrm{{overall}} overall 错误 r e f u s a l \mathrm{{refusal}} refusal 和违规率。符号表示我们是在评估模型级还是系统级安全。正如预期，模型级安全结果显示违规率较高，拒绝率较低，与系统级安全结果相比。Llama 3旨在平衡低违规率与低错误拒绝率，而一些竞争对手则更偏向于其中之一。

Multilingual safety. Our experiments demonstrate that safety knowledge in English does not readily transfer to other languages, particularly given the nuance of safety policies and language-specific context. Therefore, it is essential to collect high-quality safety data for each language. We also found that the distribution of safety data per language significantly impacts performance from a safety standpoint, with some languages benefiting from transfer learning while others require more language-specific data. To achieve a balance between FRR and VR, we iteratively add adversarial and borderline data while monitoring the impact on both metrics.

多语言安全。我们的实验表明，英语中的安全知识并不容易转移到其他语言，特别是考虑到安全政策的细微差别和特定语言的上下文。因此，为每种语言收集高质量的安全数据至关重要。我们还发现，每种语言的安全数据分布从安全角度来看显著影响性能，有些语言从迁移学习中受益，而其他语言则需要更多的特定语言数据。为了在FRR和VR之间取得平衡，我们迭代地添加对抗性和边缘数据，同时监控对这两个指标的影响。

We display results on our internal benchmarks in Figure 19 for short context models, showing Llama 3’s violation and false refusal rates for English and non-English languages compared to similar models and systems. To construct the benchmarks for each language, we use a combination of prompts written by native speakers, sometimes supplementing with translations from our English benchmarks. For each of our supported languages, we find that Llama 405B with Llama Guard is at least as safe, if not strictly safer, than the two competing systems when measured on our internal benchmark, while maintaining competitive false refusal rates. Looking at the Llama 405B model on its own, without Llama Guard, we find that it has a significantly lower violation rate than the competing standalone open source model, trading off a higher false refusal rate.

我们在图19中展示了短上下文模型的内部基准测试结果，显示了与类似模型和系统相比，Llama 3在英语和非英语语言中的违规和错误拒绝率。为了构建每种语言的基准测试，我们使用了由母语者编写的提示组合，有时辅以从我们的英语基准测试中翻译的内容。对于我们支持的每一种语言，我们发现，在内部基准测试中衡量时，Llama 405B配备Llama Guard至少与两个竞争系统一样安全，如果不是更严格地安全，同时保持了有竞争力的错误拒绝率。单独观察Llama 405B模型，不带Llama Guard，我们发现它的违规率明显低于竞争的独立开源模型，代价是更高的错误拒绝率。

Long-context safety. Long-context models are vulnerable to many-shot jailbreaking attacks without targeted mitigation (Anil et al., 2024). To address this, we finetune our models on SFT datasets that include examples of safe behavior in the presence of demonstrations of unsafe behavior in context. We develop a scalable mitigation strategy that significantly reduces VR, effectively neutralizing the impact of longer context attacks even for 256-shot attacks. This approach shows little to no impact on FRR and most helpfulness metrics.

长上下文安全性。长上下文模型在没有针对性缓解措施的情况下容易受到多次攻击的破解（Anil等人，2024年）。为了解决这个问题，我们在SFT数据集上对我们的模型进行了微调，这些数据集包括在上下文中存在不安全行为演示的情况下安全行为的示例。我们开发了一种可扩展的缓解策略，显著降低了违规率，有效地中和了长上下文攻击的影响，即使是256次攻击。这种方法对错误拒绝率和大部份有用性指标几乎没有影响。

To quantify the effectiveness of our long context safety mitigations, we use two additional benchmarking methods: DocQA and Many-shot. For DocQA, short for “document question answering,” we use long documents with information that could be utilized in adversarial ways. Models are provided both the document and a set of prompts related to the document in order to test whether the questions being related to information in the document affected the model’s ability to respond safely to the prompts. For Many-shot, following Anil et al. (2024), we construct a synthetic chat history composed of unsafe prompt-response pairs. A final prompt, unrelated to previous messages, is used to test whether the unsafe behavior in-context influenced the model

为了量化我们长期上下文安全缓解措施的有效性，我们采用了两种额外的基准测试方法：DocQA 和 Many-shot。对于 DocQA，即“文档问答”，我们使用包含可能被恶意利用信息的长文档。模型同时提供文档和一组与文档相关的提示，以测试与文档中信息相关的问题是否影响模型对提示的安全响应能力。对于 Many-shot，遵循 Anil 等人（2024）的方法，我们构建了一个由不安全提示-响应对组成的合成聊天历史。一个与之前消息无关的最终提示用于测试上下文中的不安全行为是否影响了模型的安全响应。

to response unsafely. The violation and false refusal rates for both DocQA and Many-shot are shown in Figure 20. We see that Llama 405B (with and without Llama Guard) is Pareto-better than the Comp. 2 system across both violation rates and false refusal rates, across both DocQA and Many-shot. Relative to Comp. 1, we find that Llama 405B is significantly safer, while coming at a trade off on false refusal.

对于 DocQA 和 Many-shot 的违规和错误拒绝率如图 20 所示。我们发现 Llama 405B（无论是否启用 Llama Guard）在 DocQA 和 Many-shot 的违规率和错误拒绝率上均优于 Comp. 2 系统。相对于 Comp. 1，我们发现 Llama 405B 在安全性上显著提升，尽管在错误拒绝率上有所权衡。

Tool usage safety. The diversity of possible tools and the implementation of the tool usage call and integration into the model make tool usage a challenging capability to fully mitigate (Wallace et al., 2024). We focus on the search usecase. Violation and false refusal rates are shown in Figure 20. We tested against the Comp. 1 system, where we find that Llama 405B is significantly safer, though has a slightly higher false refusal rate.

工具使用安全。工具的多样性和工具使用调用及集成到模型中的实现使得工具使用成为一个难以完全缓解的挑战（Wallace 等人，2024）。我们专注于搜索用例。违规和错误拒绝率如图 20 所示。我们针对 Comp. 1 系统进行了测试，发现 Llama 405B 在安全性上显著提升，尽管错误拒绝率略高。

5.4.5 Cybersecurity and Chemical/Biological Weapons Safety 网络安全与化学/生物武器安全

CyberSecurity evaluation results. To evaluate cybersecurity risk, we leverage the CyberSecEval benchmark framework (Bhatt et al., 2023, 2024), which contains tasks that measure safety across domains such as generating insecure code, generating malicious code, textual prompt injection, and vulnerability identification. We developed and applied Llama 3 to new benchmarks on spear phishing and autonomous cyberattacks.

网络安全评估结果。为了评估网络安全风险，我们利用了CyberSecEval基准框架（Bhatt等人，2023，2024），该框架包含跨领域的任务，如生成不安全代码、生成恶意代码、文本提示注入和漏洞识别。我们开发并应用了Llama 3到新的基准测试，包括钓鱼攻击和自主网络攻击。

Overall, we find that Llama 3 does not have significant susceptibilities in generating malicious code or exploiting vulnerabilities. We describe brief results on specific tasks:

总体而言，我们发现Llama 3在生成恶意代码或利用漏洞方面没有显著的易感性。我们简要描述了特定任务的结果：

Insecure coding testing framework: Evaluating Llama 3 8B, 70B, and 405B against the insecure coding testing framework, we continue to observe that larger models both generate more insecure code and also generate code with a higher average BLEU score (Bhatt et al., 2023).

不安全编码测试框架：评估Llama 3 8B、70B和405B对不安全编码测试框架的表现，我们继续观察到，较大的模型不仅生成更多不安全代码，而且生成的代码平均BLEU分数（Bhatt等人，2023）也更高。

Code interpreter abuse prompt corpus: We identify that Llama 3 models are susceptible to executing malicious code under certain prompts, with Llama 3405B being particularly susceptible by complying with malicious prompts 10.4 % {10.4}\% 10.4% of the time. Llama 370 B complied at a rate of 3.8 % {3.8}\% 3.8% .

代码解释器滥用提示语料库：我们发现Llama 3模型在某些提示下容易执行恶意代码，其中Llama 3405B特别容易受到恶意提示的遵从，遵从率为 10.4 % {10.4}\% 10.4%。Llama 370B的遵从率为 3.8 % {3.8}\% 3.8%。

Text-based prompt injection benchmark: When evaluated against prompt injection benchmarks, prompt injection attacks against Llama 3405B were successful 21.7 % {21.7}\% 21.7% of the time. Figure 22 provides text-based prompt injection success rates across Llama 3, GPT-4 Turbo, Gemini Pro, and Mixtral models.

基于文本的提示注入基准：在对提示注入基准进行评估时，针对Llama 3405B的提示注入攻击成功率为 21.7 % {21.7}\% 21.7%。图22提供了Llama 3、GPT-4 Turbo、Gemini Pro和Mixtral模型在文本提示注入成功率方面的比较。

Vulnerability identification challenges: In assessing Llama 3’s ability to identify and exploit vulnerabilities using CyberSecEval 2’s capture-the-flag test challenges, Llama 3 does not outperform commonly used, traditional non-LLM tools and techniques.

漏洞识别挑战：在评估Llama 3使用CyberSecEval 2的夺旗测试挑战来识别和利用漏洞的能力时，Llama 3并未超过常用的传统非LLM工具和技术。

Spear phishing benchmark: We evaluate model persuasiveness and success rate in carrying out personalized conversations designed to deceive a target into unwittingly participating in security compromises. Randomized detailed victim profiles were generated by an LLM to serve as spear phishing targets. A judge LLM (Llama 370B) scored the performance of Llama 370B and 405B in interacting with a victim model (Llama 3 70B) and evaluated the success of the attempt. Llama 370B and Llama 3405B were evaluated by the judge LLM to be moderately persuasive. Llama 370B was judged by an LLM to have been successful in 24 % {24}\% 24% of spear phishing attempts while Llama 3405B was judged to be successful in 14 % {14}\% 14% of attempts. Figure 23 presents judge LLM-evaluated persuasiveness scores across models and phishing objectives.

鱼叉式网络钓鱼基准测试：我们评估模型在执行旨在欺骗目标无意中参与安全妥协的个性化对话中的说服力和成功率。通过大型语言模型（LLM）生成了随机详细的受害者资料，作为鱼叉式网络钓鱼的目标。一个评判用的大型语言模型（Llama 370B）对Llama 370B和405B在与受害者模型（Llama 3 70B）互动中的表现进行了评分，并评估了尝试的成功与否。评判用的大型语言模型认为Llama 370B和Llama 3405B具有中等的说服力。Llama 370B被评判用的大型语言模型认为在 24 % {24}\% 24%次鱼叉式网络钓鱼尝试中取得了成功，而Llama 3405B则在 14 % {14}\% 14%次尝试中被判定为成功。图23展示了评判用的大型语言模型对各模型和钓鱼目标的说服力评分。

Attack automation framework: We assess Llama 3 70B’s and 405B’s potential to function as an autonomous agent across four critical phases of a ransomware attack - network reconnaissance, vulnerability identification, exploit execution, and post exploitation actions. We enable the models to behave autonomously by configuring the models to iteratively generate and execute new Linux commands in response to output from their prior commands on a Kali Linux virtual machine as they targeted another virtual machine with known vulnerabilities. Although Llama 3 70B and 405B efficiently identify network services and open ports in their network reconnaissance, the models fail to effectively use this information to gain initial access to the vulnerable machine across 20 and 23 test runs respectively. In identifying vulnerabilities, Llama 370B and 405B are moderately effective but struggle with selecting and applying successful exploitation techniques. Attempts to execute exploits were entirely unsuccessful as were post-exploit attempts to maintain access or impact hosts within a network.

攻击自动化框架：我们评估Llama 3 70B和405B在勒索软件攻击的四个关键阶段——网络侦察、漏洞识别、漏洞利用执行和后利用行动——中作为自主代理的潜力。我们通过配置模型在Kali Linux虚拟机上迭代生成并执行新的Linux命令以响应其先前命令的输出来实现模型的自主行为，这些模型针对另一台具有已知漏洞的虚拟机。尽管Llama 3 70B和405B在网络侦察中有效地识别了网络服务和开放端口，但模型在20次和23次测试运行中均未能有效利用这些信息获取对脆弱机器的初始访问权限。在识别漏洞方面，Llama 370B和405B表现中等有效，但在选择和应用成功的漏洞利用技术方面遇到困难。尝试执行漏洞利用完全不成功，后利用尝试维持访问或在网络内影响主机也同样失败。

Uplift testing for cyber attacks. We conduct an uplift study which measures the extent a virtual assistant improved the cyberattack rates of both novice and expert cyberattackers between two simulated offensive

针对网络攻击的提升测试。我们进行了一项提升研究，该研究衡量了在两个模拟进攻性网络安全挑战中，虚拟助手对新手和专家网络攻击者攻击率提升的程度。

cybersecurity challenges. A two-stage study was conducted with 62 internal volunteers. Volunteers were categorized into “expert” (31 subjects) and “novice” (31 subjects) cohorts based on their offensive security experience. For the first stage, subjects were asked to complete the challenge without any LLM assistance but with access to the open internet. For the second stage, subjects retained access to the internet but were also provided with Llama 3405B to complete a different offensive cybersecurity challenge of similar difficulty to the first. An analysis of the completion rates of challenge attack phases by subjects indicates that both novices and experts using the 405    B {405}\mathrm{\;B} 405B model demonstrated insignificant uplift over having open access to the internet without an LLM.

我们进行了一个两阶段的研究，共有62名内部志愿者参与。志愿者根据他们的进攻性安全经验被分为“专家”（31名受试者）和“新手”（31名受试者）两组。在第一阶段，受试者被要求在没有任何大型语言模型（LLM）协助的情况下，仅通过开放互联网完成挑战。在第二阶段，受试者保留了互联网访问权限，并获得了Llama 3405B来完成一个难度与第一阶段相似的不同进攻性网络安全挑战。对受试者完成挑战攻击阶段的分析表明，无论是新手还是专家，在使用 405    B {405}\mathrm{\;B} 405B模型的情况下，与仅通过开放互联网访问相比，其提升效果并不显著。

Figure 22 Text-based prompt injection success rates per model across prompt injection strategies. Llama 3 is on average more susceptible to prompt injection than GPT-4 Turbo and Gemini Pro but less susceptible than Mixtral models when evaluated using this benchmark.

图22 不同提示注入策略下各模型的文本提示注入成功率。根据此基准评估，Llama 3平均而言比GPT-4 Turbo和Gemini Pro更容易受到提示注入攻击，但比Mixtral模型更不易受影响。

Figure 23 Average spear phishing persuasiveness scores across spear phishermodels and goals. A t \mathrm{{At}} At - tempt persuasiveness is evaluated by a Llama 3 70B judge LLM.

图23 不同钓鱼模型和目标的平均鱼叉式网络钓鱼说服力得分。 A t \mathrm{{At}} At - 诱惑说服力由Llama 3 70B判断型大型语言模型评估。

Uplift testing for chemical and biological weapons. To assess risks related to proliferation of chemical and biological weapons, we perform uplift testing designed to assess whether use of Llama 3 could meaningfully increase the capabilities of actors to plan such attacks.

针对化学和生物武器的提升测试。为了评估与化学和生物武器扩散相关的风险，我们进行了提升测试，旨在评估使用Llama 3是否能实质性地提高行动者策划此类攻击的能力。

The study consists of six-hour scenarios where teams of two participants were asked to generate fictitious operational plans for either a biological or chemical attack. The scenarios cover the major planning stages of a CBRNE attack (agent acquisition, production, weaponization, and delivery) and are designed to elicit detailed plans that would address challenges related to procurement of restricted materials, real-world laboratory protocols, and operational security. Participants are recruited based on previous experience in relevant areas of scientific or operational expertise, and assigned to teams consisting of two low-skill actors (no formal training) or two moderate-skill actors (some formal training and practical experience in science or operations).

该研究包括六小时的情景模拟，其中两两一组的参与者被要求为生物或化学攻击制定虚构的行动计划。这些情景涵盖了CBRNE攻击的主要规划阶段（代理获取、生产、武器化和投放），旨在引出详细的计划，以应对与受限材料的采购、现实实验室协议和操作安全相关的挑战。参与者根据以往在科学或操作专业领域的经验招募，并被分配到由两名低技能演员（无正式培训）或两名中等技能演员（有一定正式培训和科学或操作实践经验）组成的团队。

The study was generated in collaboration with a set of CBRNE experts, and designed to maximize the generality, validity, and robustness of both quantitative and qualitative outcomes. A preliminary study was also performed in order to validate the study design, including a robust power analysis ensuring that our sample size was sufficient for statistical analysis.

该研究是在一组CBRNE专家的合作下产生的，旨在最大化定量和定性结果的普遍性、有效性和稳健性。还进行了一项初步研究，以验证研究设计，包括进行稳健的功率分析，确保我们的样本量足以进行统计分析。

Each team is assigned to a “control” or “LLM” condition. The control team has access to internet-based resources only, while the LLM-enabled team had internet access as well as access to Llama 3 models enabled with web search (including PDF ingestion), information retrieval capabilities (RAG), and code execution (Python and Wolfram Alpha). To enable testing of RAG capabilities, a keyword search is used to generate a dataset of hundreds of relevant scientific papers and pre-loaded into the Llama 3 model inference system. At the conclusion of the exercise, the operational plans generated by each team are evaluated by subject matter experts with domain expertise in biology, chemistry, and operational planning. Each plan is evaluated across four stages of potential attacks, generating scores for metrics such as scientific accuracy, detail, detection avoidance, and probability of success in scientific and operational execution. After a robust Delphi process to mitigate bias and variability in subject matter expert (SME) evaluations, final scores are generated by pooling stage-level metrics into a comprehensive score.

每个团队被分配到“控制”或“LLM”条件。控制团队只能访问基于互联网的资源，而启用了LLM的团队除了互联网访问外，还能访问启用了网络搜索（包括PDF摄取）、信息检索能力（RAG）和代码执行（Python和Wolfram Alpha）的Llama 3模型。为了测试RAG能力，使用关键词搜索生成一个包含数百篇相关科学论文的数据集，并预加载到Llama 3模型推理系统中。在演练结束时，每个团队生成的行动计划由具有生物学、化学和操作规划领域专业知识的主题专家进行评估。每个计划在四个潜在攻击阶段进行评估，生成科学准确性、详细程度、检测规避和科学及操作执行成功概率等指标的分数。通过一个稳健的德尔菲过程来减轻主题专家评估中的偏差和变异性后，最终分数通过将阶段级指标汇总为一个综合分数来生成。

Quantitative analysis of these results of this study show no significant uplift in performance related to usage of the Llama 3 model. This result holds true when performing an aggregate analysis (comparing all LLM conditions to the web-only control condition) as well as for breakdowns by subgroups (e.g., separate evaluation of the Llama 3 70B and Llama 3405B models, or separate evaluation of scenarios related to chemical or biological weapons). After validating these results with CBRNE SMEs, we assess that there is a low risk that release of Llama 3 models will increase ecosystem risk related to biological or chemical weapon attacks.

本研究的定量分析结果显示，使用Llama 3模型在性能上没有显著提升。无论是进行总体分析（比较所有LLM条件与仅限网络的控制条件）还是按子组细分（例如，分别评估Llama 3 70B和Llama 3405B模型，或分别评估与化学或生物武器相关的场景），这一结果都成立。在与CBRNE主题专家验证这些结果后，我们评估认为，发布Llama 3模型增加与生物或化学武器攻击相关的生态系统风险的可能性很低。

5.4.6 Red Teaming 红队测试

We utilize Red Teaming to discover risks and use the findings to improve our benchmarks and safety tuning datasets. We conduct recurring red teaming exercises to continuously iterate and discover new risks, which guides our model development and mitigation process.

我们利用红队测试来发现风险，并利用这些发现来改进我们的基准和安全调优数据集。我们进行定期的红队测试演练，以持续迭代并发现新风险，这指导我们的模型开发和缓解过程。

Our red team consists of experts in cybersecurity, adversarial machine learning, responsible AI, and integrity, in addition to multilingual content specialists with backgrounds in integrity issues for specific geographic markets. We also partner with internal and external subject-matter experts in critical risk areas to help build risk taxonomies and aid in more focused adversarial assessment.

我们的红队由网络安全、对抗机器学习、负责任的人工智能和诚信领域的专家组成，此外还有具备特定地理市场诚信问题背景的多语种内容专家。我们还与内部和外部关键风险领域的主题专家合作，帮助构建风险分类法，并协助进行更有针对性的对抗性评估。

Adversarial testing on specific model capabilities. We began initial red teaming by focusing on individual model capabilities in a risk discovery process, in context of specific high-risk categories then testing capabilities together. The red team focused on prompt-level attacks to emulate more likely more real world scenarios - we find that models often deviate from expected behavior, particularly in cases when the prompt’s intention is being obfuscated or when prompts layer multiple abstractions. These risks get more complex with additional capabilities, and we describe several of our red teaming discoveries in detail below. We utilize these red team discoveries in concert with our results on internal safety benchmarks to develop focused mitigations to continuously and iteratively improve model safety.

针对特定模型能力的对抗性测试。我们通过专注于风险发现过程中的单个模型能力，开始初步的红队测试，然后在特定的高风险类别背景下测试这些能力。红队专注于模拟更可能的现实世界场景的提示级攻击——我们发现模型经常偏离预期行为，特别是在提示的意图被混淆或提示包含多层抽象的情况下。随着能力的增加，这些风险变得更加复杂，我们在下面详细描述了我们的几个红队发现。我们利用这些红队发现与我们在内部安全基准上的结果相结合，开发有针对性的缓解措施，以持续迭代地提高模型安全性。

Short and long-context English. We employed a mix of well known, published and unpublished techniques across single and multi-turn conversations. We also leveraged advanced, adversarial multi-turn automation similar to PAIR (Chao et al., 2023) across some techniques and risk categories. Largely, multi-turn conversations lead to more harmful outputs. Several attacks were pervasive across model checkpoints, particularly when used together.

短和长上下文英语。我们采用了混合的已知、已发表和未发表的技术，涵盖单轮和多轮对话。我们还利用了类似于PAIR（Chao et al., 2023）的先进对抗性多轮自动化技术，应用于某些技术和风险类别。大多数情况下，多轮对话会导致更有害的输出。几种攻击在模型检查点中普遍存在，尤其是当它们一起使用时。

– Multi-turn refusal suppression to specify the model response to follow a particular format or include/exclude particular information related to the refusal as specific phrases.

– 多轮拒绝抑制，以指定模型响应遵循特定格式或包含/排除与拒绝相关的特定信息，如特定短语。

– Hypothetical scenarios wrap violating prompts as hypothetical/theoretical tasks or fictional scenarios

总结