A Comprehensive Overview of Large Language Models

In the very first stage, the model is trained in a self-supervised manner on a large corpus to predict the next tokens given the input. The design choices of LLMs vary from encoder-decoder to decoder-only architectures with different building blocks and loss functions in sections 2.5, 2.4,  2.10.

There are different styles to fine-tune an LLM. This section briefly discusses fine-tuning approaches.
  Transfer Learning: The pre-trained LLMs perform well for various tasks GPT-3 ; PaLM . However, to improve the performance for a downstream task, pre-trained models are fine-tuned with the task-specific data T5 ; mT5 , known as transfer learning.
  Instruction-tuning: To enable a model to respond to user queries effectively, the pre-trained model is fine-tuned on instruction formatted data i.e., instruction and an input-output pair. Instructions generally comprise multi-task data in plain natural language, guiding the model to respond according to the prompt and the input. This type of fine-tuning improves zero-shot generalization and downstream task performance. Details on formatting instruction data and its various styles are available in Flan ; Survey_LLM ; OPT_IML .
  Alignment-tuning: LLMs are prone to generating false, biased, and harmful text. To make them helpful, honest, and harmless, models are aligned using human feedback. Alignment involves asking LLMs to generate unexpected responses and then updating their parameters to avoid such responses instructgpt ; llama_2 ; self_align .
It ensures LLMs operate according to human intentions and values. A model is defined to be an “aligned” model if the model fulfills three criteria of helpful, honest, and harmless or “HHH” askell2021general .
Researchers employ reinforcement learning with human feedback (RLHF) ziegler2019fine for model alignment. In RLHF, a fine-tuned model on demonstrations is further trained with reward modeling (RM) and reinforcement learning (RL), shown in Figure 6. Below we briefly discuss RM and RL pipelines in RLHF.

Prompting is a method to query trained LLMs for generating responses, as illustrated in Figure 6. LLMs can be prompted in various prompt setups, where they can be adapted to the instructions without fine-tuning and in other cases with fine-tuning on data containing different prompt styles Flan ; cot_collection ; zero_to_hero . A good guide on prompt engineering is available at prompt_eng_guide . Below, we will discuss various widely used prompt setups.
 Zero-Shot Prompting: LLMs are zero-shot learners and capable of answering queries never seen before. This style of prompting requires LLMs to answer user questions without seeing any examples in the prompt.
 In-context Learning: Also known as few-shot learning, here, multiple input-output demonstration pairs are shown to the model to generate the desired response. This adaptation style is also called few-shot learning. A discussion on formatting in-context learning (ICL) templates is available in survey_incontext_learning ; Survey_LLM ; Tk-INSTRUCT ; Flan .
  Reasoning in LLMs: LLMs are zero-shot reasoners and can be provoked to generate answers to logical problems, task planning, critical thinking, etc. with reasoning. Generating reasons is possible only by using different prompting styles, whereas to improve LLMs further on reasoning tasks many methods Flan ; OPT_IML train them on reasoning datasets. We discuss various prompting techniques for reasoning below.
Chain-of-Thought (CoT): A special case of prompting where demonstrations contain reasoning information aggregated with inputs and outputs so that the model generates outcomes with step-by-step reasoning. More details on CoT prompts are available in survey_reasoning ; wei2022chain ; cot_collection .
Self-Consistency: Improves CoT performance by generating multiple responses and selecting the most frequent answer self_consistency .
Tree-of-Thought (ToT): Explores multiple reasoning paths with possibilities to look ahead and backtrack for problem-solving ToT .
Single-Turn Instructions: In this prompting setup, LLMs are queried only once with all the relevant information in the prompt. LLMs generate responses by understanding the context either in a zero-shot or few-shot setting.
Multi-Turn Instructions: Solving a complex task requires multiple interactions with LLMs, where feedback and responses from the other tools are given as input to the LLM for the next rounds. This style of using LLMs in the loop is common in autonomous agents.

Numerous hand-crafted instruction-tuning datasets with different design choices are proposed in the literature to instruction-tune LLMs. The performance of fine-tuned LLMs depends on multiple factors, such as dataset, instruction diversity, prompting templates, model size, and training objectives. Kee** this in view, diverse fine-tuned models have emerged in the literature using manually created datasets.
The models T0 T0 and mT0 (multi-lingual) mT0andBLOOMZ employ templates to convert existing datasets into prompt datasets. They have shown improvements in generalization to zero-shot and held-out tasks. Tk-Instruct Tk-INSTRUCT fine-tuned the T5 model with in-context instructions to study generalization on unseen tasks when given in-context instructions during test time. The model outperformed Instruct-GPT, despite being smaller in size, i.e., 11B parameters as compared to 175B of GPT-3.
Increasing Tasks and Prompt Setups: Zero-shot and few-shot performance improves significantly by expanding task collection and prompt styles. OPT-IML OPT_IML and Flan Flan curated larger 2k and 1.8k task datasets, respectively. While increasing task size alone is not enough, OPT-IML and Flan add more prompting setups in their datasets, zero-shot, few-shot, and CoT. In continuation, CoT Collection cot_collection fine-tunes Flan-T5 further on 1.88M CoT samples. Another method zero_to_hero uses symbolic tasks with tasks in T0, Flan, etc.

Generating an instruction-tuning dataset requires carefully writing instructions and input-output pairs, which are often written by humans, smaller in size, and less diverse. To overcome this, self-instruct ft_self_instruct proposed an approach to prompt available LLMs to generate instruction-tuning datasets. Self-instruct outperformed models trained on manually created dataset SUPER-NATURALINSTRUCTIONS (a dataset with 1600+ tasks) Tk-INSTRUCT by 33%. It starts with a seed of 175 tasks, 1 instruction, and 1 sample per task and iteratively generates new instructions (52k) and instances (82k input-output pairs) using GPT-3 GPT-3 . Contrary to this, Dynosaur dynosaur uses the meta-data of datasets on Huggingface to prompt LLMs to generate multiple task instruction-tuning datasets.
LLaMA Tuned: Various models in the literature instruction-tune LLaMA gao2023llama with GPT-3 GPT-3 or GPT-4 GPT-4 generated datasets. Among these, Alpaca alpaca , Vicuna vicuna , and LLaMA-GPT-4 llama_gpt_4 are a few general-purpose fine-tuned models, where Alpaca is trained on 52k samples from text-davinci-003, Vicuna on 70k samples from ShareGPT.com, and LLaMA-GPT-4 by re-creating Alpaca instructions from GPT-4. Goat goat fine-tunes LLaMA for arithmetic tasks (1 million samples) by generating data from ChatGPT and outperforms GPT-4, PaLM, BLOOM, OPT, etc., attributing its success to the LLaMA’s consistent tokenization of numbers. HuaTuo huatuo is a medical knowledge model, fine-tuned with a generated QA dataset of 8k instructions.
Complex Instructions: Evol-Instruct wizardlm ; wizardcoder prompts LLMs to convert given instructions into a more complex set. The instructions are iteratively evolved with re-writing instructions in complex wording and creating new instructions. With this style of automated instruction generation, WizardLM wizardlm (fine-tuned LLaMA on 250k instructions), outperforms Vicuna and Alpaca, and WizardCoder wizardcoder (fine-tuned StarCoder) beats Claude-Plus, Bard, and others.

Incorporating human preferences into LLMs presents a significant advantage in mitigating undesirable behaviors and ensuring accurate outputs. The initial work on alignment, such as InstructGPT instructgpt aligns GPT-3 using a 3-step approach, instruction-tuning, reward modeling, and fine-tuning with reinforcement learning (RL). The supervised fine-tuned GPT-3 on demonstrations is queried to generate responses, which human labelers rank according to human values, and a reward model is trained on the ranked data. Lastly, the GPT-3 is trained with proximal policy optimization (PPO) using rewards on the generated data from the reward model. LLaMA 2-Chat llama_2 improves alignment by dividing reward modeling into helpfulness and safety rewards and using rejection sampling in addition to PPO. The initial four versions of LLaMA 2-Chat are fine-tuned with rejection sampling and then with PPO on top of rejection sampling.
  Aligning with Supported Evidence: This style of alignment allows the model to generate responses with proofs and facts, reduces hallucination, and assists humans more effectively, which increases trust in the model’s output. Similar to the RLHF training style, a reward model is trained to rank generated responses containing web citations in answers to questions, which is later used to train the model, as in GopherCite gopher_cite , WebGPT nakano2021webgpt , and Sparrow sparrow . The ranking model in Sparrow sparrow is divided into two branches, preference reward and rule reward, where human annotators adversarial probe the model to break a rule. These two rewards together rank a response to train with RL.
  Aligning Directly with SFT: The PPO in the RLHF pipeline is complex, memory-intensive, and unstable, requiring multiple models, reward, value, policy, and reference models. Avoiding this sophisticated alignment pipeline is possible by incorporating minimal changes in the supervised fine-tuning (SFT) pipeline as in DPO ; raft ; rrhf , with better or comparable performance to PPO. Direct preference optimization (DPO) DPO trains a model directly on the human-preferred responses to maximize the likelihood of preferred against unpreferred responses, with per-sample importance weight. Reward ranked fine-tuning RAFT raft fine-tunes the model on ranked responses by the reward model. Preference ranking optimization (PRO) PRO and RRHF rrhf penalize the model to rank responses with human preferences and supervised loss. On the other hand, chain-of-hindsight (CoH) CoH provides feedback to the model in language rather than reward, to learn good versus bad responses.
 Aligning with Synthetic Feedback: Aligning LLMs with human feedback is slow and costly. The literature suggests a semi-automated process to align LLMs by prompting LLMs to generate helpful, honest, and ethical responses to the queries, and fine-tuning using the newly created dataset. Constitutional AI constitutional_ai replaces human feedback in RLHF with AI, calling it RL from AI feedback (RLAIF). AlpacaFarm alpacafarm designs prompts to imitate human feedback using LLMs APIs. Opposite to constitutional AI, AlpacaFarm injects noise in feedback to replicate human mistakes. Self-Align self_align prompts the LLM with ICL examples, instructing the LLM about what the response should contain to be considered useful and ethical. The same LLM is later fine-tuned with the new dataset.
  Aligning with Prompts: LLMs can be steered with prompts to generate desirable responses without training align_with_prompts ; aling_with_prompts_2 . The self-correction prompting in aling_with_prompts_2 concatenates instructions and CoT with questions, guiding the model to answer its instruction following a strategy to ensure moral safety before the actual answer. This strategy is shown to reduce the harm in generated responses significantly.
  Red-Teaming/Jailbreaking/Adversarial Attacks: LLMs exhibit harmful behaviors, hallucinations, leaking personal information, and other shortcomings through adversarial probing. The models are susceptible to generating harmful responses even though they are aligned for safety jailbroken ; red_team_lessons_learned . Red-teaming is a common approach to address illicit outputs, where the LLMs are prompted to generate harmful outputs red_team_lessons_learned ; red_team_explore . The dataset collected through red-teaming is used to fine-tune models for safety. While red-teaming largely relies on human annotators, another work red_team_language_models red-team LLMs to find prompts that lead to harmful outputs for other LLMs.

Although fine-tuning boosts a model’s performance, it leads to catastrophic forgetting of previously learned information. Concatenating fine-tuning data with a few randomly selected pre-training samples in every iteration avoids network forgetting cont_learn ; xuanyuan . This is also effective in adapting LLMs for cases where fine-tuning data is small and the original capacity is to be maintained. Prompt-based continued pre-training (PCP) cont_learn_dont_stop_pretrain trains the model with text and instructions related to tasks and then finally instruction-tunes the model for downstream tasks.

While fine-tuning data is generally many-fold smaller than the pre-training data, it still has to be large enough for acceptable performance Flan ; OPT_IML ; Tk-INSTRUCT and requires proportional computing resources. Studying the effects on performance with less data, existing literature sample_eff_inst_quick_learner ; sample_eff_maybe_less_data finds that models trained on less data can outperform models trained with more data. In sample_eff_inst_quick_learner , 25% of the total downstream data is found enough for state-of-the-art performance. Selecting coreset-based 0.5% of the total instruction-tuning data improves the model performance by 2% in sample_eff_maybe_less_data , as compared to the complete data tuning. Less is more for alignment (LIMA) lima uses only 1000 carefully created demonstrations to fine-tune the model and has achieved comparable performance to GPT-4.

LLMs may have limited memory and outdated information, leading to inaccurate responses. Retrieving relevant information from external up-to-date storage enables the LLMs to accurately answer with references and utilize more information. With retrieval augmentation, smaller models have been shown to perform at par with larger models. For instance, the 11B model can become competitive to 540B PaLM in atlas and 7.5B to 280B Gopher in retro . Retrieval augmented language modeling (RALM) has two major components, shown in Figure 12, namely: 1) retriever and 2) language model. In RALM, the retriever plays a crucial role in driving LLM response, where incorrect information can steer LLMs to false behavior. This leads to the development of various methods to retrieve accurate information and fuse with the query for better performance.
Zero-Shot Retrieval Augmentation: This kind of augmentation keeps the original LLM architecture and weights unchanged and uses BM25 BM25 , nearest neighbors, or frozen pre-trained models like Bert Bert as a retriever. The retrieved information is provided as input to the model for response generation, shown to improve performance over LLMs without retrieval in_context_ralm ; rationale_aug . In some scenarios, multiple retrieval iterations are required to complete the task. The output generated in the first iteration is forwarded to the retriever to fetch similar documents. Forward-looking active retrieval (FLARE) flare initially generates the response and corrects the output by retrieving relevant documents if the response contains low-confidence tokens. Similarly, RepoCoder repocoder fetches code snippets recursively for code completion.
Training with Retrieval Augmentation: To reduce failures in retrieval augmentation generation (RAG), researchers train or fine-tune retrievers and LLMs with a retrieval augmentation pipeline. We discuss the literature below based on their focus on the respective training processes of the pipeline.
Training LLM: Retrieval-enhanced transformer (RETRO) retro shows pre-training smaller LLMs with RAG pipeline outperforms larger LLMs, such as GPT-3 trained without RAG. RETRO uses a 2-trillion token subset of MassiveText as a database. The retrieval pipeline divides the input query into subsets and retrieves relevant chunks from the database for each subset, encoded together with input intermediate representations for generating tokens. It uses cross-chunked attention to attend to previous chunks auto-regressively. A study on RETRO retro_study shows models pre-trained without RAG but fine-tuned using RAG lack the performance gains obtained by pre-training with RAG.
Training Retriever: Quality of responses generated by LLMs is highly dependent on the in-context examples. Therefore, retrieve_icl_train ; retrieve_icl_GPT_3 ; retrieve_prompts_learning ; replug train retrievers to retrieve accurate few-shot samples while kee** the LLM frozen for generation. Retrieved samples are ranked to build ground-truth data to train retrievers with contrastive learning in retrieve_icl_train ; retrieve_prompts_learning . RoBERTa is trained for downstream tasks in retrieve_icl_GPT_3 for ICL samples retrieval. REPLUG replug trains the retriever with supervised signals from the frozen LLM-generated outputs.
Training Retriever and LLM: Further benefits are achieved by training both the retriever and the model in atlas ; rag_train_long ; realm . In this case, the error propagates back to the retriever, updating both the language model and the retriever. While masked language modeling (MLM) is a common pre-training objective atlas ; realm , retrieval pre-trained transformer (RPT) rag_train_long used document chunk prediction as a pre-training objective for long text modeling.
Encoded Context Augmentation: Concatenating retrieved documents with the query becomes infeasible as the sequence length and sample size grow. Encoding the context and fusing it with the decoder (Fusion-in-Decoder) using cross-attention makes it possible to augment more samples without increasing computation costs significantly fid ; retro ; rag_train_long ; atlas .
Web Augmented: Locally stored memory, but external to LLM, has limited information. However, a large amount of information is available on the internet, which gets updated regularly. Rather than storing information locally, various methods retrieve query-related context through a web search and forward it to LLMs internet_aug ; internet_aug_2 ; nakano2021webgpt .

While RAG relies on the retriever to provide context to the LLM to answer queries, tool augmented LLMs capitalize on the reasoning abilities of LLMs to iteratively plan by dividing tasks into sub-tasks, selecting necessary tools, and taking actions to complete the task assistgpt ; chameleon ; ART ; talm . A generic pipeline of tool-augmented LLMs is shown in Figure 13, where different modules in Figure 13 are selected in a loop until the task completion.

Zero-Shot Tool Augmentation: LLMs in-context learning and reasoning abilities enable them to interact with tools without training. Automatic reasoning and tool-use (ART) ART builds a task library with demonstrations of reasoning steps and calling external tools. It retrieves similar task examples and provides the context to the LLM for inference. Aside from this, tool_aug_doc shows tool documentation is enough to teach LLMs to use tools without demonstrations. RestGPT RestGPT integrates LLMs with RESTful APIs by decomposing tasks into planning and API selection steps. The API selector understands the API documentation to select a suitable API for the task and plan the execution. ToolkenGPT ToolkenGPT uses tools as tokens by concatenating tool embeddings with other token embeddings. During inference, the LLM generates the tool tokens representing the tool call, stops text generation, and restarts using the tool execution output.
Training with Tool Augmentation: LLMs are trained to interact with diverse tools, enhancing planning abilities to overcome the limitations of zero-shot tool augmentation gorilla ; talm ; tool_manipulation_cap ; toolllm . Gorilla gorilla instruction-tunes LLaMA with information retrieval from API documentation. It uses the self-instruct ft_self_instruct data generation pipeline with GPT-4 by providing in-context examples retrieved from API documentation. Tool augmented language model (TALM) talm fine-tunes T5 T5 for tool use with a self-play approach, where it iteratively completes tool manipulation tasks and includes them back in the training set. ToolLLM toolllm collects 16k APIs from RapidAPI. It samples APIs from the list to generate an instruction-tuning dataset using ChatGPT in single-tool and multi-tool scenarios. For high-quality datasets, ToolLLM suggested a depth-first search-based decision tree (DFSDT) method to generate ground-truths with diverse reasoning and planning.
Multimodal Tool Augmentation: The compositional reasoning capacity of LLMs allows them to manipulate tools in multimodal settings assistgpt ; chameleon ; hugginggpt . Following the pipeline shown in Figure 13, the LLM outlines a plan, generally executing in a sequence: Plan $\,\to\,$ Tool selection $\,\to\,$ Execute $\,\to\,$ Inspect $\,\to\,$ Generate, to respond to the user query. Here, the database of tools is rich in modalities, including text, images, etc. Many of the multimodal tool augmentation systems employ multimodal LLMs gpt4tools ; taskmatrix ; hugginggpt ; chameleon , while others utilize single modality LLMs and generate a plan on using different modality tools to solve multimodal queries vipergpt .

Fine-tuning LLMs with tens or hundreds of billions of parameters, such as GPT-3 (175B), BLOOM (176B), MT-NLG (540B), etc., is computationally intensive and time-consuming. To avoid complete model fine-tuning, numerous parameter-efficient fine-tuning (PEFT) techniques Prompt_Tuning ; Prompt_Tuning_2 ; Prefix_Tuning ; unified_peft ; LMAdapter_3 try to achieve acceptable model fine-tuning performance at reduced costs. As compared to full fine-tuning revisiting_peft , PEFT performs better in low-resource setups, achieves comparable performance on medium-resource scenarios, and performs worse than full fine-tuning under high-resource availability. An overview of different PEFT approaches is shown in Figure 14.

Adapter Tuning: Adds a few trainable parameters within the transformer block. The adapter layer is a sequence of feature downscaling, non-linearity, and upscaling LMAdapter . Variants of adapter tuning inject adapter layers sequentially LMAdapter and in parallel unified_peft , whereas the mixture of adapter (AdaMix) adamix employs multiple adapter modules in a single layer. AdaMix routes input instances randomly to one of the multiple downscale and upscale modules. The mixture of adapters is averaged out for inference to avoid additional latency. Low-Rank Adaptation (LoRA) hu2021lora learns low-rank decomposed matrices to freeze original weights. The learned weights are fused with the original weights for inference, avoiding latency.
Prompt Tuning: Prompting is an effective way to adapt a pre-trained LLM for the downstream task. However, manual prompts bring uncertainty in the model’s prediction, where a change in a single word drops the performance Prompt_Tuning_2 . Prompt tuning alleviates this problem by fine-tuning only 0.001%-3% additional parameters p_tuning_v2 . It concatenates trainable prompt parameters with the model embeddings Prompt_Tuning_2 ; Prompt_Tuning ; p_tuning_v2 . Task-specific fixed discrete prompts are concatenated with input embeddings in Prompt_Tuning . As discrete prompts bring instability, prompts are encoded through a learnable map** in P-Tuning Prompt_Tuning_2 , naming continuous prompts, which are appended with the discrete prompts. Only the prompt encoder is trainable in the model. In an extension of P-Tuning, continuous prompts are concatenated with each layer of the network in p_tuning_v2 . Progressive prompts progressive_prompts avoid catastrophic forgetting and transfer previously learned knowledge by sequentially adding trainable prompt embeddings to the previously frozen task embeddings.

LLMs require extensive computing and memory for inference. Deploying a 175B parameter GPT-3 model needs at least 5x80GB A100 GPUs and 350GB of memory to store in FP16 format SmoothQuant . Such demanding requirements for deploying LLMs make it harder for smaller organizations to utilize them. Model compression is an effective solution but comes at the cost of degraded performance, especially at large scales greater than 6B. These models exhibit very large magnitude outliers that do not exist in smaller models 8_bit_LLM , making it challenging and requiring specialized methods for quantizing LLMs SmoothQuant ; OPTQ .
Post-Training Quantization: Minimal or no training is required in this type of quantization, without significantly compromising the model performance. LLM-8-bit 8_bit_LLM uses full-precision matrix multiplication for weights associated with outlier features and 8-bit for remaining features. The lower precision multiplication outputs are converted to FP-16 and concatenated with others. The quantized models have homogenous word embeddings, which may degrade their performance. To fix this, token-level knowledge distillation is employed in compression_PLM_quant along with independent quantization scaling factors for each module due to varying weight distribution. Feature distributions are asymmetric and appear in different channels; outlier suppression outlier_suppression shifts and scales per-channel activation distributions for effective quantization. SmoothQuant SmoothQuant quantizes activations and weights to INT8 format by smoothing activations and migrating the quantization difficulty toward weights. It multiplies the inverse of the smoothing factor with weights, which introduces a few outliers in the weights but is easier to quantify than unsmoothed activations. OPTQ OPTQ uses the optimal brain compression (OBC) OBC algorithm to quantize the model layer-by-layer and update weights to compensate for quantization error. To improve speed and performance, OPTQ updates weights in arbitrary order, employs lazy updates, and uses better Cholesky kernels. Outlier-aware weight quantization (OWQ) OWQ uses the OPTQ algorithm for quantization but assigns higher precision to vulnerable weights, causing outliers and lower precision for others.
Quantization-Aware Training: To compensate for performance degradation, a quantized model is fine-tuned in quantization-aware training (QAT) alphatuning ; qlora ; llm_qat . Alpha Tuning quantizes the model using binary coding quantization (BCQ) BCQ and fine-tunes only quantization scaling factors. This approach improves performance over parameter-efficient fine-tuning of the pre-trained model. Similarly, parameter-efficient and quantization-aware adaptation (PEQA) PEQA reduces the precision of fully-connected layers and fine-tunes only quantization scaling parameters. LLM-QAT llm_qat generates training data from the pre-trained network and trains a quantized student model with knowledge distillation. QLoRA qlora fine-tunes 4-bit quantized pre-trained LLM with LoRA hu2021lora using a 4-bit normal float, which shows better performance over a 4-bit integer and float.

Pruning is an alternative approach to quantization to compress model size, thereby reducing LLMs deployment costs significantly. Compared to task-agnostic pruning, task-specific pruning is easily achievable with good performance, where a model is fine-tuned on the downstream task and pruned for faster inference. It is possible to prune LLMs for individual tasks, but the cost of pruning and deploying task-specific models is high. To overcome this, many structured and unstructured pruning methods for LLMs have been proposed to maintain reasonable performance across all tasks while shrinking the model size wanda ; llm_pruner ; structured_p_llms .
Unstructured Pruning: This kind of pruning removes less important weights without maintaining any structure. Existing LLM pruning methods take advantage of the unique characteristics of LLMs, uncommon for smaller models, where a small subset of hidden states are activated with large magnitude 8_bit_LLM . Pruning by weights and activations (Wanda) wanda prunes weights in every row based on importance, calculated by multiplying the weights with the norm of input. The pruned model does not require fine-tuning, thereby saving computational costs. Outlier weighed layerwise sparsity (OWL) OWL extends Wanda with non-uniform layer pruning. It shows that the number of outliers varies for different layers; therefore, the model should have variable pruning ratios for better performance for every layer. Contrastive pruning (CAP) contrastive_pruning iteratively prunes the model by training the sparse model using contrastive loss between pre-trained, fine-tuned, and snapshots of previous sparse models to learn task-specific and task-agnostic knowledge.
Structured Pruning: Here, the parameters are removed in groups, rows, columns, or matrices, which speeds up the inference because of effective hardware tensor core utilization wanda . LLM-Pruner llm_pruner employs a 3-stage structured pruning strategy, identifying the groups of hidden states causing each other to activate during the forward-pass, kee** important groups and removing less important ones, and fine-tuning the pruned model with LoRA. Sparsity-induced mask learning (SIMPLE) SIMPLE prunes the network using learnable masks. Similarly, another method prunes LLMs by learning masks and removing unimportant rank-1 components of the factorized weight matrix structured_p_llms .

Due to the gigantic scale of LLMs, minor changes in architecture and training strategies have a big impact on performance and stability. Here, we summarize key architectural modules used in various LLMs, leading to better performance, reduced training time and memory, and better training stability.
Layer Normalization: The performance and training stability of LLMs are affected significantly by layer normalization. Pre-norm, that is normalizing inputs rather than outputs, is more common among LLMs stabilizing the training GPT-3 ; touvron2023llama ; PanGU_alpha . BLOOM BLOOM and AlexaTM soltan2022alexatm utilize an additional layer normalization before embedding layer to stabilize the training of large-scale models, while the model’s zero-shot generalization ability can be negatively impacted BLOOM . However, another study GLM-130B finds that pre-norm degrades fine-tuned model performance as compared to post-norm, and there are no stability benefits of pre-norm beyond the 100B scale. Therefore, GLM-130B GLM-130B used deep-norm which is a variant of post-norm for better downstream task performance after fine-tuning.
Positional Encoding: Like other building blocks of the model, positional encoding also affects the performance and training stability of LLMs. BLOOM BLOOM finds ALiBi outperforms learned and rotary positional encodings. Contrary to this, GLM-130B GLM-130B identifies rotary positional encoding as being better than ALiBi. So, there is no conclusion in the literature about positional encodings yet.
Parallel Attention: In this type of attention, feed-forward and attention layers are parallel to each other rather than sequential in a transformer block. It has been shown to reduce training time by 15%. There is no evidence of performance drop due to this change in the literature and it is used by the models PaLM PaLM , GPT-NeoX GPT_NeoX , and CodeGen CodeGen .
Multi-Query Attention It has shared key and value attention heads in a transformer block while query attention heads are projected as usual. This reduces memory usage and speeds up sampling in autoregressive decoding. No performance degradation has been observed with this change and it makes the training efficient allowing larger batch sizes. Multi-query attention is used in PaLM ; li2022competition .
Mixture of Experts: This type of architecture enables easily scaling models to trillions of parameters PanGu_sigma ; du2022glam . Only a few experts are activated during the computation making them compute-efficient. The performance of MoE models is better than dense models for the same amount of data and requires less computation during fine-tuning to achieve performance similar to dense models as discussed in du2022glam . MoE architectures are less prone to catastrophic forgetting, therefore are more suited for continual learning PanGu_sigma . Extracting smaller sub-models for downstream tasks is possible without losing any performance, making MoE architecture hardware-friendly PanGu_sigma .
Sparse vs Dense Activated: GPT-3 GPT-3 uses sparse transformers sparse_transformer whereas GLaM du2022glam and PanGu-$\sum$ PanGu_sigma use MoE shazeer2017outrageously architectures to lower computational costs and increase the model size and capacity. According to the literature, sparse modules do not degrade the model’s performance sparse_transformer . However, more experiments are required to verify this statement.

Training models at a huge scale require tricks to reduce training costs, avoid loss divergence, and achieve better performance. We summarize and discuss some of these key tricks used in different LLMs.
Mixed Precision: It is a famous method for LLMs to reduce memory usage and improve training efficiency. In mixed precision, forward and backward passes are performed in FP16 format whereas optimizer states and master weights are kept in FP32 format Mixed_Precision . A drawback associated with this format change is training instability due to a smaller value range resulting in loss spikes GLM-130B . An alternative to FP16 is BF16 which has a comparatively larger range and performs precision-sensitive operations like gradient accumulation and softmax in FP32 BLOOM . BF16 has better performance and training stability but uses more memory and is supported on specific hardware, for example, A100 GPUs. Therefore, its adoption in LLMs is limited.
Training Instability: Loss divergence or spiking is a common issue in LLMs that occurs multiple times during training. This happens in the presence of gradient clip** PaLM . To mitigate this problem, many approaches suggest restarting training from an earlier checkpoint PaLM ; GLM-130B ; du2022glam , skip** 200-500 earlier data batches at the point of divergence in PaLM and re-shuffling batches in du2022glam . The embedding layer gradient shrink proves to further stabilize the training as its gradient norm is significantly larger than the other layers GLM-130B . Another suggestion to improve training stability for larger models is not to use biases in dense and norm layers as in PaLM .
Weight Initialization: It plays a significant role in model convergence and training stability. GPT-NeoX GPT_NeoX initializes feed-forward layers before residuals with $\frac{2}{L\sqrt{d}}$ as in Mesh_Transformer_JAX and other layers with the small initialization scheme small_init . This avoids activations growing exponentially with increasing depth. MT-NLG mtnlg found higher variance for weight initialization leads to unstable training, hence validating small initialization scheme small_init . Various models perform random weight initialization which can cause bad initialization, Galactica galactica suggests a longer warmup to negate the effect.
Learning Rate: A suitable learning rate is important for stable training. It is suggested to use a lower value BLOOM ; PaLM ; U-PaLM with warmup and decay (cosine or linear). Usually, the learning rate is within the range $1e^{-4}$ to $8e^{-4}$. Moreover, MT-NLG (530B) mtnlg and GPT-NeoX (20B) GPT_NeoX suggest interpolating learning rates based on the model size using the GPT-3 GPT-3 models ranging between 13B and 175B. This avoids tuning the learning rate hyperparameter.
Training Parallelism: 3D parallelism, a combination of data, pipeline, and tensor parallelism, is the most utilized training parallelism approach in LLMs GLM-130B ; PaLM ; OPT ; BLOOM ; mtnlg ; wu2021yuan ; lieber2021jurassic . In addition to 3D parallelism, BLOOM BLOOM uses a zero optimizer ZeroOpt to shard optimizer states. PanGu-$\alpha$ PanGU_alpha and PanGu-$\Sigma$ PanGu_sigma go beyond 3D parallelism and apply 5D parallelism which additionally contains optimizer parallelism and rematerialization.
Mode Switching: It adds task-related tokens at the beginning of the text during training. These tokens refer to the natural language understanding and natural language generation tasks which are shown to improve downstream task performance in UL2 ; U-PaLM ; soltan2022alexatm . During fine-tuning and inference, tokens are appended based on the downstream tasks.
Controllable Text Generation: Generating credible and controlled text from a pre-trained model is challenging. GPT-3 GPT-3 and other LLMs use in-context learning to control generated text. While in-context learning helps in controlling the generated text, ERNIE 3.0 Titan ernie3titan suggests using adversarial loss to rank its generated text for credibility and soft prompts such as genre, topic, keywords, sentiment, and length for better control on generated text.

Although generalized models are capable of performing diverse tasks with good performance they have not yet outperformed models trained in supervised settings. The supervised trained models are still state-of-the-art in various NLP tasks by a large margin as shown in GPT-3 ; PaLM ; Tk-INSTRUCT .

LLMs perform well in zero-shot and few-shot settings. But the performance difference between zero-shot and few-shot is large for pre-trained models GPT-3 ; PaLM , naming LLMs as meta-learners GPT-3 . LLMs zero-shot evaluations underperform unsupervised methods in neural machine translation GPT-3 . The literature shows pre-training is not enough for good zero-shot performance PaLM ; Flan . To improve the zero-shot performance the literature suggests using instruction fine-tuning that improves the zero-shot performance significantly and outperforms baselines. Instruction fine-tuning has also been shown to improve zero-shot generalization to unseen tasks. Another model, Flan-PaLM Flan , unlocks zero-shot reasoning with CoT training.

Traditionally, these architectures perform well for different tasks, for example, encoder-only for NLU tasks, decoder-only for NLG, and encoder-decoder for sequence2sequence modeling. Encoder-only models are famous for smaller models such as Bert Bert , RoBERTa roberta , etc., whereas LLMs are either decoder-only GPT-3 ; GPT_NeoX ; BLOOM or encoder-decoder T5 ; mT5 ; soltan2022alexatm . While decoder-only models are good at NLG tasks, various LLMs, PaLM PaLM , OPT OPT , GPT-3 GPT-3 , BLOOM BLOOM , LLaMA gao2023llama , are decoder-only models with significant performance gains on both NLU and NLG tasks. In contradiction to this, T5 T5 and UL2 UL2 identify encoder-decoder models out-performing decoder-only models. In another study, PaLM PaLM finds increasing the size of decoder-only models can reduce the performance gap between decoder-only and encoder-decoder architectures.
Although decoder-only architectures have become a trend for LLMs, many recently proposed approaches UL2 ; soltan2022alexatm use mode-switching tokens in text with encoder-decoder architectures to enable task-specific modes. Similarly, CodeT5+ codet5+ uses an encoder-decoder architecture with multiple training objectives for different tasks, activating the encoder, decoder, or both according to the tasks. These variations in architecture and training objectives allow a model to perform well in different settings. Because of this dynamic configuration, the future of LLMs can be attributed to encoder-decoder architectures.