Large Language Models

Training

Objectives

Pretraining

  • Improving Language Understanding by Generative Pre-Training

  • Universal Language Model Fine-tuning for Text Classification

Domain-Adaptation

Instruction Fine-Tuning (IFT)

Datasets: NaturalInstructions: https://github.com/allenai/natural-instructions/

Supervised Fine-Tuning (SFT)

Datasets: UltraChat: https://github.com/thunlp/UltraChat

Preference Optimisation (PO)

Reinforcement Learning with Human Feedback (RLHF)/Proximal Policy Optimisation (PPO)
Direct Preference Optimisation (DPO)

Long Context LLMs

Category

Model

Max sequence length

Full Attention

Flash Attention

Not specified

Augmented Attention

Transformer-XL

Up to 16k tokens (depends on the segment length)

Augmented Attention

Longformer

Up to 4k tokens

Recurrence

RMT

Not specified

Recurrence

xLSTM

Not specified

Recurrence

Feedback Attention

Not specified

State Space

Mamba

Not specified

State Space

Jamba

Not specified

Optimized Full Attention

  • Flash Attention

Augmented Attention

  • Receptive Field Modification: Transformer-xl

  • Sparse Attention: Longformer

Recurrence

  • RMT: Recurrent Memory Transformer

  • Feedback Attention

Non Transformer

Retrieval Augmented

  • Bidirectional Attention for encoder: BERT, T5, Electra, Matryoshka, Multimodal

    • Approximate Nearest Neighbour Search

  • Causal attention for decoder: GPT, Multimodal generation

Pruning

  • LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference

Special Techniques

Low-Rank Approximations (LoRA)

Mixture of Experts

Logit Bias

Goal: Influence the output probabilities of a language model (LLM) to steer it towards a desired output, such as a “yes” or “no” answer.

  1. Logit Adjustment

  • Each token in the vocabulary has an associated logit value.

  • By adding a bias to the logits of specific tokens, you can increase or decrease the likelihood that those tokens will be selected when the model generates text.

  1. Softmax Function

  • After adjusting the logits, the softmax function is applied to convert these logits into probabilities.

  • Tokens with higher logits will have higher probabilities of being selected.

Steps

  1. Identify Token IDs

  • Determine the token IDs for “yes” and “no” in the model’s vocabulary. For instance, suppose “yes” is token ID 345 and “no” is token ID 678.

  1. Apply Bias

  • Adjust the logits for these tokens. Typically, you would add a positive bias to both “yes” and “no” tokens to increase their probabilities and/or subtract a bias from all other tokens to decrease their probabilities.

  1. Implementing the Bias

  • If using an API or library that supports logit bias (e.g., OpenAI GPT-3), you can specify the bias directly in the request.

Example
{
  "prompt": "Is the sky blue?",
  "logit_bias": {
  "345": 10,  // Bias for "yes"
  "678": 10   // Bias for "no"
  }
}
Practical Considerations

  1. Magnitude of Bias

  • The magnitude of the bias determines how strongly the model will favor “yes” or “no.”

  • A larger bias will make the model more likely to choose these tokens.

  1. Context Sensitivity

  • The model may still consider the context of the prompt. If the context strongly indicates one answer over the other, the model may lean towards that answer even with a bias.

  1. Balanced Bias

  • If you want the model to have an equal chance of saying “yes” or “no,” you can apply equal positive biases to both tokens. If you want to skew the response towards one answer, apply a larger bias to that token.

Example in Practice

  • Consider a scenario where you want the model to respond with “yes” or “no” to the question “Is the sky blue?”

  • This setup ensures that the model will highly favor “yes” and “no” as possible outputs. The prompt and biases are designed so that “yes” or “no” are the most likely completions.

API Implementation Example Here's a pseudo-code example of how you might implement this with an API:
import openai

response = openai.Completion.create(
  engine="text-davinci-003",
  prompt="Is the sky blue?",
  max_tokens=1,
  logit_bias={"345": 10, "678": 10}
)

print(response.choices[0].text.strip())

In this example: - The prompt is set to “Is the sky blue?” - The logit_bias dictionary adjusts the logits for the “yes” and “no” tokens to be higher. - The max_tokens is set to 1 to ensure only one word is generated. - By using logit bias in this way, you can guide the LLM to produce a “yes” or “no” answer more reliably.

Applied LLMs

Prompt Engineering

Practical

In Context Learning (ICL)

  1. [prompthub.us] The Few Shot Prompting Guide

  2. [prompthub.us] In Context Learning Guide

Academic

Embeddings for Retrieval

Evaluation

  • [openreview.net] BEIR

  • [arxiv.org] MTEB

  • For speech and vision, refer to the guide above from TechTarget.

Tech

Vector DB

Retrieval Augmented Generation (RAG)

Modeling Choices

  1. Frozen RAG

  1. Trained RAG

Notes: Modeling

Note

  • x = query

  • z = doc

  • y = output

Frozen RAG

In-context

Important

RALM

  • Retrieve k documents Z_k.

  • Rerank the docs using (1) zero-shot LM or (2) dedicated trained ranker.

  • Select top doc Z_top.

  • Prepend top doc in textual format as-is to the query as a part of the prompt for the LM to generate.

  • What we pass to the decoder: prompt with Z_top in it.

  • Issues: problematic for multiple docs (!)

In-context/Seq2Seq/Decoder

Important

RePLUG

  • Retrieve k documents.

  • Use cosine similarity score to compute p(Z_k | X).

  • What we pass to the decoder: concat{Z_k, X} or prompt with Z_k in it.

  • Make k forward passes in the decoder for each token to compute the likelihood over vocab using softmax p(Y_i | concat{Z_k, X}, Y_1..{i-1}).

  • Rescale the softmax with p(Z_k | X) and marginalize.

  • Pass the marginalized softmax to the decoder.

  • Issues: k forward passes at each token.

Decoder Only

Important

kNN-LN

  • For the current token consider X = encode(Y_1…Y_{i-1}).

  • Retrieve k documents Z_k matching X.

  • Make k forward passes in the decoder with the matching doc p_k(Y_i | Z_1..{i-1}).

  • Rescale p_k(Y_i | Z_1..{i-1}) over k and marginalize over the next token Y_i.

  • Do the same in the original sequence p_decode(Y_i | Z_1..{i-1}).

  • Interpolate between these using a hyperparameter.

  • Issues: k forward passes + retrieval at each token.

Retriever trainable RAG

Seq2Seq

Important

RePLUG-LSR

  • Uses the parametric LM’s output to update the retriever.

  • Loss: KL div between p(Z_k | X) and the posterior p(Z_k | X, Y_1..Y_N) works well.

E2E trainable RAG

Seq2Seq

Important

  • RAG

    • Per token: same as RePLUG - output probability is marginalised at the time of generation of each token, pass it to beam decoder.

    • Per sequence: output probability is marginalised for the entire sequence.

      • Results in #Y generated sequences.

      • Might require additional passes.

    • Training - NLL loss across predicted tokens.

    • Issues: E2E training makes doc index update problematic, solution: just update the query encoder.

  • Atlas

    • Multiple choice for updating the retriever - simple RePLUG-LSR type formulation based on the KL div between p(Z_k | X) and the posterior p(Z_k | X, Y_1..Y_N) works well.

    • Pre-training: same objective as the Seq2Seq (prefixLM or MLM) or decoder-only objective works well.

    • Training:

    • Issues:

Notes: Index Choice

Graph RAG

Important

  • Baseline rag struggles

    • answering a question requires traversing disparate pieces of information through their shared attributes

    • holistically understand summarized semantic concepts over large data collections or even singular large documents.

  • Graph RAG: https://microsoft.github.io/graphrag/

    Note

    • Source documents -> Text Chunks: Note: Tradeoff P/R in chunk-size with number of LLM calls vs quality of extraction (due to lost in the middle)

    • Text Chunks -> Element Instances:

      • Multipart LLM prompt for (a) Entity and then (b) Relationship. Extract descriptions as well.

      • Tailor prompt for each domain with FS example.

      • Additional extraction covariates (e.g. events).

      • Multiple rounds of gleaning - detect additional entities with high logit bias for yes/no. Prepend “MANY entities were missed”.

    • Element Instances -> Element Summaries

    • Element Summaries -> Graph Communities

    • Graph Communities -> Community Summaries

      • Leaf level communities

      • Higher level communities

    • Community Summaries -> Community Answers -> Global Answer

      • Prepare community summaries: Shuffle and split into chunks to avoid concentration of information and therefore lost in the middle.

      • Map-Reduce community summaries

    • Summarisation tasks

      • Abstractive vs extractive

      • Generic vs query-focused

      • Single document vs multi-document

    • The LLM processes the entire private dataset, creating references to all entities and relationships within the source data, which are then used to create an LLM-generated knowledge graph.

    • This graph is then used to create a bottom-up clustering that organizes the data hierarchically into semantic clusters This partitioning allows for pre-summarization of semantic concepts and themes, which aids in holistic understanding of the dataset.

    • At query time, both of these structures are used to provide materials for the LLM context window when answering a question.

    • Eval:

      • Comprehensiveness (completeness within the framing of the implied context of the question)

      • Human enfranchisement (provision of supporting source material or other contextual information)

      • Diversity (provision of differing viewpoints or angles on the question posed)

      • Selfcheckgpt

Notes: RAG vs Long Context

  • RAG FTW: Xu et al (NVDA): RETRIEVAL MEETS LONG CONTEXT LARGE LANGUAGE MODELS (Jan 2024)

    • Compares between 4k+RAG and 16k/32k LC finetuned with rope trick with 40B+ models

    • Scroll and long bench

  • LC FTW: Li et al (DM): Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach (Jul 2024)

    • Systematized the eval framework using infty-bench EN.QA (~150k) and EN.MC (~142k) and 7 datasets from long-bench (<20k)

    • 60% of the cases RAG and LC agrees (even makes the same mistakes)

    • Cases where RAG fails

      1. multi-hop retrieval

      2. general query where semantic similarity doesn’t make sense

      3. long and complex query

      4. implicit query requiring a holistic view of the context

    • Key contribution: Proposes self-reflectory approach with RAG first with an option to respond “unanswerable”, then LC

  • RAG FTW: Wu et al (NVDA): In Defense of RAG in the Era of Long-Context Language Models (Sep 2024)

    • Same eval method as the above

    • Key contribution: keep the chunks in the same order as they appear in the original text instead of ordering them based on sim measure

Notes: LLM and KG

See also

  • Unifying Large Language Models and Knowledge Graphs: A Roadmap

  • QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering

  • SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models

KG-enhanced LLMs

  • pre-training:

    • ERNIE: Enhanced language representation with informative entities

    • Knowledge-aware language model pretraining

  • inference time:

    • Retrieval-augmented generation for knowledge intensive nlp tasks

  • KG for facts LLM for reasoning:

    • Language models as knowledge bases?

    • KagNet: Knowledgeaware graph networks for commonsense reasoning

LLM enhanced KGs: KG completion and KG reasoning

  • LLMs for Knowledge Graph Construction and Reasoning

  • Pretrain-KGE: Learning Knowledge Representation from Pretrained Language Models

  • From Discrimination to Generation: Knowledge Graph Completion with Generative Transformer

Synergized KG LLM

  • KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

  • Search: LaMDA: Language Models for Dialog Applications

  • RecSys: Is chatgpt a good recommender? a preliminary study

  • AI Assistant: ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Known Issues

Hallucination

Detection & Mitigation

Supervised

Applicable: translation, summarization, image captioning

  • n-gram (bleu/rouge, meteor)

    • reference dependent, usually only one reference

    • often coarse or granular

    • unable to capture semantics: fail to adapt to stylistic changes in the reference

  • ask gpt (selfcheckgpt, g-eval)

    • evaluate on (a) adherence (b) correctness

    • blackbox, unexplainable

    • expensive

Unsupervised

  • perplexity-based (gpt-score, entropy, token confidence) - good second order metric to check

  • too granular, represents confusion - not hallucination in particular, often red herring

  • not always available

Sycophany

Monosemanticity

  • many neurons are polysemantic: they respond to mixtures of seemingly unrelated inputs.

  • neural network represents more independent “features” of the data than it has neurons by assigning each feature its own linear combination of neurons. If we view each feature as a vector over the neurons, then the set of features form an overcomplete linear basis for the activations of the network neurons.

  • towards monosemanticity:

    1. creating models without superposition, perhaps by encouraging activation sparsity;

    2. using dictionary learning to find an overcomplete feature basis in a model exhibiting superposition; and

    3. hybrid approaches relying on a combination of the two.

  • developed counterexamples which persuaded us that the

    • sparse architectural approach (approach 1) was insufficient to prevent polysemanticity, and that

    • standard dictionary learning methods (approach 2) had significant issues with overfitting.

  • use a weak dictionary learning algorithm called a sparse autoencoder to generate learned features from a trained model that offer a more monosemantic unit of analysis than the model’s neurons themselves.