• How your LLM will function in deployment

  • inference speed

  • compute budget

  • trade off model performance for improved inference speed or lower storage?

• Additional resources your model may need

  • Interaction with external data or other applications? If so, how will you connect to those resources?

• How your model will be consumed

  • what will the intended application or API interface that your model will be consumed through look like?

Model optimizations to improve application performance

• Reduce the size of the LLM. Challenge: maintaining model performance

  • Distillation ( Knowledge distillation)

  • Post training quantization

  • Pruning : Reduce model parameters that contribute little to the model performance

Knowledge distillation:

• Total loss = distillation loss + student loss. Distillation loss between soft predictions of teach and student. Student loss hard predictions of student and hard labels (ground truth).

• In practice, distillation is effective for encoder based models. Not that effective for decoder based generative models.

  

  Post-training Quantization (PTQ)

• PTQ transforms model weights into a lower precision representation such 16-bit floating point or 8-bit integer.

• May result in small model metrics reduction.

  Pruning

• Remove model weights with values close or equal to zero

• Pruning methods:

  • Full model -retraining

  • Peft/LoRA

  • Post-training

• In theory, reduces model size and improves model performance

• In practice, only small % in LLMs are zero-weights

Using the LLM in applications

Other challenges:

1. Internal knowledge of model cutoff → can’t answer questions related to events after model cutoff

2. LLM do not carry out mathematical calculation on its own

3. Hallucinations

Techniques to overcome this challenges:

• RAG: helps with knowledge cutoff and hallucinations

  

  

• Data preparations for vector store for RAG

• Data must fit inside context window ( prompt context limit). Single document too large to fit in window → split long sources into short chunks.

• Data must be available in format that allows its relevance to be assessed at inference time: Embedding vectors which are stored in vector stores for efficient identification of semantically related text.

• Vector databases are a particular implementations of a vector store where each vector is also identified by a key.

Interacting with external applications

Processing a return example:

• Helping LLMs reason and plan with chain-of-thought by providing reasoning steps in the prompt

• LLMs can struggle with mathematics: Provide access to code interpreter to the LLM. Program-aided language (PAL) models.

• 

  

  ReAct: Combining reasoning and action

  

  

LLM application architectures

The three Hs to ask about model outputs: HHH - Helpful? Honest? Harmless?

Fine-tuning with human feedback helps mitigate these issues. Foe example, research indicates that fine-tuning with human feedback for text summarization task produces better responses compared to reference summaries, models with initial fine-tuning (without human feedback), or no fine-tuning at all.

A widely used method for fine-tuning large language models with human feedback is known as Reinforcement Learning from Human Feedback (RLHF). As the name implies, RLHF leverages reinforcement learning (RL) to fine-tune the large language model using human feedback data, resulting in a model that is more closely aligned with human preferences. RLHF can be used to ensure the model generates outputs that maximize relevance and usefulness to the input prompt. Most importantly, RLHF helps minimize the potential for harm by training the model to include disclaimers acknowledging its limitations and to avoid toxic language or topics.

Reinforcement learning is a machine learning approach where an agent learns to make goal-oriented decisions by interacting with an environment and taking actions to maximize a cumulative reward.

In this framework, the agent learns iteratively by taking actions, observing the changes in the environment, and receiving rewards or penalties based on the outcomes. Through repeated iterations, the agent refines its strategy or policy to make more effective decisions and improve its overall performance.

Using Tic-Tac-Toe as an example to explain how RL works

In reinforcement learning (RL), an agent learns to achieve a goal by interacting with an environment and receiving feedback. Using Tic-Tac-Toe as an example, let’s break down how RL works:

1. Components in RL for Tic-Tac-Toe:

• Agent: The player (model) that tries to learn an optimal strategy to win the game.

• Environment: The Tic-Tac-Toe board, where the agent and an opponent take turns placing Xs and Os.

• State (St​): The current configuration of the board at time t, such as which cells are filled and by whom.

• Action (at​): The agent’s move at time t, such as placing an X in an empty cell.

• Action Space: The set of all possible actions the agent can take at a given state (e.g., choosing any of the empty cells).

• Reward (Rt​): The feedback received after an action. For example:

  • +1 if the agent wins.

  • -1 if the agent loses.

  • 0 for a draw.

2. Objective:

The objective is to learn a policy that maximizes the cumulative reward, which means winning the game as frequently as possible.

3. How RL Works in Tic-Tac-Toe:

1. Initial State: The game starts with an empty board (S0​).

2. Taking Actions: The agent chooses an action at​ (e.g., placing an X in an empty cell) based on its current policy, which determines which actions to take in different states.

3. Environment Response: The opponent makes a move, changing the state of the board to St+1​. This step is repeated until the playout/rollout is over.

4. Reward: The agent receives a reward Rt​ based on the resulting final state:

   • If the agent wins, it gets a reward of +1.

   • If the agent loses, it gets a reward of -1.

   • If the game is ongoing, the reward is 0.

5. Learning: The agent updates its policy based on the observed reward to improve its decision-making for future states.

6. Iteration: The agent continues to play multiple games, refining its policy through repeated interaction and feedback.

4. Learning a Winning Strategy:

The agent explores different moves and learns which sequences of actions lead to winning states, updating its strategy to favor moves that maximize the likelihood of winning. Over time, it becomes capable of anticipating the opponent’s moves and making decisions that maximize its chances of victory.

Through repeated gameplay and feedback, the agent learns an optimal policy that guides it to win the game more often by selecting actions that lead to favorable states and avoiding moves that result in losing positions.

Fine-tuning LLMs using RL

• Objective: The goal is to generate text that is aligned with human preferences—text that is coherent, contextually relevant, and free from harmful or undesirable content.

  

• Agent: The agent is the language model itself, fine-tuned to generate desirable text outputs based on a given prompt.

• Environment: The environment is the context of the conversation or the prompt provided as input. The model’s output updates the state of the environment by generating text, which is then evaluated based on alignment criteria.

• State (St​): The state is the prompt combined with the text generated so far. Each time the LLM generates new text, the state changes.

• Action (at​): An action is the next word, sentence, or paragraph generated by the model in response to the prompt and context. The action space here is the probability distribution of the next word.

• Reward (Rt​): The reward is provided based on the quality of the generated text. Rewards are usually derived from human feedback or automatic evaluation metrics. For example:

  • +1 if the generated text is highly relevant, coherent, and aligned with human preferences.

  • -1 if the text is incoherent, misleading, or toxic.

  Learning Process

  1. Initial State: The initial state is the input prompt (e.g., a user question or command).

  2. Taking Actions: The agent generates the next word or sentence based on the current context.

  3. Evaluating the Action: The generated text is evaluated using human feedback or pre-defined metrics (e.g., fluency, correctness, and relevance). Human evaluators can score the output, providing feedback on whether it was aligned or not.

  4. Learning from the Outcome: The model updates its parameters through reinforcement learning techniques like Proximal Policy Optimization (PPO), reinforcing generation patterns that lead to high-reward text (aligned and relevant responses).

  • Note that in the context of language modeling, the sequence of actions and states is called a rollout, instead of the term playout that's used in classic reinforcement learning.

  • When using Reinforcement Learning with Human Feedback (RLHF) to align large language models with human preferences, what is the role of human labelers? In RLHF, human labelers score a dataset of completions by the original model based on alignment criteria like helpfulness, harmlessness, and honesty. This dataset is used to train the reward model that scores the model completions during the RLHF process.

  Challenges with determining the reward

• There is a wide variation in human preferences → This makes determining the reward challenging. One of doing this is for a human to evaluate all the completions of the model against some alignment methods such as determining whether the generated text is toxic or not. There reward could be a 0 or 1 and the LLM updates its weights to improve alignment and maximize reward. However, obtaining human feedback can be time consuming and expensive. One way of alleviating this bottleneck is to use an additional model called reward model which will classify the completion and evaluate the degree of alignment with human preferences.

  Why RL for LLMs?

  LLMs learn to generate aligned text through feedback:

  • Human feedback indicates which text outputs are preferred.

  • The model adjusts its policy to generate outputs that maximize the probability of receiving a positive reward.

  • Over many iterations, the LLM learns to produce text that maximizes human preferences and minimizes undesirable outputs, similar to how an agent learns to optimize its performance in a given environment.

RLHF: Obtaining feedback from humans

1. Prompt dataset: consists of prompt samples → Instructor LLM → Set of completions

2. Collect human feedback:

   1. ‘Define your model alignment criterion

   2. For the prompt-response sets that you just generated, obtain human feedback through labeler workforce. Usually assign a sample to multiple labelers (3) to establish consensus and minimize effects from poor labelers.

3. Prepare labeled data for training

   1. Convert ranking into pairwise training data for the reward model. (y_j, y_k). If you have three ranked completions, you can have 3 pairs ( combinations). y_j is always the preferred completion.

   2. Train the reward model

      Given (prompt x, completion yj) and prompt x, completion yk) minimize the log sigmoid of the difference between the rewards.

   3. Use the reward model as a binary classifier to provide a reward value for each prompt-completion pair, in this case the reward value will be a logit.

      

      

      RLHF: Fine-tuning with reinforcement learning

For each iteration, for a given prompt the LLM generates a completion. The reward model generates a reward value for the completion’s degree of human alignment. The RL algorithm then updates the LLM weights based on the reward value. This will continue until the LLM is updated to a human-aligned LLM or after pre-defined number of iterations.

RL algorithm: Proximal Policy Optimization (PPO)

PPO updated the Policy - in this case the LLM to be more aligned with human preferences. The updated are small and within a bounded region.

PPO Phase 1:

• Create completions to a number of prompts. These are experiments to assess the outcome of the current model, e.g. how helpful, harmless, hones the model is.

• Calculate the reward for each prompt, completion pair using the reward model.

• The Value function estimates future total reward a given generated next token.

  

  The goal is to minimize the difference between the its approximation to the value function (1.23) and the actual future total reward (1.87)

PPO phase 2: Calculate the policy loss

The PPO algorithm uses a surrogate objective function that is modified by a clipping mechanism. This clipping serves to restrict the size of policy updates to a safe range, which stabilizes training and prevents large updates that could negatively impact performance.

In PPO, the objective function is typically represented as:

Purpose of Clipping

The clipping serves the following purposes:

1. Prevent Large Policy Updates: Clipping limits the change in the probability ratio rt​(θ). This prevents the new policy from making large updates to the action probabilities, which can destabilize learning.

2. Reduce Variance: By controlling the policy change, clipping reduces the variance in updates, leading to more stable learning.

3. Ensure Monotonic Improvement: By avoiding large policy changes, clipping helps ensure that each policy update improves performance incrementally, avoiding sudden performance drops.

How it Works in Practice

• If the policy change results in rt​(θ) being too far from 1 (indicating a large policy shift), the clipped version of the ratio is used instead, which restricts the impact of the policy update.

• The objective function L_CLIP(θ) takes the minimum of the original objective and the clipped objective. This ensures that updates only occur if they improve the objective within a safe range, effectively discouraging large deviations from the current policy.

PPO Phase 2: Calculate the entropy loss

• Entropy allows the model to maintain creativity. Higher entropy means LLM is more creative. This is like the Temperate during inference. Entropy loss how every is during training.

• PPO Phase: Total objective function. C1 and C2 are HPs. For each PPO cycle the LLM is updated to a new LLM.

  

  • Q-learning is an alternate technique for fine-tuning LLMs through RL.

  • DPO (Direct Preference Optimization) by Stanford is a simpler alternate to RLHF.

RLHF: Reward hacking

• The agent learns to cheat the system by favoring actions that maximize the reward received even if those actions don’t align well with the original objective. In such scenarios we can use the original LLM as performance reference by computing KL divergence between the reference and the intermediate LLM probability distributions.

Scaling Human Feedback

• Building a reward model requires 10’s of thousands of human reference labels.

• Solution: Model Self-supervision → Constitutional AI. Model is trained using guiding rules/principles. Along with proper prompting forms the constitution.

  

  • Prompt the model in ways that try to get it to generate harmful responses, this process is called red teaming.

  • You then ask the model to critique its own harmful responses according to

  the constitutional principles and revise them to comply with those rules.

  • Once done, you'll fine-tune the model using the pairs of red team prompts and the revised constitutional responses.

References:

1. https://www.coursera.org/learn/generative-ai-with-llms/

2. https://huggingface.co/blog/trl-peft

A basic neuron get data from a previous layer, applies some form of computation to transform the inputs and outputs the transformed data.

When the activation function is a sigmoid function, a neuron is simply a “unit” of logistic regression. Weight <=> coefficients, input <=> variables , bias term <=> constant term.

Derivative of a sigmoid function.

• We need a larger network because a single neuron like logistic regression only permits a linear decision boundary.

• The net input to the first hidden layer is sum of weighted inputs before activation function.

• Activations are output of neurons to the next layer

Matrix representation

Gradients at each layer to use for backpropagation.

Vanishing gradients:

From the above gradient equation, we can see that as we have more layers, the gradient gets very small at the early layers. This is known as the “vanishing gradient” problem.

Other activation functions to the rescue:

• Hyperbolic tangent function ( tanh)

• Rectified Linear Unit (RELU)

• “Leaky” ReLU

How often do we need to updated the weights using gradient descent?

• Foundation model - Pre-trained LLM

  • Pre-training: GB → TB → PB side data, use of GPUs, data quality filter.

  • Model architecture: encoder only, encoder decoder, decoder only models.

  • Encoder only models: Masked Language Modeling(MLM), objective: denoising: predict the masked token, bidirectional. Use cases: sentiment analysis, named entity recognition, word classification. Examples: BERT, ROBERTA

  • Decoder only models: Autoregressive models, objective: Predict next token, Causal Language modeling, unidirectional context. Use cases: Text generation, other emergent behavior dependent on model size, examples: GPT, Bloom

  • Encoder Decoder ( sequence-to-sequence) models: objective: vary from model to model. Example objectives ( in T5 , BART model)s: Span corruption, mask a reange of contiguous words. A sentinel token is a special palceholder used to represent and mark the beginning of a missing or masked span in the input sequence. Objective: Reconstruct the span. Good use cases: Translation, text summarization, question answering. Example models: T5, BART

    

• Summary

  

The Computational Challenges of training LLMs

Approximate GPU RAM needed to store 1B parameters.

• 1 parameter = 4 bytes ( 32-bit float)

• 1 B parameters = 4*10^9 byes = 4 GB

Memory needed to store a model: 4Gb@32-bit full precision, Memory needed to train model 24GB @ 32-bit full precision.

Options to reduce the memory:

• Quantization : reduce the precision to 16-bit floating point or 8-bit integer

  

  

  GPU RAM needed to train larger models

• As model sizes get larger, you will need to split your model across multiple GPUs for training.

  Efficient multi-GPU compute strategies

• If model fits in a single GPU there may not be a need for using multi-GPUs. But even if your model does fit onto a single GPU, there are benefits to using multiple GPUs to speed up your training.

• Distributed Data Parallel (DDP): replicates LLM on each GPU, only parallelizes data.

• Zero Redundancy Optimizer: (ZERO): Reduces memory by distributing (sharding) the model parameters, gradients, and optimizer states across GPUs.

• Fully Sharded Data Parallel (FSDP): Distibutes data, Optimizer States, gradients and parameters. Must gets weights before and after the forward pass. Helps reduce overall GPU memory utilization, configure level of sharing via Sharind Factor, supports offloading to CPU if needed.

Scaling Laws and compute-optimal models

Pre-training for domain Adaptation

• Legal language

• Medical language

During the training of BloombergGPT, the authors used the Chinchilla Scaling Laws to guide the number of parameters in the model and the volume of training data, measured in tokens. The recommendations of Chinchilla are represented by the lines Chinchilla-1, Chinchilla-2 and Chinchilla-3 in the image above, and we can see that BloombergGPT is close to it.

• Scales efficiently across multiple GPUs

• Parallel process data

• Gives attention to input meaning

• Pays attention to all the words in a given input sequence.

• Attention map

• Self-attention

How does the transformer work?

• Split into Encoder and Decoder

• Tokenizer ( each number represents a position in the vocabulary dictionary). Token may represent a word or part of a word

• Embedding layer → Trainable vector embedding network which maps each tokenID to a high dimensional vector space which learns the meaning and context of each token in the input sequence. In the original Transformer arch, embedding size was 512.

• Positional Encoding mapped to positional embeddings. Input embedding to the self-attention layer is the sum of the input sequence embedding and the positional embeddings. Positional embeddings are required to preserve the order of the tokens in the sequence.

• The Transformer model also uses residual connections and layer normalization to facilitate training and prevent overfitting. In addition, the authors introduce a positional encoding scheme that encodes the position of each token in the input sequence, enabling the model to capture the order of the sequence without the need for recurrent or convolutional operations.

• Self attention layers pays attention to different parts of the sequence to better capture the dependency between the tokens. It stores the importance of each token to all other tokens in the sequence. This doesn’t happen just one, there multiple heads, multi-attention heads. Each each head will learn a different aspect of language. One head entity, another word rhythm, etc.

• output of attention layer is passed to a Feed forward network. The output of the feedforward network in the decoder are logits proportional to the probability of each token in the vocabulary. This will then pass to a softmax output which normalizes the probability over the entire vocabulary. The most likely predicted token will have the highest probability.

Generating text with transformers:

• Sequence-to-sequence task

• The encoder output is a deep representation of the input sequence structure and meaning of each token. This is fed to the decoder. The decoder has information from the encoder and the current output of the decoder back to the decoder.

  

  Source: Generative AI with LLMs, Coursera/AWS course.

  Summary:

  • Encoder encodes inputs (“prompts”) with contextual understanding and produced one vector per input token

  • Decoder accepts input tokens and generate new tokens. It uses the information from encoder outputs, “cross-attention”.

Additional notes on how the transformer is trained:

In the original Transformer architecture, the encoder and decoder are trained simultaneously rather than sequentially. The training process doesn't involve training the encoder first and then the decoder later. Instead, both the encoder and decoder work together as part of the overall model during the training phase, and they are updated jointly as the model learns from data. Here's how the training process works:

Transformer Architecture Overview:

• The encoder processes the input sequence (e.g., a sentence) and generates a sequence of continuous representations.

• The decoder takes this encoded information, along with the previous words it has already generated (during training, this includes ground-truth words), to predict the next word in the output sequence (e.g., a translated sentence or another language task).

Training Process:

1. Input Sequence (to the Encoder):

   • The input sentence is tokenized (split into words or subword units) and embedded into vectors, which are passed to the encoder layers.

   • The encoder consists of multiple layers, each with a multi-head self-attention mechanism and feedforward networks. The output of the encoder is a set of context-rich representations of the input sequence.

2. Target Sequence (to the Decoder):

   • The decoder gets the encoded input representations from the encoder, and during training, it also receives the correct previous words (ground-truth) of the output sentence. This is done in a process called teacher forcing, where the decoder is provided with the true previous words instead of its own predictions.

   • The decoder has multiple layers similar to the encoder but also includes multi-head attention over the encoder’s output (in addition to self-attention over its own input).

3. Loss Calculation:

   • The model computes the loss based on the difference between the predicted next word and the actual word from the ground-truth sequence.

   • This is typically done using a cross-entropy loss function, which measures how well the predicted probability distribution matches the actual target word distribution.

4. Backpropagation and Joint Weight Updates:

   • The error signals (gradients) from the loss are propagated backward through both the decoder and encoder.

   • The weights in both the encoder and decoder are updated together during each training step through backpropagation and gradient descent (or similar optimization algorithms like Adam).

   • The entire model — encoder, decoder, and all attention mechanisms — learns jointly.

Key Insights:

• Parallelism: The Transformer does not process inputs sequentially like RNNs. Both the encoder and decoder layers leverage self-attention, which allows parallel computation across all tokens in a sequence, significantly speeding up training.

• Training Objective: During training, both the encoder and decoder are optimized to minimize the loss, and the decoder’s predictions are improved by looking at both its own past outputs and the encoded representations of the input.

Inference:

• During inference (when generating sequences), the decoder generates words one by one, feeding each generated word back into the decoder until the entire output sequence is produced.

Variations of Transformers:

• Encoder only models: Classification tasks, example: BERT

• Encoder Decoder models: Sequence-to-sequence tasks like translation, general text generation, Example: BART, T5

• Decoder only models: Generalize to most tasks. Example: Llama, GPT, Bloom

Question: How are subwords reconstructed after token decoding

When words are tokenized into subwords during the encoding process, these subwords need to be reconstructed back into full words after decoding. This process typically happens as follows:

1. Tokenization into Subwords:

• Many tokenizers (like Byte-Pair Encoding (BPE), WordPiece, or SentencePiece) split words into smaller units, called subwords, to handle out-of-vocabulary words and reduce the overall size of the vocabulary.

• For example, the word “unhappiness” could be split into un, happiness, or further into un, happi, ness.

2. Decoding Process:

• After decoding, the model generates a sequence of subword tokens. This could be a mix of complete words and parts of words (subwords).

• These subwords are then reconstructed back into the original words during post-processing.

3. Subword Reconstruction:

• Concatenation rules: Most subword tokenization algorithms use special markers to denote whether a token is a standalone word or part of a word. This helps in the reconstruction process:

  • In BPE or WordPiece, a common convention is to use a continuation marker (like ## in BERT or just not adding space) to indicate that a subword token should be attached to the preceding subword.

  • For example, the tokens happi, ##ness would be reconstructed as happiness since the ## marker tells the model to attach the subword ness to the preceding token without a space.

  • SentencePiece uses a slightly different method, where spaces before tokens are explicitly encoded, so the model knows which tokens should be concatenated.

4. Example:

Let’s say a sentence is tokenized into subwords like this:

Original sentence: "I am running fast" 
Subwords: ["I", "am", "runn", "##ing", "fast"]

After decoding, the subwords might look like:

["I", "am", "runn", "##ing", "fast"]

To reconstruct the sentence:

• The subword ##ing is concatenated with runn to form the word "running."

• Other tokens are already full words, so no changes are needed for them.

The final output would be:

"I am running fast"

In-Context learning(ICL)

• Zero shot inference: No examples in the prompt.

• One shot inference: One example in the prompt.

• Few shot inference: 5 or 6 examples in the prompt.

Generative Configuration - Inference parameters:

• Max new tokens: Maximum number of tokens to generate. The model can also stop generating if it predicts the stop token as the next word.

• Sample top K: select an output from the top-k results after applying random-weighted strategy using the probabilities.

• Sample top P: select an output using the random-weighted strategy with the top-ranked consecutive results by probability and with cumulative probability <=P

• Greedy decoding: The word/token with the highest probability is selected.

• Random (-weighted) sampling: select a token using random-weighted strategy across the probabilities of all tokens. do_sample=True. Sample to K and Top P are used to limit the options in the random sampling.

• Temperature: The higher the temperature , the higher the randomness. Impact the probability distribution.

Generative AI project lifecycle

1. Define the use case as narrowly and as specifically as possible. Good at many tasks? Good at a specific task?

2. Choose an existing model or work on a model from scratch? Most case start with pre-trained model

3. Assess its performance and carry out additional training if needed.

   1. Start with in-context learning: no shot, one shot, few shot

   2. Fine-tuning if ICL doesn’t have acceptable performance

   3. Align with human feedback

4. Application integration

   1. Optimize and deploy model for inference: efficient use of resources and offering best user experience

   2. Augment model and build LLM-powered applications

• Analogy: Serving food in a restaurant.

1. Data preparation:

• Two steps: Upload data, Engineer features

• Data types: Streaming vs batch data, structured vs unstructured data

  2. Model development

  3. Model Serving: Deploy the model, monitor the model

AutoML with Vertex AI

• AutoML : no code, through UI, user friendly

• Data sources: Cloud storage, BigQuery, Local machine

  Suports:

• Image → Objectives: Image classification ( single or multi-label), image object detection, image segmentation

• Tabular: Regression/classification, Forecasting

• text : Classification, entity extraction, sentiment analysis

• video: video action recognition, video classification, video object tracking

• AutoML helps you work using multiple data-types to solve a given ML/AI problem.

Vertext AI feature store

A centralized repository to manage, serve, and share features. It Aggregates features from different sources in BigQuery.

Vertex AI features stores is ready for the challenge of generative AI. It can manage and serve embeddings, which is the crucial data type in gen AI. It also supports retrieving similar items in real time, ensuring low latency.

• Real time (online) serving

• Batch (offline) serving

• At scale, low latency

Steps

Benefits of feature store

Features are: shareable for training and serving, reusable, scalable, Easy to use

Model Development ( experiment with Recipes: cook then taste)

• Train the model

• Evaluate the model

Evaluation metrics:

Confusion matrix:

Vertex AI provides: confusion matrix, precision-recall curve, feature importance, Explainable AI

Model Serving

• “Ready to serve the meal”

• Deploy model : serve the meal

• Monitor model: Attend while guests are eating

- Inference types: Endpoint ( real time), Batch predictions ( does not use endpoint)

• Vertex AI Pipeline: Automate, Monitor, Govern

• Vertex AI workbench → define your own pipeline with prebuilt pipeline components

MLOps and workflow automation

Practicing MLOps: Automating and monitoring each step of the ML system construction: Continuous integration(CI), continuous training (CT), continuous delivery (CD)

Vertex AI pipelines supports Kubeflow Pipelines (KFP), TensorFlow Extended (TFX)

Pipeline components: Custom component - accomplishes a specific task to tailor to your ML workflow. Pre-built component - reuse or customize standard components to suit your needs.

Phase of ML automation:

• Phase 0: The starting point, use a GUI-based workflow such as AutoML for training, deployment and serving.

• Phase 1: Automate parts of the workflow such as the training pipeline, develop building blocks for future use

• Phase 2: Integrate components to form an entire pipeline

• Achieve CI, CT, and CD

Vertex AI offerings

Data Science Steps for ML (ref: https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning)

After the business case is defined and a success criteria is established, the process of delivering an ML model to production involves the following steps.

1. Data extraction: You select and integrate the relevant data from various data sources for the ML task.

2. Data analysis: You perform exploratory data analysis (EDA) to understand the available data for building the ML model. This process leads to the following:

   • Understanding the data schema and characteristics that are expected by the model.

   • Identifying the data preparation and feature engineering that are needed for the model.

3. Data preparation: The data is prepared for the ML task. This preparation involves data cleaning, where you split the data into training, validation, and test sets. You also apply data transformations and feature engineering to the model that solves the target task. The output of this step are the data splits in the prepared format.

4. Model training: The data scientist implements different algorithms with the prepared data to train various ML models. In addition, you subject the implemented algorithms to hyperparameter tuning to get the best performing ML model. The output of this step is a trained model.

5. Model evaluation: The model is evaluated on a holdout test set to evaluate the model quality. The output of this step is a set of metrics to assess the quality of the model.

6. Model validation: The model is confirmed to be adequate for deployment—that its predictive performance is better than a certain baseline.

7. Model serving: The validated model is deployed to a target environment to serve predictions. This deployment can be one of the following:

   • Microservices with a REST API to serve online predictions.

   • An embedded model to an edge or mobile device.

   • Part of a batch prediction system.

8. Model monitoring: The model predictive performance is monitored to potentially invoke a new iteration in the ML process.

Figure 1: GAN - with backpropagation in dscriminator training ( highlighted).

Key Components:

1. Generator (G):

   • Its goal is to generate data that looks like the real data (e.g., images, audio).

   • It starts by creating random data (e.g., noise) and learns to produce more realistic samples over time.

2. Discriminator (D):

   • Its task is to distinguish between real data (from the dataset) and fake data (generated by the generator).

   • It outputs a probability that a given sample is real or fake.

How GANs Work:

• The generator tries to fool the discriminator by generating data that looks real, while the discriminator tries to correctly classify real vs. fake data.

• Over time, the generator improves, generating more realistic data, and the discriminator becomes better at distinguishing real from fake data.

• The training continues until the generator produces data that is nearly indistinguishable from real data.

Training the discriminator:

• Input to the discriminator comes from real data instances, e.g. images of real horses and fake data instances generated by the generator to be used as negative examples during training.

• Focuses only on discriminative loss. Loss includes both types of misclassifications real as fake or fake as real.

• Weights get updated using back-propagation.

Training the Generator:

• Generator training involves: the random input, the generator network, the discriminator network, discriminator output and generator loss.

• The most basic form of GAN takes random noise sampled from a target distribution as its input.

• The generator loss penalizes the generator for producing a sample that the discriminator network classifies as fake.

• Training steps:

  1. Calculate loss from the discriminator classification.

  2. Backpropagate through both the discriminator and generator to obtain gradients.

  3. Update only hte generator weights using gradients.

Training GANs:

While training GANs, we alternate training the two networks, taking turns after certain epochs.

1. Train the discriminator for one or more epochs.

2. Train the generator for one or more epochs.

3. Repeat steps 1 & 2 to continue training both networks.

Loss function:

• In the minimax loss function the generator tries to minimize the function while the discriminator tries to maximize it.

  Where:

  • D(x) is the discriminator’s estimate of the probability that real data instance x is real

  • G(z) is the genrator’s output for input noise z

  • D(G(z)) is the discriminator’s estimate of the probability that fake instance z is real

Use Cases:

• Image generation: Generating realistic images (e.g., deepfake images, art, or animations).

• Text-to-image: Generating images from text descriptions.

• Super-resolution: Enhancing the resolution of images.

• Data augmentation: Creating synthetic data for model training.

References:

1. https://developers.google.com/machine-learning/gan/

2. https://aws.amazon.com/what-is/gan/

3. https://github.com/christianversloot/machine-learning-articles/blob/main/building-a-simple-vanilla-gan-with-pytorch.md

A Simple RNN is a type of neural network designed to process sequential data. Unlike traditional feedforward neural networks, RNNs have internal memory, allowing them to retain information from previous inputs. This memory is implemented using a hidden state, which is updated at each time step based on the current input and the previous hidden state.

Mathematical Representation

• Input: A sequence of vectors, x1​,x2​,...,xT​.

• Hidden State: A vector, ht​, representing the network's memory at time step t.

• Output: A vector, yt​, representing the network's prediction at time step t.

The equations governing an RNN are:

h_t = tanh(W_{ih}x_t + W_{hh}h_{t-1} + b_h)
y_t = softmax(W_{ho}h_t + b_o)

where:

• W_{ih} → input weight​, W_{hh} → hidden state weight​, and W_{ho} → output weight​ are weight matrices.

• b_h​ → bias to hidden layer and b_o​ bias to output layer are bias vectors.

• tanh is the hyperbolic tangent activation function.

Pros and Cons

Pros:

• Can handle sequential data.

• Can learn “long-term” dependencies.

Cons:

• Can suffer from vanishing gradient problems, making it difficult to learn long-term dependencies.

• Can be computationally expensive for long sequences.

Weight Computation and Update

Weights are typically initialized randomly and then updated during training using backpropagation. The gradient of the loss function with respect to the weights is calculated using the chain rule. The weights are then updated using gradient descent or a variant like Adam.

Backpropagation Through Time (BPTT)

BPTT is a modified version of backpropagation specifically designed for RNNs. It involves unfolding the RNN over time, creating a computational graph that can be used to calculate gradients. The gradients are then propagated backward through time to update the weights.

Key steps in BPTT:

1. Forward pass: Compute the hidden states and outputs for each time step.

2. Backward pass: Calculate the gradients of the loss function with respect to the outputs, hidden states, and weights.

3. Weight update: Update the weights using the calculated gradients.

By using BPTT, RNNs can learn long-term dependencies and improve their performance on sequential tasks.

Simple RNN example

Here is the notebook I created which demonstrates how a simple RNN can be trained to classify tweets into binary sentiments.

Dropout is a popular regularization technique used in neural networks to prevent overfitting by randomly dropping (setting to zero) a fraction of the neurons in a layer during training. This technique reduces the model's reliance on any specific neurons, encouraging it to learn more robust and generalized features.

What is Dropout?

Dropout is a form of regularization that helps prevent overfitting in neural networks by:

• Randomly dropping neurons: During each training iteration (batch), a certain percentage of neurons (e.g., 20% or 50%) are randomly “dropped out” or temporarily deactivated. These neurons do not contribute to the forward pass or backpropagation.

• Reducing co-adaptation of neurons: By forcing the network to operate with a subset of its neurons, dropout reduces the likelihood that the network will become overly reliant on specific neurons, promoting redundancy and diversity in feature learning.

How Dropout Works During Training and Testing

• During Training:

  • Each neuron is dropped (set to zero) with a probability p, known as the dropout rate (e.g., 0.5 means 50% of neurons are dropped).

  • The remaining neurons are “kept” with probability 1−p.

  • The weight updates are computed only for the neurons that remain active.

  • Mathematically, the output of a neuron h_i​ at layer l during training becomes:

    where:

    • ri is a Bernoulli random variable (1 with probability 1−p, and 0 with probability p).

    • $w_ij$ are the weights, x_j is the input from the previous layer, and $b_i$ is the bias.

  • This effectively means that each forward pass during training uses a different "thinned" network, with only a random subset of neurons active.

• During Testing:

  • No neurons are dropped. Instead, all neurons are used, but their outputs are scaled by a factor equal to the dropout rate (1−p).

  • This scaling compensates for the fact that more neurons are active during testing than in any single forward pass during training.

Why Dropout is Used for Regularization

• Prevents Overfitting: By randomly dropping neurons during training, dropout prevents the network from becoming overly specialized to the training data, making it less likely to overfit.

• Encourages Redundancy and Robustness: Since neurons cannot rely on specific other neurons being active, they must learn more robust features that are useful in combination with many different subsets of neurons.

• Acts as an Implicit Ensemble Method: Training with dropout can be thought of as training many different, smaller networks that share weights. At test time, using the full network is like taking an average prediction from this ensemble, which is often more robust.

Early Stoping

• Choose some conditions after which to stop training

  • Example: Check the validation log-loss every 5 epoch. If it’s higher than the previous loss value, stop and use the previous model.

• Regularization penalty in cost function

• Dropout

• Early stopping

• Stochastic/ mini-batch gradient descent (to some degree)

  

Penalized cost function

A penalized cost function helps in neural network regularization by adding a penalty term to the original cost function, which discourages the model from fitting too closely to the training data. This approach helps prevent overfitting, where the neural network learns noise and random fluctuations in the training data rather than the underlying patterns.

How Penalized Cost Functions Work

A penalized cost function modifies the original cost function (often the mean squared error or cross-entropy loss) by adding a regularization term. The two most common types of regularization terms are:

1. L1 Regularization (Lasso):

- Adds the sum of the absolute values of the weights to the cost function.

- Penalized cost function:

- Encourages sparsity in the weight matrix, which can lead to feature selection (some weights become exactly zero).

2. L2 Regularization (Ridge):

- Adds the sum of the squared values of the weights to the cost function.

- Penalized cost function:

- Encourages smaller weight values, reducing the complexity of the model.

Here, Joriginal(θ) is the original loss (cost) function, λ is the regularization hyperparameter that controls the strength of the penalty, and wi represents the individual weights of the neural network.

3. Elastic Net Regularization

   The elastic net combines both L1 and L2 regularizations. It’s used when both sparsity and small weights are desired.

How Regularization Works

- Reduces Model Complexity: By penalizing large weights, the model is less likely to have high variance (overfit). Large weights often indicate that the model is trying to fit the training data too closely.

- Encourages Generalization: The added penalty encourages the model to find a simpler decision boundary or function that generalizes better to unseen data.

Pros and Cons of Penalized Cost Functions

- Pros:

- Reduces Overfitting: Helps the model generalize to new data by preventing it from becoming too complex.

- Improves Model Robustness: Especially useful in high-dimensional datasets where there is a risk of overfitting.

- Cons:

- Computational Cost: May increase the computational complexity, especially in large networks.

- Requires Hyperparameter Tuning: Choosing the right regularization strength (λ) is critical; too high may lead to underfitting, and too low may not effectively prevent overfitting.

When to Use Penalized Cost Functions

- When Dealing with High-Dimensional Data: Regularization is particularly useful when you have many features (parameters) and the risk of overfitting is high.

- When the Model is Overfitting: If your model performs well on the training data but poorly on the test data, regularization can help improve its generalization.

- When Sparse or Simpler Models are Desired: L1 regularization is useful when you want a sparse model, while L2 is preferred when you want to control the magnitude of the weights.

By using penalized cost functions, neural networks become more resistant to overfitting, thus improving their ability to generalize to unseen data.

• Have little ML expertise or no intention to train a model? Use Pre-trained APIs

• Familiar with SQL and have data in BigQuery? BigQuery ML

• Want to build custom models with your own training data with minimal coding? AutoML

• Want full control of the ML workflow? Custom training

Pre-trained APIs

• Offered as a service

  

  Natural Language API

• Entity analysis

• Automatic tagging of keywords across many documents

• Classification of documents into different categories based on key information

• Extraction of information to generate summaries bases on key entities

• The salience score for an entity provides information about the importance or centrality of that entity to the entire document text. Scores closer to 0 are less salient, while scores closer to 1.0 are highly salient.

  Sentiment Analysis

• Customer feedback

• Social network comments

• Conversations

  Syntax

Parts of speech, dependency, Lemma, Morphology, Parse label

Categories

Generative AI APIs

Vertex AI

AutoML

Custom training

Layer 2 Data and AI products

• AI Solutions: Vertical Solutions and horizontal solutions

• AI Development Predictive and Generative AI: Pre-trained APIs, BigQuery ML, AutoML, Custom training

Layer 1 Compute | Storage

• AI foundations: Cloud essentials, Compute and Storage (decoupled), Data and AI products

Layer 0 Networking and security

Google Cloud Infrastructure

Compute Resources

Compute Engine: IaaS offering, compute, storage and network infra, maximum flexibility

Google Kubernetes Engine: Containerized applications running on cloud environment rather than individual machines.

App Engine: fully managed PaaS offering, binds code to libraries, focused on application logic

Cloud functions: Code execution in response to events, serverless offering, function as a service

Cloud run: fully managed platform, lets you focus on writing code, automatically scales up and down

Processing power: TPUs

• Domain specific hardware

• Allows for higher efficiency

• Faster and more energy efficient

Storage

Unstructured data storage: Cloud Storage

Source: GCP documentation

Structured data storage options:

Source: GCP documentation

Data and AI products

Data-to-AI workflow

1. Ingestion and process (gather data from multiple sources

   

2. Storage (store data in different types of storage)

   

3. Analytics (analyze the data and visualize the results)

   

4. AI/ML(train an ML model to predict future trends or generate new content)

   AI Development

AI Solutions

BigQuery ML

• Fully managed storage facility for datasets

• Fast SQL-based analytics engine

• Capabilities to build ML model:

  • Manage the tabular data and execute ML models in one place with just a few steps.

  • Steps:

    ETL data into BigQuery → Select and preprocess features → Create the model inside BigQuery → Evaluate the performance of the trained model → Use the model to make predictions.

• Model types and ML Ops

  

Query syntax:

Create model

CREATE MODEL `your_dataset.your_model`

OPTIONS(

  model_type='LINEAR_REGRESSION',

  labels=['target_column']

) AS

SELECT

  feature_column1,

  feature_column2,

  ...

  target_column

FROM

  `your_dataset.your_table`;

2. Evaluate model:

   SELECT
   
   FROM roc_auc, accuracy, precision, recall FROM
   
     ML.EVALUATE(MODEL `your_dataset.your_model`)

3. Make predictions:

   SELECT
   
     ML.PREDICT(MODEL `your_dataset.your_model`,
   
       (feature_column1, feature_column2, ...)) AS prediction
   
   FROM
   
     `your_dataset.new_data`;

1. be socially beneficial

2. avoid creating or reinforcing unfair bias

3. be built and tested for safety

4. be accountable to people

5. incorporate privacy and design principles

6. uphold high standards of scientific excellence

7. be made available for uses that accord with these principles

#GCP #Google #ResponsibleAI

What is a Confidence Interval?

A confidence interval is a range of values, derived from a sample, that is likely to contain the true population parameter. It’s often expressed in the form:

This interval gives you a range where you expect the true value (e.g., the true population mean or proportion) to lie, with a certain level of confidence (typically 95% or 99%).

Key Terms Defined

- Population: The entire group you want to study (e.g., all voters in an election).

- Sample: A subset of the population selected for study (e.g., 100 likely voters).

- Point Estimate: A single value estimate of a population parameter (e.g., sample mean or proportion).

- Margin of Error: The range of uncertainty around the point estimate, reflecting the precision of your sample.

- Significance Level (\(\alpha\)): The probability that the true parameter is not within the confidence interval (e.g., \(\alpha = 0.05\) for a 95% confidence level).

- Z-Score (\(Z_{1-\alpha/2}\)): The critical value that defines the confidence interval’s bounds, derived from the standard normal distribution.

The Central Limit Theorem (CLT) and Confidence Intervals

The Central Limit Theorem (CLT)is a fundamental concept in statistics that states that the distribution of sample means approaches a normal distribution as the sample size increases, regardless of the population's distribution, provided the sample size is sufficiently large (usually \( n \geq 30 \)).

This is crucial for confidence intervals because it allows us to use the normal distribution to estimate how much sample means or proportions will vary around the true population mean or proportion. Thanks to the CLT, even if the original data is not normally distributed, the sample means will be, allowing us to apply the Z-score for calculating confidence intervals.

Calculating Confidence Intervals: Sample Mean

Let’s start by calculating a confidence interval for a sample mean.

Suppose you want to estimate the average height of adult women in a city. You take a random sample of 100 women and find that the average height is 65 inches with a standard deviation of 3 inches.

For a 95% confidence interval:

1. Determine the Z-Score: For 95% confidence, \( Z_{1-\alpha/2} \) is approximately 1.96.

2. Calculate the Margin of Error:

3. Determine the Confidence Interval:

This means you can be 95% confident that the true average height of all adult women in the city is between 64.412 inches and 65.588 inches.

Calculating Confidence Intervals: Sample Proportion

Now, let’s calculate a confidence interval for a sample proportion, such as the proportion of voters who intend to vote for a candidate.

Suppose in a random sample of 100 likely voters, 56 say they intend to vote for you.

For a 95% confidence interval:

1. Determine the Z-Score: Again, for 95% confidence, \( Z_{1-\alpha/2} \) is approximately 1.96.

2. Calculate the Sample Proportion:

3. Calculate the Margin of Error:

4. Determine the Confidence Interval:

This means you can be 95% confident that the true proportion of voters who intend to vote for you is between 46.29% and 65.71%.

Conclusion

Confidence intervals are powerful tools for making inferences about a population based on sample data. By understanding the key concepts and how they relate to the Central Limit Theorem, you can better appreciate the precision and reliability of your estimates. Whether you’re estimating means, proportions, or other parameters, knowing how to calculate and interpret confidence intervals will help you make more informed decisions.

Stay tuned for more posts on statistical concepts, and feel free to share your thoughts or questions in the comments below!

Step 1: Install Miniconda for windows.

There are two options to install Miniconda

a) Download and install Minconda executable file from here. Select the file corresponding to the windows platform.

b) Run the shell command on Windows command prompt.

curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Windows-x86_64.exe -o miniconda.exe
start /wait "" miniconda.exe /S
del miniconda.exe

Step 2: Setup virtual environment

1. Open Anaconda Prompt (MiniConda3). Search for anaconda at the search bar to find it.

2. Create a python virtual environment using the following command.

   conda create -n playground python=3.10

   The above command creates a virtual environment named playground. Change the python version as needed. To see where the environment is created, run conda env list.

3. Activate the virtual environment using conda activate playground

Once the environment is active, you can install python packages using pip install [package name]. Example, to install Jupyter packages, run pip install jupyter

Step 3: Setup visual studio code

1. Download and install visual studio code from here. Download the windows version.

2. Point visual studio code to a folder. Create a new folder if you don’t one already.

3. VS Code will ask to install a python extension, please follow the steps and installl the extension.

4. Create a new .py or jupyter (.ipynb) file. VS code will prompt you to install python and jupyter extensions. Install the extensions.

5. Select the kernel corresponding to the environment you have created earlier. There is a select kernel button on the VS code studio UI. When you click it, it will show you the list of virtual environments available to select.

6. Now you are ready to write your code. print(“hello world!”)

If you have found any of the steps wrong, please let me know I will be happy to update the post based on your feedback. Happy coding and may the compiler be with you!