Paul Borrego

1.3

generative AI application lifecycle

  • Defining use case
  • Selecting a foundation model
  • Improving performance of FM
  • Evaluating performance of FM
  • Deployment and impact

Use Case

Parts of use case

  • Name:
  • Description
  • Actors - Entities or stakeholders
  • Preconditions - What must be true before starting
  • Basic flow - Basic description of the actions and interactions that occur when the use case is completed from start to finish
  • Alternative flows - Extensions that might be used if things go wrong
  • Postconditions - What must be true after a successful completion of the use case
  • Business rules - Any business policies constraints or regulations
  • Nonfunctional requirements - Performance, security, or other cases needed to be relevant
  • Assumptions - Assumptions about system, environment, or context that are ncessary
  • Notes
    Reasons to use gen AI:
  • Cost savings
  • time savings
  • Quality improvement
  • Customer satisfaction
  • Productivity gains

Selecting an FM

Considerations:

  • Cost
  • Modality (Text generation, image generation, audio...)
  • Latency
  • Multi-linugal support
  • Model size
  • Model complexity
  • ustomization
  • I/O length
  • Integration

Improving performance

  • Prompt engineering
    • Design: Craft clear and context-rich prompts that effectively communicate the desired task or output of the model
    • Augmentation: Add additional or constraints such as examples or task-specific instructions
    • Tuning: Iterate the prompt and test against to see what is effective
    • Ensembling: Combining multiple prompts or gen strategies to improve robustness or quality
  • RAG (Retrieval-Augmented Generation)
    • Retrieval system the retrieves relevant information from knowledge bases, web pages, or other text based sources which will give the AI a strong context
    • Used for building intelligent question-answering systems
    • Expands knowledge bases
    • Generate higher-quality content
  • Agents
    • By simplifying tasks and coordinating then more specific AI agents can do more efficient work and have a higher transparency
    • Can run multiple tasks at once
  • Fine Tuning
    • Data Curation: By having higher quality data and culling low value data the AI will improve
    • Labeling
    • Governance and compliance: Ensure data is pointed towards your specific proffesion
    • Representative and Bias Checking
    • Feedback integration: For methods like RLHF, incorporating user or expert feedback directly into the training process is crucial.

Evaluating an FM

  • Human Evaluation
  • Benchmark datasets
  • Automated metrics
  • Recall-Oriented Understudy for Gisting Evaluation (ROUGE) - Tests to see how accurate summaries or translations are
  • BERTScore is a metric that evaluates the semantic similarity between a generated text and one or more reference texts

Deploying the Application

  • Cost
  • Regions
  • Quotas
  • Security