Amazon Bedrock — reinvent 2024

Amazon Bedrock is a fully managed Generative AI Service providing you with all the capabilities (models, guardrails, tools, knowledge bases, agents etc..) required to build diverse Gen AI Applications. To learn more about Amazon Bedrock capabilities refer to the blog here — https://medium.com/@meenakshisundaram-t/amazon-bedrock-an-enterprise-generative-ai-platform-f8d62c814829

In this post, we will focus on the updates to Amazon Bedrock post re-invent 2024. We will dive deep in to how these enhancements reinforce Amazon’s vision of Bedrock as the go-to platform for businesses seeking to harness the power of generative AI. In addition to the new capabilities added to Bedrock, you can now access Bedrock using code (SDK/ API), low code (Bedrock Flows) and No Code (Bedrock Studio) making this a Gen AI platform for all Personas.

The below picture shows (marked in Orange) details out all the new advancements to Amazon Bedrock announced at reinvent 2024.

Model Hub: Amazon Bedrock model hub hosts 1st party models (models built by amazon), proprietary models (models built by partners of AWS) and Open Source / Open Weights models. With reinvent 2024, you now have access to over 170 models (and increasing)..

Bedrock hosts ~50 models on your behalf. This includes

• 1st Party Models: Models built by Amazon (Nova and Titan Models)
• Proprietary Models: Models from Anthropic, AI21, Cohere & Stability AI
• Open Source / Open Weights models: Llama Models from Meta and Mistral family of models.

Nova Models: with reinvent, AWS released Nova family of Models. Andy Jassy in his keynote referred to these models as “Any to Any Models” meaning these models will understand and generate content in any combination of all primary modalities (text, code, documents, image, video, audio, speech). Nova Lite, Micro and Pro are language models, while Canvas and Reel focusses on image and video modalities respectively. Nova models are Bedrock hosted and managed.

Bedrock Marketplace: With reinvent, Amazon Bedrock provides “Bedrock Marketplace”. You are no longer restricted to Bedrock hosted models, you can now browse this market place for partner and open source models, deploy them on Amazon Bedrock for use in just a few clicks.

Customizations: Enterprises see a need for customizing models for your domain. Bedrock provides Fine tuning and continuous pre-training as options to build your Domain trained models. The customization is managed by Amazon Bedrock and you just provide your domain dataset as input in the required format.

Distillation: with reinvent, you can now do model distillation. Small Language models (SLMs) are gaining a lot of traction as agentic ai (single agent and multi agent architectures) is taking center stage. Cost, Latency, Domain centricity are key for agentic ai applications. Here is where SLMs shine. With reinvent, you can now build your SLM with Distillation. Distillation uses a teacher model (usually a large model) to generate synthetic data, such as similar prompts or golden example responses. This data is now used to fine tune a student model (smaller model). This distilled model now provides the best price — performance trade off (performance of a large model at a small model cost) for domain centric use cases

Inference Options: Having model options is great but promoting them for production use requires you to address Customer experience and ROI. This translated to NFR requirements of cost, latency and throughput. Amazon Bedrock now provides you with configurable inference options addressing the NFR requirements for your use cases. This includes Cross regions inference, Application Inference profiles, Prompt Routers, latency optimized end points and much more. For more information on how to configure these options for your use case refer to this detailed blog — https://medium.com/@meenakshisundaram-t/amazon-bedrock-inference-options-d46d668b8ef0

Evaluation: Bedrock Evaluations is transforming from model evaluation to evaluating all aspects of Gen AI application.

RAG Evaluations: You can now run an automatic knowledge base evaluation to assess and optimize Retrieval Augmented Generation (RAG) applications. The evaluation process uses a large language model (LLM) to compute the metrics for the evaluation. With RAG evaluations, you can compare different configurations and tune your settings to get the results you need for your use case.

Agent Evaluation: The next obvious step, with the hype around agentic ai, is to evalaute agents. This is in the works and NOT released. However if you want to take a deeper dive on this, refer to the work here from awslabs — https://github.com/awslabs/agent-evaluation

Guardrails: Responsible AI is a key aspect for productionizing Gen AI applications. For a deeper dive on responsible AI using Amazon Bedrock refer to this blog post — https://medium.com/@meenakshisundaram-t/responsible-ai-lens-gen-ai-poc-to-prod-part-1-0e7efde9da83

However hallucinations remain a concern for Gen AI applications in production. Impact of hallucination amplifies for applications in regulated industries like HCM, Life sciences etc. With reinvent, Amazon Bedrock offers “Automated Reasoning” as a guardrail for detecting hallucinations in responses for RAG based applications.

Automated Reasoning: Automated reasoning is the field of computer science that attempts to provide assurance about what a system or program will do or will never do. Think about answering questions from a policy document like a W2. I intend to write a separate blog on automated reasoning in the next few weeks. If you are interested to dive in to more details, listen from Byron Cook (https://www.linkedin.com/in/byron-cook-8765205/), an industry leader in automated reasoning below

Knowledge Bases: Amazon Bedrock now offers an ability to build your knowledge base from structured data store. This addresses one of the common Text2SQL Gen AI solution pattern.

Knowledge Bases with structured data store: Support for SQL data bases is currently restricted to Amazon Redshift and Athena with more to follow later this year.

Agents: Amazon Bedrock now supports 2 new capabilities providing you to build complex agentic architectures.

Multi agent Orchestration: Ability to have multiple agents working cohesively to solve a complex workflows.

Custom Orchestration: A common issue with Agentic ai (specifically using ReACT based agents) is the multiple calls to the LLMs for thought, observation and action. This increases the overall latency and cost of operations. With custom orchestration strategy you can now control how agents handle multistep tasks, make decisions, and execute complex workflows. Define your custom orchestration logic for your agents using AWS Lambda to tailor agent’s behavior to fit your use cases.

For more information refer to this blog post — https://aws.amazon.com/blogs/machine-learning/getting-started-with-amazon-bedrock-agents-custom-orchestrator/

Prompt Management: In addition to managing prompts, versions, test results within the prompt database, Amazon Bedrock can help you optimize prompts and cache aspects of the prompts for operational excellence.

Prompt Optimization: Prompt engineer spend countless hours on optimizing the prompt for a specific model and their use case. Amazon Bedrock has made this easier with Prompt optimizer. You define your prompt and ask Bedrock to Optimize for a specific model, Bedrock will follow the Prompt Anatomy and best practices to structure the prompt. Prompt engineers can now take the optimized prompt, tune it for cost and accuracy

Prompt Caching: Another feature of Amazon Bedrock that addresses operational excellence — cost and latency in gen ai applications. Prompt caching helps in caching aspects of the prompt [think context, document, conversation history] for a time period thus reducing the overall cost on tokens and time for processing the information.

body={
        "anthropic_version": "bedrock-2023-05-31",
        "system":"Reply concisely",
        "messages": [
            {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe the best way to learn programming."
                },
                {
                    "type": "text",
                    "text": "Add additional context here for the prompt that meets the minimum token requirement for your chosen model.",
                    "cache_control": {
                        "type": "ephemeral"
                    }
                }
            ]
            }
        ],
        "max_tokens": 2048,
        "temperature": 0.5,
        "top_p": 0.8,
        "stop_sequences": [
            "stop"
        ],
        "top_k": 250
}

response = bedrock_client.invoke_model(
    body=body,
    modelId=modelId,
    accept=accept,
    contentType=contentType,
    explicitPromptCaching='enabled')

Closing thoughts: Amazon Bedrock represents a significant leap forward in the democratization of generative AI. By providing a unified platform Bedrock empowers businesses of all sizes to harness the power of Gen AI. Its seamless integration with AWS services, strong focus on security and privacy, and pay-as-you-go pricing model make it easier for you to innovate without the need for extensive AI expertise or infrastructure investments. As the AI landscape continues to evolve rapidly, Amazon Bedrock stands poised to play a crucial role in shaping the future of Gen AI-driven business solutions.