Responsible AI Lens: Gen AI PoC to Prod— Part 1
In this blog we will dive in to details on how Amazon Bedrock helps you build and deploy Gen AI applications responsibly.
Moving generative AI applications from PoC to production requires you to balance / trade-off across accuracy, cost, and latency. The interplay between creates an optimization challenge. Improving one aspect might require compromising another — for instance, reducing latency through model compression might affect accuracy, while maintaining high accuracy (say with larger / powerful) models might increases both costs and latency.
One way to address cost and latency for models hosted on Amazon Bedrock is to leverage diverse inference strategies. Refer to this blog to understand inference strategies. In addition, there are Benchmark frameworks available. One such benchmarking tool is FMBench a home grown solution built by my colleague, Amit Arora used by considerable number of strategic customers of AWS.
We will focus on one of the pillars — Accuracy / Precision. Multiple dimensions exists when dealing with Accuracy / Precision. This includes but not limited to
- Type of Gen AI Application: Accuracy of LLM generation when using just a LLM in your Gen AI application, Retrieval accuracy and Retrieval and Generation accuracy in the case of RAG application and adding reasoning accuracy for Agentic workflows and applications
- Gen AI app lifecycle: Evaluations done prior to deployment of the application (offline / batch evaluation) and real time evaluations (evaluating before sending the response)
- Functional and Responsible AI: Trusted response from the Gen AI application and the Responsible AI dimension (Fairness, Bias, Ethics, Privacy, security and much more
Let us narrow our focus down to the Responsible AI for Gen AI applications
Layered model for Responsible AI
Built-in Guardrails: Most model providers have some sort of built in guardrails for responsible AI. For example, Anthropic Claude does Constitutional AI to reduce harmlessness from model usage — https://www.anthropic.com/research/constitutional-ai-harmlessness-from-ai-feedback
Prompt driven Guardrails: the second layer of guardrails are enforced as a best practice while prompting the model (prompt engineering). AWS provides best practices on this — https://docs.aws.amazon.com/prescriptive-guidance/latest/llm-prompt-engineering-best-practices/introduction.html
Custom Guardrails: In addition to the above, it is important to have set of custom guardrails. Amazon Bedrock provides “Guardrails capabilities” to define and customize it for your use case requirements.
Amazon Bedrock Guardrails:
Amazon Bedrock help you enforce guardrails both at the prompt and at the response level. Some of the key aspects of Amazon Bedrock include the following.
- ability to define denied topics, apply content filters to block harmful responses, and redact personally identifiable information (PII) from outputs, context relevance and reasoning your LLM response
- implementation customizable safeguards tailored to your specific use cases and responsible AI policies. All configurations are optional
- Bedrock guardrails are MODEL-AGNOSTIC. you can use the guardrails irrespective of the model you use in your gen ai applications. When using guardrails on Bedrock models, you can add the guardrail directly in the invoke_model call. When using it on models hosted outside of Bedrock, you should use “apply_guardrail” api.
Now we will jump in to code snippets on creating the guardrail and using the guardrail on models hosted on Amazon Bedrock and models hosted outside of Bedrock.
Create the Guardrail
We will now create a simple guardrail that does the following. Guardrail configurations are as simple as a json structure with sections
- any discussions related to Politics.
- “content filets” to filter any discussions related to violence, hate, misconduct
- word policy filters to filter out profanity.
- A custom message to display when the guardrails intervene
response = bedrock_client.create_guardrail(
name='political_guardrail',
description='Prevents the model from discussing anything related to politcs.',
topicPolicyConfig={
'topicsConfig': [
{
'name': 'Politics',
'definition': 'Discussions related to Politics, Elections, Political figures',
'examples': [
'Who is Joe Biden',
],
'type': 'DENY'
}
]
},
contentPolicyConfig={
'filtersConfig': [
{
'type': 'SEXUAL',
'inputStrength': 'HIGH',
'outputStrength': 'HIGH'
},
{
'type': 'VIOLENCE',
'inputStrength': 'HIGH',
'outputStrength': 'HIGH'
},
{
'type': 'HATE',
'inputStrength': 'HIGH',
'outputStrength': 'HIGH'
},
{
'type': 'INSULTS',
'inputStrength': 'HIGH',
'outputStrength': 'HIGH'
},
{
'type': 'MISCONDUCT',
'inputStrength': 'HIGH',
'outputStrength': 'HIGH'
},
{
'type': 'PROMPT_ATTACK',
'inputStrength': 'HIGH',
'outputStrength': 'NONE'
}
]
},
wordPolicyConfig={
'managedWordListsConfig': [
{
'type': 'PROFANITY'
}
]
},
blockedInputMessaging='I apologize. I cannot discuss Politics',
blockedOutputsMessaging='I apologize I cannot discuss Politics',
}You can now use the Guardrail ID and Guardrail version created for the above guardrail for the model invocation
Using Amazon Guardrails for Amazon Bedrock Models:
In the below code sample, we will use the created guardrail to invoke a model hosted on Amazon Bedrock. The guardrail id and guardrail version are passed on as a parameter in the invoke_model command. you can chose to use Bedrock converse api.
## boto3 client
runtime_client = boto3.client('bedrock-runtime')
## guardrail information
guardrail_id = '7l2cg7arccsk'
guardrail_version = 'DRAFT'
## model information
modelID = 'anthropic.claude-3-haiku-20240307-v1:0'
## payload information
payload = {
"modelId": modelID,
"contentType": "application/json",
"accept": "application/json",
"body": {
"anthropic_version": "bedrock-2023-05-31",
"max_tokens": 1000,
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": prompt
}
]
}
]
}
}
body_bytes = json.dumps(payload['body']).encode('utf-8')
# Invoke the model
response = runtime_client.invoke_model(
body = body_bytes,
contentType = payload['contentType'],
accept = payload['accept'],
modelId = payload['modelId'],
guardrailIdentifier = guardrail_id,
guardrailVersion =guardrail_version,
trace = "ENABLED"
)
# Print the response
response_body = response['body'].read().decode('utf-8')
print(json.dumps(json.loads(response_body), indent=2))Using Amazon Guardrails for Models NOT hosted in Amazon Bedrock
Model: We will use the model hosted on Amazon SageMaker for this. You can log in to Amazon SageMaker Studio -> Jumpstart -> Llama 3.1 model and deploy the model as an end point
Guardrail application: For using the guardrail on a model that is now hosted on Amazon SageMaker, you will use the “apply_guardrail” api for both the prompt and the response.
def perform_guardrail_checks(text, source, guardrail_id, guardrail_version):
input_content = [
{
"text": {
"text": text,
}
}]
response = runtime_client.apply_guardrail(
guardrailIdentifier=guardrail_id,
guardrailVersion=guardrail_version,
source=source,
content=text
)
if response['action'] == 'GUARDRAIL_INTERVENED':
is_blocked = is_policy_assessement_blocked(response['assessments'])
#alternate_text = ' '.join([output['text'] for output in response['output']])
print(is_blocked)
print(response['assessments'])
print(response['outputs'])
return is_blocked, response
else:
# Return the default response in case of no guardrail intervention
return False, response
## Call guardrail for prompts
prompt = 'Who won the 2024 election?'
is_blocked, guardrail_response = perform_guardrail_checks(prompt, "INPUT", guardrail_id, guardrail_version)
## If successful invoke model hosted outside Amazon bedrock (in this case, SageMaker)
from sagemaker.predictor import retrieve_default
predictor = retrieve_default(endpoint_name)
payload = {
"inputs": prompt,
"parameters": {
"max_new_tokens": 256,
"temperature": 0.0,
"stop": "<|eot_id|>"
}
}
response = predictor.predict(payload)
completions = response.get('generated_text', '')
## call guardrail checks for response
perform_guardrail_checks(completions, "OUTPUT" ,guardrail_id, guardrail_version)
Using Amazon Bedrock Guardrails with Langchain
Amazon Bedrock supports open source framework. In the below code snippet, you can use Amazon Bedrock Guardrails with Langchain.
REGION_NAME = "us-east-1"
MODEL_ID = "anthropic.claude-3-sonnet-20240229-v1:0"
## Guardrail information
guardrail_id = '7l2cg7arccsk'
guardrail_version = 'DRAFT'
## prompt
prompt_template = ChatPromptTemplate.from_messages([
("system", "You are a helpful assistant."
"You answer the questions based on your knowledge."
"Return only the answer without any prefix or suffix."),
("human", "Question : ```{imput}```")
])
## pointer to the bedrock model
llm = ChatBedrock(
model_id=MODEL_ID,
streaming=True,
model_kwargs={"max_tokens": 1000,
"temperature": 0.7,
"top_p": 0.9,
},
guardrails={
'guardrailIdentifier': guardrail_id,
'guardrailVersion': guardrail_version,
'trace': True
},
)
## langchain chain
chain = prompt_template | llm | StrOutputParser()
prompt = 'Who won the 2024 elections?'
result = chain.invoke(input = prompt)
print(result)Using Amazon Bedrock Guardrails with streaming output:
Amazon Bedrock guardrails supports streaming output. Note that applying guardrails on a streaming output, you should consider buffering the output to a text unit size. This will help in keeping up with the latency as well applying guardrails meaningfully to a chunk of context.
In the below code snippet, the buffer is set at 1000 characters/ tokens..
TEXT_UNIT = 1000 # characters
response = runtime_client.converse_stream(
modelId=modelID,
messages=messages,
system=system_prompts
)
stream = response.get('stream')
full_text = ""
buffer_text = ""
applied_guardrails = []
if stream:
for event in stream:
if 'messageStart' in event:
print(f"\nRole: {event['messageStart']['role']}")
if 'contentBlockDelta' in event:
new_text = event['contentBlockDelta']['delta']['text']
if len(buffer_text + new_text) > TEXT_UNIT:
is_blocked, alt_text, guardrail_response = apply_guardrail(buffer_text, "OUTPUT", guardrail_id, guardrail_version)
if is_blocked:
event['messageStop'] = {
'stopReason': guardrail_response['action'],
'output': alt_text,
'assessments': guardrail_response['assessments'],
}
full_text = alt_text
else:
full_text += alt_text
print(alt_text, end="")
applied_guardrails.append(guardrail_response)
buffer_text = new_text
else:
buffer_text += new_textClosing Thoughts:
In this blog, we looked at the challenges for taking a gen ai application to production from the lens of reponsible ai. Futher we looked at how you use Amazon Bedrock guardails for guarding your prompts and responses.
In part 2, we will dive deeper in to guardrails fro RAG application from context relevance to answer relevance and finally in part 3 we will jump in to fascinating area of Automated Reasoning and its impact on Reponsible AI.
