Amazon Bedrock now provides access to Meta’s Llama 2 Chat 13B model

Immediately, we’re asserting the supply of Meta’s Llama 2 Chat 13B massive language mannequin (LLM) on Amazon Bedrock. With this launch, Amazon Bedrock turns into the primary public cloud service to supply a totally managed API for Llama 2, Meta’s next-generation LLM. Now, organizations of all sizes can entry Llama 2 Chat fashions on Amazon Bedrock with out having to handle the underlying infrastructure. This can be a step change in accessibility.

Amazon Bedrock is a totally managed service that gives a selection of high-performing basis fashions (FMs) from main AI corporations, together with AI21 Labs, Anthropic, Cohere, Stability AI, Amazon, and now Meta, together with a broad set of capabilities to construct generative AI purposes, simplifying the event whereas sustaining privateness and safety. You’ll be able to learn extra about Amazon Bedrock in Antje’s put up right here.

Llama 2 is a household of publicly out there LLMs by Meta. The Llama 2 base mannequin was pre-trained on 2 trillion tokens from on-line public information sources. In keeping with Meta, the coaching of Llama 2 13B consumed 184,320 GPU/hour. That’s the equal of 21.04 years of a single GPU, not accounting for bissextile years.

Constructed on high of the bottom mannequin, the Llama 2 Chat mannequin is optimized for dialog use instances. It’s fine-tuned with over 1 million human annotations (a method referred to as reinforcement studying from human suggestions or RLHF) and has undergone testing by Meta to establish efficiency gaps and mitigate doubtlessly problematic responses in chat use instances, comparable to offensive or inappropriate responses.

To advertise a accountable, collaborative AI innovation ecosystem, Meta established a spread of sources for all who use Llama 2: people, creators, builders, researchers, lecturers, and companies of any dimension. Specifically, I just like the Meta Accountable Use Information, a useful resource for builders that gives finest practices and concerns for constructing merchandise powered by LLMs in a accountable method, protecting varied levels of growth from inception to deployment. This information suits properly within the set of AWS instruments and sources to construct AI responsibly.

Now you can combine the LLama 2 Chat mannequin in your purposes written in any programming language by calling the Amazon Bedrock API or utilizing the AWS SDKs or the AWS Command Line Interface (AWS CLI).

Llama 2 Chat in motion
These of you who learn the AWS Information weblog recurrently know we like to indicate you the applied sciences we write about. So let’s write code to work together with Llama2.

I used to be fortunate sufficient to speak on the AWS UG Perú Conf a couple of weeks in the past. Jeff and Marcia had been there too. Jeff opened the convention with an inspiring speak about generative AI, and he used a wall of generated pictures of llamas, the emblematic animal from Perú. So what higher topic to speak about with Llama 2 Chat than llamas?

(And earlier than writing code, I can’t resist sharing two pictures of llamas I took throughout my go to to Machu Picchu)

To get began with a brand new mannequin on Bedrock, I first navigate to Amazon Bedrock on the console. I choose Mannequin entry on the underside left pane, then choose the Edit button on the highest proper facet, and allow entry to the Llama 2 Chat mannequin.

Within the left navigation bar, underneath Playgrounds, I choose Chat to work together with the mannequin with out writing any code.

Now that I do know I can entry the mannequin, I open a code editor on my laptop computer. I assume you will have the AWS Command Line Interface (AWS CLI) configured, which can enable the AWS SDK to find your AWS credentials. I exploit Python for this demo, however I wish to present that Bedrock might be referred to as from any language. I additionally share a public gist with the identical code pattern written within the Swift programming language.

Returning to Python, I first run the ListFoundationModels API name to find the modelId for Llama 2 Chat 13B.

import boto3

bedrock = boto3.consumer(service_name="bedrock", region_name="us-east-1")
listModels = bedrock.list_foundation_models(byProvider="meta")
print("n".be part of(record(map(lambda x: f"{x['modelName']} : { x['modelId'] }", listModels['modelSummaries']))))

Operating this code produces the record:

Llama 2 Chat 13B : meta.llama2-13b-chat-v1

I choose meta-llama2-chat-13b mannequin ID and write the code to ship a immediate to the LLama 2 Chat 13B mannequin.

import boto3
import json

llamaModelId = 'meta.llama2-13b-chat-v1' 
immediate = "What's the distinction between a llama and an alpaca?"

llamaPayload = json.dumps({ 
	'immediate': immediate,
    'max_gen_len': 512,
	'top_p': 0.9,
	'temperature': 0.2
})

bedrock_runtime = boto3.consumer(
    service_name="bedrock-runtime", 
    region_name="us-east-1"
)
response = bedrock_runtime.invoke_model(
    physique=llamaPayload, 
    modelId=llamaModelId, 
    settle for="software/json", 
    contentType="software/json"
)
physique = response.get('physique').learn().decode('utf-8')
response_body = json.hundreds(physique)
print(response_body['generation'].strip())

The response is printed:

Llamas and alpacas are each members of the camelid household, 
however they're completely different species with distinct bodily and behavioral traits. 
Listed here are among the primary variations: (...)

I redacted the reply for brevity.

You’ll be able to tune the completely different request parameters to see how they affect the response.

Optimizing your prompts
To get the perfect outcomes out of the Llama 2 Chat mannequin, you may experiment with a few immediate engineering strategies, comparable to utilizing roles or chain-of-thought prompts.

Utilizing roles, you inform the mannequin how you can behave by giving it a job and directions earlier than asking your query. For instance:

Immediate: You're a instructor. You clarify issues for six-year previous youngsters.
What's the distinction between a llama and an alpaca?

Reply: Nice query! Each llamas and alpacas are animals that stay in South America, they usually look a bit bit comparable. However there are some variations that we will speak about.
First, let's speak about their fur. Llamas have longer fur than alpacas. It is like they've an enormous, fluffy coat that helps them keep heat within the chilly climate. Alpacas, alternatively, have shorter fur that is a bit bit smoother.
One other distinction is their ears. Llamas have banana-shaped ears which might be actually massive, whereas alpacas have small, triangular ears.
Lastly, let's speak about their dimension. Llamas are typically greater than alpacas. They will develop as much as 6 ft tall and weigh lots, whereas alpacas are often smaller, solely rising as much as 3 ft tall.
So, these are among the primary variations between llamas and alpacas. They're each actually cool animals, and it is enjoyable to find out about them!

With chain-of-thought prompts, you ask the mannequin to clarify every step that led to the reply. For instance:

Immediate: You're a a really clever bot with distinctive important pondering.
I went to the market and acquired 10 apples. I gave 2 apples to your good friend and a couple of to the helper. I then went and acquired 5 extra apples and ate 1. What number of apples did I stay with?
Let's assume step-by-step.

Reply:
1. You went to the market and acquired 10 apples.
2. You gave 2 apples to your good friend.
3. You gave 2 apples to the helper.
4. You went again to the market and acquired 5 extra apples.
5. You ate 1 apple.
Now, let's depend what number of apples you will have:
10 apples (from the primary journey to the market) - 2 apples (given to your good friend) - 2 apples (given to the helper) = 6 apples
6 apples + 5 apples (from the second journey to the market) = 11 apples
11 apples - 1 apple (that you just ate) = 10 apples
Subsequently, you stay with 10 apples.
I hope this helps! Do you will have any extra questions or would you prefer to play once more?

Meta shared extra examples in part 4.2 of their attending to know Llama pocket book.

Availability
The Llama 2 Chat mannequin is offered right this moment for all AWS clients in two of the AWS Areas the place Bedrock is offered: US East (N. Virginia) and US West (Oregon).

You’ll be charged for mannequin inference. You’ll be able to select to be charged on a pay-as-you-go foundation, with no upfront or recurring charges; AWS costs per processed enter and output tokens. Or you may provision adequate throughput to fulfill your software’s efficiency necessities in change for a time-based time period dedication. The Bedrock pricing web page has the small print.

With this info, you’re ready to start out utilizing Amazon Bedrock and the Llama 2 Chat mannequin in your purposes.

Go construct!

— seb