d S r SSKJr SSKJr SSKJr Sr " S S\R 5 r
" S S \R 5 r " S
S \R 5 r " S S
\R 5 r
" S S\R 5 r " S S\R 5 r " S S\R 5 r " S S\R 5 r " S S\R 5 r " S S\R 5 r " S S\R 5 r " S S \R 5 r " S S \R 5 r " S S!\R 5 r " S" S#\R 5 r " S$ S%\R 5 r " S& S'\R 5 r " S( S)\R 5 r " S* S+\R 5 r " S, S-\R 5 r " S. S/\R 5 r " S0 S1\R 5 r " S2 S3\R 5 r " S4 S5\R 5 r! " S6 S7\R 5 r" " S8 S9\R 5 r# " S: S;\R 5 r$ " S< S=\R 5 r% " S> S?\R 5 r& " S@ SA\R 5 r' " SB SC\R 5 r( " SD SE\R 5 r) " SF SG\R 5 r* " SH SI\R 5 r+\RX " \(SJSK5 \RZ " \(R\ SLSM5 \RZ " \(R\ SNSO5 gP)QzNGenerated message classes for gkerecommender version v1.
GKE Recommender API
)absolute_import)messages)encodinggkerecommenderc \ rS rSrSr\R " S\R R S9r \R " S5 r
Sr g)Amount aW Represents an amount of money in a specific currency.
Fields:
nanos: Output only. Number of nano (10^-9) units of the amount. The value
must be between -999,999,999 and +999,999,999 inclusive. If `units` is
positive, `nanos` must be positive or zero. If `units` is zero, `nanos`
can be positive, zero, or negative. If `units` is negative, `nanos` must
be negative or zero. For example $-1.75 is represented as `units`=-1 and
`nanos`=-750,000,000.
units: Output only. The whole units of the amount. For example if
`currencyCode` is `"USD"`, then 1 unit is one US dollar.
variant N) __name__
__module__ __qualname____firstlineno____doc__ _messages IntegerFieldVariantINT32nanosunits__static_attributes__r Ylib/googlecloudsdk/generated_clients/apis/gkerecommender/v1/gkerecommender_v1_messages.pyr r s7
I,=,=,C,C
D%
#%r r c \ rS rSrSr\R " SS5 r\R " SS5 r\R " S\R R S9r \R " S5 rS rg
) Cost" ab Cost for running a model deployment on a given instance type. Currently,
only USD currency code is supported.
Fields:
costPerMillionInputTokens: Optional. The cost per million input tokens.
$/input token = ($/output token) / output-to-input-cost-ratio.
costPerMillionOutputTokens: Optional. The cost per million output tokens,
calculated as: $/output token = GPU $/s / (1/output-to-input-cost-ratio
* input tokens/s + output tokens/s)
outputInputCostRatio: Optional. The output-to-input cost ratio. This
determines how the total GPU cost is split between input and output
tokens. If not provided, `4.0` is used, assuming a 4:1 output:input cost
ratio.
pricingModel: Optional. The pricing model used to calculate the cost. Can
be one of: `3-years-cud`, `1-year-cud`, `on-demand`, `spot`. If not
provided, `spot` will be used.
r r
r
r r N)r r r r r r MessageFieldcostPerMillionInputTokenscostPerMillionOutputTokens
FloatFieldr FLOAToutputInputCostRatio StringField pricingModelr r r r r r " s^ $ ( 4 4Xq A ( 5 5h B "--a9J9J9P9PQ&&q),r r c \ rS rSrSr\R " S5 r\R " SS5 r \R " S5 r
\R " SS5 r \R " S 5 r S
r
g ) FetchBenchmarkingDataRequest; a! Request message for GkeInferenceQuickstart.FetchBenchmarkingData.
Fields:
instanceType: Optional. The instance type to filter benchmarking data.
Instance types are in the format `a2-highgpu-1g`. If not provided, all
instance types for the given profile's `model_server_info` will be
returned. Use GkeInferenceQuickstart.FetchProfiles to find available
instance types.
modelServerInfo: Required. The model server configuration to get
benchmarking data for. Use GkeInferenceQuickstart.FetchProfiles to find
valid configurations.
pricingModel: Optional. The pricing model to use for the benchmarking
data. Defaults to `spot`.
servingStack: Optional. The serving stack to filter benchmarking data by,
e.g. `llm-d/0.3`. If not provided, benchmarking data for all serving
stacks that support the given model and model server will be returned.
useCase: Optional. The use case to filter benchmarking data by. If not
provided, all benchmarking data for the given profile's
`model_server_info` will be returned.
r
ModelServerInfor
r ServingStackr! r N)r r r r r r r( instanceTyper" modelServerInfor) servingStackuseCaser r r r r+ r+ ; s^ * &&q), **+B " "9a$ ?'r rX c ^ \ rS rSrSr\R " S5 r\R " SSS9rSr g) !FetchServingStackVersionsResponse a Response message for GkeInferenceQuickstart.FetchServingStackVersions.
Fields:
nextPageToken: Output only. A token which may be sent as page_token in a
subsequent `FetchServingStackVersionsResponse` call to retrieve the next
page of results. If this field is omitted or empty, then there are no
more results to return.
servingStackVersions: Output only. A list of available serving stack
versions.
r
r
Tr8 r N)
r r r r r r r( r? servingStackVersionsr r r r r^ r^ s, ''*-"..q4@r r^ c ` \ rS rSrSr\R " S5 r\R " SSSS9r Sr
g )
FetchServingStacksResponse a Response message for GkeInferenceQuickstart.FetchServingStacks.
Fields:
nextPageToken: Output only. A token which may be sent as page_token in a
subsequent `FetchServingStacksResponse` call to retrieve the next page
of results. If this field is omitted or empty, then there are no more
results to return.
servingStacks: Output only. List of available serving stacks.
r
r. r
Tr8 r N) r r r r r r r( r? r"
servingStacksr r r r rb rb s- ''*-(( TJ-r rb c \ rS rSrSrSrg)FetchUseCasesRequest z9Request message for GkeInferenceQuickstart.FetchUseCases.r N)r r r r r r r r r rf rf s Br rf c < \ rS rSrSr\R " SSSS9rSrg) FetchUseCasesResponse zResponse message for GkeInferenceQuickstart.FetchUseCases.
Fields:
workloadSpecs: Output only. The workload specifications supported by the
service.
rN r
Tr8 r N) r r r r r r r"
workloadSpecsr r r r ri ri s (( TJ-r ri c \ rS rSrSr\R " S5 r\R " S5 r\R " SS5 r
\R " SS5 r \R " S S
5 r \R " S S 5 r
\R " S
5 rSrg) GenerateOptimizedManifestRequesti a Request message for GkeInferenceQuickstart.GenerateOptimizedManifest.
Fields:
acceleratorType: Required. The accelerator type. Use
GkeInferenceQuickstart.FetchProfiles to find valid accelerators for a
given `model_server_info`.
kubernetesNamespace: Optional. The kubernetes namespace to deploy the
manifests in.
modelServerInfo: Required. The model server configuration to generate the
manifest for. Use GkeInferenceQuickstart.FetchProfiles to find valid
configurations.
performanceRequirements: Optional. The performance requirements to use for
generating Horizontal Pod Autoscaler (HPA) resources. If provided, the
manifest includes HPA resources to adjust the model server replica count
to maintain the specified targets (e.g., NTPOT, TTFT) at a P50 latency.
Cost targets are not currently supported for HPA generation. If the
specified targets are not achievable, the HPA manifest will not be
generated.
servingStack: Optional. The serving stack to use for generating the
manifest. If not provided, the latest serving stack that supports the
given model and model server will be used.
storageConfig: Optional. The storage configuration for the model. If not
provided, the model is loaded from Huggingface.
useCase: Optional. The use case of the workload. Can be one of: `advanced
costumer support`, `code completion, `text summarization`, `chatbot`,
`text generation`, `deep research`. If not provided, `chatbot` is used.
r
r
r- r rK r! r. r/
StorageConfigrL rM r N)r r r r r r r( acceleratorTypekubernetesNamespacer" r1 rU r2
storageConfigr3 r r r r rm rm s 8 ))!,/!--a0 **+ a A GkerecommenderModelServerVersionsFetchRequest object.
Fields:
model: Required. The model for which to list model server versions. Open-
source models follow the Huggingface Hub `owner/model_name` format. Use
GkeInferenceQuickstart.FetchModels to find available models.
modelServer: Required. The model server for which to list versions. Open-
source model servers use simplified, lowercase names (e.g., `vllm`). Use
GkeInferenceQuickstart.FetchModelServers to find available model
servers.
pageSize: Optional. The target number of results to return in a single
response. If not specified, a default value will be chosen by the
service. Note that the response may include a partial list and a caller
should only rely on the response's next_page_token to determine if there
are more instances left to be queried.
pageToken: Optional. The value of next_page_token received from a previous
`FetchModelServerVersionsRequest` call. Provide this to retrieve the
subsequent page in a multi-page list of results. When paginating, all
other parameters provided to `FetchModelServerVersionsRequest` must
match the call that provided the page token.
r
r
r r r! r Nr r r r r r r( rP rQ r r r rS rT r r r r rx rx > W ,
"%%%a(+
#
#Ay/@/@/F/F
G(##A&)r rx c \ rS rSrSr\R " S5 r\R " S\R R S9r \R " S5 r Sr
g) &GkerecommenderModelServersFetchRequesti[ a A GkerecommenderModelServersFetchRequest object.
Fields:
model: Required. The model for which to list model servers. Open-source
models follow the Huggingface Hub `owner/model_name` format. Use
GkeInferenceQuickstart.FetchModels to find available models.
pageSize: Optional. The target number of results to return in a single
response. If not specified, a default value will be chosen by the
service. Note that the response may include a partial list and a caller
should only rely on the response's next_page_token to determine if there
are more instances left to be queried.
pageToken: Optional. The value of next_page_token received from a previous
`FetchModelServersRequest` call. Provide this to retrieve the subsequent
page in a multi-page list of results. When paginating, all other
parameters provided to `FetchModelServersRequest` must match the call
that provided the page token.
r
r
r r r N)r r r r r r r( rP r r r rS rT r r r r r| r| [ sG $
"%
#
#Ay/@/@/F/F
G(##A&)r r| c \ rS rSrSr\R " S\R R S9r \R " S5 r Sr g) GkerecommenderModelsFetchRequestis a A GkerecommenderModelsFetchRequest object.
Fields:
pageSize: Optional. The target number of results to return in a single
response. If not specified, a default value will be chosen by the
service. Note that the response may include a partial list and a caller
should only rely on the response's next_page_token to determine if there
are more instances left to be queried.
pageToken: Optional. The value of next_page_token received from a previous
`FetchModelsRequest` call. Provide this to retrieve the subsequent page
in a multi-page list of results. When paginating, all other parameters
provided to `FetchModelsRequest` must match the call that provided the
page token.
r
r r
r N)
r r r r r r r r r rS r( rT r r r r r~ r~ s s7
#
#Ay/@/@/F/F
G(##A&)r r~ c \ rS rSrSr\R " S5 r\R " S5 r\R " S\R R S9r \R " S5 r
\R " S5 rS rg
) .GkerecommenderServingStackVersionsFetchRequesti a A GkerecommenderServingStackVersionsFetchRequest object.
Fields:
model: Optional. The model to filter serving stack versions by.
modelServer: Optional. The model server to filter serving stack versions
by.
pageSize: Optional. The target number of results to return in a single
response. If not specified, a default value will be chosen by the
service. Note that the response may include a partial list and a caller
should only rely on the response's next_page_token to determine if there
are more instances left to be queried.
pageToken: Optional. The value of next_page_token received from a previous
`FetchServingStackVersionsRequest` call. Provide this to retrieve the
subsequent page in a multi-page list of results. When paginating, all
other parameters provided to `FetchServingStackVersionsRequest` must
match the call that provided the page token.
servingStack: Required. The serving stack to list versions for.
r
r
r r r! r/ r N)r r r r r r r( rP rQ r r r rS rT r2 r r r r r r sg &
"%%%a(+
#
#Ay/@/@/F/F
G(##A&)&&q),r r c \ rS rSrSr\R " S5 r\R " S5 r\R " S\R R S9r \R " S5 r
Srg )
'GkerecommenderServingStacksFetchRequesti a A GkerecommenderServingStacksFetchRequest object.
Fields:
model: Optional. The model for which to list serving stacks. Open-source
models follow the Huggingface Hub `owner/model_name` format. Use
GkeInferenceQuickstart.FetchModels to find available models.
modelServer: Optional. The model server for which to list serving stacks.
Open-source model servers use simplified, lowercase names (e.g.,
`vllm`). Use GkeInferenceQuickstart.FetchModelServers to find available
model servers.
pageSize: Optional. The target number of results to return in a single
response. If not specified, a default value will be chosen by the
service. Note that the response may include a partial list and a caller
should only rely on the response's next_page_token to determine if there
are more instances left to be queried.
pageToken: Optional. The value of next_page_token received from a previous
`FetchServingStacksRequest` call. Provide this to retrieve the
subsequent page in a multi-page list of results. When paginating, all
other parameters provided to `FetchServingStacksRequest` must match the
call that provided the page token.
r
r
r r r! r Nry r r r r r rz r r c \ rS rSrSr\R " S5 r\R " S5 r\R " S5 r Sr
g)rt i zA Kubernetes manifest.
Fields:
apiVersion: Output only. Kubernetes API version.
content: Output only. YAML content.
kind: Output only. Kubernetes resource kind.
r
r
r r N) r r r r r r r(
apiVersioncontentkindr r r r rt rt s9 $$Q'* ! !! $' q !$r rt c \ rS rSrSr\R " S\R R S9r \R " S\R R S9r
Sr g)MillisecondRangei zRepresents a range of latency values in milliseconds.
Fields:
max: Output only. The maximum value of the range.
min: Output only. The minimum value of the range.
r
r r
r N r r r r r r r r r maxminr r r r r r E q)*;*;*A*AB# q)*;*;*A*AB#r r c h \ rS rSrSr\R " S\R R S9r \R " S\R R S9r
\R " S\R R S9r \R " S\R R S9r \R " S\R R S9r
\R " S \R R S9r\R " S
\R R S9r\R " S \R R S9rS rg
) ModelInfoi a Model information for a model deployment.
Fields:
attentionHeadsCount: Output only. The number of attention heads in the
model.
headDimensions: Output only. The number of dimensions in the model's
hidden layers.
hiddenLayersCount: Output only. The number of hidden layers in the model.
kvCacheSizePerToken: Output only. The size of the key value cache per
token.
kvHeadsCount: Output only. The number of key value heads in the model.
maxContextLength: Output only. The maximum context length of the model.
modelSizeGb: Output only. The size of the model in gigabytes.
parametersBillionsCount: Output only. The number of parameters in
billions.
r
r r
r r! r/ rL rM rO r N)r r r r r r r r r attentionHeadsCountheadDimensionshiddenLayersCountkvCacheSizePerToken kvHeadsCountmaxContextLengthr% r& modelSizeGbparametersBillionsCountr r r r r r s " "..q):K:K:Q:QR ))!Y5F5F5L5LM. ,,Q 8I8I8O8OP!..q):K:K:Q:QR''93D3D3J3JK, ++Ay7H7H7N7NO$$Q 0A0A0G0GH+ % 0 0IO>O>U>U V$11!Y=N=N=T=TUr rK c B \ rS rSrSr\R " SSSS9r\R " S\R R S9r \R " S \R R S9r \R " S
\R R S9r
\R " S \R R S9r\R " S \R R S9r\R " S
\R R S9r\R " S\R R S9rSrg)PerformanceStatsiB a Performance statistics for a model deployment.
Fields:
cost: Output only. The cost of running the model deployment.
inputTokensPerSecond: Output only. The input tokens per second. This is
the throughput measured as the number of input tokens processed by the
server divided by the elapsed time in seconds.
itlMilliseconds: Output only. The inter-token latency (ITL) in
milliseconds. This is the latency between consecutive tokens being
generated.
ntpotMilliseconds: Output only. The Normalized Time Per Output Token
(NTPOT) in milliseconds. This is the request latency normalized by the
number of output tokens, measured as request_latency /
total_output_tokens.
outputTokensPerSecond: Output only. The number of output tokens per
second. This is the throughput measured as
total_output_tokens_generated_by_server / elapsed_time_in_seconds.
queriesPerSecond: Output only. The number of queries per second. Note:
This metric can vary widely based on context length and may not be a
reliable measure of LLM throughput.
totalTokensPerSecond: Output only. The total tokens per second. This is
the throughput measured as the sum of input and output tokens processed
by the server divided by the elapsed time in seconds.
ttftMilliseconds: Output only. The Time To First Token (TTFT) in
milliseconds. This is the time it takes to generate the first token for
a request.
r r
Tr8 r
r r r! r/ rL rM rO r N)r r r r r r r" costr r r inputTokensPerSeconditlMillisecondsntpotMillisecondsoutputTokensPerSecondr% r& queriesPerSecondtotalTokensPerSecondttftMillisecondsr r r r r r B s 8
D 9$"//9;L;L;R;RS **1i6G6G6M6MN/ ,,Q 8I8I8O8OP#00IDQ(( !<-'':,%%a(+'':,r r7 c b \ rS rSrSr\R " S\R R S9r Sr
g)r i zResources used by a model deployment.
Fields:
acceleratorCount: Output only. The number of accelerators (e.g., GPUs or
TPUs) used by the model deployment on the Kubernetes node.
r
r r N) r r r r r r r r r acceleratorCountr r r r r r s( ++Ay7H7H7N7NOr r c ` \ rS rSrSr\R " S5 r\R " S5 rSr g)r. i zServing stack information.
Fields:
name: Required. The name of the serving stack.
version: Optional. The version of the serving stack.
r
r
r N)
r r r r r r r( nameversionr r r r r. r. s)
q !$ ! !! $'r r. c \ rS rSrSr " S S\R 5 r " S S\R 5 r\R " SS5 r
\R " S5 r \R " SS S
S 9r
\R " S 5 r\R " S
5 r\R " S5 r\R " S5 r\R$ " SSS 9r\R " S5 r\R " S5 r\R " S5 r\R " S5 rSrg)StandardQueryParametersi a Query parameters accepted by all methods.
Enums:
FXgafvValueValuesEnum: V1 error format.
AltValueValuesEnum: Data format for response.
Fields:
f__xgafv: V1 error format.
access_token: OAuth access token.
alt: Data format for response.
callback: JSONP
fields: Selector specifying which fields to include in a partial response.
key: API key. Your API key identifies your project and provides you with
API access, quota, and reports. Required unless you provide an OAuth 2.0
token.
oauth_token: OAuth 2.0 token for the current user.
prettyPrint: Returns response with indentations and line breaks.
quotaUser: Available to use for quota purposes for server-side
applications. Can be any arbitrary string assigned to a user, but should
not exceed 40 characters.
trace: A tracing token of the form "token:" to include in api
requests.
uploadType: Legacy upload protocol for media (e.g. "media", "multipart").
upload_protocol: Upload protocol for media (e.g. "raw", "multipart").
c $ \ rS rSrSrSrSrSrSrg)*StandardQueryParameters.AltValueValuesEnumi zData format for response.
Values:
json: Responses with Content-Type of application/json
media: Media download with context-dependent Content-Type
proto: Responses with Content-Type of application/x-protobuf
r r
r
r N) r r r r r jsonmediaprotor r r r AltValueValuesEnumr s
D
E
Er r c \ rS rSrSrSrSrSrg)-StandardQueryParameters.FXgafvValueValuesEnumi zFV1 error format.
Values:
_1: v1 error format
_2: v2 error format
r r
r N)r r r r r _1_2r r r r FXgafvValueValuesEnumr s
B
Br r r
r
r r )defaultr! r/ rL rM rO Tr
r N)r r r r r r Enumr r EnumFieldf__xgafvr( access_tokenaltcallbackfieldskey oauth_token BooleanField prettyPrint quotaUsertrace
uploadTypeupload_protocolr r r r r r s 4
9>>
inn
!8!
<(&&q), 0!VD#
"
"1
%( #& a #%%a(+&&q$7+##A&)
#%$$R(* ))"-/r r c ` \ rS rSrSr\R " S5 r\R " S5 rSr g)rn i am Storage configuration for a model deployment.
Fields:
modelBucketUri: Optional. The Google Cloud Storage bucket URI to load the
model from. This URI must point to the directory containing the model's
config file (`config.json`) and model weights. A tuned GCSFuse setup can
improve LLM Pod startup time by more than 7x. Expected format: `gs:///`.
xlaCacheBucketUri: Optional. The URI for the GCS bucket containing the XLA
compilation cache. If using TPUs, the XLA cache will be written to the
same path as `model_bucket_uri`. This can speed up vLLM model
preparation for repeated deployments.
r
r
r N)
r r r r r r r( modelBucketUrixlaCacheBucketUrir r r r rn rn s* ((+. ++A.r rn c \ rS rSrSr\R " S\R R S9r \R " S\R R S9r
Sr g)r i zRepresents a range of throughput values in tokens per second.
Fields:
max: Output only. The maximum value of the range.
min: Output only. The minimum value of the range.
r
r r
r Nr r r r r r r r r c \ rS rSrSr\R " S\R R S9r \R " S\R R S9r
\R " S5 r Sr
g) rN i a Workload specification for a workload.
Fields:
averageInputLength: Optional. The average input length of the workload.
Only works alongside average output length.
averageOutputLength: Optional. The average output length of the workload.
Only works alongside average input length.
useCase: Optional. The use case of the workload. Can be one of: `advanced
costumer support`, `code completion, `text summarization`, `chatbot`,
`text generation`, `deep research`.
r
r r
r r N)r r r r r r r r r averageInputLengthaverageOutputLengthr( r3 r r r r rN rN sW
!--a9J9J9P9PQ!..q):K:K:Q:QR ! !! $'r rN r z$.xgafvr 1r 2N)/r
__future__r apitools.base.protorpcliter r apitools.base.pyr packageMessager r r+ r5 r< rA rE rI rX r^ rb rf ri rm rs rx r| r~ r r rt r r r- rZ rK r r7 r r. r rn r rN AddCustomJsonFieldMappingAddCustomJsonEnumMappingr r r r r s ' < % $Y
$$*9 *2%9#4#4 %:@I$5$5 @
+y'8'8
+
+ 1 1
+
+)++
+ +;9,, +;\@I - - @*
A (9(9
A K!2!2 K C9,, CKI - - K#%y'8'8 #%L- (9(9 -('I4E4E ':'Y->-> '0'y'8'8 '(*Y5F5F *6'i.?.? ': "** " Cy(( CU !! U80i'' 0* PI%% P %9$$ %<.i / / <.~/I%% /$ C9,, C%9$$ %$ " "Z4 ! !114> ! !114>r