qSrSSKJr SSKJr SSKJr SSKrSSKrSSKrSSKrSSK r SSK J r SSK J r SSKJr SS KJr SS KJr SS KJr SSKJ r SS KJr SS KJr SSKJr SSKJr SSKrSrSrSrSr Sr!Sr"S#Sjr#"SS5r$"SS5r%"SS5r&"SS5r'"SS 5r("S!S"5r)g)$z4Helpers for interacting with the Cloud Dataflow API.)absolute_import)division)unicode_literalsN)encoding) exceptions)cloudbuild_util) storage_api) storage_util)apis) submit_util)log) properties)filesdataflowv1b3z us-central1c@[R"[[5$N)r GetMessagesModuleDATAFLOW_API_NAMEDATAFLOW_API_VERSION+lib/googlecloudsdk/api_lib/dataflow/apis.pyrr.   13G HHrc@[R"[[5$r)r GetClientInstancerrrrrrr2rrcd[RRRR SS9$)NT)required)rVALUEScoreprojectGetrrr GetProjectr#6s(      ' ' + +T + ::rc(U(dSUS3$SUS3$)z/Returns full the image path of the given image.zgcr.io/dataflow-templates-base/z-template-launcher-base:latestz)-template-launcher-base-distroless:latestr)image is_distrolesss r_GetBaseImagePathr':s)  )%0NO +5'1Z [[rc@\rSrSrSr\"5R r\"5Rr \"5Rr \"5Rr \S5r\S Sj5r\S Sj5r\SSj5r\SSj5r\SS j5r\SS j5rS rg)JobsCz'The Jobs set of Dataflow API functions.c*[5R$rrprojects_locations_jobsrrr GetServiceJobs.GetServiceM   6 66rNc"U=(d [5nU=(d [n[5RXXS9n[R 5R U5$![Ran[R"U5eSnAff=f)a3Calls the Dataflow Jobs.Get method. Args: job_id: Identifies a single job. project_id: The project which owns the job. region_id: The regional endpoint where the job lives. view: (DataflowProjectsJobsGetRequest.ViewValueValuesEnum) Level of information requested in response. Returns: (Job) )jobIdlocation projectIdviewN) r#DATAFLOW_API_DEFAULT_REGIONr'DataflowProjectsLocationsJobsGetRequestr)r.r"apitools_exceptions HttpErrorr HttpException)job_id project_id region_idr5requesterrors rr"Jobs.GetQs~+z|J88I!IIJJKG, __  " "7 ++  ( (,  $ $U ++,"AB3B  BcHU=(d [5nU=(d [nSnU(aL[5RR [5RRR SSS9/S9n[5RU[5RR RS9n[5RXX%S9n[R5RU5$![Ran[R"U5eSnAff=f)zCancels a job by calling the Jobs.Update method. Args: job_id: Identifies a single job. force: True to forcibly cancel the job. project_id: The project which owns the job. region_id: The regional endpoint where the job lives. Returns: (Job) Nforce_cancel_jobtruekeyvalueadditionalProperties)labelsrequestedStater2r3r4job)r#r6rJob LabelsValueAdditionalPropertyRequestedStateValueValuesEnumJOB_STATE_CANCELLED*DataflowProjectsLocationsJobsUpdateRequestr)r.Updater8r9rr:)r;forcer<r=rJrMr>r?s rCancel Jobs.Cancelis+z|J88I F  "&&22   ! ! - - @ @$F A 4I2 f   ! !)+//MM,, " /C !LLJMIG, __  % %g ..  ( (,  $ $U ++,s"C22D!DD!cBU=(d [5nU=(d [n[5R[5R UUU(aSOUS9S9n/nUbUR S5 UbUR S5 UcU(aUR S5 SR U5n [5RUUUUU S9n [R5RU 5$![Ran [R"U 5eSn A ff=f) aUpdate pipeline options on a running job. You should specify at-least one (or both) of min_num_workers and max_num_workers. Args: job_id: ID of job to update project_id: Project of the job region_id: Region the job is in min_num_workers: Lower-bound for worker autoscaling max_num_workers: Upper-bound for worker autoscaling worker_utilization_hint: Target CPU utilization for worker autoscaling unset_worker_utilization_hint: Unsets worker_utilization_hint value Returns: The updated Job N) minNumWorkers maxNumWorkersworkerUtilizationHint)runtimeUpdatableParamsz(runtime_updatable_params.min_num_workersz(runtime_updatable_params.max_num_workersz0runtime_updatable_params.worker_utilization_hint,)r2r3r4rM updateMask)r#r6rrNRuntimeUpdatableParamsappendjoinrSr)r.rTr8r9rr:) r;r<r=min_num_workersmax_num_workersworker_utilization_hintunset_worker_utilization_hintrMupdate_mask_pieces update_maskr>r?s r UpdateOptionsJobs.UpdateOptionss&8+z|J88I   ! !02II))1, J  " C" JK" JK+ ( <((-.K!LL  MG, __  % %g ..  ( (,  $ $U ++,s "C//DDDcU=(d [5nU=(d [n[5R[5RRR S9n[5R XXS9n[R5RU5$![Ran[R"U5eSnAff=f)zDrains a job by calling the Jobs.Update method. Args: job_id: Identifies a single job. project_id: The project which owns the job. region_id: The regional endpoint where the job lives. Returns: (Job) )rKrLN)r#r6rrNrQJOB_STATE_DRAINEDrSr)r.rTr8r9rr:)r;r<r=rMr>r?s rDrain Jobs.Drains+z|J88I   ! !)+//MM** " -C !LLJMIG, __  % %g ..  ( (,  $ $U ++,s9"BC 0CC cU=(d [5nU=(d [n[5RU/S9n[5R US9n[5R XX%S9n[ R5RU5$![Ran[R"U5eSnAff=f)a5Resumes a job by calling the Jobs.Update method. Args: job_id: Identifies a single job. experiment_with_token: The resume token unique to the job prefixed with the experiment key. project_id: The project which owns the job. region_id: The regional endpoint where the job lives. Returns: (Job) ) experiments) environmentrLN) r#r6r EnvironmentrNrSr)r.rTr8r9rr:)r;experiment_with_tokenr<r=rprMr>r?s rResumeUnsupportedSDKJobs.ResumeUnsupportedSDKs"+z|J88I#%11*+2-K   ! !k ! :C!LLJMIG, __  % %g ..  ( (,  $ $U ++,s+"BB="B88B=c RU=(d [5nU=(d [n[5RUUU[5R X#US9S9n[ R 5RU5$![Ran[R"U5eSnAff=f)a>Takes a snapshot of a job via the Jobs.Snapshot method. Args: job_id: Identifies a single job. project_id: The project which owns the job. region_id: The regional endpoint where the job lives. ttl: The ttl for the snapshot. snapshot_sources: If true, the sources will be snapshotted. Returns: (Snapshot) )r3ttlsnapshotSources)r2r3r4snapshotJobRequestN) r#r6r,DataflowProjectsLocationsJobsSnapshotRequestSnapshotJobRequestr)r.Snapshotr8r9rr:)r;r<r=rvsnapshot_sourcesr>r?s rr{ Jobs.Snapshots$+z|J88I!NN,.AA9IBK OG, __  ' ' 00  ( (,  $ $U ++,s"A77B& B!!B&rNNN)FNN)NNNNNN)NN)NN604800sF)__name__ __module__ __qualname____firstlineno____doc__rr7 GET_REQUEST(DataflowProjectsLocationsJobsListRequest LIST_REQUEST%DataflowProjectsJobsAggregatedRequestAGGREGATED_LIST_REQUESTrSUPDATE_REQUEST staticmethodr.r"rVrhrlrsr{__static_attributes__rrrr)r)Cs/!#KK+"$MM,-))$..77,,.,,@"$(A,A,F,,0'+%),,8 % ,,rr)c^\rSrSrSr\"5R r\S5r \SSj5r Sr g)Metricsi&z*The Metrics set of Dataflow API functions.c*[5R$rr,rrrr.Metrics.GetService,r0rNc&U=(d [5nU=(d [n[5RUUUUS9n[R 5R U5$![Ran[R"U5eSnAff=f)a]Calls the Dataflow Metrics.Get method. Args: job_id: The job to get messages for. project_id: The project which owns the job. region_id: The regional endpoint of the job. start_time: Return only metric data that has changed since this time. Default is to return all information about all metrics for the job. Returns: (MetricUpdate) )r2r3r4 startTimeN) r#r6r.DataflowProjectsLocationsJobsGetMetricsRequestrr. GetMetricsr8r9rr:)r;r<r= start_timer>r?s rr" Metrics.Get0s+z|J88I44 5 ,    ! , ,W 55  ( (,  $ $U ++,s"A!!B5B  Brr~) rrrrrrrrrr.r"rrrrrr&s@2!2277,,rrc\rSrSrSrSrSrSrSrSr Sr Sr Sr Sr SrSrSrSrSrSrSrSrSrSrSrSrSrSrSrSrSrSSjrSr g)TemplateArgumentsiMz%Wrapper class for template arguments.NcRXlX lX0lX@lXPlX`lXplXlXlXl Xl Xl Xl Xl XlUUlUUlUUlUUlUUlUUlUUlUUlUUlUUlUUlgr)r<r=job_name gcs_locationzone max_workers num_workersnetwork subnetworkworker_machine_typelauncher_machine_typestaging_location temp_location kms_key_namedisable_public_ips parametersservice_account_email worker_region worker_zoneenable_streaming_engineadditional_experimentsadditional_pipeline_optionsadditional_user_labelsstreaming_updatetransform_name_mappings flexrs_goal)selfr<r=rrrrrrrrrrrrrrrrrrrrrrrrs r__init__TemplateArguments.__init__ks6!ONM$I""L O2!6,&$0 DO!6D&D"D#:D "8D'BD$"8D,D#:D "Dr)rrrrrrrrrrrrrrr<r=rrrrrrrrrr)NNNNNNNNNNNNNNNNNNNNNNNNNN)!rrrrrr<r=rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrMs-*), ( $++ '*-,*-+  $ + #'%) $! "&%)!'+&*+/&* $'+54#rrc\rSrSrSr\"5R r\"5Rr \ Rr \"5Rr \Rr\"5Rr\R"r\ R&r\"5R*r\Rr\R0r\"5RR4r\"5RR8r\"5R<r\"5R@r!\"5R@RDr#\"5RHr%\&"S5r'\&"S5r(\&"S5r)\&"S5r*\&"SS5r+\&"SS5r,\&"SS5r-\&"SS5r.\&"S5r/\&"S 5r0\&"S S5r1S r2\3Rh\3Rj-\3Rl-r7\3Rp\3Rr-\3Rt-\3Rv-r<\3Rh\3Rz-\3Rt-\3R|-\3Rv-r?\@S 5rA\@S 5rB\@S#Sj5rC\@S#Sj5rD\@S5rE\@S5rF\@S5rG\@S5rH\@S5rI\@S5rJ\@S5rK\@S5rL\@S5rM\@S5rN\@S5rO\@S5rP\@S5rQ\@S$Sj5rR\@S5rS\@S5rT\@S 5rU\@S#S!j5rVS"rWg )% Templatesiz,The Templates set of Dataflow API functions.java11java17java21java25Tpython3goz4gs://dataflow-templates-{}/latest/flex/Yaml_Templatec*[5R$r)rprojects_locations_templatesrrrr.Templates.GetService   ; ;;rc*[5R$r)r projects_locations_flexTemplatesrrrGetFlexTemplateService Templates.GetFlexTemplateService   ? ??rNcV/nURnU(a[R"U5O0H1up4UR[R R X4S95 M3 UR=(d [n[5RRnURnUR(aUOSn[5RRn [RUR U 5n [R#UR$UR&U[5RUR(UR*UR,UR.UR0UR2UR4UR6UR8UUR:UR<UR>U (aU "U S9OSUR@(a UR@O/URB(a URBO/S9U(a[R US9OSS9n [5REURF=(d [I5UU S9n [RK5RMU 5$![NRPan [RRT"U 5eSn A ff=f)z|Calls the Dataflow Templates.CreateFromJob method. Args: template_args: Arguments for create template. Returns: (Job) rENrH)serviceAccountEmailr maxWorkers numWorkersrr machineType tempLocation kmsKeyNameipConfiguration workerRegion workerZoneenableStreamingEngineadditionalUserLabelsadditionalExperimentsadditionalPipelineOptions)gcsPathjobNamer3rpr)r4r3createJobFromTemplateRequest)+rsix iteritemsr`rPARAMETERS_VALUErPr=r6rRuntimeEnvironmentIpConfigurationValueValuesEnumWORKER_IP_PRIVATErAdditionalUserLabelsValue _Templates__ConvertDictArgumentsrCREATE_REQUESTrrrrrrrrrrrrrrrr/DataflowProjectsLocationsTemplatesCreateRequestr<r#r.Creater8r9rr:) template_args params_listrkvr=ip_configuration_enum ip_privateip_configurationuser_labels_valueuser_labels_listbodyr>r?s rrTemplates.CreatesjK))J-7 j)R?  $ $ 7 7A 7 GI@ ''F+FI-77&88J%2%E%Ez4)22 77,,  # #**&&%':: - C C##$00$00!))$//%99&77$11,&44$00"/"G"G "3%5"7%)!7744!<<993;82<--!,..AEA $ GDB 55**:jl%)6+ ,  ! ! # * *7 33  ( (,  $ $U ++,s"I99J( J##J(cX/nURnU(a[R"U5O0H1up4UR[R R X4S95 M3 [RUR[R5nSnSnUR(a&URnU(a[RUS9nUR=(d [n[5RRn U R n UR"(aU OSn [R%[5RUR&UR(UR*UR,UR.UR0UR2UR4UR6U UR8UR:UR<UR>(a UR>O/S9UR@U(a[R US9OSUUS9n [5RCURDUU URF=(d [I5SS9n [RK5RMU 5$![NRPan[RRT"U5eSnAff=f)zCalls the Dataflow Templates.LaunchTemplate method on a dynamic template. Args: template_args: Arguments to create template. gcs_location must point to a Json serialized DynamicTemplateFileSpec. Returns: (LaunchTemplateResponse) rENrH)rrrrrrrrrrrrrr)rprrupdatetransformNameMappingF)rr3launchTemplateParametersr4 validateOnly)+rrrr`r LAUNCH_TEMPLATE_PARAMETERS_VALUErPrr-DYNAMIC_TEMPLATE_TRANSFORM_NAME_MAPPING_VALUErr=r6rrrrrLAUNCH_TEMPLATE_PARAMETERSrrrrrrrrrrrrrr/DataflowProjectsLocationsTemplatesLaunchRequestrr<r#r.Launchr8r9rr:)rrrrrtransform_mapping_listtransform_mappingsrr=rrrrr>r?s rLaunchDynamicTemplateTemplates.LaunchDynamicTemplate(srK))J-7 j)R?  4 4 G G H @ '==--??%%&77   C C%; D  ''F+FI-77&88J%2%E%Ez4  / /%':: - C C##$00$00!))$//%99&77$11,&44$00"/"G"G!7744#; (&&1<==!,>.BF/3 0 D8 KK!..%)#..>*, L  ,  ! ! # * *7 33  ( (,  $ $U ++,s"I::J)J$$J)c/nU(a=[R"U5H#up4URURX4S95 M% U$)zConvert dictionary arguments to parameter list . Args: arguments: Arguments for create job using template. value_message: the value message of the arguments Returns: List of value_message.AdditionalProperty rE)rrr`rP) arguments value_messagerrrs r__ConvertDictArguments Templates.__ConvertDictArguments{sEK-- *$!=;;;KL+ rcSnSnSUS'[UR55VVs/sHupVSRXV5PM nnnSRU5nSRU5n SRU 5n [R "U5R[ RU5UU SRU5S 9n U $s snnf) a*Builds Dockerfile contents for java flex template image. Args: flex_template_base_image: SDK version or base image to use. pipeline_paths: List of paths to pipelines and dependencies. env: Dictionary of env variables to set in the container image. Returns: Dockerfile contents as string. B FROM {base_image} {env} {copy} {commands} z /template/*FLEX_TEMPLATE_JAVA_CLASSPATH ENV {}={}  COPY {} /template/ base_imageenvcopycommands)sorteditemsformatratextwrapdedentr_GetFlexTemplateBaseImage) flex_template_base_imagepipeline_pathsr dockerfile_templater rrenvsenv_listpaths copy_commanddockerfile_contentss rBuildJavaImageDockerfile"Templates.BuildJavaImageDockerfilesH*7C&'17 1D E1DK  q $1DD EyyH HH^ $E'..u5L"//*=>EE66 $&  8$ F&  FsCc $Sn/SQnSUS3US'SU;aiUSRS5nU(aNUVs/sHnSU3PM nnSRU5US'SRU5nURSU35 S U;a3SUS 3US 'URS US 35 URS 5 S U;a SUS 3US '[UR 55V V s/sHupS R X5PM n n n SRU 5n SRU5n SR U 5n[ R"U5R [RU5U USSRU5-S9nU$s snfs sn n f)a,Builds Dockerfile contents for python flex template image. Args: flex_template_base_image: SDK version or base image to use. pipeline_paths: List of paths to pipelines and dependencies. env: Dictionary of env variables to set in the container image. Returns: Dockerfile contents as string. r)zapt-get updatez!apt-get install -y libffi-dev gitzrm -rf /var/lib/apt/lists/*z /template/FLEX_TEMPLATE_PYTHON_PY_FILE#FLEX_TEMPLATE_PYTHON_EXTRA_PACKAGESr]rz pip install &FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILEz!pip install --no-cache-dir -U -r a(pip check || (e=$?; echo "Building a container with incompatible dependencies is prevented by default. If you are sure you want to proceed, you need to create your own container image. See: https://cloud.google.com/dataflow/docs/guides/templates/configuring-flex-templates"; exit $e))FLEX_TEMPLATE_PYTHON_SETUP_FILErrrzRUN z && r) splitrar`r rrrrrr)rrr rr  package_listpackage packages_path package_argrrrrrrrs rBuildPythonImageDockerfile$Templates.BuildPythonImageDockerfilesH S789:&'-3>?EEcJl ?KL|G:gY/| L58XXm5L 12hh}- ;- ( 036 sCDE F 23oo :; < >oo )C/ s<=> ? +,28 1D E1DK  q $1DD EyyH HH^ $E'..u5L"//*=>EE66 $&  &++h// F1 KM4 Fs F'F cSnSRUS5US'SRU5nSRU5n[UR55VVs/sHupgSRXg5PM nnnSRU5n [R "U5R[ RU5U US9n U $s snnf) aBuilds Dockerfile contents for go flex template image. Args: flex_template_base_image: SDK version or base image to use. pipeline_paths: Path to pipeline binary. env: Dictionary of env variables to set in the container image. Returns: Dockerfile contents as string. z2 FROM {base_image} {env} {copy} z /template/{}FLEX_TEMPLATE_GO_BINARYrrrr)r r r )rrar rrrrr) rrr rrrvarvalrrrs rBuildGoImageDockerfile Templates.BuildGoImageDockerfiles&4%:%: %&&(C!" HH^ $E'..u5L6EE66 $&   F  sCcUS:Xa[RUX5$US:Xa[RUX5$US:Xa[RUX5$g)aXBuilds Dockerfile contents for flex template image. Args: flex_template_base_image: SDK version or base image to use. pipeline_paths: List of paths to pipelines and dependencies. env: Dictionary of env variables to set in the container image. sdk_language: SDK language of the flex template. Returns: Dockerfile contents as string. JAVAPYTHONGON)rrr'r-)rrr  sdk_languages rBuildDockerfileTemplates.BuildDockerfile$srv  / /0H0>EE  !  1 12J2@GG    - -.F.<CC rcUHnUR(d[SRU55eUR(d[SRU55eUR(aMl[SRU55e g)zValidates ParameterMetadata objects in template metadata. Args: parameters: List of ParameterMetadata objects. Raises: ValueError: If is any of the required field is not set. zGInvalid template metadata. Parameter name field is empty. Parameter: {}zHInvalid template metadata. Parameter label field is empty. Parameter: {}zKInvalid template metadata. Parameter helpText field is empty. Parameter: {}N)name ValueErrorrlabelhelpText)r parameters r_ValidateTemplateParameters%Templates._ValidateTemplateParameters<s ^^ #VI.0 0__ #VI.0 0    #VI.0 0 rcUS:XaSU;a [S5eUS:XaSU;a [S5eUS:XaSU;a [S 5eg ) aBuilds and validates Flex template environment values. Args: env: Dictionary of env variables to set in the container image. sdk_language: SDK language of the flex template. Returns: True on valid env values. Raises: ValueError: If is any of parameter value is invalid. r0FLEX_TEMPLATE_JAVA_MAIN_CLASSzXFLEX_TEMPLATE_JAVA_MAIN_CLASS environment variable should be provided for all JAVA jobs.r1rzYFLEX_TEMPLATE_PYTHON_PY_FILE environment variable should be provided for all PYTHON jobs.r2r*zPFLEX_TEMPLATE_GO_BINARY environment variable should be provided for all GO jobs.T)r8)r r3s r__ValidateFlexTemplateEnv#Templates.__ValidateFlexTemplateEnvTsuv"A"L @ BB  !&DC&O B DD  ";3"F > @@ rc[R"[RU5n[R5nUR(d[ SR U55eURUlUR(aURUlUR(a0[RUR5 URUlUR(aURUl U$)zBuilds and validates TemplateMetadata object. Args: template_metadata_json: Template metadata in json format. Returns: TemplateMetadata object on success. Raises: ValueError: If is any of the required field is not set. zEInvalid template metadata. Name field is empty. Template Metadata: {}) r JsonToMessagerTEMPLATE_METADATAr7r8r descriptionrr<yamlDefinition)template_metadata_jsontemplate_metadatatemplate_metadata_objs r_BuildTemplateMetadata Templates._BuildTemplateMetadatams!..y/J/J/EG%779  ! ! 0067H0I KK!2!7!7$$*;*G*G'##++,=,H,HI):)E)E&''->-M-M*  rc>UR(a UR$UR(a UR$UR(a [R UR5$[R [5$![ R a N/f=f)+Returns the image path for a YAML template.)r% yaml_imagerr_ExtractYamlTemplateImagerr:r6)argss rGetYamlTemplateImageTemplates.GetYamlTemplateImages{ zz ZZ  __  2243E3EFF  . ./J KK % %   s BBBcn[RRU5n[R"5n[ R RU5n[R"URU55nUS$![a+n[R"SRU55UeSnAff=f)rMzSUnable to read file {0} due to incorrect file path or insufficient read permissionsNr%)rYAML_TEMPLATE_GCS_LOCATIONrr StorageClientr ObjectReferenceFromUrljsonload ReadObject Exceptionrr:)r=yaml_gcl_template_pathstorage_clientobj_refgeneric_template_definitiones rrO#Templates._ExtractYamlTemplateImages'AAHH!..0N**223IJG$(II  # #G ,%! 'w //   $ $ $f%;< s%A?? B4 &B//B4cBUS:Xa[R$US:Xa[R$US:Xa[R$US:Xa[R$US:Xa[R $US:Xa[R $US:Xa[R$US:Xa[R$US :Xa&[R"S 5 [R$US :Xa[R$US :Xa[R$US :Xa[R$U$)aReturns latest base image for given sdk version. Args: flex_template_base_image: SDK version or base image to use. Returns: If a custom base image value is given, returns the same value. Else, returns the latest base image for the given sdk version. JAVA11JAVA17JAVA21JAVA25JAVA11_DISTROLESSJAVA17_DISTROLESSJAVA21_DISTROLESSJAVA25_DISTROLESSJAVA8z]JAVA8 is deprecated and redirected to JAVA11. This option will be removed in a future releasePYTHON3r2 GO_DISTROLESS)rFLEX_TEMPLATE_JAVA11_BASE_IMAGEFLEX_TEMPLATE_JAVA17_BASE_IMAGEFLEX_TEMPLATE_JAVA21_BASE_IMAGEFLEX_TEMPLATE_JAVA25_BASE_IMAGE*FLEX_TEMPLATE_JAVA11_DISTROLESS_BASE_IMAGE*FLEX_TEMPLATE_JAVA17_DISTROLESS_BASE_IMAGE*FLEX_TEMPLATE_JAVA21_DISTROLESS_BASE_IMAGE*FLEX_TEMPLATE_JAVA25_DISTROLESS_BASE_IMAGEr warning FLEX_TEMPLATE_PYTHON3_BASE_IMAGEFLEX_TEMPLATE_GO_BASE_IMAGE&FLEX_TEMPLATE_GO_DISTROLESS_BASE_IMAGE)rs rr#Templates._GetFlexTemplateBaseImages 8+  6 66 !X -  6 66 !X -  6 66 !X -  6 66 !%8 8  A AA !%8 8  A AA !%8 8  A AA !%8 8  A AA !W , kk 0 6 66 !Y .  7 77 !T )  2 22 !_ 4  = == ##rcUS:Xa+[R[RRS9$US:Xa+[R[RRS9$US:Xa+[R[RR S9$US:Xa+[R[RR S9$g)zlBuilds SDKInfo object. Args: sdk_language: SDK language of the flex template. Returns: SDKInfo object r0)languager1YAMLr2N)rSDK_INFO SDK_LANGUAGEr0r1r}r2)r3s r _BuildSDKInfoTemplates._BuildSDKInfosv   )?)?)D)D  EE  !   )?)?)F)F  GG     )?)?)D)D  EE     )?)?)B)B  CC rcV[R"5n[RR US5n[R "X1SS9 [ R"5n[RRU5nURX55sSSS5 $!,(df  g=f)a"Stores flex template container spec file in GCS. Args: template_file_gcs_location: GCS location to store the template file. container_spec_json: Container spec in json format. Returns: Returns the stored flex template file gcs object on success. Propagates the error on failures. ztemplate-file.jsonr)newlineN) rTemporaryDirectoryospathraWriteFileContentsr rUr rVrW CopyFileToGCS)template_file_gcs_locationcontainer_spec_jsontemp_dir local_pathr]r^s r_StoreFlexTemplateFile Templates._StoreFlexTemplateFilesv  ! ! #x77<<*>?j jtL"002n,,445OPg  ) )* > $ # #s A:B B(c Sn U(a[RU5n [RU5n Sn U(GaU[RUR[R 5n [R Rn UR(aU OSnUR(aSOSn[RURURURURURUR UR"(a UR"O UR$UR$UR&UUR(UR*UUR,(a UR,O/U (a[R U S9OSS9n [R/UU U U UUUS9n[0R2"U5n[4R6"[4R8"U5SSSS9nU(aU$[R;UU5 [<R>RAS RCUU55 g![DRFan[HRJ"U5eSnAff=f) aBuilds container spec and stores it in the flex template file in GCS. Args: template_file_gcs_location: GCS location to store the template file. image: Path to the container image. template_metadata_json: Template metadata in json format. sdk_language: SDK language of the flex template. print_only: Only prints the container spec and skips write to GCS. template_args: Default runtime parameters specified by template authors. image_repository_username_secret_id: Secret manager secret id for username to authenticate to private registry. image_repository_password_secret_id: Secret manager secret id for password to authenticate to private registry. image_repository_cert_path: The full URL to self-signed certificate of private registry in Cloud Storage. Returns: Container spec json if print_only is set. A success message with template file GCS path and container spec otherewise. NTrH)rrrrrrrstagingLocationrrrrrrr)r%metadatasdkInfodefaultEnvironmentimageRepositoryUsernameSecretIdimageRepositoryPasswordSecretIdimageRepositoryCertPath)r]z: ) sort_keysindent separatorszkSuccessfully saved container spec in flex template file. Template File GCS Location: {} Container Spec: {})&rrJrrrFLEX_TEMPLATE_USER_LABELS_VALUEIP_CONFIGURATION_ENUM_VALUErrrFLEX_TEMPLATE_ENVIRONMENTrrrrrrrrrrrrCONTAINER_SPECr MessageToJsonrXdumpsloadsrr statusPrintrr8r9rr:)rr%rGr3 print_onlyr#image_repository_username_secret_id#image_repository_password_secret_idimage_repository_cert_pathrHsdk_infodefault_environmentrrrrcontainer_specrcontainer_spec_pretty_jsonr?s rBuildAndStoreFlexTemplateFile'Templates.BuildAndStoreFlexTemplateFiles><#:: "&&|4H"99  . .  3 3588JJj%88*d 77$T&??+AA".."..''"--#77  ( (%22.;.L.L'88"//*$22".. 7!.!E!E"/!E!EKM )HH#3 I 5#''@)(--".(K(K :. >>KHHTrc [R"U5Htupn[RU[R5 UHDn[R[R R X5[R5 MF Mv g)zWalks a dir to chmod itself and its contents with the configured access. Args: top_dir_path: The path to the top-level directory. Raises: OSError: If the chmod fails. N)rwalkrrDIR_PERMISSIONS_MASKrraFILE_PERMISSIONS_MASK) top_dir_pathdirpath_ filenamesfilenames r_ChmodRWorldReadableTemplates._ChmodRWorldReadableqsb"$!6I)G)GH(!! GGLL +Y-L-L  "7rc [RXV5 [R"5n [R R SRU 55 /n Un U(aUn O U(aU/n U GHn [RRU 5n [RRU 5(a<[R"X5n[RU[R5 Oh[R$"U [RR'U [RR)U 555n[R+U5 U R-[RR/U 5S5 GM [R R S5 [R1UXU5n[RR'U S5n[R2"UU5 [R R SRU55 [4R6"5n[8R:"S!0SU_S S _S U_S S _SS_SS_SS _SU _SS _SS _SU_SS _SS _SS _SS _SS _SS _SS _SU(aUOS _SS _6n[R R S 5 [8R<"US U5 S S S 5 g![ a) [R""SRU55 GNf=f![ a) [R""SRU55 GNf=f!,(df  g =f)"a(Builds the flex template docker container image and stores it in GCR. Args: image_gcr_path: GCR location to store the flex template container image. flex_template_base_image: SDK version or base image to use. jar_paths: List of jar paths to pipelines and dependencies. py_paths: List of python paths to pipelines and dependencies. go_binary_path: Path to compiled Go pipeline binary. env: Dictionary of env variables to set in the container image. sdk_language: SDK language of the flex template. gcs_log_dir: Path to Google Cloud Storage directory to store build logs. cloud_build_service_account: Service account to be used by Cloud Build to build the image. Returns: True if container is built and store successfully. Raises: ValueError: If the parameters values are invalid. z$Copying files to a temp directory {}z/Could not adjust permissions for copied file {}z4Could not adjust permissions for copied directory {}zCGenerating dockerfile to build the flex template container image... Dockerfilez"Generated Dockerfile. Contents: {}tagno_cacheFmessages substitutionsN arg_configzcloudbuild.yamlis_specified_sourceT no_sourcesourcegcs_source_staging_dir ignore_filearg_gcs_log_dirarg_machine_type arg_disk_sizearg_worker_poolarg_dir arg_revisionarg_git_source_dirarg_git_source_revisionarg_service_account buildpackz/Pushing flex template container image to GCR...r)r#_Templates__ValidateFlexTemplateEnvrrr rrrrrabspathisfileshutilcopy2rrOSErrorrvcopytreerabasenamerr`r"r4rrrr CreateBuildConfigBuild)image_gcr_pathr jar_pathspy_pathsgo_binary_pathr r3 gcs_log_dircloud_build_service_accountrpipeline_filesrr absl_path copy_filecopy_dirrdockerfile_pathr build_configs rBuildAndStoreFlexTemplateImage(Templates.BuildAndStoreFlexTemplateImages!@'':  ! ! #x jj=DDXNOne   $GGOOD) 77>>) $ $ll97)   % %9:: __ggll8RWW%5%5i%@A (    * *8 4 bggmmI6q9:;> jj OQ%556N6D6BD X|UR>UR@UURBURDURFU URH(a URHO/U(a[RUS9OSURJ(a URJO/S9UUU(a[RUS9OSS9S9n [M5ROURP=(d [S5UU S9n [RU5RWU 5$![XRZan [\R^"U 5eSn A ff=f) zwCalls the create job from flex template APIs. Args: template_args: Arguments for create template. Returns: (Job) NrHSPEED_OPTIMIZEDCOST_OPTIMIZED)rrrrrrlauncherMachineTyperrrrrrr flexrsGoalrrr)rcontainerSpecGcsPathrprtransformNameMappingsr)launchParameter)r4r3launchFlexTemplateRequest)0rrrFLEX_TEMPLATE_PARAMETERS_VALUEr*FLEX_TEMPLATE_TRANSFORM_NAME_MAPPING_VALUErrrr=r6rrrrFLEXRS_GOAL_ENUM_VALUEFLEXRS_SPEED_OPTIMIZEDFLEXRS_COST_OPTIMIZEDLAUNCH_FLEX_TEMPLATE_REQUESTFLEX_TEMPLATE_PARAMETERrrrrrrrrrrrrrrrrrrr3DataflowProjectsLocationsFlexTemplatesLaunchRequestr<r#rrr8r9rr:) rrrrrrr=rrrrr>r?s rCreateJobFromFlexTemplate#Templates.CreateJobFromFlexTemplates22  )"J"JLK&==--<<>%%&77   @ @%; A   !77,,113 ''F+FI66HHJ%2%E%Ez4K   " "&7 766MM  $ $(8 866LL  1 1!99!**!.!;!;!;;$1$G$G(44(44%--(33)==$1$G$G >(55 0*88(44&3&K&K&= level start_time: If specified, return only messages with timestamps >= start_time. The default is the job creation time (i.e. beginning of messages). end_time: Return only messages with timestamps < end_time. The default is now (i.e. return up to the latest messages available). page_size: If specified, determines the maximum number of messages to return. If unspecified, the service may choose an appropriate default, or may return an arbitrarily large number of results. page_token: If supplied, this should be the value of next_page_token returned by an earlier call. This will cause the next page of results to be returned. Returns: (ListJobMessagesResponse) )r2r3r4rendTimeminimumImportancepageSize pageTokenN) r#r6r0DataflowProjectsLocationsJobsMessagesListRequestrr.Listr8r9rr:) r;r<r=minimum_importancerend_time page_size page_tokenr>r?s rr$ Messages.List\s>+z|J88I66,7 ,  " ' ' 00  ( (,  $ $U ++,s"A%%B9BBr)NNNNNNN) rrrrrrr#rrr.r$rrrrrrRsW3"44@@".,.,rrcd\rSrSrSr\S5r\S Sj5r\S Sj5r\S Sj5r Sr g) SnapshotsizCloud Dataflow snapshots api.c*[5R$r)rprojects_locations_snapshotsrrrr.Snapshots.GetServicerrNc"U=(d [5nU=(d [n[5RXUS9n[R 5R U5$![Ran[R"U5eSnAff=f)zCalls the Dataflow Snapshots.Delete method. Args: snapshot_id: The id of the snapshot to delete. project_id: The project that owns the snapshot. region_id: The regional endpoint of the snapshot. Returns: (DeleteSnapshotResponse)  snapshotIdr3r4N) r#r6r/DataflowProjectsLocationsSnapshotsDeleteRequestr+r.Deleter8r9rr: snapshot_idr<r=r>r?s rr3Snapshots.Deletes+z|J88I55j6J ,  ! ! # * *7 33  ( (,  $ $U ++,rAc"U=(d [5nU=(d [n[5RXUS9n[R 5R U5$![Ran[R"U5eSnAff=f)zCalls the Dataflow Snapshots.Get method. Args: snapshot_id: The id of the snapshot to get. project_id: The project that owns the snapshot. region_id: The regional endpoint of the snapshot. Returns: (GetSnapshotResponse) r0N) r#r6r,DataflowProjectsLocationsSnapshotsGetRequestr+r.r"r8r9rr:r4s rr" Snapshots.Gets+z|J88I!NNjOJG,  ! ! # ' ' 00  ( (,  $ $U ++,rAc"U=(d [5nU=(d [n[5RXUS9n[R 5R U5$![Ran[R"U5eSnAff=f)aCalls the Dataflow Snapshots.List method. Args: job_id: If specified, only snapshots associated with the job will be returned. project_id: The project that owns the snapshot. region_id: The regional endpoint of the snapshot. Returns: (ListSnapshotsResponse) )r2r3r4N) r#r6r-DataflowProjectsLocationsSnapshotsListRequestr+r.r$r8r9rr:)r;r<r=r>r?s rr$Snapshots.Lists+z|J88I!OOJP@G,  ! ! # ( ( 11  ( (,  $ $U ++,rArr~) rrrrrrr.r3r"r$rrrrr+r+sS%<<,,,,,*,,rr+)F)*r __future__rrrrXrrrrapitools.base.pyrrr8!googlecloudsdk.api_lib.cloudbuildrgooglecloudsdk.api_lib.storager r googlecloudsdk.api_lib.utilr !googlecloudsdk.command_lib.buildsr googlecloudsdk.corer rgooglecloudsdk.core.utilrrrrr6rrr#r'r)rrrrr+rrrrEs;&' %>=67,29#** +II;\`,`,F$,$,NR#R#jm,m,`9,9,xI,I,r