%SrSSKJr SSKJr SSKJr SSKrSSKrSSKrSSKrSSK r SSK J r SSK J r SSK Jr SS KJr SS KJr SS KJr SS KJr SS KJr SS KJr SSKJr SSKJr SSKJr SSKJ r SSK!r!Sr"Sr#S/Sjr$Sr%"SS5r&S0Sjr'Sr(Sr)Sr*Sr+Sr,Sr-Sr.S r/S!r0S"r1S#r2S$r3S%r4S&r5S'r6S(r7S)r8S*r9S+r:S,r;S-rr#c\rSrSrSrSrSrg) IPAddressesz/Worker is a holder for the worker IP addresses.cXlX lgN) ip_addressinternal_address)selfr@rAs r!__init__IPAddresses.__init__s  O,r#)rAr@N)__name__ __module__ __qualname____firstlineno____doc__rC__static_attributes__r#r!r<r<s 7-r#r<c:[SU5n[U5VVs/sHupVXT-S:XdMUPM nnn[U5n[R"U5R 5S:Xa[ [U55n GO [5n URS5n U Hn U (dM SU ;aU RS5n [U 5S:wdU S(a U S(d&[R"SSRU 55e[U S5[U S5pX:a&[R"SSRU 55eU R[XS-55 MU R[U 55 M U (d[R"SS 5e[U 5nX:a)[R"SS RXS- 55e0nU HvnUUR"nU(dLUUR$(a8UUR$R&(aUUR$R&nOUn[)UU5UU'Mx U$s snnf![ a' [R"SS RU 55ef=f) zGParses the --worker flag into a dict of worker indexes to IP addresses.rALL,-rzfound malformed range "{}".z6unable to parse worker ID {}. Please only use numbers.zAno worker specified, or none were parsed from the argument value.zworker index {} is larger than the valid worker indexes on this TPU VM. Please only use indexes in the range [0, {}], inclusive.)max enumeratelensix text_typeupperlistrangesetsplitr InvalidArgumentExceptionr(intupdateadd ValueError ipAddress accessConfig externalIpr<) worker_flagnetwork_endpointsuse_internal_ipsnum_nets max_num_netsiendpointprimary_network_endpointsn_vmsindexesrangesrboundsstartendmx worker_ipsworkerrAr@s r!ParseWorkerFlagrvsp Q!,##455+! Q 5 ' (%]];%%'505< GeG   s #F    v;! 6!9F1I337>>qA  ^S^s <337>>qA   uU!G,- ++c!f #4   - -K   7|"[  - - VB *  *f08BB  %f - : : %f - : : E E,V4AALLj#j$Z1ABJv E< 33FMM   sI#I#.I))1Jc [R"U5R5S:XaUS:a[R"S5 U$[ U5S:a[ [ U5U5$[5e![aEn[R"[R"SSRU55U5 SnAgSnAff=f)aParses the --batch-size flag and validates the flag value. Args: batch_size_flag: str, batch-size flag argument. num_ips: int, number of ip-addresses for the ssh command execution. Returns: int, batch-size value capped at number of workers in num_ips. Raises: InvalidArgumentException, if the batch_size_flag is neither a positive integer nor equal to the `all` keyword. rNdzExecuting ssh command on too many workers simultaneously is prone to error. Command may fail. Please consider using `--batch-size` flag if the command fails, for example, --batch-size=100.rrzounable to parse the batch size value {}. Please use a positive integer not more than the number of TPU workers.N) rUrVrWrr'r]minr` raise_fromr r\r()batch_size_flagnum_ipserrors r!ParseBatchSizer~s ]]?#))+u4} kk  N _  !3'11l   nn  - -AAG!B     s#A6, A66 C;CCc/nUHQnURRH4nURS:XdMURURSS5 MO MS U$)zReturns the host key suffixes. ssh-ed25519N) queryValueitemskeyappendvalue)guest_attributeshost_key_suffixesworker_guest_attributesitems r!_ParseHostKeySuffixesr/sX!1'2288 ] "  BC1 9"2 r#cS/U-nUSRRH'nURS:XdMURSSX2' U$ U$)zEReturns a list with only a single worker's host key suffix populated.rrrNrrrr)r worker_countrusuffixesrs r!_ParseSingleHostKeySuffixr:sVTL (q!,,22d xx= BCh / 3 /r#c~XRRH!nURU:XdMURs $ gr?r)rrurrs r!GetFromGuestAttributesrDs5&1177d xx3 ZZ8 r#cU(a8[U5Sn[UR[UR5U5$[ UR5$)z9Retrieves the host key suffixes from the GuestAttributes.r)rXrguestAttributesrTnetworkEndpointsr)guest_attributes_responsesingle_pod_workerrtnode worker_ids r!%GetHostKeySuffixesFromGuestAttributesrKsQZ #I $!11 D ! !"  !!:!J!J KKr#cU(a7[U5SnURX4[R"U55nU$URX45nU$![a gf=f![a gf=f)zRetrieves the GuestAttributes.rN)rXGetGuestAttributesrUrVr) tpu_helperrrtr9zonerrs r!rrZsZ #I","?"? #--4# #" ","?"?"O #"    s#&AA# A A # A0/A0c[R"U5nSU;a"SUS;aUSSR5S:H$g)z-Returns true if the node has OSLogin enabled.metadatazenable-osloginTRUEN)r MessageToDictrW)r node_dicts r!TpuHasOsLoginEnabledrpsI++D1)9!1Yz5J!J Z !1 2 8 8 :f DD r#cU(aUR(dgURVs/sH$o"R[:XdMURPM& nnU(dgXS;$s snf)zReturns true if the metadata has the SSH key. Args: metadata: Project metadata. public_key: The SSH key. Returns: True if present, False if not present. Fr)rrSSH_KEYS_METADATA_KEYr)r public_keyrmatching_valuess r!_MetadataHasSSHKeyrxs[ x~~ %^^+Txx;P/Pjdjj^   q) )) s A$A$c[URU5(a [RR S5 g[ R "U5nSnSU;a[US;a US[nXW;a[R"S5 gUSU-- nURRURUR[US9S9nURUUUSS 5 g![a gf=f) zCVerifies that instance has SSH key, and adds it in case it doesn't.z9SSH key found in project metadata; not updating instance.Nrrz:SSH key found in instance metadata; not updating instance. )rrr)rz-Propagating SSH public key to all TPU workers)rcommonInstanceMetadatarstatusPrintrrrdebugmessagesNodeUpdateMetadataKeyr UpdateNoder) projectrrr9rrrssh_keysnode_for_updates r!AddSSHKeyIfNeededrs66 CCJJC ++D1) (9!6)J:O!O$%:;HIIJK  dZ('',,++==&;8,-/   7    sC$$ C10C1cxSRX5nUb%[U5U:aUSRX!5- nU$)Nz tpu.{}-{}z-{})r(rT)node_idrur instance_ids r! GetInstanceIDrsB""73+"s+<'='F5<< 1 9::K r#cSSU/n[R"5n[R"5n[R"USURURS9nUS:waJ[ R "SRUR555 [ R"S5 gUR5n[UR55S ::a&[ R "S RU55 gUR5S nS S /n[R"5n[R"USURURS9nUS:waJ[ R "S RUR555 [ R"S5 gXgR5;a[R"U5eg)z.Verifies that the ssh-agent holds the SSH key.z ssh-keygenz-lfT)no_exitout_funcerr_funcrzssh-keygen exited with error {}z:Cannot generate fingerprint of SSH key. Command may stall.NrMz=ssh-keygen returned fingerprint entry in invalid format: "{}"zssh-addz-lzssh-add exited with error {}z5Cannot retrieve keys in ssh-agent. Command may stall.)ioStringIOrExecwriterrr(getvaluer'rTr[r2SSHKeyNotInAgent) identity_filecmd keygen_outerrretcodefingerprint_entry fingerprintouts r!VerifyKeyInAgentrsl um,#{{}* #   4*"2"2SYY ' \II/66s||~FGKKLM  ))+  "#q(IIGNN    !'')!,+ D# #   4#))cii ' \II,33CLLNCDKKGH  &  ) )- 88'r#cURS5(dg[R"5nURUlUR UlX5lXElU$)zDConstruct an SshTunnelArgs object from command line args and values.tunnel_through_iapN) IsKnownAndSpecifiedr SshTunnelArgsprefixtracknamerrinstance)argsrrrrress r!CreateSshTunnelArgsrsL  ! !"6 7 7   "#ll#) #+ (, *r#cUHnUR5 M UH"nU(dM [R"U5 M$ g)aTWaits for all the running ssh threads to complete. Exits with a nonzero code, if there are any non-zero exit status in ssh command execution. This ensures that if any command failed on a worker, we don't end up returning 0 for a value. Args: ssh_threads: List of ssh threads. exit_statuses: List of exit status of each ssh execution. N)joinsysexit) ssh_threads exit_statuses ssh_threadrs r!WaitForBatchCompletionrs5 jOO f v hhvr#c SnSn [U5Hn [RRSR X55 U"XCU5n U (aOU(a2[RRSR X55 XU'[ R "U 5 g g ![Ran XS- :Xa U(aSX!'U eU(a[RRS5 [RRSR U[R"U 555 [R"U 5 S n A GMES n A ff=f) z1Attempts to connect to a worker using SSH or SCP. z){}: Attempting to connect to worker {}...zL##### Command execution on worker {} failed with exit status {}. Continuing.rMzFailed to execute command on multiple workers. This may have happened if you have not added your SSH key to your ssh-agent using "ssh-add ~/.ssh/google_compute_engine".zRetrying: {} command error: {}N) rYrrrr(rrr CommandErrorrUrVtimesleep) command_namerurrenv output_filemultiple_workersrun_cmd max_attemptssleep_intervalri exit_statuses r!AttemptRunWithRetriesrs*,.  a( jj 5 < <  Ck2k   **  , #.  , U *    Q  #&-    -  jj * 1 1CMM!,   jj )sB B&&E:BEEc SnSnUR(deURRRnURR [ UR X!R5URS5n[R"URUR5n[R"XURUR5nSnURS5(aMUR (a<UR"Un [%UUR&UR(UR*U 5n[R,"UUUR.UUUR0US9$)aNPrepares the SSH command used to SSH into the worker. Args: args: The arguments passed in by the user. prepped_node: The object that contains all the necessary information to ssh into the node. worker: the worker to ssh into. ips: The ips of the worker Returns: ssh.SSHCommand that can be used to execute SSH command. Nr)remoterremote_command extra_flagsoptions remainderiap_tunnel_args)plain ssh_helperkeyskey_file GetConfigridrstrict_host_key_checkingrRemoter@userParseAndSubstituteSSHFlagsrArrinstance_namesr release_trackrr SSHCommand command_listr) r prepped_noderuipsrrrrr instance_names r! SSHPrepCmdr As-- '  ++0099M%%//loov/M/MN %% G ::cnnl&7&7 8&.. CNNC$8$8+/ 2338O8O //7M) ""  O  !!..&&% r#c$URUUUS9$)Returns a function to run.)explicit_output_fileexplicit_error_fileRun)rroutput_file_writers r! SSHRunCmdrts!  -,  r#c  [R"XX45n[R"U5n U RXR5n [R "U 5(d[ R"SS5e[U RU RRRU5 [U S5(dSO[U R5n [!UR"U R$UR&U 5Ul[UR(5S:a(UR*(d[ R,"SS5eU R.U RRR0R2:Xa[4R6"S5 [U R$5S:=(a [UR(5S:Hn [9XUR(XR5n U cUR;S 5(a~UR<(am[4R>"S 5 [4R@RCS 5 [4R@RC[D5 [FRH"5e[4R>"S 5 SUl%O[MU U UR(U 5Ul%[NRP"5Ul)URRRUU5 URRRVRY5R[5n[R\"X8RR5Ul/UR`(d[NRb"U5unn[dRf"SUR^UUUUU[iU 5[jRl"U5RnRS 9 nURp=Ul8nSRsX5nUR`(d3URt(d"[wUR^XXRU5 UR*(aUR*RyS5OSUl=/Ul>UR~(a%UR|RUR~5 SnUR(a.[4R@RCSRsU55 0UlBUR;S 5(aUR<(aUR(HnU (aSOUn[U RUS5nUcW[4R@RCS 5 [4R@RC[D5 [FRH"5eUURU'M U RUlEXU'U$)aPrepares TPU VM Node for SSH. Args: tpu_name: The unqualified name of the tpu instance to prepare for SSH. user: The username to use for SSH. args: The command line arguments used for SSH. release_track: The release track/version of client protos to use. enable_batching: A bool, True if the user opts into batching requests. username_requested: A bool, True if the user has passed a specific username in the args. prepped_nodes: The list to put resulting prepared nodes into. idx: The index specifying which slot in the 'prepped_nodes' list to put the output node. Returns: The prepared node that is now ready for SSH. Raises: BadArgumentException: If the node retrieved is not a TPU VM. Non TPU VMs are not supported. InvalidArgumentException: If there is no command specified, but multiple workers are targeted. IapTunnelingUnavailable: If IAP Tunneling is not available, the node cannot be SSHed into. TPUthis command is only available for Cloud TPU VM nodes. To access this node, please see https://cloud.google.com/tpu/docs/creating-deleting-tpus.networkConfigsrMrzdll0054,, ]]!!$--0 JJFMM !  !#, 2338O8O))%a&e, # 3 3UJm   45 124466,9l!!&)*GG,/# r#c/nSn/nUH,nURURR55 M. [U5nS/U-n[R R SRU[U5U55 UGHnURR5GHup[XX5n UR(aS[RR SRU RURR555 MvSn Sn UR(a/[!SRU ["R$"U 555n [UR5S:d[U5S:aUR'[(R*"[,SU UU URRU S[.4S 95 US R15 US- nUR2(aXB:Xa[5X85 Sn/nGMjGMmGMp[-SU UU URRU S [.5 GM GM US:aU(a [5X85 ggg) zSSH's into the prepped nodes. Args: prepped_nodes: The list of prepared nodes to be SSHed into. args: The list of arguments passed in to SSH with. total_ssh_batch_size: The final parsed batch size to SSH into the workers' nodes. rNzWUsing ssh batch size of {}. Attempting to SSH into {} nodes with a total of {} workers.r!z {}/{}.logrMSSHTtargetrF)r<rtrrTrrrr(r r:rrBuildrrr=rrUrVr threadingThreadrrrqrr)r>rtotal_ssh_batch_sizercurrent_batch_sizertr r|rrur rrrEs r!SSHIntoPreppedNodesrR&s+*#ll--3356$  O'&7"-**V0#m2DgN $l#..446  t6 7c   chhsyy)@)@)D)DEFG"  '   4cmmF6K L  \ $ $ % )S-?!-C   ,! ++//&    Ba  ( (":  < + ; )       # # ' '    W7$n q[[;6![r#c [R"UR5nUR(aURURlO(URUR SRlSnUR(dEURR[URX!R5URS5nSnURS5(aMUR(a<UR Un[#UUR$UR&UR(U5n[*R,"UR UUR.UUR0UR2UR4US9$)aNPrepares the SCP command used to SCP into the worker. Args: args: The arguments passed in by the user. prepped_node: The object that contains all the necessary information to scp into the node. worker: the worker to scp into. ips: The ips of the worker Returns: ssh.SCPCommand that can be used to execute SCP command. rNr)rr recursivecompressrr)copydeepcopydstrr@hostsrcsrrrrrrrrrrrrrrr SCPCommandrrecurserUr)rr rur  worker_dstrrr s r! SCPPrepCmdr^}s3}}\--.* ^^J (+~~La$ ' %%//loov/M/MN %% G / 2338O8O //7M) ""  O  .. }}**%  r#c&AURU5$)rr)rrrs r! SCPRunCmdr`s r#c  [R"USX#Xx5n [R"U5n U RXR5n [R "U 5(d[ R"SS5e[U S5(dSO[U R5n [URU RURU 5U l[U R5S:a5U R SR"(a[ R$"SS5e['U R(U R*R,R.U5 U R0U R*R,R2R4:Xa[6R8"S 5 [U R5S:=(a [U R5S:Hn[;XU RXR5nUcUR=S 5(a~UR>(am[6R@"S 5 [6RBRES 5 [6RBRE[F5 [HRJ"5e[6R@"S 5 SU l&O[OUUU RU 5U l&[PRR"5U l*U RTRWU5 U RTRXR[5R]5n[R^"X*RT5U l0URb(d[PRd"U5unn[fRh"SU R`U RjUUUU[mU 5[nRp"U5RrR*S9 nURj=U l5U l5SRuUU Rj5nURb(d3URv(d"[yU R`XXRU5 SU l=URb(ddU RTRXR|U l=URv(d.[U R5S:a[U Rz5 /U l@UR(a%U RRUR5 0U lCUR=S 5(aUR>(aU RHnU(aSOUn[URUS5nUcW[6RBRES 5 [6RBRE[F5 [HRJ"5eUU RU'M U RU lFXU'U $)aPrepares TPU VM Node for SCP. Args: tpu_name: The unqualified name of the tpu instance to prepare for SCP. args: The command line arguments used for SCP. release_track: The release track/version of client protos to use. enable_batching: A bool, True if the user opts into batching requests. username_requested: A bool, True if the user has passed a specific username in the args. prepped_nodes: The list to put resulting prepared nodes into. idx: The index specifying which slot in the 'prepped_nodes' list to put the output node. srcs: The list of sources for the file to send from. dst: The destination to put the SCP-ed file. remote: The remote location for the file to be SCP-ed. Returns: The prepared node that is now ready for SCP. Raises: BadArgumentException: If the node retrieved is not a TPU VM. Non TPU VMs are not supported. InvalidArgumentException: If there are multiple workers targeted. IapTunnelingUnavailable: If IAP Tunneling is not available, the node cannot be SCPed into. NrrrrMrrz=cannot target multiple workers while copying files to client.rrrrrrr r")Gr#SCPPreppedNoder%r&rr'r r(r*rTrrvrurr+rtrZrr\r:r7rrr)r-r.r/rr'rrrrrrr0r2r1rrr r2rrrr3r4r5rrr6rr7rrr r8r9r(r:rrrrrscp_flagr<rrrr)r9rrrrr>r?rZrXrr r@rrArrrrBrCrDrurFr s r!PrepareNodeForSCPrdsL))md, -(# 99 %$   t $ $  ) )  D t-..aC8K8K4L , kk4(($*:*:FD,  !A%,*;*;A*>*E*E  - -G  4::s||00EExP kk    0 0 F FGKK  $   1$J\-D-D)E)J1 l55x&   !566  # # iiLM jj23 jj/0  2 2 44IIOP%)L"%J! &L"&668,d#&&++88:BBD*"--,,, %@@FA''  - 4T :..}=DDMM M'4&8&88L ##J <* DLLc99j $, !-!8!8!=!=!F!FL <rtotal_scp_batch_size scp_threadsrQrtr r|rrur rs r!SCPIntoPreppedNodesrigs+*#ll--3356$  O'&7"-**V0#m2DgN $l#..446  t6 7c   chhsyy)@)@)D)DEFG \ $ $ % )S-?!-C   ,! ++//    Ba  ( (":  < + ; )       # # ' '    K7$b _q[;6)r#)Fall)rM)>rI __future__rrrrVrrrNrapitools.base.pyrapitools.base.py.exceptionsrrgooglecloudsdk.api_lib.computer googlecloudsdk.callioper "googlecloudsdk.command_lib.computer r .googlecloudsdk.command_lib.compute.tpus.tpu_vmr2r r##googlecloudsdk.command_lib.util.sshrgooglecloudsdk.corerrgooglecloudsdk.core.util.filesrrUrr0r"r:r<rvr~rrrrrrrrrrrrrr rrGrRr^r`rdrirKr#r!rus3&'  ,917.98WL3/#5 "HMOd&;R--@AHV'T L#,*(" J$9N  &< ~0ffRT7n7t gTN7r#