gS\SrSSKJr SSKJr SSKJr SSKrSSKrSSKrSSKrSSK r SSK r SSK r SSK J r SSKJr SSKJr SS KJr SS KJr SS KJr SSKrSS KJr S r"SS\R45r"SS\5r"SS\5r"SS\5r"SS\5r"SS\5r "SS\5r!"SS\5r"Sr#Sr$\RJS 5r&"S!S"\RN"\RP\)55r*"S#S$\*5r+"S%S&\*5r,S'r-S(r.S)r/S*r0S-S+jr1S.S,jr2g)/zUtilities for job submission preparation. The main entry point is UploadPythonPackages, which takes in parameters derived from the command line arguments and returns a list of URLs to be given to the AI Platform API. See its docstring for details. )absolute_import)division)unicode_literalsN) storage_util)uploads) exceptions)execution_utils)log)files)mapzfrom setuptools import setup, find_packages if __name__ == '__main__': setup( name='{package_name}', packages=find_packages(include=['{package_name}']) ) c\rSrSrSrSrg)UploadFailureError7z0Generic error with the packaging/upload process.N__name__ __module__ __qualname____firstlineno____doc____static_attributes__r5lib/googlecloudsdk/command_lib/ml_engine/jobs_prep.pyrr7s8rrc,^\rSrSrSrU4SjrSrU=r$)SetuptoolsFailedError<z/Error indicating that setuptools itself failed.cr>SRU5nU(aUS- nOUS- n[[U]U5 g)Nz8Packaging of user Python code failed with message: {} zTTry manually writing a setup.py file at your package root and rerunning the command.zTry manually building your Python code by running: $ python setup.py sdist and providing the output via the `--packages` flag (for example, `--packages dist/package.tar.gz,dist/package2.whl)`)formatsuperr__init__)selfoutput generatedmsg __class__s rr SetuptoolsFailedError.__init__?sM VF^ '(c MNc /4rrrrrrrr r __classcell__r%s@rrr<s7 5 5rrc,^\rSrSrSrU4SjrSrU=r$)SysExecutableMissingErrorMz/Error indicating that sys.executable was empty.cT>[[U] [R"S55 g)Nz No Python executable found on path. A Python executable with setuptools installed on the PYTHONPATH is required for building AI Platform training jobs. )rr+r textwrapdedent)r!r%s rr "SysExecutableMissingError.__init__Ps$ #T3  rrr'r)s@rr+r+Ms7rr+c,^\rSrSrSrU4SjrSrU=r$)MissingInitErrorXzCError indicating that the package to build had no __init__.py file.cr>[[U] [R"S5R U55 g)Nz [{}] is not a valid Python package because it does not contain an `__init__.py` file. Please create one and try again. Also, please ensure that --package-path refers to a local directory. )rr2r r.r/r)r! package_dirr%s rr MissingInitError.__init__[s/ D*8??< , VK "rrr'r)s@rr2r2XsK""rr2c\rSrSrSrSrg)UncopyablePackageErrorczError with copying the package.rNrrrrr8r8cs'rr8c,^\rSrSrSrU4SjrSrU=r$)DuplicateEntriesErrorgzFError indicating that multiple files with the same name were provided.ch>[[U] SRSR U555 g)NzrrCc,^\rSrSrSrU4SjrSrU=r$)InvalidSourceDirErrortz6Error indicating that the source directory is invalid.cJ>[[U] SRU55 g)Nz/Source directory [{}] is not a valid directory.)rrFr r)r! source_dirr%s rr InvalidSourceDirError.__init__ws" /9@@LNrrr'r)s@rrFrFts>NNrrFc,[RRU5(d [U5e[R "U5nU(aU$[R"X5(a[SRX55e[RRUS5n[R"SX5 [R"X5 U$![ a [U5ef=f![a [SRU55ef=f)aXReturns a writable directory with the same contents as source_dir. If source_dir is writable, it is used. Otherwise, a directory 'dest' inside of temp_dir is used. Args: source_dir: str, the directory to (potentially) copy temp_dir: str, the path to a writable temporary directory in which to store any copied code. Returns: str, the path to a writable directory with the same contents as source_dir (i.e. source_dir, if it's writable, or a copy otherwise). Raises: UploadFailureError: if the command exits non-zero. InvalidSourceDirError: if the source directory is not valid. zVCannot copy directory since working directory [{}] is inside of source directory [{}].destz+Copying local source tree from [%s] to [%s]z%Cannot write to working location [{}])ospathisdirrFr HasWriteAccessInDir ValueErrorIsDirAncestorOfr8rr?r debugCopyTreeOSError)rItemp_dirwritabledest_dirs r_CopyIfNotWritablerY|s& z " "  ++,((4H  :00  !!'!= ??WW\\(F +()) 9:PB NN:( /% ,  ++, B /66x@ BBBsC=C.C+.%Dc*[R"SU5 [RR U5(a[R "SU5 g[ RUS9n[R "SU5 [R"X5 g)aLGenerates a temporary setup.py file if there is none at the given path. Args: setup_py_path: str, a path to the expected setup.py location. package_name: str, the name of the Python package for which to write a setup.py file (used in the generated file contents). Returns: bool, whether the setup.py file was generated. z!Looking for setup.py file at [%s]z$Using existing setup.py file at [%s]F) package_namez&Generating temporary setup.py file: %sT) r rSrMrNisfileinfoDEFAULT_SETUP_FILErr WriteFileContents) setup_py_pathr[setup_contentss r_GenerateSetupPyIfNeededrbsk)) /?WW^^M""HH 3]C %,,,,G.(( 4nE-8 rc#<# [R"5nUR5nUv U(a#UR"[ R "56 gg![a SnUnN@f=f!U(a#UR"[ R "56 ff=f7f)avYields a temporary directory or a backup temporary directory. Prefers creating a temporary directory (which will be cleaned up when the context manager is closed), but falls back to default_dir. There are systems where users can't write to temp, but we still need to copy. Args: default_dir: str, the backup temporary directory. Yields: str, the temporary directory. N)r TemporaryDirectory __enter__rU__exit__sysexc_info) default_dirrVrNs r_TempDirOrBackuprjs '')H    D ) J( H D(s8B%AA-+BA*'B)A**B-,BBcl\rSrSrSrSr\RS5r\RS5r Sr Sr g) _SetupPyCommandaA command to run setup.py in a given environment. Includes the Python version to use and the arguments with which to run setup.py. Attributes: setup_py_path: str, the path to the setup.py file setup_py_args: list of str, the arguments with which to call setup.py package_root: str, path to the directory containing the package to build (must be writable, or setuptools will fail) c(XlX lX0lgN)r` setup_py_args package_root)r!r`rprqs rr _SetupPyCommand.__init__s&&$rc[5e)zEReturns arguments to use for execution (including Python executable).NotImplementedErrorr!s rGetArgs_SetupPyCommand.GetArgs   rc[5e)z?Returns the environment dictionary to use for Python execution.rtrvs rGetEnv_SetupPyCommand.GetEnvryrc [R"UR5SURURURUR 5S9$)zRun the configured setup.py command. Args: out: a stream to which the command output should be written. Returns: int, the return code of the command. T)no_exitout_funcerr_funccwdenv)r Execrwwriterqr{)r!outs rExecute_SetupPyCommand.Executes@    syy399   4;;= 22r)rqrpr`N) rrrrrr abcabstractmethodrwr{rrrrrrlrlsE %      2rrlc$\rSrSrSrSrSrSrg)_CloudSdkPythonSetupPyCommandi zA command that uses the Cloud SDK Python environment. It uses the same OS environment, plus the same PYTHONPATH. This is preferred, since it's more controlled. cn[R"UR/URQ7S[ 506$)Npython)r ArgsForPythonToolr`rpGetPythonExecutablervs rrw%_CloudSdkPythonSetupPyCommand.GetArgss:  , ,T-?-? K.2.@.@ K4G4I KKrc[RR5n[RR [ R 5US'U$)N PYTHONPATH)rMenvironcopypathsepr?rgrN)r!exec_envs rr{$_CloudSdkPythonSetupPyCommand.GetEnvs2zz HZZ__SXX6H\ OrrNrrrrrrwr{rrrrrr sK rrc$\rSrSrSrSrSrSrg)_SystemPythonSetupPyCommandizA command that uses the system Python environment. Uses the same executable as the Cloud SDK. Important in case of e.g. a setup.py file that has non-stdlib dependencies. cH[5UR/UR-$ro)rr`rprvs rrw#_SystemPythonSetupPyCommand.GetArgs's!  !4#5#5 69K9K KKrcgrorrvs rr{"_SystemPythonSetupPyCommand.GetEnv*s rrNrrrrrrsLrrcfSn[R"5nU$![a [5ef=fro)r rrQr+)python_executables rrr.s>&';;=  & # %%&s0c [U5nSSU/nSSUSU/nSSU/nXe-U-XT-U4n/nUH9n UR[XU55 UR[XU55 M; UH2n [R "5n U R U 5n U (aM2 O [W R55eSSS5 [R"U5V s/sH"n [RRX-5PM$ nn [R"S S RU55 U$!,(df  Ny=fs sn f) aExecutes the setuptools `sdist` command. Specifically, runs `python setup.py sdist` (with the full path to `setup.py` given by setup_py_path) with arguments to put the final output in output_dir and all possible temporary files in a temporary directory. package_root is used as the working directory. May attempt to run setup.py multiple times with different environments/commands if any execution fails: 1. Using the Cloud SDK Python environment, with a full setuptools invocation (`egg_info`, `build`, and `sdist`). 2. Using the system Python environment, with a full setuptools invocation (`egg_info`, `build`, and `sdist`). 3. Using the Cloud SDK Python environment, with an intermediate setuptools invocation (`build` and `sdist`). 4. Using the system Python environment, with an intermediate setuptools invocation (`build` and `sdist`). 5. Using the Cloud SDK Python environment, with a simple setuptools invocation which will also work for plain distutils-based setup.py (just `sdist`). 6. Using the system Python environment, with a simple setuptools invocation which will also work for plain distutils-based setup.py (just `sdist`). The reason for this order is that it prefers first the setup.py invocations which leave the fewest files on disk. Then, we prefer the Cloud SDK execution environment as it will be the most stable. package_root must be writable, or setuptools will fail (there are temporary files from setuptools that get put in the CWD). Args: package_root: str, the directory containing the package (that is, the *parent* of the package itself). setup_py_path: str, the path to the `setup.py` file to execute. output_dir: str, path to a directory in which the built packages should be created. Returns: list of str, the full paths to the generated packages. Raises: SysExecutableMissingError: if sys.executable is None RuntimeError: if the execution of setuptools exited non-zero. sdistz --dist-dirbuildz --build-basez --build-tempegg_infoz --egg-baseNz!Python packaging resulted in [%s]r>)rjappendrrioStringIOr RuntimeErrorgetvaluerMlistdirrNr?r rS)rqr` output_dir working_dir sdist_args build_args egg_info_argssetup_py_arg_setssetup_py_commandsrpsetup_py_commandr return_coderel_file local_pathss r_RunSetupToolsr7sWd %<4J nkKJ {;M"Z/ * <   67:   67+ . KKMc$,,S1k [ .  (( ;&D"$J!79!7Xj3!79)) /;1GH I&%B9sBD0D0)E0 D>c X[RRU5n[RRU5n[ U5n[ X#5n[RR [RRUS55(d [U5e[RRUS5n[RRU5n[XE5n[X$U5U(aA[RRUS5nXG4Hn[R"U5 M sSSS5 $![a [R"SU5 MLf=f![ a%n [#[$R&"U 5U5eSn A ff=f!U(aj[RRUS5nXG4HBn[R"U5 M![a [R"SU5 M@f=f ff=f!,(df  g=f)aJBuilds Python packages from the given package source. That is, builds Python packages from the code in package_path, using its parent directory (the 'package root') as its context using the setuptools `sdist` command. If there is a `setup.py` file in the package root, use that. Otherwise, use a simple, temporary one made for this package. We try to be as unobstrustive as possible (see _RunSetupTools for details): - setuptools writes some files to the package root--we move as many temporary generated files out of the package root as possible - the final output gets written to output_dir - any temporary setup.py file is written outside of the package root. - if the current directory isn't writable, we silenly make a temporary copy Args: package_path: str. Path to the package. This should be the path to the directory containing the Python code to be built, *not* its parent (which optionally contains setup.py and other metadata). output_dir: str, path to a long-lived directory in which the built packages should be created. Returns: list of str. The full local path to all built Python packages. Raises: SetuptoolsFailedError: If the setup.py file fails to successfully build. MissingInitError: If the package doesn't contain an `__init__.py` file. InvalidSourceDirError: if the source directory is not valid. z __init__.pyzsetup.pyz setup.pycz;Couldn't remove file [%s] (it may never have been created).N)rMrNabspathdirnamerjrYexistsr?r2basenamerbrunlinkrUr rSrrsix text_type) package_pathrrqrr`r[r#pyc_filerNerrs r BuildPackagesrsB.,., %%l@L 77>>"'',,|]C D D \ **GGLLz:M77##L1L(EI L D  77<< k:"-D  IIdO.3&%8  IIM   A !#--"4i @@A  77<< k:"-D  IIdO  IIM  . '&%s B#H. E39-H'E =H !E0 ,H/E0 0H3 F"= FF""F%%.HG,*H,!H HH HH H)cU(d [5e[R"[[ [ R RU555n[R"U5VVs/sHup4US:dM UPM nnnU(a [U5eUVs/sH#of[ R RU54PM% nn[R"XqRUR5$s snnfs snf)z;Uploads files after validating and transforming input type.)rC collectionsCounterlistr rMrNrr iteritemsr;r UploadFiles bucket_refname)pathsstaging_locationcounterrcountr@rN upload_pairss r_UploadFilesByPathrs  ""   S)9)95%A B C'(+ g(>L(>%!)(>*L  ++=BCUT))$/0U,C   \+F+F-22 44 MDs, C2<C2*C8c n/n/nUHKn[RRU5(aURU5 M:URU5 MM U(a[R R [R RU55n[U5nUR[U[R RUS555 UR[XB55 SSS5 U$U(aUR[XB55 U$!,(df  U$=f)aUploads Python packages (if necessary), building them as-specified. An AI Platform job needs one or more Python packages to run. These Python packages can be specified in one of three ways: 1. As a path to a local, pre-built Python package file. 2. As a path to a Cloud Storage-hosted, pre-built Python package file (paths beginning with 'gs://'). 3. As a local Python source tree (the `--package-path` flag). In case 1, we upload the local files to Cloud Storage[1] and provide their paths. These can then be given to the AI Platform API, which can fetch these files. In case 2, we don't need to do anything. We can just send these paths directly to the AI Platform API. In case 3, we perform a build using setuptools[2], and upload the resulting artifacts to Cloud Storage[1]. The paths to these artifacts can be given to the AI Platform API. See the `BuildPackages` method. These methods of specifying Python packages may be combined. [1] Uploads are to a specially-prefixed location in a user-provided Cloud Storage staging bucket. If the user provides bucket `gs://my-bucket/`, a file `package.tar.gz` is uploaded to `gs://my-bucket///package.tar.gz`. [2] setuptools must be installed on the local user system. Args: packages: list of str. Path to extra tar.gz packages to upload, if any. If empty, a package_path must be provided. package_path: str. Relative path to source directory to be built, if any. If omitted, one or more packages must be provided. staging_location: storage_util.ObjectReference. Cloud Storage prefix to which archives are uploaded. Not necessary if only remote packages are given. Returns: list of str. Fully qualified Cloud Storage URLs (`gs://..`) from uploaded packages. Raises: ValueError: If packages is empty, and building package_path produces no tar archives. SetuptoolsFailedError: If the setup.py file fails to successfully build. MissingInitError: If the package doesn't contain an `__init__.py` file. DuplicateEntriesError: If multiple files with the same name were provided. ArgumentError: if no packages were found in the given path or no staging_location was but uploads were required. r"N) rObjectReference IsStorageUrlrrMrNrrrjextendrr?r)packagesrr remote_pathsrpackagerqrs rUploadPythonPackagesrsl,+g##0099'"!  77??277??<#@AL , ';|')ww||K'JLM,[KL (  *;IJ  ( ' s $AD%% D4c NSnU(a![RRX5nU$U(an[RRURSR UR RS5S4Vs/sHnU(dM UPM sn55nU$s snf)zEGet the appropriate staging location for the job given the arguments.N/r)rr FromBucketRefFromNamebucketr?rrstrip)job_idstaging_bucketjob_dirrfs rGetStagingLocationr/s#33AA   #33<<gll.A.A#.F.8.:"@.:=>#$.:"@AB "@s B"B")rNN)NNN)3r __future__rrrrr contextlibrrMrgr.googlecloudsdk.api_lib.storager$googlecloudsdk.command_lib.ml_enginergooglecloudsdk.corerr r googlecloudsdk.core.utilr r six.movesr r^Errorrrr+r2r8r;rCrFrYrbcontextmanagerrjwith_metaclassABCMetaobjectrlrrrrrrrrrrrrs9''  78*/#* )) 5.5" 2")"(/($.$?/?N.N+\, ))@(2c((f=(2VO& / VrBJ 4IX r