Advanced gsutil

Previously, we have seen some simple basic commands that we can import using gsutil. But with this tool we can do something else:

• gsutil supports wildcards (and ?) and limits a wildcard to files. To include folders in the wildcard target double the sign: gsutil ll gs:///**.txt will list all the text files in subdirectories.

• gsutil requires use of UTF-8 character encoding. For Windows: to use Unicode characters you need to run this command in the command shell before the first time you use gsutil in that shell: chcp 65001.

• The .boto file in your local machine user path is the configuration file for the gsutil CLI. You can edit it directly or via the gsutilconfig command. Some interesting parameters in the .boto file are as follows:

  • parallel_composite_upload_threshold: This is used to specify the maximum size of a file to be uploaded in a single stream. Files larger than this threshold will be uploaded in parallel. The parallel_composite_upload_threshold parameter is for the moment disabled by default.
  • check_hashes: This is used to enforce integrity checks when downloading data, always, never, or conditionally.
  • prefer_api: This parameter value is JSON or XML.
  • aws_access_key_id: This is used for interoperability with S3.
  • aws_secret_access_key: This is used for interoperability with S3.