[A-00118]Azure CliでAzure Storage/Azure Filesを操作する

azure cliを使用してAzure Storageを操作したいと思います。

Azureではサービスを使用するためにまずResourceGroupを作成する必要があります。

作成してない場合は下記を参照

[A-00117]Azure CliでResource Groupを操作してみる

1 Azure Filesを使用してみる

1.1 ストレージアカウントを作成する

azureストレージを使用するため、まずはstorage accountを作成します。

az storage account create --name <account-name> --resource-group <resource-grp-name> --kind <storage-type> --location <location-name> --account-type <account-type>

MacBook-Pro:~$ az storage account create --name azstrgtest20230815 --resource-group jp-az-gnm-20230815 --kind StorageV2 --location japaneast --account-type Account
The public access to all blobs or containers in the storage account will be disallowed by default in the future, which means default value for --allow-blob-public-access is still null but will be equivalent to false.
......

下記のとおり作成されました。

・ストレージアカウントを削除する方法(補足)

az storage account delete --name <account-name> --resource-group <resource-grp-name>

MacBook-Pro:test1$ az storage account delete --name azstrgtest20230815 --resource-group jp-az-gnm-20230815
Command group 'az storage' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
Are you sure you want to perform this operation? (y/n): y

下記のとおり削除されました。

1.2 ファイル共有を作成する

後述するディレクトリを作成する為にファイル共有をまず作成する必要があります。

az storage share-rm create \
 --resource-group <resource-grp-name> \
 --storage-account <storage-acct-name> \
 --name <share-file-name> \
 --access-tier "TransactionOptimized" \
 --quota 1024 \
 --output none

MacBook-Pro:~$ az storage share-rm create \
 --resource-group jp-az-gnm-20230815 \
 --storage-account azstrgtest20230815 \
 --name share20230815 \
 --access-tier "TransactionOptimized" \
 --quota 1024 \
 --output none

下記のとおり作成されました。

1.3 ディレクトリを作成する

az storage directory create \
 --name <dir-name> \
 --share-name <share-file-name> \
 --account-name <storage-acct-name> \

MacBook-Pro:~$ az storage directory create \
 --name azdir20230815 \
 --share-name share20230815 \
 --account-name azstrgtest20230815 \
 

There are no credentials provided in your command and environment, we will query for account key for your storage account.
It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.

You also can add `--auth-mode login` in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.

In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
{
  "created": true
}

ディレクトリが作成される場所はファイル共有の中のようです。

ファイル共有を選択して[Browse…]を選択すると下記のように作成されたことが確認できます。

1.4 ファイルをアップロードする

適当なファイルを作成します。

touch test-upload.txt
echo "Hello,world" > test-upload.txt 

az storage file upload \
    --account-name <account-name> \
    --share-name <share-file-name> \
    --source "<upload-file-name>" \
    --path "<az-dir-file-path>"

MacBook-Pro:test1$ az storage file upload \
 --account-name azstrgtest20230815 \
 --share-name share20230815 \
 --source "test-upload.txt" \
 --path "azdir20230815/test-upload.txt"

Command group 'az storage' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
The behavior of this command has been altered by the following extension: storage-preview

There are no credentials provided in your command and environment, we will query for account key for your storage account.
It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.

You also can add `--auth-mode login` in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.

In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
Finished[#############################################################]  100.0000%
{
  "content_md5": "0x760xf80xa90xc70x9f0x2a0x8a0x2c0xd0x330x950xe00xbd0xbf0x660x2b",
  "date": "2023-08-16T19:21:23+00:00",
  "etag": "\"0x8DB9E8DFB06AFA0\"",
  "file_last_write_time": "2023-08-16T19:21:24.2153888Z",
  "last_modified": "2023-08-16T19:21:24+00:00",
  "request_id": "8beddb41-601a-00a4-1776-d01f9a000000",
  "request_server_encrypted": true,
  "version": "2022-11-02"
}

下記のとおり、アップロードされていることが確認できました。

・ファイルの一覧を取得する(補足)

上記のアップロードしたファイルを一覧として取得してみます。

az storage file list \
    --account-name <account-name> \
    --share-name <share-file-name> \
    --path "<directory-name>" \
    --output table

MacBook-Pro:test1$ az storage file list \
 --account-name azstrgtest20230815 \
 --share-name share20230815 \
 --path "azdir20230815" \
 --output table

Command group 'az storage' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
The behavior of this command has been altered by the following extension: storage-preview

There are no credentials provided in your command and environment, we will query for account key for your storage account.
It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.

You also can add `--auth-mode login` in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.

In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
Name             Content Length    Type    Last Modified
---------------  ----------------  ------  -------------------------
test-upload.txt  12                file    2023-08-16T19:21:24+00:00

1.5 ファイルをダウンロードする

先ほどアップしたファイルをダウンロードして中身が変わっていないことを確認します。

az storage file download \
    --account-name <account-name> \
    --share-name <share-file-name> \
    --path "<az-file-path>" \
    --dest "<local-download-path>" \
    --output none

MacBook-Pro:test1$ az storage file download \
 --account-name azstrgtest20230815 \
 --share-name share20230815 \
 --path "azdir20230815/test-upload.txt" \
 --dest "./test-download.txt" \
 --output none
Command group 'az storage' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus
The behavior of this command has been altered by the following extension: storage-preview

There are no credentials provided in your command and environment, we will query for account key for your storage account.
It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.

You also can add `--auth-mode login` in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.

In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
Finished[#############################################################]  100.0000%
MacBook-Pro:test1$ ls
test-download.txt
MacBook-Pro:test1$ cat test-download.txt 
Hello,world

上記の通り、中身は変わらずにダウンロードできました。

2 Azure Blobを使用してみる

次はAzure Blobを使用してみます。AWSでいうS3のような何でも入れられるストレージです。

こちらは静的webサイトのホスティングも可能ですが今回はFilesと同様にBlobをアップロード、ダウンロードだけ確認します。

ストレージアカウントの作成については省略します。

2.1 コンテナを作成する

MacBook-Pro:~$ az storage container create \
 --name <container-name> \
 --account-name <account-name> \
 --resource-group <resource-grp-name> \

MacBook-Pro:~$ az storage container create \
 --name azcntner20230815 \
 --account-name azstrgtest20230815 \
 --resource-group jp-az-gnm-20230815 \
 
Argument 'resource_group_name' has been deprecated and will be removed in a future release.

There are no credentials provided in your command and environment, we will query for account key for your storage account.
It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.

You also can add `--auth-mode login` in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.

In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
{
  "created": true
}

コンテナの作成は[Storage Browser]から[Blob Container]を選択して確認可能です。

2.2 Blobファイルをアップロードする

適当なファイルを作成します。

touch blob && echo "Hello,Az-Blob" >> blob

次にファイルをアップロードします。

az storage blob upload \
    --account-name <storage-account> \
    --container-name <container> \
    --name <blob-file-name> \
    --file <blob-file-name>

MacBook-Pro:test1$ az storage blob upload --account-name azstrgtest20230815 --container-name azcntner20230815 --name blob --file blob
Command group 'az storage' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus

There are no credentials provided in your command and environment, we will query for account key for your storage account.
It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.

You also can add `--auth-mode login` in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.

In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
Finished[#############################################################]  100.0000%
{
  "client_request_id": "ba9d04f4-3e4e-11ee-8858-685b35adcbfd",
  "content_md5": "xvILZgBEI5mMRqgL2huFGA==",
  "date": "2023-08-19T05:10:35+00:00",
  "encryption_key_sha256": null,
  "encryption_scope": null,
  "etag": "\"0x8DBA0729EE3BE7D\"",
  "lastModified": "2023-08-19T05:10:35+00:00",
  "request_id": "65a815d1-201e-003a-3e5b-d22d01000000",
  "request_server_encrypted": true,
  "version": "2022-11-02",
  "version_id": null
}

下記のとおりアップロードされました。

2.3 コンテナのBlob一覧を取得

az storage blob list \
    --account-name <storage-account> \
    --container-name <container> \
    --output table

MacBook-Pro:test1$ az storage blob list --account-name azstrgtest20230815 --container-name azcntner20230815 --output table
Command group 'az storage' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus

There are no credentials provided in your command and environment, we will query for account key for your storage account.
It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.

You also can add `--auth-mode login` in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.

In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
Name    Blob Type    Blob Tier    Length    Content Type              Last Modified              Snapshot
------  -----------  -----------  --------  ------------------------  -------------------------  ----------
blob    BlockBlob    Hot          14        application/octet-stream  2023-08-19T05:10:35+00:00

2.3 Blobをダウンロードする

az storage blob download \
    --account-name <storage-account> \
    --container-name <container> \
    --name <blob-file-name> \
    --file <~/destination/path/for/file>

MacBook-Pro:test1$ az storage blob download \
> --account-name azstrgtest20230815 \
> --container-name azcntner20230815 \
> --name blob \
> --file ./blob_down
Command group 'az storage' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus

There are no credentials provided in your command and environment, we will query for account key for your storage account.
It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.

You also can add `--auth-mode login` in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.

In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
Finished[#############################################################]  100.0000%
{
  "container": "azcntner20230815",
  "content": "",
  "contentMd5": null,
  "deleted": false,
  "encryptedMetadata": null,
  "encryptionKeySha256": null,
  "encryptionScope": null,
  "hasLegalHold": null,
  "hasVersionsOnly": null,
  "immutabilityPolicy": {
    "expiryTime": null,
    "policyMode": null
  },
  "isAppendBlobSealed": null,
  "isCurrentVersion": null,
  "lastAccessedOn": null,
  "metadata": {},
  "name": "blob",
  "objectReplicationDestinationPolicy": null,
  "objectReplicationSourceProperties": [],
  "properties": {
    "appendBlobCommittedBlockCount": null,
    "blobTier": null,
    "blobTierChangeTime": null,
    "blobTierInferred": null,
    "blobType": "BlockBlob",
    "contentLength": 14,
    "contentRange": "bytes None-None/14",
    "contentSettings": {
      "cacheControl": null,
      "contentDisposition": null,
      "contentEncoding": null,
      "contentLanguage": null,
      "contentMd5": "xvILZgBEI5mMRqgL2huFGA==",
      "contentType": "application/octet-stream"
    },
    "copy": {
      "completionTime": null,
      "destinationSnapshot": null,
      "id": null,
      "incrementalCopy": null,
      "progress": null,
      "source": null,
      "status": null,
      "statusDescription": null
    },
    "creationTime": "2023-08-19T05:10:35+00:00",
    "deletedTime": null,
    "etag": "\"0x8DBA0729EE3BE7D\"",
    "lastModified": "2023-08-19T05:10:35+00:00",
    "lease": {
      "duration": null,
      "state": "available",
      "status": "unlocked"
    },
    "pageBlobSequenceNumber": null,
    "pageRanges": null,
    "rehydrationStatus": null,
    "remainingRetentionDays": null,
    "serverEncrypted": true
  },
  "rehydratePriority": null,
  "requestServerEncrypted": true,
  "snapshot": null,
  "tagCount": null,
  "tags": null,
  "versionId": null
}
MacBook-Pro:test1$ ls
blob		blob_down
MacBook-Pro:test1$ cat blob_down 
Hello,Az-Blob

2.4(補足) azcopyを使用してコンテナにファイル(Blob)をアップロードする

先ほど作成したコンテナにファイルをアップロードしてみます。

touch test1.txt && echo "Hello,AZ" >> test1.txt

az storage azcopy blob upload \
 --container <container-name> \
 --account-name <account-name> \
 --source <file-path> \
 --destination <blob-name>

azcopyを初めて使用する際は下記のとおり、自動でインストールしてくれるようです。

MacBook-Pro:test1$ az storage azcopy blob upload \
 --container azcntner20230815 \
 --account-name azstrgtest20230815 \
 --source ./test1.txt \
 --destination NewBlob
The command requires the extension storage-preview. Do you want to install it now? The command will continue to run after the extension is installed. (Y/n): Y
Run 'az config set extension.use_dynamic_install=yes_without_prompt' to allow installing extensions without prompt.
The installed extension 'storage-preview' is in preview.
Command group 'az storage' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus

There are no credentials provided in your command and environment, we will query for account key for your storage account.
It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.

You also can add `--auth-mode login` in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.

In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
Azcopy command: ['/Users/ikegawayouichi/.azure/cliextensions/storage-preview/azext_storage_preview/azcopy/azcopy_darwin_amd64_10.5.0/azcopy', 'copy', './test1.txt', 'https://azstrgtest20230815.blob.core.windows.net/azcntner20230815/NewBlob?se=2023-08-15T23%3A01%3A14Z&sp=rwdlacup&sv=2018-03-28&ss=b&srt=sco&sig=A%2Bmyh3KTPJM8h6KkDFl5FqS7m0mrqG0L%2BtNQt7dagwU%3D']
INFO: Scanning...
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

Job f16d37c6-be59-824d-5c66-046c12ab0d62 has started
Log file is located at: /Users/ikegawayouichi/.azcopy/f16d37c6-be59-824d-5c66-046c12ab0d62.log

INFO: azcopy: A newer version 10.20.0 is available to download

0.0 %, 0 Done, 0 Failed, 1 Pending, 0 Skipped, 1 Total, 


Job f16d37c6-be59-824d-5c66-046c12ab0d62 summary
Elapsed Time (Minutes): 0.0334
Number of File Transfers: 1
Number of Folder Property Transfers: 0
Total Number of Transfers: 1
Number of Transfers Completed: 1
Number of Transfers Failed: 0
Number of Transfers Skipped: 0
TotalBytesTransferred: 9
Final Job Status: Completed

下記のとおりファイルがアップロードされました。

2.5(補足) azcopyを使用してコンテナからBlobをダウンロードする

先ほど作成したBlobをダウンロードします。

az storage azcopy blob download --container <container-name> --account-name <account-name> --source * --destination "<download-path>" --recursive

MacBook-Pro:test1$ az storage azcopy blob download --container azcntner20230815 --account-name azstrgtest20230815 --source * --destination "./down.txt" --recursive
Command group 'az storage' is in preview and under development. Reference and support levels: https://aka.ms/CLI_refstatus

There are no credentials provided in your command and environment, we will query for account key for your storage account.
It is recommended to provide --connection-string, --account-key or --sas-token in your command as credentials.

You also can add `--auth-mode login` in your command to use Azure Active Directory (Azure AD) for authorization if your login account is assigned required RBAC roles.
For more information about RBAC roles in storage, visit https://docs.microsoft.com/azure/storage/common/storage-auth-aad-rbac-cli.

In addition, setting the corresponding environment variables can avoid inputting credentials in your command. Please use --help to get more information about environment variable usage.
Namespace(_log_verbosity_verbose=False, _log_verbosity_debug=False, _log_verbosity_only_show_errors=False, _output_format='json', _command_package='storage', _subcommand='download', account_name='azstrgtest20230815', account_key='xxxx==', connection_string=None, sas_token=None, _subscription=None, cmd=<azure.cli.core.commands.AzCliCommand object at 0x10d9282b0>, source='https://xxxx.blob.core.windows.net/azcntner20230815/test1.txt', destination='./down.txt', recursive=True, func=<azure.cli.core.commands.AzCliCommand object at 0x10d74ee90>, command='storage azcopy blob download', _cmd=<azure.cli.core.commands.AzCliCommand object at 0x10d9282b0>, _command_validator=None, _argument_validators=[<function validate_client_parameters at 0x10d756ef0>, <function validate_azcopy_download_source_url at 0x10d757250>], _parser=AzCliCommandParser(prog='az storage azcopy blob download', usage=None, description='', formatter_class=<class 'argparse.HelpFormatter'>, conflict_handler='error', add_help=True))
Azcopy command: ['/Users/anonymous/.azure/cliextensions/storage-preview/azext_storage_preview/azcopy/azcopy_darwin_amd64_10.5.0/azcopy', 'copy', 'https://xxxx.blob.core.windows.net/azcntner20230815/test1.txt?se=2023-08-15T23%3A17%3A22Z&sp=rwdlacup&sv=2018-03-28&ss=b&srt=sco&sig=rFNsUKVM%2B/C7Glj64bgbV9SrdDkYgM8dCAnMNtnlHEU%3D', './down.txt', '--recursive']
INFO: Scanning...
INFO: Any empty folders will not be processed, because source and/or destination doesn't have full folder support

Job 01163320-84b0-9348-414c-68a4db3d2b2b has started
Log file is located at: /Users/anonymous/.azcopy/01163320-84b0-9348-414c-68a4db3d2b2b.log




Job 01163320-84b0-9348-414c-68a4db3d2b2b summary
Elapsed Time (Minutes): 0
Number of File Transfers: 0
Number of Folder Property Transfers: 0
Total Number of Transfers: 0
Number of Transfers Completed: 0
Number of Transfers Failed: 0
Number of Transfers Skipped: 0
TotalBytesTransferred: 0
Final Job Status: Completed

・Appendix

公式ドキュメントはこちら

https://learn.microsoft.com/ja-jp/cli/azure/storage?view=azure-cli-latest

https://learn.microsoft.com/ja-jp/azure/storage/files/storage-how-to-use-files-portal?tabs=azure-cli

https://learn.microsoft.com/ja-jp/azure/storage/blobs/storage-quickstart-blobs-portal

参考文献はこちら

https://business.ntt-east.co.jp/content/cloudsolution/column-120.html

Azure Portalを介したAzure Blob Storageの使い方（初心者向け）

Azure Files を試す！