minio
文章目录
- minio
-
- 1. 基础介绍
-
- 1.1 简介
- 1.2 基础概念
- 1.3 纠删码EC
- 1.4 存储形式
- 2. minio部署
-
-
- non-erasure code mode
- **erasure code mode**
- 2.1 用户名密码设置
- 2.2 服务启动
- 2.3 登录控制台
-
- 3. 基础操作
-
- 3.1 控制台基础操作
-
- 3.1.1 创建bucket
- 3.1.2 上传文件
- 3.1.3 文件操作
- 3.2 客户端的使用
-
- 3.2.1 服务配置
- 3.2.2 文件操作
- 3.2.3 bucket管理
- 3.2.4 http跟踪
- 3.2.5 扫描修复
- 3.2.6 文件查找
- 4. JAVA
-
- 4.1 环境准备
- 4.2 代码示例
- 4.3 API操作
-
- 4.3.1 bucket操作
-
- 4.3.1.1 bucket是否存在
- 4.3.1.2 bucket创建
- 4.3.1.3 bucket移除
- 4.3.1.4 设置存储周期
- 4.3.1.5 多版本设置
- 4.3.2 对象操作
-
- 4.3.2.1 上传对象
- **4.3.2.2 获取对象**
- 4.3.2.3 复制对象
- 4.3.2.4 删除对象
- 4.3.2.5 对象信息查询
- 4.4 Springboot集成
-
- 4.4.1 环境搭建
- 4.4.2 代码示例
-
- 4.4.2.1 配置参数
- 4.4.2.2 配置文件修改
- 4.4.2.3 MinioClient创建
- 4.4.2.4 Minio API操作
- 4.4.2.5 接口提供
- 4.4.2.6 调用示例
1. 基础介绍
1.1 简介
MinIO 是一个基于Apache License v2.0开源协议的对象存储服务。它兼容亚马逊S3云存储服务接口,非常适合于存储大容量非结构化的数据,例如图片、视频、日志文件、备份数据和容器/虚拟机镜像等,而一个对象文件可以是任意大小,从几kb到最大5T不等。
MinIO是一个非常轻量的服务,可以很简单的和其他应用的结合,类似 NodeJS, Redis 或者 MySQL。目前支持JavaScript 、Java、Python、Golang、.NET。
官网:https://min.io/ http://www.minio.org.cn/
MinIO的优点如下:
- 部署简单: 一个 single 二进制文件即是一切,还可支持各种平台。
- minio 支持海量存储,可按 zone 扩展(原 zone 不受任何影响),支持单个对象最大 5TB;
- 低冗余且磁盘损坏高容忍,标准且最高的数据冗余系数为 2(即存储一个 1M 的数据对象,实际占用磁盘空间为 2M)。但在任意 n/2 块 disk 损坏的情况下依然可以读出数据(n 为一个纠删码集合(Erasure Coding Set)中的 disk 数量)。并且这种损坏恢复是基于单个对象的,而不是基于整个存储卷的。
- 读写性能优异
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-YjEABJWK-1671378607578)(E:\Typora\548e9d729925e15a05c21c314e8e3c79.png)]
1.2 基础概念
MinIO中涉及以下几个基础概念:
Object:存储到MinIO的基本对象,如文件、字节流等
Bucket:用来存储Object的逻辑空间。每个Bucket之间的数据是相互隔离的。对于客户端而言,就相当于一个存放文件的顶层文件夹
Drive:即存储数据的磁盘,在MinIO启动时,以参数的方式传入。MinIO中的所有对象数据都会存储在Drive里
Set:即一组Drive的集合。分布式部署根据集群规模自动划分一个或多个Set,每个Set中的Drive分布在不同位置。
- 一个对象存储在一个Set上
- 一个Object存储在一个Set上
- 一个集群划分为多个Set
- 一个Set包含的Drive数量是固定的,默认由系统根据集群规模自动计算得出
- 一个Set中的Drive尽可能分布在不同节点上
1.3 纠删码EC
纠删码(erasure coding,EC)是一种数据保护方法,它将数据分割成片段,把冗余数据块扩展、编码,并将其存储在不同的位置,比如磁盘、存储节点或者其它地理位置。
Minio采用Reed-Solomon code将对象拆分成N/2数据和N/2 奇偶校验块。 这就意味着如果是12块盘,一个对象会被分成6个数据块、6个奇偶校验块,你可以丢失任意6块盘(不管其是存放的数据块还是奇偶校验块),你仍可以从剩下的盘中的数据进行恢复。
1.4 存储形式
文件对象上传到 MinIO ,会在对应的数据存储磁盘中,以 Bucket 名称为目录,文件名称为下一级目录,文件名下是 part.1 和 xl.meta(老版本,最新版本如下图),前者是编码数据块及检验块,后者是元数据文件。
2. minio部署
minio 支持多种 server 启动模式:
non-erasure code mode
在此启动模式下,对于每一份对象数据,minio 直接在 data 下面存储这份数据,不会建立副本,也不会启用纠删码机制。因此,这种模式无论是服务实例还是磁盘都是“单点”,无任何高可用保障,磁盘损坏就表示数据丢失。
erasure code mode
此模式为 minio server 实例传入多个本地磁盘参数。一旦遇到多于一个磁盘参数,minio server 会自动启用 erasure code mode。erasure code 对磁盘的个数是有要求的,如不满足要求,实例启动将失败。 erasure code 启用后,要求传给 minio server 的 endpoint(standalone 模式下,即本地磁盘上的目录)至少为 4 个。
2.1 用户名密码设置
set MINIO_ROOT_USER=minio //用户名
set MINIO_ROOT_PASSWORD=minio //密码
2.2 服务启动
单节点单目录部署
minio.exe server ./data
单节点多目录部署
minio.exe server ./data{1...64} [生成data1-data64 共64个文件存储数据]
多节点部署
//顺序部署需要节点名,目录名顺序化
minio.exe server http://node{1...32}.example.com/mnt/export{1...32}
//自定义ip,目录部署
minio.exe server http://192.168.1.1/opt/minio/data1 http://192.168.1.1/opt/minio/data2 \http://192.168.1.2/opt/minio/data1 http://192.168.1.2/opt/minio/data2 \http://192.168.1.3/opt/minio/data1 http://192.168.1.3/opt/minio/data2 \http://192.168.1.4/opt/minio/data1 http://192.168.1.4/opt/minio/data2
参数设置:
--address value bind to a specific ADDRESS:PORT, ADDRESS can be an IP or hostname (default: ":9000") [%MINIO_ADDRESS%]--console-address value bind to a specific ADDRESS:PORT for embedded Console UI, ADDRESS can be an IP or hostname [%MINIO_CONSOLE_ADDRESS%]--certs-dir value, -S value path to certs directory (default: "C:\\Users\\ZF\\.minio\\certs")--quiet disable startup and info messages--anonymous hide sensitive information from logging--json output logs in JSON format--help, -h show help
启动成功输出以下日志:
2.3 登录控制台
3. 基础操作
3.1 控制台基础操作
3.1.1 创建bucket
3.1.2 上传文件
3.1.3 文件操作
可进行下载,分享,预览,删除等操作
3.2 客户端的使用
MinIO提供了客户端,可以让我们以类似linux命令的方式操作MinIO
PS F:\MINIO> ./mc.exe -help
────────────────────────────────────────────────────────────
NAME:mc - MinIO Client for object storage and filesystems.USAGE:mc [FLAGS] COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]COMMANDS:alias manage server credentials in configuration filels list buckets and objectsmb make a bucketrb remove a bucketcp copy objectsmv move objectsrm remove object(s)mirror synchronize object(s) to a remote sitecat display object contentshead display first 'n' lines of an objectpipe stream STDIN to an objectfind search for objectssql run sql queries on objectsstat show object metadatatree list buckets and objects in a tree formatdu summarize disk usage recursivelyretention set retention for object(s)legalhold manage legal hold for object(s)support support related commandslicense license related commandsshare generate URL for temporary access to an objectversion manage bucket versioningilm manage bucket lifecycleencrypt manage bucket encryption configevent manage object notificationswatch listen for object notification eventsundo undo PUT/DELETE operationsanonymous manage anonymous access to buckets and objectstag manage tags for bucket and object(s)diff list differences in object name, size, and date between two bucketsreplicate configure server side bucket replicationadmin manage MinIO serversupdate update mc to latest releaseready checks if the cluster is ready or notping perform liveness checkod measure single stream upload and downloadbatch manage batch jobsGLOBAL FLAGS:--autocompletion install auto-completion for your shell--config-dir value, -C value path to configuration folder (default: "C:\\Users\\ZF\\mc")
────────────────────────────────────────────────────────────
MinIO默认配置了以下服务列表
PS F:\MINIO> ./mc.exe config host ls
gcsURL : https://storage.googleapis.comAccessKey : YOUR-ACCESS-KEY-HERESecretKey : YOUR-SECRET-KEY-HEREAPI : S3v2Path : dnslocalURL : http://localhost:9000AccessKey :SecretKey :API :Path : autoplayURL : https://play.min.ioAccessKey : Q3AM3UQ867SPQQA43P2FSecretKey : zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TGAPI : S3v4Path : autos3URL : https://s3.amazonaws.comAccessKey : YOUR-ACCESS-KEY-HERESecretKey : YOUR-SECRET-KEY-HEREAPI : S3v4Path : dns
3.2.1 服务配置
PS F:\MINIO> ./mc.exe config -h
────────────────────────────────────────────────────────────
NAME:mc config - configure MinIO clientUSAGE:mc config COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...]COMMANDS:host add, remove and list hosts in configuration fileFLAGS:--config-dir value, -C value path to configuration folder (default: "C:\\Users\\ZF\\mc")--quiet, -q disable progress bar display--no-color disable color theme--json enable JSON lines formatted output--debug enable debug output--insecure disable SSL certificate verification--help, -h show help
示例【增加服务】:
PS F:\MINIO> ./mc.exe config host a local http://127.0.0.1:9000 minio 123ABCdef*
Added `local` successfully.
3.2.2 文件操作
文件预览
PS F:\MINIO> ./mc.exe ls local
[2022-12-18 16:53:47 CST] 0B 202211/
[2022-12-18 16:23:48 CST] 0B 202212/
PS F:\MINIO> ./mc.exe ls local/202212
[2022-12-18 16:32:11 CST] 106KiB STANDARD erasure-code1.jpg
文件上传下载
上传
PS F:\MINIO> ./mc.exe cp .\mc.exe local/202211
F:\MINIO\mc.exe: 24.86 MiB / 24.86 MiB [==========================================================] 244.25 MiB/s 0s
下载
PS F:\MINIO> ./mc.exe cp .\mc.exe local/202211
F:\MINIO\mc.exe: 24.86 MiB / 24.86 MiB [==========================================================] 244.25 MiB/s 0s
文件删除
PS F:\MINIO> ./mc.exe cp .\mc.exe local/202211
F:\MINIO\mc.exe: 24.86 MiB / 24.86 MiB [==========================================================] 244.25 MiB/s 0s
3.2.3 bucket管理
//创建
PS F:\MINIO> .\mc.exe mb local/202210
Bucket created successfully `local/202210`.
//无文件时删除
PS F:\MINIO> .\mc.exe rb local/202210
Removed `local/202210` successfully.
//有文件时删除
PS F:\MINIO> .\mc.exe --force rb local/202210
Removed `local/202210` successfully.
3.2.4 http跟踪
PS F:\MINIO> .\mc.exe admin trace local
2022-12-18T20:51:14.994 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/202212/?location= 127.0.0.1 37µs ↑ 98 B ↓ 128 B
2022-12-18T20:51:14.994 [200 OK] s3.ListObjectsV2 127.0.0.1:9000/202212/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix= 127.0.0.1 1.134ms ↑ 98 B ↓ 626 B
2022-12-18T20:51:14.997 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/202212/?location= 127.0.0.1 0s ↑ 98 B ↓ 128 B
2022-12-18T20:51:14.997 [200 OK] s3.GetBucketVersioning 127.0.0.1:9000/202212/?versioning= 127.0.0.1 0s ↑ 98 B ↓ 99 B
2022-12-18T20:51:15.010 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/202212/?location= 127.0.0.1 0s ↑ 98 B ↓ 128 B
2022-12-18T20:51:15.011 [404 Not Found] s3.GetBucketPolicy 127.0.0.1:9000/202212/?policy= 127.0.0.1 0s ↑ 98 B ↓ 288 B
2022-12-18T20:51:15.012 [404 Not Found] s3.GetBucketTagging 127.0.0.1:9000/202212/?tagging= 127.0.0.1 514µs ↑ 98 B ↓ 275 B
2022-12-18T20:51:15.030 [200 OK] s3.ListObjectsV2 127.0.0.1:9000/202212/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix=%2F 127.0.0.1 1.026ms ↑ 98 B ↓ 626 B
2022-12-18T20:51:21.205 [200 OK] s3.GetBucketLocation 127.0.0.1:9000/202212/?location= 127.0.0.1 511µs ↑ 98 B ↓ 128 B
2022-12-18T20:51:21.206 [200 OK] s3.PutObject 127.0.0.1:9000/202212/erasure-code1.jpg 127.0.0.1 32.825ms ↑ 106 KiB ↓ 0 B
2022-12-18T20:51:21.251 [200 OK] s3.ListObjectsV2 127.0.0.1:9000/202212/?delimiter=%2F&encoding-type=url&fetch-owner=true&list-type=2&prefix= 127.0.0.1 11.408ms ↑ 98 B ↓ 626 B
3.2.5 扫描修复
mc admin heal 命令用来扫描损坏的对象并修复这些对象。mc admin heal 是资源密集型的,即使在磁盘故障或损坏事件之后通常也不需要。
相反,MinIO会自动修复因bitrot损坏、磁盘故障或其他POST/GET问题而损坏的对象。MinIO 还执行周期性的后台对象修复。
语法如下:
C:\>mc admin heal -h
NAME:mc admin heal - [DEPRECATED] heal disks, buckets and objects on MinIO serverUSAGE:mc admin heal [FLAGS] TARGETFLAGS:--scan value [DEPRECATED] select the healing scan mode (normal/deep) (default: "normal")--recursive, -r [DEPRECATED] heal recursively--dry-run, -n [DEPRECATED] only inspect data, but do not mutate--force-start, -f [DEPRECATED] force start a new heal sequence--force-stop, -s [DEPRECATED] force stop a running heal sequence--remove [DEPRECATED] remove dangling objects in heal sequence--config-dir value, -C value path to configuration folder (default: "C:\\Users\\Administrator\\mc")--quiet, -q disable progress bar display--no-color disable color theme--json enable JSON lines formatted output--debug enable debug output--insecure disable SSL certificate verification--help, -h show helpSCAN MODES:normal (default): Heal objects which are missing on one or more disks.deep : Heal objects which are missing or with silent data corruption on one or more disks.DEPRECATED:MinIO server now supports auto-heal, this command will be removed in future.
命令应在存储桶或存储桶前缀的完整路径上执行对象修复,将已配置的 MinIO 部署的alias指定为路径的前缀。例如:
mc admin heal play``/mybucket/myprefix
如果 TARGET 存储桶或存储桶前缀具有活动的修复扫描,则该命令将返回该扫描的状态
3.2.6 文件查找
mc find 命令用来查询对象。语法如下:
PS D:\minio> .\mc.exe find -h
NAME:mc.exe find - search for objectsUSAGE:mc.exe find [FLAGS] TARGETFLAGS:--exec value spawn an external process for each matching object (see FORMAT)--ignore value exclude objects matching the wildcard pattern--name value find object names matching wildcard pattern--newer-than value match all objects newer than value in duration string (e.g. 7d10h31s)--older-than value match all objects older than value in duration string (e.g. 7d10h31s)--path value match directory names matching wildcard pattern--print value print in custom format to STDOUT (see FORMAT)--regex value match directory and object name with PCRE regex pattern--larger value match all objects larger than specified size in units (see UNITS)--smaller value match all objects smaller than specified size in units (see UNITS)--maxdepth value limit directory navigation to specified depth (default: 0)--watch monitor a specified path for newly created object(s)--config-dir value, -C value path to configuration folder (default: "C:\\Users\\admin\\mc")--quiet, -q disable progress bar display--no-color disable color theme--json enable JSON lines formatted output--debug enable debug output--insecure disable SSL certificate verification--help, -h show helpUNITS--smaller, --larger flags accept human-readable case-insensitive numbersuffixes such as "k", "m", "g" and "t" referring to the metric units KB,MB, GB and TB respectively. Adding an "i" to these prefixes, uses the IECunits, so that "gi" refers to "gibibyte" or "GiB". A "b" at the end isalso accepted. Without suffixes the unit is bytes.--older-than, --newer-than flags accept the string for days, hours and minutesi.e. 1d2h30m states 1 day, 2 hours and 30 minutes.FORMATSupport string substitutions with special interpretations for following keywords.Keywords supported if target is filesystem or object storage:{} --> Substitutes to full path.{base} --> Substitutes to basename of path.{dir} --> Substitutes to dirname of the path.{size} --> Substitutes to object size of the path.{time} --> Substitutes to object modified time of the path.Keywords supported if target is object storage:{url} --> Substitutes to a shareable URL of the path.EXAMPLES:01. Find all "foo.jpg" in all buckets under "s3" account.C:\> mc.exe find s3 --name "foo.jpg"02. Find all objects with ".txt" extension under "s3/mybucket".C:\> mc.exe find s3/mybucket --name "*.txt"03. Find only the object names without the directory component under "s3/mybucket".C:\> mc.exe find s3/mybucket --name "*" -print {base}04. Find all images with ".jpg" extension under "s3/photos", prefixed with "album".C:\> mc.exe find s3/photos --name "*.jpg" --path "*/album*/*"05. Find all images with ".jpg", ".png", and ".gif" extensions, using regex under "s3/photos".C:\> mc.exe find s3/photos --regex "(?i)\.(jpg|png|gif)$"06. Find all images with ".jpg" extension under "s3/bucket" and copy to "play/bucket" *continuously*.C:\> mc.exe find s3/bucket --name "*.jpg" --watch --exec "mc cp {} play/bucket"07. Find and generate public URLs valid for 7 days, for all objects between 64 MB, and 1 GB in size under "s3" account.C:\> mc.exe find s3 --larger 64MB --smaller 1GB --print {url}08. Find all objects created in the last week under "s3/bucket".C:\> mc.exe find s3/bucket --newer-than 7d09. Find all objects which were created are older than 2 days, 5 hours and 10 minutes and exclude the ones with ".jpg"extension under "s3".C:\> mc.exe find s3 --older-than 2d5h10m --ignore "*.jpg"10. List all objects up to 3 levels sub-directory deep under "s3/bucket".C:\> mc.exe find s3/bucket --maxdepth 3
PS D:\minio>
4. JAVA
4.1 环境准备
最低要求:
Java 1.8 或更高版本。
Maven:
<dependency><groupId>io.minio</groupId><artifactId>minio</artifactId><version>8.4.6</version>
</dependency>
jar包资源:
Central Repository: io/minio/minio/8.4.6 (maven.org)
4.2 代码示例
import io.minio.*;
import io.minio.errors.MinioException;import java.io.IOException;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;public class MinIOExamples {public final static String bucketName = "my-bucketname";public static void main(String[] args)throws IOException, NoSuchAlgorithmException, InvalidKeyException {try {// create minioClientMinioClient minioClient =MinioClient.builder()// minio服务端地址URL.endpoint("http://127.0.0.1:9000")// 用户名及密码(访问密钥/密钥).credentials("minio", "123ABCdef*").build();// 检查 bucket 'my-bucketname' 是否存在不存在进行创建boolean found =minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build());if (!found) {minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).build());System.out.println("my-bucketname no found , create success");}// 上传文件minioClient.uploadObject(UploadObjectArgs.builder().bucket(bucketName).object("minio-8.4.6.jar").filename("F:\\lib\\minio-8.4.6.jar").build());System.out.println("上传文件成功");// 下载文件minioClient.downloadObject(DownloadObjectArgs.builder().bucket(bucketName).object("minio-8.4.6.jar").filename("F:\\minio-8.4.6.jar").build());// 删除文件minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucketName).object("minio-8.4.6.jar").build());// 删除bucketminioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucketName).build());} catch (MinioException e) {System.out.println("Error occurred: " + e);}}
}
4.3 API操作
官方API参考文档:Java Client API Reference — MinIO Object Storage for Linux
4.3.1 bucket操作
4.3.1.1 bucket是否存在
bucketExists(BucketExistsArgs args)
public boolean bucketExists(BucketExistsArgs args)
[Javadoc]
Checks if a bucket exists.
Parameters
Parameter | Type | Description |
---|---|---|
bucketName |
BucketExistsArgs | Arguments. |
Returns |
---|
boolean – True if the bucket exists. |
Example
// Check whether 'my-bucketname' exists or not.
boolean found =minioClient.bucketExists(BucketExistsArgs.builder().bucket("my-bucketname").build());
if (found) {System.out.println("my-bucketname exists");
} else {System.out.println("my-bucketname does not exist");
}
4.3.1.2 bucket创建
makeBucket(MakeBucketArgs args)
public void makeBucket(MakeBucketArgs args)
[Javadoc]
Creates a bucket with given region and object lock feature enabled.
Parameters
Parameter | Type | Description |
---|---|---|
args |
MakeBucketArgs | Arguments to create bucket |
Example
// Create bucket with default region.
minioClient.makeBucket(MakeBucketArgs.builder().bucket("my-bucketname").build());// Create bucket with specific region.
minioClient.makeBucket(MakeBucketArgs.builder().bucket("my-bucketname").region("us-west-1").build());// Create object-lock enabled bucket with specific region.
minioClient.makeBucket(MakeBucketArgs.builder().bucket("my-bucketname").region("us-west-1").objectLock(true).build());
4.3.1.3 bucket移除
removeBucket(RemoveBucketArgs args)
public void removeBucket(RemoveBucketArgs args)
[Javadoc]
Removes an empty bucket.
Parameters
Parameter | Type | Description |
---|---|---|
args |
RemoveBucketArgs | Arguments. |
Example
minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucketName).build());
4.3.1.4 bucket查看
listBuckets()
public List<Bucket> listBuckets()
[Javadoc]
Lists bucket information of all buckets.
Returns |
---|
List<Bucket> – List of bucket information. |
Example
List<Bucket> bucketList = minioClient.listBuckets();
for (Bucket bucket : bucketList) {System.out.println(bucket.creationDate() + ", " + bucket.name());
}
4.3.1.4 设置存储周期
setBucketLifecycle(SetBucketLifecycleArgs args)
public void setBucketLifecycle(SetBucketLifecycleArgs args)
[Javadoc]
Sets lifecycle configuration to a bucket.
Parameters
Parameter | Type | Description |
---|---|---|
args |
SetBucketLifecycleArgs | Arguments. |
Example
// 配置生命周期规则
rules.add(new LifecycleRule(Status.ENABLED, // 开启状态null,new Expiration((ZonedDateTime) null, 365, null), // 保存365天new RuleFilter("logs/"), // 目录配置"rule2",null,null,null));
LifecycleConfiguration lifecycleConfiguration = new LifecycleConfiguration(rules);
// 添加生命周期配置
minioClient.setBucketLifecycle(SetBucketLifecycleArgs.builder().bucket("my-bucketname").config(lifecycleConfiguration).build());
4.3.1.5 多版本设置
若开启了多版本控制,上传对象时,OBS自动为每个对象创建唯一的版本号。上传同名的对象将以不同的版本号同时保存在OBS中。
若未开启多版本控制,向同一个文件夹中上传同名的对象时,新上传的对象将覆盖原有的对象。
某些功能(例如版本控制、对象锁定和存储桶复制)需要使用擦除编码分布式部署 MinIO。开启了版本控制后,允许在同一密钥下保留同一对象的多个版本。
设置存储桶的版本控制配置。
public void setBucketVersioning(SetBucketVersioningArgs args)
获取存储桶的版本控制配置。
public VersioningConfiguration getBucketVersioning(GetBucketVersioningArgs args)
示例代码:
// 版本配置// 'my-bucketname'启用版本控制minioClient.setBucketVersioning(SetBucketVersioningArgs.builder().bucket("my-bucketname").config(new VersioningConfiguration(VersioningConfiguration.Status.ENABLED, null)).build());System.out.println("Bucket versioning is enabled successfully");// 'my-bucketname'暂停版本控制minioClient.setBucketVersioning(SetBucketVersioningArgs.builder().bucket("my-bucketname").config(new VersioningConfiguration(VersioningConfiguration.Status.SUSPENDED, null)).build());System.out.println("Bucket versioning is suspended successfully");
4.3.2 对象操作
4.3.2.1 上传对象
- PutObject 调用PutObject接口上传文件(Object)。
public ObjectWriteResponse putObject(PutObjectArgs args)
注意事项:
添加的Object大小不能超过5 GB。
默认情况下,如果已存在同名Object且对该Object有访问权限,则新添加的Object将覆盖原有的Object,并返回200 OK。OSS没有文件夹的概念,所有资源都是以文件来存储,但您可以通过创建一个以正斜线(/)结尾,大小为0的Object来创建模拟文件夹。
示例1,InputStream上传:
// 1. 创建InputStream上传File file = new File("D:\\deploy\\nacos-server-2.0.3.tar.gz");InputStream bais = new FileInputStream(file);long start = System.currentTimeMillis();minioClient.putObject(PutObjectArgs.builder().bucket("my-bucketname").object(file.getName()).stream(bais, bais.available(), -1).build());bais.close();System.out.println("my-objectname is uploaded successfully 耗时:" + (System.currentTimeMillis() - start));
- uploadObject 将文件中的内容作为存储桶中的对象上传。
public void uploadObject(UploadObjectArgs args)
示例:
//将本地文件上传到minio
minioClient.uploadObject(UploadObjectArgs.builder() .bucket("my-bucketname").object("start.sh").filename("D:\\deploy\\service\\general-task\\start.sh").build());
System.out.println("my-filename is uploaded to my-objectname successfully");
4.3.2.2 获取对象
- getObject 用于获取某个文件(Object)。此操作需要对此Object具有读权限。获取对象的数据。InputStream使用后返回必须关闭以释放网络资源。
public InputStream getObject(GetObjectArgs args)
示例:
// 2. 获取对象的InputStream,并保存为文件
InputStream stream =minioClient.getObject(GetObjectArgs.builder().bucket("my-bucketname").object("my-objectname").build());
// 读流
File targetFile = new File("D:\\deploy\\targetFile.tmp");
FileUtils.copyInputStreamToFile(stream, targetFile);
stream.close();
- downloadObject 将对象的数据下载到文件。
public void downloadObject(DownloadObjectArgs args)
示例:
// 4. 下载对象到本地文件
minioClient.downloadObject(DownloadObjectArgs.builder().bucket("my-bucketname").object("my-objectname").filename("D:\\deploy\\service\\general-task\\aaa.tmp").build());
System.out.println("my-objectname is successfully downloaded to my-filename");
- getPresignedObjectUrl
获取一个指定了 HTTP 方法、到期时间和自定义请求参数的对象URL地址,也就是返回带签名的URL,这个地址可以提供给没有登录的第三方共享访问或者上传对象。
public String getPresignedObjectUrl(GetPresignedObjectUrlArgs args)
示例:
// 指定一个GET请求,返回获取文件对象的URL,此URL过期时间为一天
String url =minioClient.getPresignedObjectUrl(GetPresignedObjectUrlArgs.builder().method(Method.GET).bucket("my-bucketname").object("start.sh").expiry(60 * 60 * 24).build());
System.out.println(url);
4.3.2.3 复制对象
- copyObject 通过服务器端从另一个对象复制数据来创建一个对象。
public ObjectWriteResponse copyObject(CopyObjectArgs args)
示例:
// 5. 将my-bucketname中的aa.tmp文件,复制到aaaaa桶下的bb.tmp对象minioClient.copyObject(CopyObjectArgs.builder().bucket("aaaaa").object("bb.tmp").source(CopySource.builder().bucket("my-bucketname").object("aa.tmp").build()).build());System.out.println("successfully");
4.3.2.4 删除对象
- removeObject 移除一个对象。
public void removeObject(RemoveObjectArgs args) 示例:
// 6. 删除单个对象minioClient.removeObject(RemoveObjectArgs.builder().bucket("my-bucketname").object("my-objectname").build());// 删除指定版本号的对象minioClient.removeObject(RemoveObjectArgs.builder().bucket("my-bucketname").object("my-versioned-objectname").versionId("my-versionid").build());
- removeObjects 懒惰地删除多个对象。它需要迭代返回的 Iterable 以执行删除。
public Iterable<Result<DeleteError>> removeObjects(RemoveObjectsArgs args)
示例:
// 7. 删除多个文件List<DeleteObject> objects = new LinkedList<>();objects.add(new DeleteObject("aa.tmp"));objects.add(new DeleteObject("my-objectname"));objects.add(new DeleteObject("nacos-server-2.0.3.tar.gz"));Iterable<Result<DeleteError>> results =minioClient.removeObjects(RemoveObjectsArgs.builder().bucket("my-bucketname").objects(objects).build());for (Result<DeleteError> result : results) {DeleteError error = result.get();System.out.println("Error in deleting object " + error.objectName() + "; " + error.message());}
4.3.2.5 对象信息查询
- listObjects列出桶的对象信息。
public Iterable<Result<Item>> listObjects(ListObjectsArgs args)
示例1,查询存储桶下文件信息:
// 8. 查询存储桶下文件信息Iterable<Result<Item>> results =minioClient.listObjects(ListObjectsArgs.builder().bucket("my-bucketname").build());for (Result<Item> result : results) {Item item = result.get();System.out.println(item.lastModified() + "\t" + item.size() + "\t" + item.objectName());}
示例2,递归查询存储桶下文件信息【存在文件夹时,使用】:
Iterable<Result<Item>> results =minioClient.listObjects(ListObjectsArgs.builder().bucket("my-bucketname").recursive(true).build());for (Result<Item> result : results) {Item item = result.get();System.out.println(item.lastModified() + "\t" + item.size() + "\t" + item.objectName());}
示例2, 条件查询,指定前缀、后缀、最大数量:
// 条件查询,指定前缀、后缀、最大数量
Iterable<Result<Item>> results =minioClient.listObjects(ListObjectsArgs.builder().bucket("my-bucketname").startAfter("ExampleGuide.pdf").prefix("E").maxKeys(100).build()); for (Result<Item> result : results) {Item item = result.get();System.out.println(item.lastModified() + "\t" + item.size() + "\t" + item.objectName());}
- 标签
public void setObjectTags(SetObjectTagsArgs args) 为对象设置标签。
示例:
Map<String, String> map = new HashMap<>();map.put("Project", "Project One");map.put("User", "jsmith");minioClient.setObjectTags(SetObjectTagsArgs.builder().bucket("my-bucketname").object("my-objectname").tags(map).build());
public Tags getObjectTags(GetObjectTagsArgs args) 获取对象的标签。
示例:
Tags tags = minioClient.getObjectTags(GetObjectTagsArgs.builder().bucket("my-bucketname").object("my-objectname").build());
System.out.println("Object tags: " + tags.get());
private void deleteObjectTags(DeleteObjectTagsArgs args) 删除对象的标签。
示例:
minioClient.deleteObjectTags(DeleteObjectTagsArgs.builder().bucket("my-bucketname").object("my-objectname").build());System.out.println("Object tags deleted successfully");
4.4 Springboot集成
4.4.1 环境搭建
<dependency><groupId>org.jetbrains.kotlin</groupId><artifactId>kotlin-stdlib</artifactId><version>1.3.70</version></dependency><dependency><groupId>com.squareup.okhttp3</groupId><artifactId>okhttp</artifactId><version>4.9.0</version></dependency><dependency><groupId>io.minio</groupId><artifactId>minio</artifactId><version>8.2.0</version><exclusions><exclusion><groupId>com.squareup.okhttp3</groupId><artifactId>okhttp</artifactId></exclusion></exclusions></dependency>
4.4.2 代码示例
4.4.2.1 配置参数
package com.minio.learning.springminio.config;import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;/*** @author Jiang*/
@Data
@Component
@ConfigurationProperties(prefix = "minio")
public class MinioProperties {private String endpoint;private String accessKey;private String secretKey;
}
4.4.2.2 配置文件修改
server:port: 8080minio:endpoint: http://localhost:9000accessKey: miniosecretKey: 123ABCdef*bucketName: 202212# byYear byMonth byDatestoragePolicy: no
4.4.2.3 MinioClient创建
package com.minio.learning.springminio.config;import io.minio.MinioClient;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;import javax.annotation.Resource;/*** @author Jiang*/
@Configuration
public class MinioConfig {@Resourceprivate MinioProperties minioProperties;@Beanpublic MinioClient minioClient(){return MinioClient.builder().endpoint(minioProperties.getEndpoint()).credentials(minioProperties.getAccessKey(),minioProperties.getSecretKey()).build();}
}
4.4.2.4 Minio API操作
package com.minio.learning.springminio.service;import com.minio.learning.springminio.entity.FileInfo;
import com.minio.learning.springminio.entity.FileVO;
import com.minio.learning.springminio.entity.RemoveFailVO;
import org.springframework.web.multipart.MultipartFile;import javax.servlet.http.HttpServletResponse;
import java.util.List;/*** @author Jiang* @date 2022-12-01 17:29*/
public interface MinioService {/*** 创建一个桶* @param bucketName bucketName*/boolean createBucket(String bucketName);/*** 上传一个文件* @param bucketName bucketName* @param files files**/List<FileVO> uploadFile(String bucketName, MultipartFile[] files);/*** 列出所有的桶**/List<String> listBuckets();/*** 列出一个桶中的所有文件和目录*/List<FileInfo> listFiles(String bucketName, String prefix);/*** 下载一个文件*/void download(String bucketName, String filePath, HttpServletResponse response);/*** 删除一个桶* @param bucketName bucketName*/boolean deleteBucket(String bucketName);/*** 删除一个对象* @param bucketName bucketName* @param filePath filePath* @return 删除结果*/boolean deleteObject(String bucketName, String filePath);/*** 删除一个或多个对象* @param bucketName bucketName* @param filePaths 文件路径* @return 删除失败列表*/List<RemoveFailVO> deleteObjects(String bucketName, List<String> filePaths);
}
实现类
package com.minio.learning.springminio.service.impl;import com.minio.learning.springminio.entity.FileInfo;
import com.minio.learning.springminio.entity.FileVO;
import com.minio.learning.springminio.entity.RemoveFailVO;
import com.minio.learning.springminio.exception.ServiceException;
import com.minio.learning.springminio.service.MinioService;
import io.minio.*;
import io.minio.messages.Bucket;
import io.minio.messages.DeleteError;
import io.minio.messages.DeleteObject;
import io.minio.messages.Item;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.apache.tomcat.util.http.fileupload.IOUtils;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;
import org.springframework.web.multipart.MultipartFile;import javax.annotation.Resource;
import javax.servlet.http.HttpServletResponse;
import java.io.InputStream;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.time.LocalDateTime;
import java.time.format.DateTimeFormatter;
import java.util.ArrayList;
import java.util.List;/*** @author Jiang* @date 2022-12-01 17:30*/
@Service
@Slf4j
public class MinioServiceImpl implements MinioService {@Resourceprivate MinioClient minioClient;@Value("${minio.storagePolicy}")private String storagePolicy;@Value("${minio.endpoint}")private String endpoint;/*** 创建一个桶** @param bucketName bucketName*/@Overridepublic boolean createBucket(String bucketName) {boolean found = bucketIsExist(bucketName);try {if (!found) {minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).build());}} catch (Exception e) {log.error("minio创建一个桶失败:{}", e.getMessage());throw new ServiceException("创建存储桶失败!");}return true;}/*** 获取文件上传前缀** @return String*/private String getFilePrefix() {String prefix = null;LocalDateTime dateTime = LocalDateTime.now();switch (storagePolicy) {case "byDate":DateTimeFormatter byDate = DateTimeFormatter.ofPattern("yyyy/MM/dd/");prefix = dateTime.format(byDate);break;case "byMonth":DateTimeFormatter byMonth = DateTimeFormatter.ofPattern("yyyy/MM/");prefix = dateTime.format(byMonth);break;case "byYear":prefix = dateTime.getYear() + "/";break;default:prefix="";break;}return prefix;}/*** 上传一个文件** @param bucketName bucketName* @param files files*/@Overridepublic List<FileVO> uploadFile(String bucketName, MultipartFile[] files) {int length = files.length;boolean result = bucketIsExist(bucketName);if (!result) {throw new ServiceException("存储桶不存在!");}List<FileVO> fileList = new ArrayList<>(length);for (MultipartFile file : files) {String uploadFileName = file.getOriginalFilename();try {PutObjectArgs args = PutObjectArgs.builder().bucket(bucketName).object(getFilePrefix() + uploadFileName).stream(file.getInputStream(), file.getSize(), -1).contentType(file.getContentType()).build();minioClient.putObject(args);String url = endpoint + "/" + bucketName + "/" + getFilePrefix() + uploadFileName;FileVO fileVO = new FileVO();fileVO.setFileName(uploadFileName);fileVO.setUrl(url);fileList.add(fileVO);} catch (Exception e) {log.error("minio上传文件失败:{}", e.getMessage());}}return fileList;}/*** 列出所有的桶*/@Overridepublic List<String> listBuckets() {List<Bucket> list;try {list = minioClient.listBuckets();} catch (Exception e) {throw new ServiceException("列出所有的桶失败!");}List<String> names = new ArrayList<>();list.forEach(b -> names.add(b.name()));return names;}/*** 判断存储桶是否存在** @param bucket bucket* @return boolean*/private boolean bucketIsExist(String bucket) {try {return minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucket).build());} catch (Exception e) {throw new ServiceException("系统中断,稍后重试!");}}/*** 列出一个桶中的所有文件和目录** @param bucketName bucketName*/@Overridepublic List<FileInfo> listFiles(String bucketName, String prefix) {boolean result = bucketIsExist(bucketName);if (!result) {throw new ServiceException("存储桶不存在!");}Iterable<Result<Item>> results;if (StringUtils.isNotBlank(prefix)) {results = minioClient.listObjects(ListObjectsArgs.builder().bucket(bucketName).prefix(prefix).recursive(false).build());} else {results = minioClient.listObjects(ListObjectsArgs.builder().bucket(bucketName).recursive(false).build());}DateTimeFormatter fmt = DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss");List<FileInfo> infos = new ArrayList<>();results.forEach(r -> {FileInfo info = new FileInfo();try {Item item = r.get();info.setFilename(item.objectName());boolean dir = item.isDir();info.setDirectory(dir);// 文件夹没有修改时间if (!dir) {info.setLastModifiedTime(item.lastModified().format(fmt));}infos.add(info);} catch (Exception e) {log.info("获取文件失败!{}", e.getMessage());}});return infos;}/*** 下载一个文件** @param bucketName bucketName* @param filePath filePath*/@Overridepublic void download(String bucketName, String filePath, HttpServletResponse response) {try {String fileName = getFileName(filePath, bucketName);// 获取对象信息StatObjectResponse statObjectResponse = minioClient.statObject(StatObjectArgs.builder().bucket(bucketName).object(fileName).build());response.setContentType(statObjectResponse.contentType());response.setHeader("Content-Disposition", URLEncoder.encode(fileName, String.valueOf(StandardCharsets.UTF_8)));InputStream stream = minioClient.getObject(GetObjectArgs.builder().bucket(bucketName).object(fileName).build());IOUtils.copy(stream, response.getOutputStream());} catch (Exception e) {throw new ServiceException("下载失败!");}}/*** 删除一个桶** @param bucketName bucketName*/@Overridepublic boolean deleteBucket(String bucketName) {boolean result = bucketIsExist(bucketName);if (!result) {throw new ServiceException("存储桶不存在!");}try {minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucketName).build());} catch (Exception e) {log.error("minio删除存储桶失败:{}", e.getMessage());throw new ServiceException("删除存储桶失败!");}return true;}/*** 删除一个对象** @param bucketName bucketName* @param filePath filePath* @return 删除结果*/@Overridepublic boolean deleteObject(String bucketName, String filePath) {String fileName = getFileName(bucketName, filePath);try {minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucketName).object(fileName).build());} catch (Exception e) {log.error("minio删除文件失败:{}", e.getMessage());throw new ServiceException("删除文件失败!");}return true;}/*** 获取文件名称** @param filePath 文件路径* @param bucketName 存储桶名称* @return 获取文件名称*/private String getFileName(String filePath, String bucketName) {String prefix = endpoint + "/" + bucketName + "/";int length = prefix.length();return filePath.substring(length);}/*** 删除一个或多个对象* 删除文件夹下文件 /fileName/fileName/sss.txt** @param bucketName bucketName* @param filePaths 文件路径* @return 删除失败列表*/@Overridepublic List<RemoveFailVO> deleteObjects(String bucketName, List<String> filePaths) {List<DeleteObject> list = new ArrayList<>();for (String filePath : filePaths) {String fileName = getFileName(filePath,bucketName);list.add(new DeleteObject(fileName));}Iterable<Result<DeleteError>> iterable = minioClient.removeObjects(RemoveObjectsArgs.builder().bucket(bucketName).objects(list).build());List<RemoveFailVO> failList = new ArrayList<>();for (Result<DeleteError> result : iterable) {DeleteError error = null;try {error = result.get();} catch (Exception e) {log.error("minio删除对象失败:{}", e.getMessage());}assert error != null;log.info("minio删除错误->bucketName={},objectName={},message={}", error.bucketName(), error.objectName(), error.message());RemoveFailVO failVO = new RemoveFailVO();failVO.setObjectName(error.objectName());failVO.setBucketName(error.bucketName());failList.add(failVO);}return failList;}
}
4.4.2.5 接口提供
package com.minio.learning.springminio.controller;import com.minio.learning.springminio.entity.FileInfo;
import com.minio.learning.springminio.entity.FileVO;
import com.minio.learning.springminio.entity.RemoveFailVO;
import com.minio.learning.springminio.resp.CommonResult;
import com.minio.learning.springminio.service.MinioService;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.validation.annotation.Validated;
import org.springframework.web.bind.annotation.*;
import org.springframework.web.multipart.MultipartFile;
import javax.annotation.Resource;
import javax.servlet.http.HttpServletResponse;
import javax.validation.constraints.NotBlank;
import javax.validation.constraints.NotEmpty;
import java.util.List;/*** @author Jiang* @date 2022-12-12 17:08*/
@Slf4j
@Validated
@RestController
@RequestMapping("/minio")
public class MinioController {@Resourceprivate MinioService minioService;@Value("${minio.bucketName}")private String bucket;@GetMapping("/createBucket")public CommonResult<?> createBucket(@RequestParam(name = "bucketName")@NotBlank(message = "存储桶名称不能为空!") String bucketName) {boolean result = minioService.createBucket(bucketName);if (result) {return CommonResult.ok("创建桶成功!");} else {return CommonResult.error("创建桶失败!");}}@GetMapping("/deleteBucket")public CommonResult<?> deleteBucket(@RequestParam(name = "bucketName")@NotBlank(message = "存储桶名称不能为空!") String bucketName) {boolean result = minioService.deleteBucket(bucketName);if (result) {return CommonResult.ok("删除桶成功!");} else {return CommonResult.error("删除桶失败!");}}@PostMapping("/uploadFile")public CommonResult<List<FileVO>> uploadFile(@RequestParam(name = "file")@NotEmpty(message = "上传文件不能为空!") MultipartFile[] files,@RequestParam(name = "bucketName",required = false)String bucketName) {if(StringUtils.isBlank(bucketName)){bucketName = bucket;}List<FileVO> fileVOS = minioService.uploadFile(bucketName, files);if (fileVOS.size() > 0) {return CommonResult.ok("上传成功!",fileVOS);} else {return CommonResult.error("上传失败!");}}@GetMapping("/download")public void uploadFile(@RequestParam(name = "filePath") @NotBlank(message = "下载路径不能为空!") String filePath,HttpServletResponse response,@RequestParam(name = "bucketName",required = false) String bucketName) {if(StringUtils.isBlank(bucketName)){bucketName = bucket;}minioService.download(bucketName, filePath, response);}@GetMapping("/deleteObjects")public CommonResult<List<RemoveFailVO>> deleteObjects(@RequestParam(name = "bucketName",required = false)String bucketName,@RequestParam("filePaths") List<String> filePaths) {if(StringUtils.isBlank(bucketName)){bucketName = bucket;}List<RemoveFailVO> removeFailVOS = minioService.deleteObjects(bucketName, filePaths);if (removeFailVOS.size() > 0) {return CommonResult.error("部分删除失败!", removeFailVOS);} else {return CommonResult.ok("删除成功");}}@GetMapping("/listBuckets")public CommonResult<List<String>> listBuckets() {List<String> buckets = minioService.listBuckets();return CommonResult.ok("查询成功!", buckets);}@GetMapping("/listFiles")public CommonResult<List<FileInfo>> listBuckets(@RequestParam(name = "bucketName",required = false) String bucketName,@RequestParam(name = "prefix", required = false) String prefix) {if(StringUtils.isBlank(bucketName)){bucketName = bucket;}List<FileInfo> fileInfos = minioService.listFiles(bucketName, prefix);return CommonResult.ok("查询成功!", fileInfos);}
}
4.4.2.6 调用示例
http://127.0.0.1:8080/minio/createBucket?bucketName=202213
bucketName = bucket;}minioService.download(bucketName, filePath, response);
}@GetMapping("/deleteObjects")
public CommonResult<List<RemoveFailVO>> deleteObjects(@RequestParam(name = "bucketName",required = false)String bucketName,@RequestParam("filePaths") List<String> filePaths) {if(StringUtils.isBlank(bucketName)){bucketName = bucket;}List<RemoveFailVO> removeFailVOS = minioService.deleteObjects(bucketName, filePaths);if (removeFailVOS.size() > 0) {return CommonResult.error("部分删除失败!", removeFailVOS);} else {return CommonResult.ok("删除成功");}
}@GetMapping("/listBuckets")
public CommonResult<List<String>> listBuckets() {List<String> buckets = minioService.listBuckets();return CommonResult.ok("查询成功!", buckets);
}@GetMapping("/listFiles")
public CommonResult<List<FileInfo>> listBuckets(@RequestParam(name = "bucketName",required = false) String bucketName,@RequestParam(name = "prefix", required = false) String prefix) {if(StringUtils.isBlank(bucketName)){bucketName = bucket;}List<FileInfo> fileInfos = minioService.listFiles(bucketName, prefix);return CommonResult.ok("查询成功!", fileInfos);
}
}
###### 4.4.2.6 调用示例
http://127.0.0.1:8080/minio/createBucket?bucketName=202213