[Ceph](https://ceph.io/en/) is an open-source storage platform that stores its data in a storage cluster based on .
It is highly scalable and, as a distributed system without a single point of failure, very reliable.
#### TIP
If you want to quickly set up a basic Ceph cluster, check out [MicroCeph](https://canonical.com/microcloud).
Ceph provides different components for block storage and for file systems.
Ceph is Ceph’s block storage component that distributes data and workload across the Ceph cluster.
It uses thin provisioning, which means that it is possible to over-commit resources.
## Terminology
Ceph uses the term *object* for the data that it stores.
The daemon that is responsible for storing and managing data is the *Ceph* .
Ceph’s storage is divided into *pools*, which are logical partitions for storing objects.
They are also referred to as *data pools*, *storage pools* or *OSD pools*.
Ceph block devices are also called *RBD images*, and you can create *snapshots* and *clones* of these RBD images.
## `ceph` driver in LXD
#### NOTE
To use the Ceph RBD driver, you must specify it as `ceph`.
This is slightly misleading, because it uses only Ceph RBD (block storage) functionality, not full Ceph functionality.
For storage volumes with content type `filesystem` (images, containers and custom file-system volumes), the `ceph` driver uses Ceph RBD images with a file system on top (see [`block.filesystem`](#storage-ceph-volume-conf:block.filesystem)).
Alternatively, you can use the [CephFS](https://canonical.com/lxd/docs/latest/reference/storage_cephfs/index.html.md#storage-cephfs) driver to create storage volumes with content type `filesystem`.
Unlike other storage drivers, this driver does not set up the storage system but assumes that you already have a Ceph cluster installed.
This driver also behaves differently than other drivers in that it provides remote storage.
As a result and depending on the internal network, storage access might be a bit slower than for local storage.
On the other hand, using remote storage has big advantages in a cluster setup, because all cluster members have access to the same storage pools with the exact same contents, without the need to synchronize storage pools.
The `ceph` driver in LXD uses RBD images for images, and snapshots and clones to create instances and snapshots.
LXD assumes that it has full control over the OSD storage pool.
Therefore, you should never maintain any file system entities that are not owned by LXD in a LXD OSD storage pool, because LXD might delete them.
Due to the way copy-on-write works in Ceph RBD, parent RBD images can’t be removed until all children are gone.
As a result, LXD automatically renames any objects that are removed but still referenced.
Such objects are kept with a `zombie_` prefix until all references are gone and the object can safely be removed.
### Limitations
The `ceph` driver has the following limitations:
Sharing custom volumes between instances
: Custom storage volumes with [content type](https://canonical.com/lxd/docs/latest/explanation/storage/index.html.md#storage-content-types) `filesystem` can usually be shared between multiple instances different cluster members.
However, because the Ceph RBD driver “simulates” volumes with content type `filesystem` by putting a file system on top of an RBD image, custom storage volumes can only be assigned to a single instance at a time.
If you need to share a custom volume with content type `filesystem`, use the [CephFS](https://canonical.com/lxd/docs/latest/reference/storage_cephfs/index.html.md#storage-cephfs) driver instead.
Sharing the OSD storage pool between installations
: Sharing the same OSD storage pool between multiple LXD installations is not supported.
Using an OSD pool of type “erasure”
: To use a Ceph OSD pool of type “erasure”, you must create the OSD pool beforehand.
You must also create a separate OSD pool of type “replicated” that will be used for storing metadata.
This is required because Ceph RBD does not support `omap`.
To specify which pool is “erasure coded”, set the [`ceph.osd.data_pool_name`](#storage-ceph-pool-conf:ceph.osd.data_pool_name) configuration option to the erasure coded pool name and the [`ceph.osd.pool_name`](#storage-ceph-pool-conf:ceph.osd.pool_name) configuration option to the replicated pool name.
## Configuration options
The following configuration options are available for storage pools that use the `ceph` driver and for storage volumes in these pools.
### Storage pool configuration
`ceph.cluster_name`
Name of the Ceph cluster in which to create new storage pools
[](#storage-ceph-pool-conf:ceph.cluster_name)
| **Key:** | `ceph.cluster_name` |
|--------------|-----------------------|
| **Type:** | string |
| **Default:** | `ceph` |
| **Scope:** | global |
`ceph.osd.data_pool_name`
Name of the OSD data pool
[](#storage-ceph-pool-conf:ceph.osd.data_pool_name)
| **Key:** | `ceph.osd.data_pool_name` |
|-------------|-----------------------------|
| **Type:** | string |
| **Scope:** | global |
`ceph.osd.pg_num`
Number of placement groups for the OSD storage pool
[](#storage-ceph-pool-conf:ceph.osd.pg_num)
| **Key:** | `ceph.osd.pg_num` |
|--------------|---------------------|
| **Type:** | string |
| **Default:** | `32` |
| **Scope:** | global |
`ceph.osd.pool_name`
Name of the OSD storage pool
[](#storage-ceph-pool-conf:ceph.osd.pool_name)
| **Key:** | `ceph.osd.pool_name` |
|--------------|------------------------|
| **Type:** | string |
| **Default:** | name of the pool |
| **Scope:** | global |
This option specifies the name of the OSD storage pool.
The OSD storage pool gets created if missing.
`ceph.osd.pool_size`
Number of RADOS object replicas. Set to 1 for no replication.
[](#storage-ceph-pool-conf:ceph.osd.pool_size)
| **Key:** | `ceph.osd.pool_size` |
|--------------|------------------------|
| **Type:** | string |
| **Default:** | `3` |
This option specifies the name for the file metadata OSD pool that should be used when
creating a file system automatically.
`ceph.rbd.clone_copy`
Whether to use RBD lightweight clones
[](#storage-ceph-pool-conf:ceph.rbd.clone_copy)
| **Key:** | `ceph.rbd.clone_copy` |
|--------------|-------------------------|
| **Type:** | bool |
| **Default:** | `true` |
| **Scope:** | global |
Enable this option to use RBD lightweight clones rather than full dataset copies.
`ceph.rbd.du`
Whether to use RBD `du`
[](#storage-ceph-pool-conf:ceph.rbd.du)
| **Key:** | `ceph.rbd.du` |
|--------------|-----------------|
| **Type:** | bool |
| **Default:** | `true` |
| **Scope:** | global |
This option specifies whether to use RBD `du` to obtain disk usage data for stopped instances.
`ceph.rbd.features`
Comma-separated list of RBD features to enable on the volumes
[](#storage-ceph-pool-conf:ceph.rbd.features)
| **Key:** | `ceph.rbd.features` |
|--------------|------------------------------------------|
| **Type:** | string |
| **Default:** | Default features defined in Ceph cluster |
| **Scope:** | global |
`ceph.replicator.`
Name of the peer Ceph site to mirror a project to
[](#storage-ceph-pool-conf:ceph.replicator.)
| **Key:** | `ceph.replicator.` |
|-------------|-------------------------------|
| **Type:** | string |
| **Scope:** | global |
This option specifies the peer site, as registered in Ceph, that the volumes of the
given project are mirrored to. One OSD pool can back several projects, each replicating
to a different peer.
`ceph.rbd.clone_copy` must also be set to `false` before any container is created on the
pool, because Ceph cannot mirror a volume that is a clone of its image.
The project must exist before this option is set. Deleting the project with `--force`
removes the option from the pool.
`ceph.user.name`
The Ceph user to use when creating storage pools and volumes
[](#storage-ceph-pool-conf:ceph.user.name)
| **Key:** | `ceph.user.name` |
|--------------|--------------------|
| **Type:** | string |
| **Default:** | `admin` |
| **Scope:** | global |
`rsync.bwlimit`
Upper limit on the socket I/O for `rsync`
[](#storage-ceph-pool-conf:rsync.bwlimit)
| **Key:** | `rsync.bwlimit` |
|--------------|-------------------|
| **Type:** | string |
| **Default:** | `0` (no limit) |
| **Scope:** | global |
When `rsync` must be used to transfer storage entities, this option specifies the upper limit
to be placed on the socket I/O.
`rsync.compression`
Whether to use compression while migrating storage pools
[](#storage-ceph-pool-conf:rsync.compression)
| **Key:** | `rsync.compression` |
|--------------|-----------------------|
| **Type:** | bool |
| **Default:** | `true` |
| **Scope:** | global |
`source.recover`
Whether to recover an existing `source`
[](#storage-ceph-pool-conf:source.recover)
| **Key:** | `source.recover` |
|--------------|--------------------|
| **Type:** | bool |
| **Default:** | `false` |
| **Scope:** | local |
Set this option to true to recover an existing source which was previously created by LXD.
`user.*`
User-provided free-form key/value pairs
[](#storage-ceph-pool-conf:user.*)
| **Key:** | `user.*` |
|-------------|------------|
| **Type:** | string |
| **Scope:** | global |
`volatile.pool.pristine`
Whether the pool was empty on creation time
[](#storage-ceph-pool-conf:volatile.pool.pristine)
| **Key:** | `volatile.pool.pristine` |
|--------------|----------------------------|
| **Type:** | string |
| **Default:** | `true` |
| **Scope:** | global |
#### TIP
In addition to these configurations, you can also set default values for the storage volume configurations. See storage-configure-vol-default.
### Storage volume configuration
`block.filesystem`
File system of the storage volume
[](#storage-ceph-volume-conf:block.filesystem)
| **Key:** | `block.filesystem` |
|----------------|---------------------------------------------------|
| **Type:** | string |
| **Default:** | same as `volume.block.filesystem` |
| **Condition:** | block-based volume with content type `filesystem` |
| **Scope:** | global |
Valid options: `btrfs`, `ext4`, `xfs`
If not set, `ext4` is assumed.
`block.mount_options`
Mount options for block-backed file system volumes
[](#storage-ceph-volume-conf:block.mount_options)
| **Key:** | `block.mount_options` |
|----------------|---------------------------------------------------|
| **Type:** | string |
| **Default:** | same as `volume.block.mount_options` |
| **Condition:** | block-based volume with content type `filesystem` |
| **Scope:** | global |
`security.shared`
Enable volume sharing
[](#storage-ceph-volume-conf:security.shared)
| **Key:** | `security.shared` |
|----------------|---------------------------------------------|
| **Type:** | bool |
| **Default:** | same as `volume.security.shared` or `false` |
| **Condition:** | virtual-machine or custom block volume |
| **Scope:** | global |
Enable this option to allow the volume to be shared across multiple instances despite the possibility of data loss.
`security.shifted`
Enable ID shifting overlay
[](#storage-ceph-volume-conf:security.shifted)
| **Key:** | `security.shifted` |
|----------------|----------------------------------------------|
| **Type:** | bool |
| **Default:** | same as `volume.security.shifted` or `false` |
| **Condition:** | custom volume |
| **Scope:** | global |
Enable this option to allow the volume to be attached to multiple isolated instances.
`security.unmapped`
Disable ID mapping for the volume
[](#storage-ceph-volume-conf:security.unmapped)
| **Key:** | `security.unmapped` |
|----------------|-----------------------------------------------|
| **Type:** | bool |
| **Default:** | same as `volume.security.unmapped` or `false` |
| **Condition:** | custom volume |
| **Scope:** | global |
`size`
Size/quota of the storage volume
[](#storage-ceph-volume-conf:size)
| **Key:** | `size` |
|----------------|-----------------------|
| **Type:** | string |
| **Default:** | same as `volume.size` |
| **Condition:** | appropriate driver |
| **Scope:** | global |
`snapshots.expiry`
Time until snapshots are deleted
[](#storage-ceph-volume-conf:snapshots.expiry)
| **Key:** | `snapshots.expiry` |
|----------------|-----------------------------------|
| **Type:** | string |
| **Default:** | same as `volume.snapshots.expiry` |
| **Condition:** | custom volume |
| **Scope:** | global |
Specify an expression like `1M 2H 3d 4w 5m 6y`.
`snapshots.pattern`
Template for the snapshot name
[](#storage-ceph-volume-conf:snapshots.pattern)
| **Key:** | `snapshots.pattern` |
|----------------|------------------------------------------------|
| **Type:** | string |
| **Default:** | same as `volume.snapshots.pattern` or `snap%d` |
| **Condition:** | custom volume |
| **Scope:** | global |
You can specify a naming template for scheduled snapshots and unnamed snapshots.
The `snapshots.pattern` option takes a Pongo2 template string to format the snapshot name.
To add a time stamp to the snapshot name, use the Pongo2 context variable `creation_date`.
Make sure to format the date in your template string to avoid forbidden characters in the snapshot name.
For example, set `snapshots.pattern` to `{{ creation_date|date:'2006-01-02_15-04-05' }}` to name the snapshots after their time of creation, down to the precision of a second.
Another way to avoid name collisions is to use the placeholder `%d` in the pattern.
If no matching snapshots exist, the placeholder is replaced with `0`.
Otherwise, it is replaced with the next snapshot index, which is one higher than the highest existing matching snapshot index.
`snapshots.schedule`
Schedule for automatic volume snapshots
[](#storage-ceph-volume-conf:snapshots.schedule)
| **Key:** | `snapshots.schedule` |
|----------------|------------------------------|
| **Type:** | string |
| **Default:** | same as `snapshots.schedule` |
| **Condition:** | custom volume |
| **Scope:** | global |
Specify either a cron expression (``), a comma-separated list of schedule aliases (`@hourly`, `@daily`, `@midnight`, `@weekly`, `@monthly`, `@annually`, `@yearly`), or leave empty to disable automatic snapshots (the default).
`user.*`
User-provided free-form key/value pairs
[](#storage-ceph-volume-conf:user.*)
| **Key:** | `user.*` |
|-------------|------------|
| **Type:** | string |
| **Scope:** | global |
`volatile.devlxd.owner`
ID of the DevLXD identity that owns the volume
[](#storage-ceph-volume-conf:volatile.devlxd.owner)
| **Key:** | `volatile.devlxd.owner` |
|--------------|---------------------------|
| **Type:** | string |
| **Default:** | DevLXD owner identity ID |
| **Scope:** | global |
`volatile.idmap.last`
JSON-serialized UID/GID map that has been applied to the volume
[](#storage-ceph-volume-conf:volatile.idmap.last)
| **Key:** | `volatile.idmap.last` |
|----------------|-------------------------|
| **Type:** | string |
| **Condition:** | filesystem |
`volatile.idmap.next`
JSON-serialized UID/GID map that has been applied to the volume
[](#storage-ceph-volume-conf:volatile.idmap.next)
| **Key:** | `volatile.idmap.next` |
|----------------|-------------------------|
| **Type:** | string |
| **Condition:** | filesystem |
`volatile.uuid`
Volume UUID
[](#storage-ceph-volume-conf:volatile.uuid)
| **Key:** | `volatile.uuid` |
|--------------|-------------------|
| **Type:** | string |
| **Default:** | random UUID |
| **Scope:** | global |