Linstor 4.22 shared storagepool support - #13910
Open
rp- wants to merge 15 commits into
Open
Conversation
Resources in a LINSTOR shared storage pool live on one shared LUN and are active on at most one node; there is no DRBD replication or dual-primary. Handle them in the KVM storage adaptor and MS driver: - resolve the actual resource-definition name (template resources can be shared under a different name) before connect/copy operations - on live migration create and activate the resource on the destination node instead of setting DRBD allow-two-primaries - on disconnect delete the local resource only if another active resource exists (during live migration two resources are active at the same time) - getDevicePath: only accept resources that expose a device path, inactive shared resources have none - createResourceBase returns the actually used resource name
Client for LINSTOR REST API 1.29.0 (shipped with linstor-server 1.35.0): make-available gained the auto_manage_dual_primary option, a new unmake-available endpoint was added and resourceSnapshotDelete grew a delete_empty_resource_definition parameter. Adapt the snapshot delete call sites to the new signature (passing null keeps the previous behavior).
LINSTOR REST API 1.29.0 moves the live-migration preparation into the controller: make-available with auto_manage_dual_primary sets DRBD allow-two-primaries (and protocol C) between the migration source and destination, or activates shared storage pool resources on both nodes; unmake-available reverts this on the source node and deletes diskless or redundant shared copies while keeping tiebreaker and diskful resources. connectPhysicalDisk now issues a single make-available with auto_manage_dual_primary set for migrations, and tryDisconnectLinstor a single unmake-available. The controller REST API version is probed once per controller URL; against controllers older than 1.29 the previous manual dual-primary, activate and delete handling is used unchanged.
- getSnapshotPath: thick LVM snapshot LVs use the same naming scheme as thin ones - backup: thick LVM snapshot LVs of inactive origins are inactive and carry the skip-activation flag; activate them (lvchange -ay -K) for the copy to secondary storage and deactivate them again afterwards - prefer the in-use node, then a node with an active resource, when selecting the diskful endpoint: dm-snapshot is not cluster aware, so shared storage pool snapshots must be read on the node the origin volume is active on
Shared storage pool resources of stopped VMs are INACTIVE and expose no device on any node. Activate them through the Linstor controller (which takes the shared-space activation lock) before sending the backup or revert copy command to the host and deactivate them again afterwards. The agent itself must only ever activate the snapshot LV, never the resource, to not bypass the shared-space locking.
The KVM start flow builds the domain XML (createVbd) before connectPhysicalDisk runs make-available, so getDevicePath must not fail for resources that are inactive on every node - the start would spin on the missing device path and never reach the activating connect. Return the deterministic storage layer path instead; make-available activates the resource before the domain is started. Also add the plugin CHANGELOG entries for shared storage pool support.
Activating a thick LVM snapshot LV for the backup copy registers it with dmeventd by default; that registration can block lvchange indefinitely (observed stuck in pipe_read waiting for a dmeventd reply), and a blocked lvm command holds the node-wide LVM lock. Pass --monitor n on the temporary activation and deactivation: monitoring only exists to auto-extend snapshots, and Linstor creates thick snapshot LVs at full origin size, so they can never overflow. Also raise the command timeout.
Shared storage pool templates are INACTIVE while no clone is running and Linstor's clone source selection skips inactive resources, failing with 'No suitable storage pools found for cloning'. Activate a diskful template resource for the duration of the clone and deactivate it again afterwards.
copyVolumeFromPrimaryToSecondary read the source device without connecting it first. Detached volumes being migrated between pools are not connected anywhere, and storage drivers that expose devices on demand (e.g. Linstor shared storage pools, where resources of stopped VMs are inactive without a device path) fail the copy. Connect the source volume for the duration of the copy, mirroring what VM start and attach flows do.
Clone requests without an explicit layer list can produce a target with a default DRBD layer stack while the source on a shared storage pool is STORAGE only, failing with 'no common clone strategy found'. Set the resource group's configured layer stack on the clone request for unencrypted volumes, mirroring what is already done for encrypted ones.
Rely on the explicit layer stack instead of the resource group default when spawning unencrypted resources. A template spawned with a DRBD stack cannot be used as a clone source for STORAGE or LUKS,STORAGE resources of a shared storage pool later on (the clone either fails with 'no common clone strategy found' or hangs in state CLONING).
Every node accessing a shared space (e.g. a LUN with a shared LVM volume group) reports the full capacity of that space, so summing all storage pools of the resource group multiplied the reported capacity by the number of nodes: a 100 GiB shared LUN on three nodes was reported as 300 GiB (and 600 GiB after the over-provisioning factor). Deduplicate storage pools by their shared space when calculating total, used and free capacity.
Thickly provisioned backends (thick LVM, ZFS with reservation, file) allocate the full volume size, so the pool capacity is a hard limit and the storage.overprovisioning.factor default of 2 makes CloudStack allocate twice the space that exists. The Linstor storage pool type has to allow over-provisioning for the thin backends, so set the factor to 1.0 per pool when all storage pools of the resource group are thick. Mixed or unknown setups keep the default.
Linstor controllers older than 1.29 reject requests with unknown properties, so a make-available carrying auto_manage_dual_primary fails against them - which broke every make-available call (and with it template copies and VM starts) even though the live-migrate API is version gated: the generated client initializes fields with their schema default (false here) and serialized everything non-null, so the property was always sent.
getDiskfulStoragePools only ever returned the storage pool of a single node, so an endpoint was only looked for on that one node - if its cloudstack host was down or disabled the operation fell back to a temporary resource or a diskless attach although other diskful copies were usable. Return all diskful storage pools ordered by how suited the copy is (in use, then active, then inactive) and walk that order when picking a host. The order matters: thick LVM snapshots on shared storage pools must be read on the node the volume is active on, so the host selection no longer shuffles candidates.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 4.22 #13910 +/- ##
==========================================
Coverage 17.71% 17.71%
- Complexity 15858 15884 +26
==========================================
Files 5926 5926
Lines 533613 533852 +239
Branches 65285 65326 +41
==========================================
+ Hits 94517 94589 +72
- Misses 428415 428574 +159
- Partials 10681 10689 +8
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds support for LINSTOR shared storage pools (thick LVM on a LUN that all
hypervisors can access, SAN-style) to the Linstor volume plugin.
Until now the plugin assumed every resource is DRBD replicated and thin provisioned. On a
shared storage pool the data exists once on the LUN, a resource is active on at most one
node at a time, there is no DRBD layer to make dual-primary, and the backing LVM is thick.
Functionally this changes:
make-availablewithauto_manage_dual_primaryprepares the destination (DRBD dual-primary, or activating theresource on both nodes for shared pools) and
unmake-availablereverts it on the source.The controller REST API version is probed once per controller; against controllers older
than 1.29 the previous manual
allow-two-primarieshandling is used unchanged, soexisting DRBD/thin deployments behave exactly as before.
now activated on demand (through the controller, so the shared-space lock is honored) for
VM start, snapshot backup, snapshot revert and template cloning, and deactivated again
afterwards.
snapshot LV is temporarily activated for the copy to secondary storage (without
registering it with
dmeventd, which could block the node wide LVM lock), and the copy isrouted to the node the volume is active on, because dm-snapshot is not cluster aware.
on three nodes was reported as 300 GiB), and thickly provisioned pools get a per pool
storage.overprovisioning.factorof 1.0 at registration, since they cannot be overprovisioned. Thin pools keep the current default.
group instead of the implicit default, which otherwise produced DRBD resources in a
STORAGE-only resource group.
One change is outside the plugin:
KVMStorageProcessor.copyVolumeFromPrimaryToSecondary()now connects the source volume before reading it and disconnects it afterwards, mirroring
what the VM start and attach flows do. Migrating a detached volume between pools
otherwise reads a device that was never connected, which fails for any storage driver that
exposes devices on demand.
Requires
java-linstor0.8.0 (LINSTOR REST API 1.29.0, shipped with linstor-server 1.35).Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
n/a - no UI or API changes.
How Has This Been Tested?
Two 3-node KVM clusters (Ubuntu 24.04, CloudStack 4.22, one management server each), so
both storage models and both controller API versions are covered:
STORAGElayer only, place-count 1lvmthin) + DRBD, 2 replicasMarvin,
test/integration/plugins/linstor:test_linstor_volumes.py(15 tests): 15/15 pass on both clusters - includes volumeattach/detach/reboot, snapshot create, template from snapshot, volume migration to the
same and to a distinct pool, and VM snapshot create/revert/delete.
test_linstor_encrypted_snapshots.py(3 tests): 3/3 pass on cluster A(revert of an encrypted root snapshot, create-volume-from-encrypted-snapshot is rejected,
and the qcow2 on secondary storage is verified to be LUKS encrypted at rest). On cluster B
the third test skips itself because it needs DB access from the test runner to locate the
backed-up file.
provider-kind detection, diskful copy ordering and the controller version gate.
Manually verified on cluster A, checking the LINSTOR side after each step: VM deploy from
template, stop/start of a VM whose resource is INACTIVE, live migration (controller log
shows make-available on the destination and unmake-available on the source; the resource
ends up active on the destination only), snapshot of a running and of a stopped VM,
template from snapshot, volume migration between pools, and capacity (the 100 GiB LUN is
now reported as 100 GB instead of 599.98 GB).
How did you try to break this feature and the system with this change?
regenerated client serialized
auto_manage_dual_primaryon every make-available (thegenerated field is initialized with its schema default), and LINSTOR rejects unknown
properties - so template copies and VM starts failed even though the feature is version
gated. Fixed by only setting the flag for migrations (and, in java-linstor, by serializing
only explicitly set properties). Full suite passes on 1.28.0 afterwards, i.e. no
regression for existing deployments.
broke template cloning, VM start and snapshots, which all assumed an active resource;
each path now activates on demand and was re-tested from a rebooted cluster.
after a clone raced other clones still using it, so the template is left active.
temporary resource; all diskful copies are considered, in preference order.
for
lvmthin(dedup keys on the shared space, thick detection requires all pools of theresource group to be thick, mixed setups keep the current default).
and on a running VM, and repeated snapshot/delete cycles to check nothing is left behind
in LINSTOR.
lvchangeregistering a snapshot LV withdmeventdcould hang andblock the node wide LVM lock, wedging all storage operations on that host - the temporary
activation now uses
--monitor n.Testing this also uncovered several LINSTOR server side issues (satellite startup probe
racing on a shared VG, snapshot rollback with an inactive copy, an io-suspend hang on
encrypted devices, a clone that never left
CLONING); those are fixed in linstor-server1.35 and are not CloudStack changes.