Solaris: remove unus­able SAN disks


Your Solaris sys­tem may have “lost” some SAN disks (for what­ev­er rea­son). If you can not get them back and arrive at the stage where you want to cleanup (if the sys­tem can not do it auto­mat­i­cal­ly), you want to have a solu­tion which does not need much think­ing about rarely exe­cut­ed tasks.


for i in $(luxadm -e port | cut -d : -f 1); do
  luxadm -e forcelip $i
  sleep 10

for i in $(cfgadm -al -o show_FCP_dev | awk '/unusable/ {print $1}' | cut -d , -f 1); do
  cfgadm -c unconfigure -o unusable_SCSI_LUN $i

devfsadm -Cv

If you still have some devices here, you may have to reboot to release some locks or such.

Send to Kin­dle

VMware: set­ting the stor­age device queue depth (HDS fibre chan­nel disks)


In 2017 we replaced an IBM stor­age sys­dem with an Hitachi Van­tara stor­age sys­tem (actu­al­ly, we replaced the com­plete SAN infra­struc­ture). We han­dled it by attach­ing both stor­age sys­tems to VMware (v5.5) and migrat­ing the data­s­tores. A rec­om­men­da­tion from Hitachi Van­tara was to set the queue depth for fibre chan­nel disks to 64.


Here is a lit­tle script which does that. Due to issues as described in a pre­vi­ous post which caused HA/FA (High Avail­abil­i­ty / Fault Tol­er­ance) reac­tions in VMware to trig­ger, we played it safe and added a lit­tle sleep after each change. The script also checks of the queue depth is already set to the desired val­ue and does noth­ing in this case. It’s small enough to just copy&paste it direct­ly into a shell on the host.

SLEEPTIME=210 # 3.5 minutes  !!!! only if all RDMs on the host are reserved!!!
for LDEV in $(esxcli storage core device list | grep "HITACHI Fibre Channel Disk" | awk '{gsub(".*\\(",""); gsub("\\).*",""); print}'); do
  echo $LDEV
  DEPTH="$(esxcli storage core device list -d $LDEV | awk '/outstanding/ {print $8}')"
  if [ "$DEPTH" -ne $TARGET_DEPTH ]; then
    echo "   setting queue depth $TARGET_DEPTH"
    esxcli storage core device set -d $LDEV -O $TARGET_DEPTH
    echo "   sleeping $SLEEPTIME"
    sleep $SLEEPTIME
    echo "    queue depth OK"
Send to Kin­dle

VMware: blank per­for­mance graph issue

Prob­lem / Sto­ry

In 2017 we replaced a stor­age sys­tem with anoth­er stor­age sys­tem (actu­al­ly, we replaced the com­plete SAN infra­struc­ture). We han­dled it by attach­ing both stor­age sys­tems to VMware (v5.5) and migrat­ing the data­s­tores. In this process we stum­bled upon issues which made some hosts unre­spon­sive in VCen­ter (while the VMs were run­ning with­out issues). Before the hosts went unre­spon­sive, the per­for­mance graphs of them start­ed to blank out. So at the moment the issue appeared until it was resolved, any graph con­tin­ued to advance, but had no val­ues list­ed in the cor­re­spond­ing time­frame (left = col­or­ful lines, mid­dle = white space, and after the issue was resolved the col­or­ful lines appeared again). Some times the issue of the blank per­for­mance graph resolved itself, some­times the hosts became unre­spon­sive and VCen­ter greyed them out and trig­gered a HA/FT (High Avail­abil­i­ty / Fault Tol­er­ance) reac­tion.

Root cause

On the cor­re­spond­ing hosts we had RDMs (Raw Device Map­pings) which are used by Microsoft Clus­ter Ser­vice (there is a knowledge-base arti­cle). The issues showed up when we did some SAN oper­a­tions in VMware (like (auto­mat­ic) scan­ning) of new disks after hav­ing pre­sent­ed new disks to VMware. VMware tried to do some­thing clever with the disks (also dur­ing the boot of a host, so if you use RDMs and boot­ing the host takes a long time, you are in the sit­u­a­tion I describe here). If only a small amount of changes hap­pened at the same time, the issues fixed itself. A large amount of changes caused a HA/FT reac­tion.

Workaround when the issue shows up

When you see that the per­for­mance graphs start to show blank space and your VMs are still work­ing, go to the clus­ter set­tings and dis­able vSphere HA (High Avail­abil­i­ty): clus­ter -> “Edit Set­tings” -> “Clus­ter Fea­tures” -> remove check­mark in front of “Turn On vSphere HA”. Wait until the graph shows some val­ues again (for all involved hosts) and then enable vSphere HA again.


To not have this issue show up at all, you need to change some set­tings for the devices on which you have the RDMs. Here is a lit­tle script (small enough to jsut copy&paste it into a shell on the host) which needs the IDs of the devices which are used for the RDMs (atten­tion, let­ters need to be low­er­case) in the “RDMS” vari­able. As we did that on the run­ning sys­tems, and each change of the set­tings caused some action in in he back­ground which made the per­fro­mance graph issue to show up, there is a “lit­tle” sleep between the mak­ing the changes. The amount of sleep depends upon your sit­u­a­tion, the more RDMs are con­fig­ured, the big­ger it needs to be. For us we had 15 of such devices and a sleep of 20 min­utes between each change was enough to not trig­ger a HA/FT reac­tion. The amount of time need­ed in the end is much low­er than in the begin­ning, but as this was more or less an one-off task, this sim­ple ver­sion was good enough (it checks if the set­ting is already active and does noth­ing in this case).

For our use case it was also ben­e­fi­cial to the the path selec­tion pol­i­cy to fixed, so this is also includ­ed in this script. Your use case may be dif­fer­ent.

SLEEPTIME=1200              # 20 minutes per LDEV!
# REPLACE THE FOLLOWING IDs   !!! lower case !!!
RDMS="1234567890abcdef12345c42000002a2 1234567890abcdef12345c42000003a3 \
1234567890abcdef12345c42000003a4 1234567890abcdef12345c42000002a5 \
1234567890abcdef12345c42000002a6 1234567890abcdef12345c42000002a7 \
1234567890abcdef12345c42000003a8 1234567890abcdef12345c42000002a9 \
1234567890abcdef12345c42000002aa 1234567890abcdef12345c42000003ab \
1234567890abcdef12345c42000002ac 1234567890abcdef12345c42000003ad \
1234567890abcdef12345c42000002ae 1234567890abcdef12345c42000002af \

for i in $RDMS; do
  echo $LDEV
  RESERVED="$(esxcli storage core device list -d $LDEV | awk '/Perennially/ {print $4}')"
  if [ "$RESERVED" = "false" ]; then
    echo "   setting prerennially reserved to true"
    esxcli storage core device setconfig -d $LDEV --perennially-reserved=true
    echo "   sleeping $SLEEPTIME"
    sleep $SLEEPTIME
    echo "   setting fixed path"
    esxcli storage nmp device set --device $LDEV --psp VMW_PSP_FIXED                    
    echo "    perennially reserved OK"

Send to Kin­dle

Solaris 10/11(.3) boot panic/crash after mov­ing rpool to a new stor­age sys­tem


The boot disks of some Solaris LDOMs were migrat­ed from one stor­age sys­tem to anoth­er one via ZFS mir­ror­ing the rpool to the new sys­tem and detach­ing the old LUN.


After reboot with on the new stor­age sys­tem Solaris 10 and 11(.3) pan­ic at boot.


  • rpool not on slice 0 but on slice 2
  • bug in Solaris when doing such a mir­ror and “just” doing a reboot <- this is the real issue, it seems Solaris can not han­dle a change of the name of the under­ly­ing device for a rpool, as just mov­ing the par­ti­tion­ing to slice 0 is not fix­ing the pan­ic.


# boot from net­work (or an alter­nate pool which was not yet moved), import/export the pools, boot from the pools
boot net -
# go to shell
# if need­ed: change the par­ti­tion­ing so that slice 0 has the same val­ues as slice 2 (respec­tive­ly make sure the rpool is in slice 0)
zpool import ‑R /tmp/yyy rpool
zpool export rpool


Send to Kin­dle

iocage: HOWTO cre­ate a base­jail from src (instead of from an offi­cial release)


So far I have used ezjail to man­age FreeB­SD jails. I use jails since years to have dif­fer­ent parts of a soft­ware stack in some kind of a con­tain­er (in a ZFS dataset for the filesys­tem side of the con­tain­er). On one hand to not let depen­den­cies of one part of the soft­ware stack have influ­ence of oth­er parts of the soft­ware stack. On the oth­er hand to have the pos­si­bil­i­ty to move parts of the soft­ware stack to a dif­fer­ent sys­tem if nec­es­sary. Nor­mal­ly I run -sta­ble or ‑cur­rent or more gen­er­al­ly speak­ing, a self-compiled FreeB­SD on those sys­tems. In ezjail I like the fact that all jails on a sys­tem have one com­mon base­jail under­ly­ing, so that I update one place for the user­land and all jails get the updat­ed code.

Since a while I heard good things about iocage and how it inte­grates ZFS, so I decid­ed to give it a try myself. As iocage does not come with an offi­cial way of cre­at­ing a base­jail (respec­tive­ly a release) from a self-compiled FreeB­SD (at least doc­u­ment­ed in those places I looked, and yes, I am aware that I can cre­ate a FreeB­SD release myself and use it, but I do not like to have to cre­ate a release addi­tion­al­ly to the build­world I use to update the host sys­tem) here now the short HOWTO achieve this.


In the fol­low­ing I assume the iocage ZFS parts are already cre­at­ed in dataset ${POOLNAME}/iocage which is mount­ed on ${IOCAGE_BASE}/iocage. Addi­tion­al­ly the build­world in /usr/src (or wher­ev­er you have the FreeB­SD source) should be fin­ished.


To have the nec­es­sary dataset-infrastructure cre­at­ed for own basejails/releases, at least one offi­cial release needs to be fetched before. So run the com­mand below (if there is no ${IOCAGE_BASE}/iocage/releases direc­to­ry) and fol­low the on-screen instruc­tions.

iocage fetch


Some vari­ables:

SRC_REV=r$(cd /usr/src; svnliteversion)

Cre­at­ing the iocage basejail-datasets for this ${SRC_REV}:

zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/bin
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/boot
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/lib
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/libexec
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/rescue
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/sbin
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/bin
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/include
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/lib
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/lib32
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/libdata
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/libexec
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/sbin
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/share
zfs create -o compression=lz4 ${POOLNAME}/iocage/base/${SRC_REV}-RELEASE/root/usr/src

Install from /usr/src (the exe­cutable “chown” is hardlinked across an iocage base­jail dataset bound­ary, this fails in the nor­mal install­world, so we have to ignore this error and install a copy of the chown bina­ry to the place where the hardlink nor­mal­ly is):

cd /usr/src
make -i installworld DESTDIR=${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root >&! iocage_installworld_base.log
cp -pv ${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root/usr/sbin/chown ${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root/usr/bin/chgrp
make distribution DESTDIR=${IOCAGE_BASE}/iocage/base/${SRC_REV}-RELEASE/root >>& iocage_installworld_base.log

While we are here, also cre­ate a release and not only a base­jail:

zfs create -o compression=lz4 ${POOLNAME}/iocage/releases/${SRC_REV}-RELEASE
zfs create -o compression=lz4 ${POOLNAME}/iocage/releases/${SRC_REV}-RELEASE/root
make installworld DESTDIR=${IOCAGE_BASE}/iocage/releases/${SRC_REV}-RELEASE/root >&! iocage_installworld_release.log
make distribution DESTDIR=${IOCAGE_BASE}/iocage/releases/${SRC_REV}-RELEASE/root >>& iocage_installworld_release.log

And final­ly make this the default release which iocage uses when cre­at­ing new jails (this is option­al):

iocage set release=${SRC_REV}-RELEASE default

Now the self-build FreeB­SD is avail­able in iocage for new jails.

Send to Kin­dle