Fight­ing the Coro­n­avirus with FreeB­SD (Folding@Home)

Pho­to by Fusion Med­ical Ani­ma­tion on Unsplash

Here is a quick HOWTO for those which want to pro­vide some FreeB­SD based com­pute resources to help find­ing vac­cines. I have not made a port out of this and do not know yet if I get the time to make one. If some­one wants to make a port, go ahead, do not wait for me.

UPDATE 2020-03-22: 0mp@ made a port out of this, it is in “biology/lin­ux-foldin­gath­ome”.

  • Down­load the lin­ux RPM of the Folding@Home client (this cov­ers fah­client only).
  • Enable the lin­ux­u­la­tor (ker­nel moduls and linux_base (first part of chap­ter 10.2) is enough).
  • Make sure linprocfs/linsysfs are mount­ed in /com­pat/lin­ux/{proc|sys}.
  • cd /compat/linux
  • tar -xf /path/to/fahclient....rpm
  • add the “fah­client” user (give it a real home direc­to­ry)
  • make sure there is no /compat/linux/dev or alter­na­tive­ly mount devfs there
  • mkdir /compat/linux/etc/fahclient
  • cp /compat/linux/usr/share/doc/fahclient/sample-config.xml /compat/linux/etc/fahclient/config.xml
  • chown -R fahclient /compat/linux/etc/fahclient
  • edit /compat/linux/fahclient/config.xml: mod­i­fy user (manda­to­ry) / team (option­al: FreeB­SD team is 11743) / passkey (option­al) as appro­pri­ate (if you want to con­trol the client remote­ly, you need to mod­i­fy some more parts, but some­how the client “los­es” a filedescrip­tor and stops work­ing as it should if you do that on FreeB­SD)
  • If you have the home direc­to­ries of the users as no-exec (e.g. seper­ate ZFS datasets with exec=off): make sure the home direc­to­ry of the fah­client user has exec per­mis­sions enabled
  • cd ~fahclient (impor­tant! it tries to write to the cur­rent work direc­to­ry when you start it)
  • Start it: /usr/sbin/daemon /compat/linux/usr/bin/FAHClient /compat/linux/etc/fahclient/config.xml --run-as fahclient --pid-file=/var/run/ >/dev/null 2>&1

Per default it will now pick up some SARS-CoV‑2 (COVID-19) relat­ed fold­ing tasks. There are some more con­fig options (e.g. how much of the sys­tem resources are used). Please refer to the offi­cial Folding@Home site for more infor­ma­tion about that. Be also aware that there is a big rise in com­pute resources donat­ed to Folding@Home, so the pool of avail­able work units may be emp­ty from time to time, but they are work­ing on adding more work units. Be patient.

Send to Kin­dle

Strate­gic think­ing, or what I think what we need to do to keep FreeB­SD rel­e­vant

Since I par­tic­i­pate in the FreeB­SD project there are from time to time some voic­es which say FreeB­SD is dead, Lin­ux is the way to go. Most of the time those voic­es are trolls, or peo­ple which do not real­ly know what FreeB­SD has to offer. Some­times those voic­es wear blind­ers, they only see their own lit­tle world (were Lin­ux just works fine) and do not see the big pic­ture (like e.g. com­pe­ti­tion stim­u­lates busi­ness, …) or even dare to look what FreeB­SD has to offer.

Some­times those voic­es raise a valid con­cern, and it is up to the FreeB­SD project to fil­ter out what would be ben­e­fi­cial. Recent­ly there were some mails on the FreeB­SD lists in the sense of “What about going into direc­tion X?”. Some peo­ple just had the opin­ion that we should stay were we are. In my opin­ion this is sim­i­lar­ly bad to blind­ly say­ing FreeB­SD is dead and fol­low­ing the mass­es. It would mean stag­na­tion. We should not hold peo­ple back in explor­ing new / dif­fer­ent direc­tions. Some­one wants to write a ker­nel mod­ule in (a sub­set of) C++ or in Rust… well, go ahead, give it a try, we can put it into the Ports Col­lec­tion and let peo­ple get expe­ri­ence with it.

This dis­cus­sion on the mail­inglists also trig­gered some kind of “where do we see us in the next years” / strate­gic think­ing reflec­tion. What I present here, is my very own opin­ion about things we in the FreeB­SD project should look at, to stay rel­e­vant in the long term. To be able to put that into scope, I need to clar­i­fy what “rel­e­vant” means in this case.

FreeB­SD is cur­rent­ly used by com­pa­nies like Net­flix, NetApp, Cis­co, Juniper, and many oth­ers as a base for prod­ucts or ser­vices. It is also used by end-users as a work-horse (e.g. mailservers, web­servers, …). Stay­ing rel­e­vant means in this con­text, to pro­vide some­thing which the user base is inter­est­ed in to use and which makes it more easy / fast for the user base to deliv­er what­ev­er they want or need to deliv­er than with anoth­er kind of sys­tem. And this in terms of time to mar­ket of a solu­tion (time to deliv­er a ser­vice like a web-/mail-/whatever-server or prod­uct), and in terms of per­for­mance (which not only means speed, but also secu­ri­ty and reli­a­bil­i­ty and …) of the solu­tion.

I have cat­e­go­rized the list of items I think are impor­tant into (new) code/features, docs, pol­ish­ing and project infra­struc­ture. Links in the fol­low­ing usu­al­ly point to documentation/HOWTOs/experiences for/with FreeB­SD, and not to the canon­i­cal entry points of the projects or tech­nolo­gies. In a few cas­es the links point to an expla­na­tion in the wikipedia or to the web­site of the top­ic in ques­tion.


The vir­tu­al­iza­tion train (Open­Stack, Open­Neb­u­la, oVirt, Cloud­Stack, Kuber­netes, Dock­er, Pod­man, …) is run­ning on full speed. The marked is as big/important, that solu­tion providers even do joint ven­tures on cross­ing bor­ders between each oth­ers, e.g. VMware is open­ing up to inte­grate their solu­tion with solu­tions from Amazon/Azure/Google. The under­ly­ing infra­struc­ture is get­ting more and more unim­por­tant, as long as the ser­vices which shall be run per­form as need­ed. Ease of use and time to mar­ket are the key-drivers (the last lit­tle piece of per­for­mance is most­ly impor­tant for com­pa­nies which go to the “edge” (both mean­ings intend­ed in a non-exclusive-or way) like Net­flix for their FreeB­SD based CDN). FreeB­SD is not real­ly par­tic­i­pat­ing in this world. Yes, we had jails way before any­one else out there had some­thing smi­lar, and some even do not have that right now. But if you are real­is­tic, FreeB­SD does not play a major role here. You can do nice things with jails and bhyve, but you have to do it “by hand” (ezjail, iocage and such are improve­ments on the ease of use side, but that is not enough as this is still lim­it­ed on a host cen­tric view). The world has moved on to admin­is­ter­ing a dat­a­cen­ter (to avoid the buzz­words “cloud” or “private-cloud”) with a single-click. In my opin­ion we would need to port sev­er­al of the ini­tialy men­tioned cloud/container man­age­ment solu­tions to FreeB­SD and have them able to han­dle their work via jails and/or bhyve. If FreeB­SD is not able to serve as a build­ing block in this big pic­ture, we will fall off the edge in this par­tic­u­lar IT-area in the long run.

With all the ready-made con­tain­ers avail­able in the inter­net, we should improve our lin­ux­o­la­tor. Our kernel-support for this is lim­it­ed to a 2.6.32-ish ABI ver­sion (it is less than 2.6.32, more like 2.6.16, we are miss­ing epoll and ino­ti­fy sup­port, among oth­ers, but this is the low­est ver­sion glibc in the Cen­tOS 7 based linux_base port is able to run on… and glibc checks the ver­sion num­ber). We need to catch-up to a more recent ver­sion if we want to be able to run those ready-made lin­ux con­tain­ers with­out issue (we can put a lin­ux sys­tem into a jail and start that). If some­one would like to work on that, a good start would be to run the Lin­ux Test Project tests via the lin­ux­u­la­tor and start fix­ing bugs. The last time I did that was in 2007 and about 16% of the test cas­es failed back then. It would be also quite nice if we could inte­grate those lin­ux­u­la­tor tests into the FreeB­SD CI. With improve­ments in the lin­ux­u­la­tor and the above men­tioned vir­tu­al­iza­tion sup­port, we should be able to run more/those linux-images … ehrm, sor­ry, docker/kubernetes/…-containers with­in the lin­ux­u­la­tor.

Fin­ish the work regard­ing ker­beros in base. Cy Schu­bert is/was work­ing on this. I do not know the sta­tus of this, but the “fix­ing 800 ports” part in his mail from May 2018 looks like some more help­ing hands would be ben­e­fi­cial. This would bring us to a bet­ter start­ing point for a more seam­less inte­gra­tion (some ports need “the oth­er” ker­beros).

We have one port (as far as I was able to deter­mine… the exact amount does not real­ly mat­ter) in terms of SDN – net/openvswitch – but the doc­u­men­ta­tion of this … leaves room for improve­ment (ker­nel sup­port / netmap sup­port and func­tion­al­i­ty / FreeB­SD spe­cif­ic HOWTO). As part of the vir­tu­al­i­sa­tion of every­thing we (yes: we – as part of the FreeB­SD hand­book, see docs cat­e­go­ry below for more on this) need to pro­vide this info so that FreeB­SD is able to par­tic­i­pate in this area. We should also have a look at port­ing some more SDN soft­ware, e.g Open­Con­trail now tung­sten­fab­ric (there is an old con­trail port­ing project), Open­Stack Neu­tron, Open­Day­light, … so that users have a choice, respec­tive­ly FreeB­SD can be inte­grat­ed into exist­ing het­ero­ge­neous envi­ron­ments. 

Sen­sors (tem­per­a­ture, volt­age, fans, …), a top­ic with his­to­ry. Short: in the Google Sum­mer of Code 2007 code was pro­duced, com­mit­ted, and then removed again due to a dis­pute. My per­son­al under­stand­ing (very sim­pli­fied) is “remove every­thing because some of the data han­dled by this frame­work shall not be han­dled by this frame­work” (instead of e.g. “remove sen­sor X, this data shall not be han­dled in this way”), and “remove every­thing as this does not han­dle sen­sors of type X which are not used in servers but in enter­prise class >99% non-IT-related sen­sors”. Noth­ing bet­ter has shown up since then. If I look at Win­dow, VMware, Solaris and Lin­ux, I can query sen­sors on my mainboard/chassis/disks/whatever (yes, I am mix­ing some apples with oranges here), plot them in mon­i­tor­ing sys­tems, and get alarms. In FreeB­SD we fail on this top­ic (actu­al­ly mul­ti­ple top­ics) which I con­sid­er to be some­thing basic and manda­to­ry. I do not sug­gest that we com­mit the Google Sum­mer of Code 2007 code. I sug­gest to have a look at what makes sense to do here. Take the exist­ing code and com­mit it, or improve on this code out­side the tree and then com­mit it, or write some­thing new. In the end it does not mat­ter (for an user) which way it is han­dled, as long as we have some­thing which users can use in the end. It sure­ly makes sense to have an OS-provided frame­work of reg­is­ter­ing sen­sors in a cen­tral place (it would sure­ly be nice if you could get the temp/fan val­ues of your graph­ics card… ooops… sor­ry… AI/HPC accel­er­a­tor togeth­er with oth­er sim­i­lar hard­ware data in your hard­ware).

To con­tin­ue play­ing well (not only) in the high-availability area, we should also have a look at get­ting an imple­men­ta­tion of MPTCP (Mut­li­path TCP) into the tree. Apple (and oth­ers) is already using it since 2013 with good ben­e­fits (most prob­a­bly not only for Siri users). There exists some code for FreeB­SD, but it is far from usable and it does not look like there is progress since 2016. We say we have the pow­er to serve, but with the cloud­i­fi­ca­tion of the recent years, all users expect that every­thing is always-on and nev­er fails, and being able to pro­vide the serv­er side of this client-server relat­ed tech­nol­o­gy for those peo­ple which have such high demands is nec­es­sary to not fall behind (do not let us rest on our lau­rels).

Secure­Boot needs also some help­ing hands. At some point oper­at­ing sys­tems which do not sup­port it will not be con­sid­ered by com­pa­nies any­more.

Anoth­er item we should have a look at is to pro­vide means to write ker­nel code in dif­fer­ent lan­guages. Not in the base sys­tem, but at least in ports. If some­one wants to write a ker­nel mod­ule in  C++ or Rust, why not? It offers pos­si­bil­i­ties to explore new areas. There are even reports of expe­ri­ences with dif­fer­ent lan­guages. It does not fit your needs? Well, ignore it and con­tin­ue writ­ing ker­nel code in C, but let oth­er peo­ple which want to use a screw­driv­er instead of a ham­mer do what they want, they will either learn that they should have used a ham­mer, or can report about ben­e­fits about the screw­driv­er.


I think we can improve our end-user docs to the next lev­el. The base sys­tem is already well cov­ered (we can sure­ly find some fea­tures which we could doc­u­ment), but an user does not use FreeB­SD to use FreeB­SD. An user sure­ly has a goal in mind which requires to set­up some kind of ser­vice (mail serv­er, web serv­er, dis­play serv­er (desk­top sys­tem), …). While one could argue that it is the 3rd par­ty project which needs to doc­u­ment how to run their soft­ware on FreeB­SD, I think we need to do our share here too. There are a lot of HOW­TOs for Lin­ux, and then you have to find some tips and tricks to make some­thing work (bet­ter) on FreeB­SD. What I have in mind here is that we should doc­u­ment how to make FreeB­SD par­tic­i­pate in a Win­dows Active Direc­to­ry envi­ron­ment, or in an LDAP envi­ron­ment (as a client), improve the Sam­ba part with FreeB­SD spe­cif­ic parts (like how to make Sam­ba use ZFS snap­shots for Win­dows Shad­ow Copies), con­fig­u­ra­tion man­age­ment tools and so on. I do not talk about pro­vid­ing in-depth docs about the 3rd par­ty soft­ware, but lit­tle HOW­TOs with FreeB­SD spe­cif­ic parts / tips and tricks, and a ref­er­ence to the 3rd par­ty docs. Peo­ple come to us for real-world needs and if we pro­vide them with a head-start of the most com­mon items (e.g. also cov­er­ing nginx or what­ev­er and not only apache httpd) and then guide them to fur­ther docs will improve the val­ue of our hand­book even more for end-users (spe­cial­ly for new­com­ers, but also for expe­ri­enced FreeB­SD users which out of a sud­den now need to do some­thing which they nev­er did before…).

We should also review our docs. The hand­book lists e.g. proc­mail (just an exam­ple…). With proc­mail not being mainained any­more since a long time and known vul­ner­a­bil­i­ties we should replace the info there with info about mail­drop (or any suit­able replace­ment). Care­ful review may also find sim­i­lar items which need some care.

One more item I have in mind in terms of docs for user is the restruc­tur­ing of some parts. Now the world is more think­ing in terms of XaaS (“some­thing as a ser­vice”) we should also have a “cloud” sec­tion (going beyond of what we have in terms of vir­tu­al­iza­tion already) in out hand­book. We can put there items like the exist­ing descrip­tion of vir­tu­al­i­sa­tion items, but also should put there new items like glus­terfs or object stor­age or the hope­ful­ly upcom­ing pos­si­bil­i­ty of how to set­up OpenStack/kubernetes/… on FreeB­SD. This goes into the same direc­tion as the first docs-item in terms of pro­vide more doc­u­men­ta­tion how to achieve goals of our users. 

In my opin­ion we are also lack­ing on the developer-documentation side. Yes, we have man pages which describe the offi­cial API (in most cas­es). Where I see room for improve­ment is the source code doc­u­men­ta­tion. Some­thing like doxy­gen (or what­ev­er the tool of the day is – which one does not real­ly mat­ter, any kind of extractable-from-source doc­u­men­ta­tion is bet­ter than no extractable doc­u­men­ta­tion) is already used in sev­er­al places in our source (search for it via: egrep ‑R ‘\\(brief|file)’ /usr/src/) and we have already some infra­struc­ture to extract and ren­der (HTML / PDF) them. The more acces­si­ble / easy it is to start devel­op­ment in FreeB­SD, the more attrac­tive it will be (addi­tion­al to the exist­ing ben­e­fits) to peo­ple / com­pa­nies to dive in. The best exam­ples about doc­u­ment­ing source code in our code I have found so far is the isci and ocs_fc device code.


Pol­ish­ing some­thing in the top­ic of stay­ing rel­e­vant? Yes! It is the details which mat­ter. If peo­ple have 2 options with rough­ly the same fea­tures (noth­ing miss­ing what you need, same price), which one do they take, the one which has every­thing con­sis­tent and well inte­grat­ed, or the one with some quirks you can cir­cum­vent with a lit­tle bit of work on their side?

We have some nice fea­tures, but we are not using it to the extend pos­si­ble. One of the items which come to my mind is DTrace. The area which I think needs pol­ish­ing is to add more probes, and to have some kind of probe-convention about com­mon top­ics. For exam­ple I/O relat­ed nam­ing con­ven­tion (maybe area spe­cif­ic, like stor­age I/O and net­work I/O) and cov­er­ing all dri­vers to com­ply. We should also look into mak­ing it more acces­si­ble by pro­vid­ing more easy inter­faces (no mat­ter if text based (thanks to Devin Teske for dwatch, more of this mag­ic please…), web based, or what­ev­er) to make it real­ly easy (= start a com­mand or click around and you get the result for a spe­cif­ic set of probes/conditions/…). Some exam­ples are statemaps, flamegraphs and most promi­nent­ly the Oracle/Sun ZFS Stor­age Ana­lyt­ics to give you an idea what is pos­si­ble with DTrace and and how to make it acces­si­ble to peo­ple with­out knowl­edge about the ker­nel inter­nals and pro­gram­ming.

Some pol­ish­ing in the ports col­lec­tion would be to revis­it the defaults options for ports with options. The tar­get here should be to have con­sis­tent default set­tings (e.g. serv­er soft­ware should not depend upon X11 by default (direct­ly or indi­rect­ly), most peo­ple should not need to build the port with non-default options). One could argue that it is the respon­s­abil­i­ty of the main­tain­er of the port, and to some extend it is, but we do not have guide­lines which help here. So a lit­tle team of peo­ple to review all ports (and mod­i­fy them if nec­es­sary) and come up with guide­lines and exam­ples would be great.

Addi­tion­al­ly we should come up with meta-ports for spe­cif­ic use case, e.g. web­serv­er (dif­fer­ent flavours… apache/nginx/…), data­base (dif­fer­ent flavours, with some use­ful tools like mytop or mysql­tuner or sim­i­lar) and then even ref­er­ence them in the hand­book (this goes along with my sug­ges­tion above to doc­u­ment real-world use cas­es instead of “only” the OS itself).

Recent­ly ‑cur­rent has seen some low lev­el per­for­mance improve­ments (via ifuncs, …). We should con­tin­ue this and even extend it to revise default set­tings / val­ues (non auto-tuned and even auto-tuned ones). I think it would be ben­e­fi­cial in the long run if we tar­get more cur­rent hard­ware (with­out los­ing the abil­i­ty to run on old­er hard­ware) and for those val­ues which can not be auto-tuned pro­vide some way of down-tuning (e.g. in a failsafe-boot set­ting in the loader or in doc­u­ment­ed set­tings for rc.conf or wher­ev­er those defaults can be changed).

Project infra­struc­ture

We have a CI (con­tin­u­ous inte­gra­tion) sys­tem, but it is not very promi­nent­ly placed. Just recent­ly it gained some more atten­tion from the devel­op­er side and we even got the first sta­tus report about it (nice! vis­i­bil­i­ty helps mak­ing it a part of the com­mu­ni­ty effort). There is a FreeBSD-wiki page about the sta­tus and the future ideas, but it was not updat­ed since sev­er­al months. There is also a page which talks in more details about using it for per­for­mance test­ing, which is some­thing peo­ple have talked about since years but nev­er became avail­able (and is not today).

I think we need to improve here. The goals I think which are impor­tant are to get var­i­ous test­ing, san­i­tiz­ing and fuzzing tech­nolo­gies inte­grat­ed into our CI. On the con­fig repos­i­to­ry I have not found any inte­gra­tion of e.g. the cor­re­spond­ing clang tech­nolo­gies (fuzzing, ASAN, UBSAN, MSAN (still exper­i­men­tal, so maybe not to tar­get before oth­er mature tech­nolo­gies)) or any oth­er such tech­nol­o­gy.

We should also make our CI more public/visible (build sta­tus linked some­where on, nag peo­ple more about issues found by it, have some docs how to add new tests (maybe from ports), so that more peo­ple can help in extend what we auto­mat­i­cal­ly test (e.g. how could I inte­grate the LTP (Lin­ux Test Project) tests to test our lin­ux­u­la­tor? This requires the down­load of a lin­ux dist port, the LTP itself, and then to run the tests). There are a lot of nice ideas float­ing around, but I have the impres­sion we are lack­ing some help­ing hands to get var­i­ous items inte­grat­ed.


Var­i­ous items I talked above are not sexy. Those are typ­i­cal­ly not the things peo­ple do just for fun. Those are typ­i­cal­ly items peo­ple get paid for. It would be nice if some of the com­pa­nies which ben­e­fit from FreeB­SD would be so nice to lend a help­ing hand for one or anoth­er item. Maybe the FreeB­SD Foun­da­tion has some con­tacts they could ask about this?

It could also be that for some of the items I men­tioned here there is more ongo­ing that I know of. This means then that the cor­re­spond­ing work could be made more known on the mail­inglists. When it is more known, maybe some­one wants to pro­vide a help­ing hand.

Send to Kin­dle

Solaris: remove unus­able SAN disks


Your Solaris sys­tem may have “lost” some SAN disks (for what­ev­er rea­son). If you can not get them back and arrive at the stage where you want to cleanup (if the sys­tem can not do it auto­mat­i­cal­ly), you want to have a solu­tion which does not need much think­ing about rarely exe­cut­ed tasks.


for i in $(luxadm -e port | cut -d : -f 1); do
  luxadm -e forcelip $i
  sleep 10

for i in $(cfgadm -al -o show_FCP_dev | awk '/unusable/ {print $1}' | cut -d , -f 1); do
  cfgadm -c unconfigure -o unusable_SCSI_LUN $i

devfsadm -Cv

If you still have some devices here, you may have to reboot to release some locks or such.

Send to Kin­dle

VMware: set­ting the stor­age device queue depth (HDS fibre chan­nel disks)


In 2017 we replaced an IBM stor­age sys­dem with an Hitachi Van­tara stor­age sys­tem (actu­al­ly, we replaced the com­plete SAN infra­struc­ture). We han­dled it by attach­ing both stor­age sys­tems to VMware (v5.5) and migrat­ing the data­s­tores. A rec­om­men­da­tion from Hitachi Van­tara was to set the queue depth for fibre chan­nel disks to 64.


Here is a lit­tle script which does that. Due to issues as described in a pre­vi­ous post which caused HA/FA (High Avail­abil­i­ty / Fault Tol­er­ance) reac­tions in VMware to trig­ger, we played it safe and added a lit­tle sleep after each change. The script also checks of the queue depth is already set to the desired val­ue and does noth­ing in this case. It’s small enough to just copy&paste it direct­ly into a shell on the host.

SLEEPTIME=210 # 3.5 minutes  !!!! only if all RDMs on the host are reserved!!!
for LDEV in $(esxcli storage core device list | grep "HITACHI Fibre Channel Disk" | awk '{gsub(".*\\(",""); gsub("\\).*",""); print}'); do
  echo $LDEV
  DEPTH="$(esxcli storage core device list -d $LDEV | awk '/outstanding/ {print $8}')"
  if [ "$DEPTH" -ne $TARGET_DEPTH ]; then
    echo "   setting queue depth $TARGET_DEPTH"
    esxcli storage core device set -d $LDEV -O $TARGET_DEPTH
    echo "   sleeping $SLEEPTIME"
    sleep $SLEEPTIME
    echo "    queue depth OK"
Send to Kin­dle

VMware: blank per­for­mance graph issue

Prob­lem / Sto­ry

In 2017 we replaced a stor­age sys­tem with anoth­er stor­age sys­tem (actu­al­ly, we replaced the com­plete SAN infra­struc­ture). We han­dled it by attach­ing both stor­age sys­tems to VMware (v5.5) and migrat­ing the data­s­tores. In this process we stum­bled upon issues which made some hosts unre­spon­sive in VCen­ter (while the VMs were run­ning with­out issues). Before the hosts went unre­spon­sive, the per­for­mance graphs of them start­ed to blank out. So at the moment the issue appeared until it was resolved, any graph con­tin­ued to advance, but had no val­ues list­ed in the cor­re­spond­ing time­frame (left = col­or­ful lines, mid­dle = white space, and after the issue was resolved the col­or­ful lines appeared again). Some times the issue of the blank per­for­mance graph resolved itself, some­times the hosts became unre­spon­sive and VCen­ter greyed them out and trig­gered a HA/FT (High Avail­abil­i­ty / Fault Tol­er­ance) reac­tion.

Root cause

On the cor­re­spond­ing hosts we had RDMs (Raw Device Map­pings) which are used by Microsoft Clus­ter Ser­vice (there is a knowledge-base arti­cle). The issues showed up when we did some SAN oper­a­tions in VMware (like (auto­mat­ic) scan­ning) of new disks after hav­ing pre­sent­ed new disks to VMware. VMware tried to do some­thing clever with the disks (also dur­ing the boot of a host, so if you use RDMs and boot­ing the host takes a long time, you are in the sit­u­a­tion I describe here). If only a small amount of changes hap­pened at the same time, the issues fixed itself. A large amount of changes caused a HA/FT reac­tion.

Workaround when the issue shows up

When you see that the per­for­mance graphs start to show blank space and your VMs are still work­ing, go to the clus­ter set­tings and dis­able vSphere HA (High Avail­abil­i­ty): clus­ter -> “Edit Set­tings” -> “Clus­ter Fea­tures” -> remove check­mark in front of “Turn On vSphere HA”. Wait until the graph shows some val­ues again (for all involved hosts) and then enable vSphere HA again.


To not have this issue show up at all, you need to change some set­tings for the devices on which you have the RDMs. Here is a lit­tle script (small enough to jsut copy&paste it into a shell on the host) which needs the IDs of the devices which are used for the RDMs (atten­tion, let­ters need to be low­er­case) in the “RDMS” vari­able. As we did that on the run­ning sys­tems, and each change of the set­tings caused some action in in he back­ground which made the per­fro­mance graph issue to show up, there is a “lit­tle” sleep between the mak­ing the changes. The amount of sleep depends upon your sit­u­a­tion, the more RDMs are con­fig­ured, the big­ger it needs to be. For us we had 15 of such devices and a sleep of 20 min­utes between each change was enough to not trig­ger a HA/FT reac­tion. The amount of time need­ed in the end is much low­er than in the begin­ning, but as this was more or less an one-off task, this sim­ple ver­sion was good enough (it checks if the set­ting is already active and does noth­ing in this case).

For our use case it was also ben­e­fi­cial to the the path selec­tion pol­i­cy to fixed, so this is also includ­ed in this script. Your use case may be dif­fer­ent.

SLEEPTIME=1200              # 20 minutes per LDEV!
# REPLACE THE FOLLOWING IDs   !!! lower case !!!
RDMS="1234567890abcdef12345c42000002a2 1234567890abcdef12345c42000003a3 \
1234567890abcdef12345c42000003a4 1234567890abcdef12345c42000002a5 \
1234567890abcdef12345c42000002a6 1234567890abcdef12345c42000002a7 \
1234567890abcdef12345c42000003a8 1234567890abcdef12345c42000002a9 \
1234567890abcdef12345c42000002aa 1234567890abcdef12345c42000003ab \
1234567890abcdef12345c42000002ac 1234567890abcdef12345c42000003ad \
1234567890abcdef12345c42000002ae 1234567890abcdef12345c42000002af \

for i in $RDMS; do
  echo $LDEV
  RESERVED="$(esxcli storage core device list -d $LDEV | awk '/Perennially/ {print $4}')"
  if [ "$RESERVED" = "false" ]; then
    echo "   setting prerennially reserved to true"
    esxcli storage core device setconfig -d $LDEV --perennially-reserved=true
    echo "   sleeping $SLEEPTIME"
    sleep $SLEEPTIME
    echo "   setting fixed path"
    esxcli storage nmp device set --device $LDEV --psp VMW_PSP_FIXED                    
    echo "    perennially reserved OK"

Send to Kin­dle