I googled a lot regard­ing the error mes­sage “pass­word is not set” when test­ing a data­source in Web­Sphere (7.0.0.21), but I did not find a solu­tion. A co-worker finally found a solu­tion (by accident?).

Prob­lem case

While hav­ing the appli­ca­tion JVMs run­ning, I cre­ated a new JAAS–J2C authen­ti­ca­tor (in my case the same login but a dif­fer­ent pass­word), and changed the data­source to use the new authen­ti­ca­tor. I saved the con­fig and syn­chro­nized it. The files config/cells/cell­name/nodes/node­name/resources.xml and config/cells/cell­name/security.xml showed that the changes arrived on the node. Test­ing the data­source con­nec­tiv­ity fails now with:

DSRA8201W: Data­Source Con­fig­u­ra­tion: DSRA8040I: Failed to con­nect to the Data­Source.  Encoun­tered java.sql.SQLException: The appli­ca­tion server rejected the con­nec­tion. (Pass­word is not set.)DSRA0010E: SQL State = 08004, Error Code = –99,999.

Restart­ing the appli­ca­tion JVMs does not help.

Solu­tion

After stop­ping every­thing (appli­ca­tion JVMs, nodeagent and deploy­ment man­ager) and start­ing every­thing again, the con­nec­tion test of the data­source works directly as expected.

I have not tested if it is enough to just stop all appli­ca­tion JVMs on one node and the cor­re­spding nodeagent, or if I really have to stop the deploy­ment man­ager too.

Recently we had a strange per­for­mance prob­lem at work. A web appli­ca­tion was hav­ing slow response times from time to time and users com­plained. We did not see an uncom­mon CPU/mem/swap usage on any involved machine. I gen­er­ated heat-maps from per­for­mance mea­sure­ments and there where no obvi­ous traces of slow behav­ior. We did not find any rea­son why the appli­ca­tion should be slow for clients, but obvi­ously it was.

Then some­one men­tioned two recent apache DoS prob­lems. Num­ber one — the cookie hash issue — did not seem to be the cause, we did not see a huge CPU or mem­ory con­sump­tion which we would expect to see with such an attack. The sec­ond one — the slow reads prob­lem (no max con­nec­tion dura­tion time­out in apache, can be exploited by a small receive win­dow for TCP) — looked like it could be an issue. The slow read DoS prob­lem can be detected by look­ing at the server-status page.

What you would see on the server-status page are a lot of worker threads in the ‘W’ (write data) state. This is sup­posed to be an indi­ca­tion of slow reads. We did see this.

As our site is behind a reverse proxy with some kind of IDS/IPS fea­ture, we took the reverse proxy out of the pic­ture to get a bet­ter view of who is doing what (we do not have X-Forwarded-For configured).

At this point we noticed still a lot of con­nec­tion in the ‘W’ state from the rev-proxy. This was strange, it was not sup­posed to do this. After restart­ing the rev-proxy (while the clients went directly to the web­servers) we had those ‘W’ entries still in the server-status. This was get­ting really strange. And to add to this, the dura­tion of the ‘W’ state from the rev-proxy tells that this state is active since sev­eral thou­sand sec­onds. Ugh. WTF?

Ok, next step: killing the offend­ers. First I ver­i­fied in the list of con­nec­tions in the server-status (extended-status is acti­vated) that all worker threads with the rev–proxy con­nec­tion of a given PID are in this strange state and no client request is active. Then I killed this par­tic­u­lar PID. I wanted to do this until I do not have those strange con­nec­tions any­more. Unfor­tu­nately I arrived at PIDs which were listed in the server-status (even after a refresh), but not avail­able in the OS. That is bad. Very bad.

So the next step was to move all clients away from one web­server, and then to reboot this web­server com­pletely to be sure the entire sys­tem is in a known good state for future mon­i­tor­ing (the big ham­mer approach).

As we did not know if this strange state was due to some kind of mis-administration of the sys­tem or not, we decided to have the rev-proxy again in front of the web­server and to mon­i­tor the systems.

We sur­vived about one and a half day. After that all worker threads on all web­servers where in this state. DoS. At this point we where sure there was some­thing mali­cious going on (some days later our man­age­ment showed us a mail from a com­pany which offered secu­rity con­sult­ing 2 months before to make sure we do not get hit by a DDoS dur­ing the hol­i­day sea­son… a coincidence?).

Next step, ver­i­fi­ca­tion of miss­ing secu­rity patches (unfor­tu­nately it is not us who decides which patches we apply to the sys­tems). What we noticed is, that the rev-proxy is miss­ing a patch for a DoS prob­lem, and for the web­servers a new fix­pack was sched­uled to be released not far in the future (as of this writ­ing: it is avail­able now).

Since we applied the DoS fix for the rev-proxy, we do not have a prob­lem any­more. This is not really con­clu­sive, as we do not really know if this fixed the prob­lem or if the attacker stopped attack­ing us.

From read­ing what the DoS patch fixes, we would assume we should see some con­tin­u­ous traf­fic going on between the rev-rpoxy and the web­server, but there was noth­ing when we observed the strange state.

We are still not allowed to apply patches as we think we should do, but at least we have a bet­ter mon­i­tor­ing in place to watch out for this par­tic­u­lar prob­lem (acti­vate the extended sta­tus in apache/IHS, look for lines with state ‘W’ and a long dura­tion (col­umn ‘SS’), raise an alert if the dura­tion is higher than the max. possible/expected/desired dura­tion for all pos­si­ble URLs).

I was fight­ing with the right way to add a recent Verisign cer­tifi­cate to a key­store for the IBM HTTP Server (IHS). I have used the ikey­man util­ity on Solaris.

The prob­lem indi­ca­tor was the error mes­sage “SSL0208E: SSL Hand­shake Failed, Cer­tifi­cate val­i­da­tion error” in the SSL log of IHS.

The IBM web­sites where not really help­ful to track down the prob­lem (the miss­ing stuff). The Verisign instruc­tions did not lead to a work­ing solu­tion either.

What was done before: the Verisign Inter­me­di­ate Cer­tifi­cates where imported as “Signer Cer­tifi­cates”, and the cer­tifi­cate for the web­server was imported within “Per­sonal Cer­tifi­cates”. With­out the signer cer­tifi­cates the per­sonal cer­tifi­cate would not import due to an inter­me­di­ate cer­tifi­cated miss­ing (no valid trust-chain).

What I did to resolve the problem:

•  I removed all Verisign certificates.
•  I added the Verisign Root Cer­tifi­cate and the Verisign Inter­me­di­ate Cer­tifi­cate A as a signer cer­tifi­cate (use the “Add” but­ton). I also tried to add the Verisign Inter­me­di­ate Cer­tifi­cate B, but it com­plained that some part of it was already there as part of the Inter­me­di­ate Cer­tifi­cate A. I skipped this part.
•  Then I con­verted the server cer­tifi­cate and key to a PKS12 file via “openssl pkcs12 –export –in server-cert.arm –out cert-for-ihs.p12 –inkey server-key.arm –name name_for_cert_in_ihs”.
• After that I imported the cert-for-ihs.p12 as a “Per­sonal Cer­tifi­cate”. The import dia­log offers 3 items to import. I selected the “name_for_cert_in_ihs” and the one con­tain­ing “cn=verisign class 3 pub­lic pri­mary cer­ti­fi­ca­tion author­ity — g5” (when I selected the 3^rd one too, it com­plained that a part of it was already imported with a dif­fer­ent name).

With this mod­i­fied key­store in place, I just had to select the cer­tifi­cate via “SSLServerCert name_for_cert_in_ihs” in the IHS con­fig and the prob­lem was fixed.

I have a lit­tle prob­lem find­ing a clean solu­tion to the fol­low­ing problem.

A machine with two net­work inter­faces and no default route. The first inter­face gets an IP at boot time and the cor­re­spond­ing sta­tic route is inserted dur­ing boot into the rout­ing table with­out prob­lems. The sec­ond inter­face only gets an IP address when the shared-IP zones on the machine are started, dur­ing boot the inter­face is plumbed but with­out any address. The net­works on those inter­faces are not con­nected and the machine is not a gate­way (this means we have a machine–admin­is­tra­tion net­work and a production-network). The sta­tic routes we want to have for the addresses of the zones are not added to the rout­ing table, because the next hop is not reach­able at the time the routing-setup is done. As soon as the zones are up (and the inter­face gets an IP), a re-run of the routing-setup adds the miss­ing sta­tic routes.

Unfor­tu­nately I can not tell Solaris to keep the sta­tic route even if the next hop is not reach­able ATM (at least I have not found an option to the route com­mand which does this).

One solu­tion to this prob­lem would be to add an address at boot to the inter­face which does not have an address at boot-time ATM (prob­a­bly with the dep­re­cated flag set). The prob­lem is, that this sub­net (/28) has not enough free addresses any­more, so this is not an option.

Another solu­tion is to use a script which re-runs the routing-setup after the zones are started. This is a prag­matic solu­tion, but not a clean solution.

As I under­stand the in.routed man-page in.routed is not an option with the default con­fig, because the machine shall not route between the net­works, and shall not change the rout­ing based upon RIP mes­sages from other machines. Unfor­tu­nately I do not know enough about it to be sure, and I do not get the time to play around with this. I have seen some inter­st­ing options regard­ing this in the man-page, but play­ing around with this and sniff­ing the net­work to see what hap­pens, is not an option ATM. Any­one with a config/tutorial for this “do not broad­cast any­thing, do not accept any­thing from outside”-case (if possible)?

Recently I switched to a new client where the Solaris Secu­rity Toolkit (JASS) is exten­sively used. I am now in the process of updat­ing some things, among them are JET and JASS. As part of this work I reeval­u­ate the local JASS mod­i­fi­ca­tions. Pre­vi­ously a cus­tom JASS pack­age was used, but in case JASS is updated by Ora­cle at some point in time (and an update is really needed, see below), this would need some amount of work to find out the dif­fer­ences and to for­ward port them to the new ver­sion. If every­thing is well doc­u­mented, this should not be hard to do, but the per­son doing the work also needs to find the up-to-date docs.

To make it more easy I decided to change this. I now install the offi­cial JASS pack­age via JET together with the lat­est patch for it, and then let JET copy our mod­i­fi­ca­tions over the installed pack­age. Instead of mod­i­fy­ing exist­ing dri­vers, I cre­ated our own dri­vers with a ref­er­ence to the dri­ver which served as a base.

While doing this I encoun­tered sev­eral short­com­ings of JASS on Solaris 10.

There are sev­eral FS based checks which do not make sense to do for the FS of zones in a global zone (at least not the way I use JASS, so maybe a con­fig­urable way of chang­ing the behav­ior should serve for every­one). If zones are installed in /zones, you do not need to check for files with­out valid UIDs (you surely find a lot of files, as the users are defined inside the zones and not in the global zone) or sim­i­lar things (even not for world writable files, as the zones are installed in a root-access-only sub­tree and inside the zones there may be other secu­rity con­straints con­fig­ured inside JASS, read: it is the respon­si­bil­ity of JASS inside the zone to do this). An easy solu­tion would be to exclude those FS which con­tain zones (and as we only have one sub­tree, I just hard­coded this in sev­eral scripts).

I also miss the pos­si­bil­ity (maybe I over­looked a sim­ple way) for the ssh check to limit the Allow­Root­Lo­gin to spe­cific hosts. JASS only checks yes or no, but can not limit it to spe­cific hosts (e.g. via “Match IP/hostname”). Often you do not need to per­mit root-logins (RBAC/sudo/…), but some­times it is the only way to han­dle a par­tic­u­lar edge-case (or to speed up an action dra­mat­i­cally), and in such cases you do not want to allow root-logins more than necessary.