Stand Aside, Sysadmin

I’ve recently been involved in a fairly high-visibility pilot deployment of Lotus Connections 3.0 for a large corporate.
Usually I don’t get involved in big heavy IBM software wherever I can avoid it but, but as my involvement in the project would mainly centre around configuration/tuning of RHEL, Apache IHS, and touching on OpenLDAP TDS, I didn’t jump back straight away. It’s also given me the chance to get to know the terribly flabby and over-complex fellow that is IBM WebSphere a whole lot better.

And so onto backup:

The first statement we received from the IBM Connections ‘Guru’ was that Connections cannot be backed up on-line. All databases and application servers must be stopped. Obviously, in a global deployment which is going to be accessed from many different time zones, nightly downtime for backups isn’t acceptable.
The DB2 consultant was perplexed at this requirement – with archive logging enabled, there is no need to shut down the database to create a backup, online backups are perfectly acceptable and widely used.
So, we pushed back and asked them to explain this logic. The response was that the dependency lies with the Files and Wikis applications – both store files on the filesystem, and the metadata for those files is stored within the DB2 database; so in order to take a solid backup, there must be no opportunity for a user to delete or otherwise modify files within the backup window for the application server’s filesystem and the database.
We were then pointed to some docs for Wikis and Files that showed how to use the jython interpreter via the wcadmin interface to pause the file deletion tasks.

Great, I thought, I know a bit of python so it should be pretty straightforward so just script these actions to be called as pre and post scripts, so that when the DB2 backup is called it will SSH to the deployment manager and pause these tasks, dump the databases and application file-storage, then start the tasks running again.
So, I setup a proof-of-concept on a couple of VM’s, tested it and it all worked swimmingly. Problem solved.

And so onto the deployment in the pre-production environment. Multiple application servers serving multiple application clusters, using shared NFS storage for Files and Wikis data, handled by a single deployment manager. Got it all configured and.. FAIL.

WASX7209I: Connected to process "dmgr" on node pgplnkdwdm0001CellManager01
using SOAP connector; The type of process is: DeploymentManager
WASX7303I: The following options are passed to the scripting environment
and are available as arguments that are stored in the argv variable:
"[/opt/IBM/WebSphere/AppServer/profiles/Dmgr01]"
1:
WebSphere:cell=lpgplnkdwdm0001Cell01,name=ActivityService,type=LotusConnections,node=lpgplnkdwas0001Node01,process=activities
2:
WebSphere:cell=lpgplnkdwdm0001Cell01,name=ActivityService,type=LotusConnections,node=lpgplnkdwas0002Node01,process=activities2
Which service do you want to connect to?

When there is more than one node serving that application (as will generally happen when you have a cluster), the script prompts the user to choose which one to connect to.
All I could find relating to this issue was this one un-resolved question on the IBM Dev forums.

So, I started digging around the python modules to see where this was coming from; as well as the main lotusConnectionsCommonAdmin module, each application has it’s own python administration module. The application administration modules are more of a bastardisation of a module and a script. They contain classes, and are generally called in the same way as a module would be, but also have a requirement when there is more than one node available to it, to interact directly with the user and ask which service to connect to.. from stdin.
This is bad programming 101. A module should be a module, and hence not interact directly with a user – if there is a requirement to interact directly with a user, this should be done by a script. Aside from this fact, I can’t believe that IBM have not thought this through at all – generally speaking, backups tend not to be carried out manually by humans. It’s not even a new issue, as it existed in LC 2.5.

So, the options are: Use expect (which is an arcane and dark art – and also quite a ‘hacky’ solution). Or, modify the ‘module’ to not be so stupid.
I chose the latter.

To avoid any future updates overwriting these modules and breaking functionality, I made a copy of the two modules to $module_unattend.py which would be called by my pre and post backup scripts. With that being said, the modifications are very minor, and safe – if they don’t detect the existence of the ‘serviceNum’ integer, they will prompt the user as normal – and so you can safely overwrite the existing module if you’re feeling brave.

As Lotus Connections code is proprietary, I’ve only uploaded patches for the admin modules, which you can find here.
Also included are pause and resume scripts for each application that query the deployment manager for all servers running those applications, and then cycle through each one, actioning the request.

Hopefully IBM fix this issue in later versions; as much as I generally dislike software that is bloated/heavy, java-based and/or proprietary; with the release of version 3.0 (at least from an end-users perspective), Connections has gone from “tries hard, could do better” to a product that almost makes you believe for a second that all of the hype surrounding “Collaboration and Social Computing in the Workplace” isn’t just a set of new clothes for the emperor.