JDF: Jama Distribution Framework
Posted by MarcKlaver on March 28, 2011
This blog describes our framework for file distribution between a SCOM agent and our central SCOM environment. After this framework is installed, you will be able to transfer files between your SCOM agent and a central location. Before you continue, be warned. I can not deliver a single file, which will do all that I just promised. I will however guide you through the work that needs to be done to setup a file distribution framework for SCOM, but you will need to change scripts, compile the management pack and install and configure a secure copy server. Changes will be minimized as much as possible and most actions are automated with scripts. But first things first, let’s describe the general idea behind the framework……
The idea behind the framework is to write an set of scripts, that would make it possible to transfer files to and from the SCOM agent. The main trigger for this wish was our installed base, which are manual installed agents only. Updating these agents manually is a long job for every CU update. If we could just schedule an update from the operations console and sit-back, that would be great. But in order to do that, we need to be able to get the required update files to the SCOM agent. Keep in mind that we do not have a single customer but multiple customers, all with (or without) their own file distribution method. Being able to distribute and update from the operations console itself would be ideal.
But transferring files should also be secure. Our solution: Secure Copy, based on private / public key pairs for data transfers.
And since we have multiple customers, we need to make sure that Customer A is not able to see or change any data we retrieve from Customer B (and vice versa). Therefor we have to implement a secure way to copy files to and from the SCOM agent and separate data based on customers. Our solution: A Secure Copy implementation, that is capable of creating virtual directory structures, based on a customer (secure copy) account.
We did not want to change any firewall rules (with the exception of adding a new management server). So we basically wanted to use communication from the SCOM agent on TCP port 5723 and only initiate communication from the SCOM agent. Our solution: A Secure Copy server configured to listen on port 5723 for secure copy clients.
Finally it needed to be implemented on all our agents, not just a subset of agents. Our solution: A Microsoft Windows Scripting Component, written in vbscript, which will run on all Windows versions we support (Windows 2000 and higher).
So if we draw this in a high overview, it would look like this:
And yes it looks very simple So the first thing we need is a secure copy server.
Secure Copy server
The secure copy server can be any implementation, as long as it is capable of implementing private/public key pairs and able to generate virtual directories. We use the bitvise winsshd service. Now before we start implementing this service, we need to make some rules.
- Only logins with private/public ssh key pairs are allowed.
- Only secure copy is allowed (no sftp or ssh).
- There will only be one general (shared) account and this account has read only access only to it own virtual directories. We call the shared account: jdfBaseDistribution This will prevent accounts from uploading data to a “shared” location. This is also the account that will be used to distribute the framework.
- Data will be compressed as much as possible, but the framework will not rely on the secure copy implementation and will “zip” the files to upload into a “package”.
- When a “package” is downloaded, it is expected to be a “zip” file, which can be extracted at the SCOM agent side by the framework.
- Files (not “packages”) that will be downloaded, must be compressed files to reduce network traffic (this is not forced by the framework).
- Customer accounts will have the naming convention: jdfCustomer_<CustomerName>
- The root (virtual) directory for every account will always be empty. All directories will be read only, with the exception of the “upload” directory. But it will not be possible to remove files from the “upload” directory. The following structure will be used for every customer
Off course you can implement your own directories and rules for read/write, but these are the rules we use to explain the framework. If you start with these, the examples and code will work correctly.
Note: The shared account (jdfBaseDistribution) will not have a /upload directory!
A seperate account for each customer
We also need an account for every customer we support. Each customer will have the same folder layout (as show above), but this will be virtual directories. This will result in the same “view” for every customer, but data will be uploaded and downloaded from customer specific locations on the Secure Copy server.
Note: Files can not be deleted from the upload directory.
As can be seen, the account jdfCustomer_JAMA has four virtual mount paths. And with the exception of the root path (which should never be used), all data is ‘re-directed’ to a directory specific for that account. So if we create a second account (jdfCustomer_JAMA00), you can see that it has its own ‘re-directed’ directories.
This results in the same view for both customers (the Virtual mount path), but data being stored and retrieved from physical different locations.
We only allow the secure copy protocol and only logins with private/public key pairs. On how to configure winsshd with private/public key pairs, see this link.
ssh key pairs
For each customer we need to create an ssh public/private key pair. We use puttygen.exe to create the key pairs. The public key is saved as “jdfCustomer_<CustomerName>.public” and the private key is saved as “jdfCustomer_<CustomerName>.private”.
Note: You can not use a password phrase for the private key file (not supported by the framework). So be sure to only distribute the private key file to the correct customer.
When creating the keys with puttygen.exe, you should use the default Paramters, as shown below:
When your implementation of the secure copy server is working correctly you can continue with the next step: configuring the framework. Just make sure you can use the secure copy server with private/public key pairs before you continue.
The JDF Framework
The JDF Framework itself needs to be configured for use in your environment. What is required is fully described in the documentation of the framework. Although I tried to create a generic framework, still some environment dependent settings are required. But first you should download the framework
If you have setup the framework, you can start testing it. The framework download includes two examples on how to use the framework. The first thing we created with this framework was the ability to update our manual installed agents from the console (we have this working in our test environment). In my next blog I will create a management pack which will update your (manual) installed agents to CU4. Just make sure you got the framework up and running If you fail to do so just write a comment and I will try to answer your questions.