Controlling Costs with TSM

How TSM is charged

TSM charging is based on three components: data transfer, data storage, offsite storage; what you back up, and how much you keep. The rates for these charges are published on the CNS Charging algorithm under the category 'NSAM'.

As of Spring of 2009, these charges added up to roughly $0.35 for each gigabyte transferred, and roughly $0.60 per terabyte-day of storage. Offsite storage is charged identically with onsite storage.

It costs something like a thousand times as much to get data to the server as it does to keep it there. Accordingly, most clients find the majority of their TSM costs to be associated with transfer (what you back up). Storage (what you keep) is much less prominent a factor.

Controlling what you back up

 

Common Problem Files.

Some of the most frequent sources of unexpected usage in TSM are logfiles.  If you have a webserver access log which is simply written to and never rolled or moved, you will end up re-sending the overwhelming bulk of the data over and over again.   More frustrating still, once your file grows beyond a certain size it becomes inevitable that the file will have changed as the backup proceeds.  TSM responds to this by helpfully retrying several times.  It's possible to send logfiles several times this way, and still not have a copy on the server.

If, on the other hand, you periodically "roll" your logs, and copy them to a filename which stays constant, such as access.log-2009-01-05 or similar, then you will get a single, succinct backup of those files, and the size of the active log will be small.  This means that 1) you might get away with sending the whole thing in between appends and 2) the total size of the wasted repeated send will be small enough to neglect.

Be certain not to roll logs to a 'generation-based' filename series, in which yesterday's log is (for example) access.log.1 and then the day before's is access.log.2... In this way you'll have all the same data in the directory, but since you renamed every file, TSM will back it all up again.  Better than the repeat-then-fail case of a single huge log, but not by much.

 

Which files get attention.

TSM grants client nodes subtle and detailed controls over what does and does not get backed up. The most coarse of these controls is the DOMAIN statement in the TSM configuration. This permits you to address or exclude entire 'filespaces' (unix: 'filesystems'; Win: 'disks' or 'shares') from consideration.

Most of the depth of expression of these controls, however, lies in the realm of the 'include/exclude' file. I will include several examples of common interest, but someone who is interested in making use of them ought to read the Client Manuals linked to from the Administrator Documentation page.

[Warning] Warning

It is easy to generate surprising results with include/exclude files. In general, I advise that you avoid fiddling with them except in clearcut cases. Always test your include/exclude configuration on data you are willing to lose. Nowhere in the system is it easier to accidentally tell TSM to simply throw everything away than in include/exclude configuration.

When designing your include/exclude statements, be aware that they constitute a series of tests applied in REVERSE ORDER from that in which they appear in your configuration files. When considering how to add a new rule, work from the bottom up, and know that the first rule which matches the file under consideration terminates include/exclude processing.

Excluding file types

You've decided that your system's users are collecting entirely too many music files, and you want to stop backing them up (at substantial cost, mind you).

 
exclude /home/.../*.mp3
Excluding directories

You have a database installation in /var/lib/pgsql/ and you know that filesystem-level backups of your database containers are useless and expensive. You want to remove that directory from consideration entirely.

 
exclude.dir /var/lib/pgsql/

How frequently files get looked at .

We often think of backups as necessarily daily, but this need not be the case. If you are finding that you need to cut costs, then one of the simplest ways to do so is simply to run incrementals less frequently. TSM schedules can be run with a great deal of flexibility, including specifying a day of the week, specifying a number of days between runs, etc. Contact osg-nsam-l@lists.ufl.edu for more information about configuring these behaviors.

Controlling how much you keep

Usually, transfer traffic will dominate your charges. However, it is possible to arrange things in such a way that storage becomes the most prominent expense.

Number of copies.

TSM retains old data according to parameters which express, among other things, how many old versions of files should be kept around. Though it is not a version-control system, TSM may be misapplied as such and in these applications you would tend to keep many copies of old files. Of course, the number of copies you retain is directly related to the size of your occupation; It is entirely possible to be storing several terabytes of data for a 20GB filespace.

Manual deletion.

Especially for database back-ends (DB2, Oracle) retention of data is managed by hand. You should carefully consider how much of what data you need to retain to satisfy your recovery needs.