Reclamation is the process of reclaiming unused space on a tape (applies to Virtual Tapes as well). Over time the TSM expiration process will expire old data, creating gaps of unused storage on the tapes.  Because Tapes are sequential media, TSM can only write data to the end of the tape, so these gaps of “Empty Space” cannot be used.

Lets see an illustration of this

Tape 1 is a new tape and data is written to it as follows (each number on the following line represents a file being backed up)

1-2-3-4-5-6-7-8-9-10-11-12-13-14-15-16

Over time the expiration runs and expires old files (The * represents an expired file)

1-2-3-*-5-*-*-8-9-*-*-*-13-14-15-16

TSM refers to this space ‘*’ as Reclaimable Space.

TSM can only write to the end of the tape and the Reclaimable Space cannot be reused until all the data on the tapes expires and the tape becomes a scratch tape again.

To view this reclaimable space

QUERY VOLUME volno F=D   (where volno is the Volume Number for the Tape ) and look at the Pct Reclaimable Space field.

When using TSM Studio open the Volumes dataview from the Libraries and Volumes Category and view the Pct Reclaim Column

To get back this reclaimable space TSM has a Reclamation process which copies all the valid information from the tape to a new scratch tape so looking at our previous illustration TSM will create the new tape with the valid files

1-2-3-5-8-9-13-14-15-16

This eliminates all the reclaimable space and the old tape becomes a scratch tape.

On each Storage Pool supporting reclamation there is a Reclaim Threshold parameter

Run the QUERY STGPOOL stgpoolname F=D and look at the Reclamation Threshold field

Or when using TSM Studio, open the Storage Pools dataview in the Storage Category and check the Reclaim column

NOTE:  For tapes with a value of READWRITE or READONLY in the ACCESS column. TSM will run the reclamation on them when they are FULL, if they are FILLING they will not be touched.

Tapes in an OFFSITE status in a copy pool will be reclaimed whether or not they are FILLING or FULL.

To Change the Reclamation Threshold for a Storage Pool use the following command

UPDATE STGPOOL stgpoolname RECLAIM=xx  ( stgpoolname is the storage pool and RECLAIM is the threshold to use )

On TSM Studio right click on the Storage Pool and select Reclamation Threshold and either choose one of the values or use the SET option to choose another value.

As there are a number of admin jobs and backup jobs that all need to use tape drives, you will need to find the right window to run your reclamation processes and decide how many processes you wish to run. Remember that each process will require 2 tapes drives, one for the input tape and one for the output tape. Automatically starting and stopping your reclamation processes can be done with Administration Schedules. ( Check back soon to see an article on Admin Schedules )

Permalink.

Permalink.

Collocation is the process of grouping your nodes together (collocating) to help improve restore performance.

Over time as you add more and more nodes to your environment, the data for these nodes will slowly become more and more dispersed across your tapes as you perform backups.  When restoring the data if only a small number of files are required, this will not be a problem but when a restore is needed for a large number of files and those files have become dispersed across a large number of tapes then restores time will become an issue. I have personally experienced an issue when restoring a large volume that took over 2 days when not using collocation.

Instead of having every node’s data being placed on tapes at TSM’s discretion, collocation allows you to define groups of nodes to be stored together on your tapes.

As with everything there is a trade-off, collocation groups will require the use of more tapes but can significantly improve restore times.

Badly configured collocation will cause the consumption of very large amount of tapes.  And in my experience this has been the number one cause of the “Where have my all my scratch tapes gone” and “Why do I have so many filling tapes” , we will explore this later in the document.

Collocation is set at the Storage Pool so it is possible to have different levels of collocation based on Storage Pools.  As Collocation is really designed to help with Physical Tapes it only applies to Primary Sequential and Copy Storage Pools.

There are a number of difference types of collocation methods that can be used

NO – Collocation is disabled for the Storage Pool

FILESPACE – TSM Attempts to put each file space on its own tape

NODE – TSM Attempts to put each Node on its own tape

GROUP – TSM will put groups of Nodes on their own tapes as defined in the collocation groups.  If no collocation groups are defined then the storage pool reverts to Node Collocation.

NOTE:  If Group Collocation is set and there are no collocation groups defined or there are nodes not defined in any collocation group then you very quickly run out of scratch tapes and see a lot of filling tapes with not much data on them

Firstly let’s look at how to check to see if collocation is set.  From the Admin CLI issue the command

Q STGPOOL F=D and check the status of the collocate field

From TSM Studio open the Storage category and open the Storage Pools Dataview and the check the value in the Collocate column

To change the collocation method being used

Admin CLI:  UPDATE STGPOOL COLLOCATE=GROUP

From TSM Studio

Go to the Storage Pools dataview and right click on the storage pool(s) you want to change

and choose Collocation

Choose the collocation method and press Ok

The next thing to do will be create the collocation groups

From TSM Studio open the Configuration category and open the Collocation dataview

Right Click and Choose Add Group

Enter the Group Details and then press Ok

Next Add Members to the Group, From the Collocation Dataview right click the Group and Choose Edit Group Members

NOTE: A Node can only be a member of one collocation group

Simply move nodes in and out of the group and press Ok

Read Here for Trouble Shooting Tips

Recommendations:  If you have plenty of tapes and will never run out then use FILESPACE or NODE.

If Tapes are at a premium and you dont care how long a restore will take then dont use collocation

Otherwise you will need to find a happy medium between tape usage and restore times using the Group method.  I have seen a number of different methods for collocating node, such as grouping by application type or OS Type.  I suggest putting nodes with small amounts of data that does not change very often together and putting nodes with large amounts of data by themselves.  For the rest of the nodes try and keep each group from using more that 50 tapes for any node in the group ( this number will depend on the number of drives you have, how fast the library can load the tapes and how fast the drive can move to the required location on the tape) to ensure fast restore times.

I have seen collocation disabled for copypools as a lot of organizations already replicate the critical data to their Business Recovery sites using other methods

As with everything I say in these articles test, test end test again

NOTE:  Virtual Tape Libraries do not have the same limitations as Physical Tapes. As the mount times and seek times are negligible collocation can be disabled without affecting restore times.

Permalink.

The reuse delay parameter is used to define how long a tape will stay in a pending state. It is specified in days and can be applied individually to primary sequential and copy storage pools (in other words storage pools that handle tapes).

Why do we need this parameter?   This parameter is used to ensure that if you had to restore your TSM Database to a previous days backup that the tapes will be in a consistent state with the TSM Database as at the point of the restore.

To Illustrate this, assume that the reuse delay parameter is set to 0 so that tapes become scratch tapes as soon as they are empty.

Monday

A TSM Database Backup is taken on Monday

Tape A1 currently has data for Node ZZ

Tuesday

The data for Node ZZ on tape A1 expires and because the reuse delay parameter is set to 0 tape A1 becomes a scratch tape

Tape A1 is now immediately used to backup node YY’s data

Wednesday 

For some reason, you have to restore to the last TSM database backup which was taken on the Monday.

Because the database is now running as at Monday TSM believes that tape A1 holds the data for Node ZZ.  A decision is made to restore data for Node ZZ which resides on A1 and TSM complains that the volume is corrupt because the data on the tape is not what TSM is expecting. In this case if the reuse delay parameter had been set to 2 days then tape A1 would not have been written on and the restore would work as expected.

How do I check for the tape status ?

Tapes that are empty but not yet available to be used as scratch tapes have a status of PENDING. To view pending tapes in TSM Studio open the Libraries and Volumes category and double click on the Volumes Dataview. As seen below check the “Status” column and look for tapes in a ‘PENDING” state.

To see when these tapes will become scratch tapes, check the Pending Date.  This is the date that the tape went into a pending state. Add the Reuse Delay value to determine when the tape will go from a Pending State to becoming a scratch tape.

To find the Reuse Delay value check the Storage Pool that the Volume is in

Then goto the Storage Pools dataview locate the same Storage Pool name and check the Reuse Delay Column

How do I set the value for this parameter ?

I recommend that you set the value of this parameter to be equal with the number of days or TSM Database backups you keep.

To change the reuse delay parameter from the command line

Update STGPOOL STGPOOL_NAME REUSEDELAY=days

Where STGPOOL_NAME is the name of the storage pool and days is the value for the REUSEDELAY parameter

UPDATE STGPOOL TAPEPOOL REUSEDELAY=2 for 2 days

To change the reuse parameter from TSM Studio goto the Storage Pools Dataview, select the storage pool that you wish to modify, right click to bring up the context menu. Select Reuse Delay and choose the value (in days) you wish to set. Press Ok

Permalink.

Because TSM Studio saves the last known position of the window, it is possible that if the resolution changes, multiple screens were used or the window was accidently dragged off the screen that the TSM Studio is now on a part of the Window Screen that is not visible. To resolve this issue use ALT+CTRL+W to reset the position of the window

Permalink.

As of TSM Studio Server version 2.9.1.0 security has been tightened up and no longer stores any security information in the registry.

The default logon to TSM Studio Server is now admin with a default password of password, (Domain field should be blank otherwise TSM Studio Server will assume it is a windows userid).

We strongly advise changing the admin password using the following procedure.

To change the password for the inbuilt admin userid use the Security–>Change Admin Password

Security in TSM Studio Server is now defined by assigning role(s) and Windows groups to a TSM Studio Server security group.

The TSM Studio Server security system will automatically create a group named Admins that has access to all functions in TSM Studio Server, the first time it is loaded it will look for the old group definitions in the registry and load them into this group otherwise it will populate with following default groups Administrators, Domains Admins and TSMStudioAdmins.

Also two roles will be created, one called Administrators which is assigned to the Admins Server group and one called Operational Report Viewer for allowing access to the Operational Report Viewer.

To access TSM Studio Server Groups select Security–>Groups

To modify the Admins Group Double Click or select edit from the Toolbar

Add Windows Group that should have admin access to TSM Studio Server, One Group per line

Permalink.

If SQL Server is running on the same Server as the TSM Studio Server service then we recommend 12GB of memory as a minimum to ensure the healthy running on TSM Studio Server.

By default SQL Server will be configured to use all available memory this will need to be changed so that there is enough free memory for  TSM Studio Server to function. It is recommended  that SQL Server be given 4GB of memory and the other 8GB is left available for TSM Studio Server.

To change the default memory settings in SQL Server. Start up SQL Server Management Server right click on the server and select properties

Select Memory and then enter 4096 into the Maximum server memory field

Permalink.

In TSM Studio Server 2.9.011 and above there are a number of added enhancements to help tune TSM Studio for you environment.

The first area to check is on the Concurrency Tab of the TSM Studio Server properties dialog which can be accessed from Configuration–>Properties as shown here



 

The main setting to check here are Concurrent Reports and Concurrent Collector Threads. Report creation can consume large amount of memory so it is recommended to keep this number low.



Also some of the collections can consume a large amount of transaction log space especially the activity log collector.  Each collector  can be individually tuned.  Goto Configuration->Collection schedules and edit the Activity Log collector and change the Concurrent Collectors to a small number (less than 5 )



Also check out the below link for SQL Server memory settings


https://spiritsoftware.biz/knowledgebase/microsoft-sql-…-studio-server

Permalink.

Alternate Row BackColor Example

Permalink.

Permalink.

This will display Line Numbers as shown below

Permalink.

Permalink.

Permalink.

TSM has the ability to encrypt data at the client node before sending the data to the TSM server. There are two methods that are available

1. Transparent Encryption

• This is where the encryption key is managed by and stored on the TSM server
• If the client node needs to be rebuilt data can be easily restored
• Data can be restored back to any node that is allowed to impersonate the original node

2. Client Side Encryption

• This is where the encryption key is manually managed and stored on the client using an encryption password
• More secure as data can only be restored if the encryption password is known
• If the password is lost then the data cannot be restored

To enable encryption at the client there are two parameters for setting up and a couple of include and exclude statements for selecting or excluding which files are to be encrypted.

ENCRYPTKEY

The ENCRYPTKEY option is used to choose either transparent encryption or client-side encryption. For client-side encryption there are two options to choose from

ENCRYPTKEY=SAVE  ( Client-Side )

This option will prompt for an encryption password on the initial backup and then store this password in the password file. The password will be retrieved from this file for each subsequent backup.

ENCRYPTKEY=PROMPT ( Client-Side)

This option will prompt for an encryption password for each backup and restore. To be able to restore the data the same password that was using when backing the data up will be required

ENCRYPTION=GENERATE (Transparent)

This option will have TSM generate an encryption key password which is stored on the TSM server and managed by the TSM server.

 ENCRYPTIONTYPE

The ENCRYPTIONTYPE parameter selects what type of encryption is used either DES56 or AES128 with the AES128 algorithm being the stronger of the two

Next is to select which file or directories to include in the backup

use the include.encrypt statement to include files and directories to be encrypted and takes the same format as any other include statement
use the exclude.encrypt statement to exclude files and directories to be encrypted and takes the same format as any other exclude statement

example

ENCRYPTKEY=GENERATE
ENCRYPTIONTYPE=AES256
INCLUDE.ENCRYPT /home/…/
EXCLUDE.ENCRYPT /home/…/test.fil
INCLUDE.ENCRYPT  C:…*
EXCLUDE.ENCRYPT  C:windows…*

When using the client-side encryption the encryption passwords are stored in the TSM.PWD files in unix or in the registry for windows

I would recommended using transparent encryption unless you have a specific requirement not to.

I am option asked how to prove that the data is encrypted. There is no way to do this with TSM and they only way to do this is use a network packet tracing tool such as wireshark. If you are interested on how to do this just send me an email  gelliott@spiritsoftware.biz

For more information see Chapter 5 of IBM Tivoli Storage Manager: Building a Secure Environment

Permalink.

In Windows you can use the Preview Include-Exclude option in the backup client.

Start the backup client and go to the utilities menu and select Preview Include-Exclude

In the Preview Include-Exclude Dialog, choose the type either Backup or Archive ( Your normal nightly jobs will usually be of type backup )

Choose either to just show included files, excluded files or both. This information is ascertained from settings in your dsm.opt file

Choose  the directories and files to be reported on. If you have a large file system and you choose a top level directory this command may take a long time to run and produce a large output file

Lastly choose the output file and select ok

When the task finishes open the output file with notepad and it will look as follows with the files on the left and the management class on the left hand side

The same preview include-exclude can be achieved with the backup-archive command line in both unix and windows as follows.  This example will produce the same out as above

the -traverse=y tells the backup-archive client to include subdirectories

Permalink.

This article describes installation and configuration of the TSM Scheduler Service on windows. The TSM Scheduler service is used by the TSM Scheduling Services to automate client node backups.

I am using the version 6 client for these examples, even though the screens look different it is still the same process for the version 5 clients

After installing the Windows Backup Archive client start the Backup-Archive GUI go to the Utilities–>Setup Wizard

Select Next

Select Next on TSM Schedule Wizard Screen

Choose Install a new or additional scheduler

Choose a Name for the Service. This the name you will in the Windows Services Manager

This is what you will see in Windows Service Manager

Choose an options file to be associated with this TSM Scheduler Service. It is possible to run multiple TSM Scheduler Service’s using different option files

Choose the node name for this TSM Service. Again it is possible to use a different node name for each TSM Scheduler Service and input the password. This node name will need to be registered on the TSM Server or the service will not start.

Choose the userid that the service will start as.  This userid will need to have access to all the files it must backup or be a member of the backup operators group in windows

Choose the location for the scheduler service log files

NOTE:  Some of these options are stored with the TSM Scheduler service and not in the options file and will override the option file settings. These values are stored in the registry at the following location

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTSM Client SchedulerParameters     (replace TSM Client Scheduler with the name you used when installing the service )

Permalink.

Expiring old backup and archive data from TSM is controlled by the EXPINTERVAL in your dsmserv.opt file.

EXPINTERVAL=24  specifies to run the expire process every 24 hours.

EXPINTERVAL=0 disables automatic expiration.

Removing old data from TSM can be run manually using the expire inventory command. Data is expired using the “Management Class” definitions that you have set on your files or if none are defined then the defaults for the policy set are used.  ( I will discuss policy sets in an upcoming article )

In version 5 expiration can only be run system wide, in version 6 you can run the expire inventory command on individual nodes, this is useful when changing management classes on individual nodes.

I recommend that you set the EXPINTERVAL to 0 so that you can control at what time this process runs and the setup an admin schedule to run the expire inventory command

Permalink.

We suggest you read the following Article on Collocation before reading this Article –> Read Here

As noted in the article on collocation. The main problems you will come across with collocation is running out of scratch tapes andor a large number of tapes in a filling status. The main cause of this is having a storage pool with a collocation method of Group and then having no collocation groups or nodes not in collocation groups.

Using TSM Studio go to the Trouble Shooting category and open the Nodes not in Collocation Group dataview.

To Add a Node(s) to a Group simply right click the select Add to A Group

After putting all your nodes in a group, You will now need to move the data off the current tapes so that TSM can collocate the nodes and their data.

There are a number of ways to do this and here are two of them

1. To do this use the Move Data command on each tape and move the data back to the primary disk pool so that the migration process can move the data back to tape using the collocation you specified.

2. Or use the Move Data command to move the data to the same storage pool as the tape is already in, TSM will then move the data from the selected tape and collocate the data with nodes from the same group on other tapes.

Using TSM Studio to run the move data command. Open the Volumes Dataview from the Libraries and Volumes category Right Click on the Tape and Choose Move Data

Choose the Storage Pool to move the data to

Permalink.

SELECT NODE_NAME, COUNT(DISTINCT VOLUME_NAME) AS VOLUME_COUNT FROM VOLUMEUSAGE GROUP BY NODE_NAME ORDER BY VOLUME_COUNT

or in reverse order

SELECT NODE_NAME, COUNT(DISTINCT VOLUME_NAME) AS VOLUME_COUNT FROM VOLUMEUSAGE GROUP BY NODE_NAME ORDER BY VOLUME_COUNT DESC

Permalink.

For TSM v 5

SELECT COUNT(*) FROM NODES

For TSM v6 and above

SELECT COUNT(*) FROM NODESVIEW

Permalink.

TSM v6 and Above

SELECT NODE_NAME FROM NODESVIEW WHERE NODE_NAME NOT IN (SELECT NODE_NAME FROM OCCUPANCY)

Permalink.

SELECT DISTINCT NODE_NAME, CLASS_NAME FROM BACKUPS WHERE NODE_NAME=’XXXXX’

Replace XXXXX with your node name

Permalink.

SELECT DOMAIN_NAME,SCHEDULE_NAME,COUNT(NODE_NAME) AS NUMBER_OF_NODES FROM ASSOCIATIONS GROUP BY DOMAIN_NAME,SCHEDULE_NAME

Permalink.