Knowledgebase: How Stuff Works
P5 Index Cleanup
Posted by Sven Koester on 29 August 2018 14:24
P5 indexes consist (among further info) of two parts:
  • The file and folder structure and change thereof over time, and
  • The media (tape) related tables with positions where the data is stored.
Both parts are independent from each other. In case for instance a file is saved on three tapes, it has three database records in the media part but just one in the file structure.

When deleting or relabeling a volume, the files of that volume can no longer be restored. In the index, they will however still be visible:
On deletion of a media (tape), the media part of the file will be marked as deleted. However, the file part is not, as that would require a check for each file whether it is on another tape. As such a test takes time, it is not done immediately.

When browsing in the Restore areas in P5, only the file and folder structure is regarded to allow fast browsing. In case the media part of the index would be tested, too, browsing would be rather slow. So it is possible that files which cannot be restored anymore are still visible in the index. They will remain visible until the index has been cleaned up.

It is always possible to check whether or not a file can be restored by opening the versions windows from the context menu. That window shows all media where the file is stored on and in which version.

Note that this is handled differently for files and folders: even if there are "versions" of of folder in the index, these are not exposed as they refer to the time stamps and other attributes of the folder, but not to its contents. A folder with a time
stamp from May can contain files that are much newer. So the "version" of a folder would be misleading.
Some folders may even appear without time stamp, these were not saved but exist only as nodes (to navigate to saved files and folders).

To figure out whether a folder contains files (without navigating down the folder tree), one can select the folder and click "restore to". P5 will then count the files and folders below and sum up the size before the actual restore is started. The restore can be cancelled before files are really restored, but the window shows size and number of the contained files.

In Backup indexes, the cleanup happens automatically after the next backup job using that index. The cleanup will appear in the index tables when 10% of the contained data is invalidated. So it may appear that if only a few files are affected, the cleanup is delayed.

In Archive indexes, there is no automated cleanup. If required it is possible to execute the cleanup manually.
The cleanup can be called through the nsdchat utility with the following commands:    
This call does the cleanup of elements no longer on tape.
Please make sure that during the cleanup, no archive or
restore jobs are running that use this index.

  /usr/local/aw/bin/nsdchat -c cli::index::purge Default-Archive true

This call removes empty folders in the index. It does so
 recursively and runs for the given number  of seconds.
In the example, these are 600 seconds or 10 minutes.
Please make sure that during the cleanup, no archive or
restore jobs are running that use this index,   

  /usr/local/aw/bin/nsdchat -c cli::index::prunedirs Default-Archive 600

(0 vote(s))
Not helpful

Comments (2)
Marc Levy
14 July 2019 21:52
this is very helpful but can you confirm the clean up does not require to read volumes?
i'm using cloud archive pools and will be charged if volumes are read upon this check.
Andre Kuehnemund
06 August 2019 20:52
P5 does not read the contents of the volumes themselves when cleaning up the index.
Post a new comment
Full Name:
CAPTCHA Verification 
Please enter the text you see in the image into the textbox below (we use this to prevent automated submissions).