The FILTR:
(File Inventory for Loading, Transfer and Recovery)
Table of Contents
| 1. Introduction |
| 2. Concepts |
| 3. Tutorial 1: backup, restore, delete and recover |
| 4. Day-to-day work with the FILTR |
| 5. Using project tags |
| 6. Using the time attribute |
| 7. Tutorial 2: advanced features with the backup control file |
| 8. Tutorial 3: setting links, mirrors and quotas |
| Links: |
| Mirrors: |
| Quotas: |
| 9. Additional notes: |
| 10. Complete notes on setting quotas |
The FILTR is an archiving utility designed to make it easy to back up and restore your files to a storage repository. It's also designed to make it easy to move the storage repository you create from computer to computer, operating system to operating system. With the FILTR, you can take your files back and forth to work and home on a disk or memory stick, and keep it all tidy. Or you can ditch your old, slow computer and get right to work on your new, fast computer, without fearing that you've left something important behind. Unlike other backup utilities, which are designed to keep a single computer up and running in case of mishap, the FILTR is designed to help you organize and mobilize your work over the course of your lifetime and the many computers and OS's you will use.
The FILTR defines two locations for your files: the “Local Store,” a folder on your hard drive where you keep all your important stuff (like the My Documents folder on a Windows computer), and the “Backup Repository,” which is where your backup copies go (but you should never have to look at or work in this location directly).
When the FILTR starts up, an explorer-type file manager appears, with a folder tree on the left, and folder contents on the right. By default a folder in your home directory named “FILTR” is created, and the file manager automatically switches to that directory and displays its contents (at the first startup it's empty). This is your local store. To see how files are backed up and restored, take the following steps:
on the menu bar, choose File>New File. A file called new_text_file.txt will appear.
right-click on this file and choose “Display text...” An editing window will appear.
type “hello world” in the editing window, and hit the Save button.
now you have a file you can back up. Right click on the file again and choose “Attributes...” A window showing the file's attributes will appear.
change the value of the attribute “backup” from 0 to 1. Hit the “Edit” button to close.
Whew, you're done! Now there's a copy of new_text_file.txt is now in the backup repository. The backup repository is in a folder called “backup” located right next to where the FILTR program is installed. How do you know there's a copy in the backup repository? You could look there, but instead, try this:
right-click on new_text_file.txt and choose “Delete...”
confirm the delete.
Now the file is deleted from the local store. But wait! The file is still visible. This is because the file manager is showing you a virtual view of the combined contents of the local store and the repository. Since a copy of the file is in the repository, you can still see it.
But this can be confusing. You want to see the current state of your active working files accurately. So do a backup of the containing folder to let the repository know that the file has been deleted:
on the menu bar, choose View>Folder up to go up a level. Now the right window pane should show the contents of your home directory.
find the folder named “FILTR” and right-click on it.
choose “Attributes...”
change the value of the attribute “backup” from 0 to 1 and hit the “Edit” button
double-click on the folder “FILTR” so its contents show again in the right pane. The pane should show the file is gone.
But don't worry. The repository keeps copies of all deleted files, which you can restore if you decide you need them again. To restore the file to the local store:
choose View>Folder up again to go back to you home directory
right click on the folder “FILTR” again and choose “Attributes...”
In the edit field for the attribute “project,” type deleted, and hit “Edit”
display the contents of the folder “FILTR” again by double-clicking on it. The file new_text_file.txt appears again.
Setting the project attribute allows you to give labels to different versions of your files. Every deleted file automatically gets the project label deleted. But the file is not yet back in the local store. You're just seeing that a copy of the deleted file new_text_file.txt exists in the repository. To finish restoring the file:
right-click on new_text_file.txt and choose “Attributes...”
note the attribute named “mtime” (modification time). This is a timestamp representing the time the file was deleted. Move the mouse cursor over the edit field of this attribute, and a popup will appear showing a more readable display of the date/time.
copy (Ctrl-C) the timestamp in the mtime field and paste it (Ctrl-V) into the edit field of the attribute named “time.”
set the value of the attribute “restore” from 0 to 1 and hit “Edit”
A copy of the file as it existed at the time specified is now copied into the local store. You can open it and make additional edits to it if you want.
Setting the value of the “time” attribute makes the repository display all its files and directories as they appeared at the time represented by the timestamp. So you can go back in time and retrieve any previously backed-up version of your files.
Move the files you work with into the local store directory. Create as many subdirectories as you want to organize your work. The project, time, backup and restore attributes can be set for both files and directories. As you make changes to your files, you can save your work by doing backups on individual files or on entire directories at a time.
You can copy all or part of the backup repository to a disk or memory stick and take your files with you. Restore files to the local store on another computer, edit them, and back them up back to the mobile repository. To sync the mobile repository with your main repository, just drag and drop the one onto the other. All file versions will be preserved.
You've seen that deleted files automatically get a project attribute tag called deleted. You can attach project tags of your own to versions of files when you do backups by filling in the edit field of the “project” attribute.
Say you want to create two versions of a financial report at work, one for your boss, and one for the accountants. Complete the report for your boss, then right-click on it in the file manager and bring up the attributes window. Set the value of “project” to boss, and “backup” to 1, then hit Edit to back up.
Then re-open the report and edit it for the accountants. When you're done, open the attribute window again and change boss to accountant in the “project” field, and back up again.
Now you can restore your choice of file version any time you want. Set “project” to boss and “restore” to 1 in the attributes window, and the version of the report for the boss will be copied to the local store. Do another restore with “project” set to accountant, and the version for the boss will be replaced in the local store with the accountants' version.
You can add more that one project tag to a file version. Just type as many tags as you want into the project attribute edit field before starting the backup.
Setting the “time” attribute allows you to go back in time with your files. When the “time” attribute is set, all the directories and files visible in the repository revert to how they appeared at that time. So you can restore a file or an entire directory to how it appeared at the time you specified.
Be careful, because the restore process will overwrite and possibly delete files in the local store in order to bring things back to the way they were. You may want to do a backup first before you do a big restore, in case you want to save any current work.
The “time” attribute will accept a wide variety of inputs. In addition to timestamps, you can put times and dates, like 09:30 PM or 11/01/06. You can also use phrases like yesterday or 10 minutes ago. For a complete description of the inputs that the “time” attribute can handle, look at the manual page for the Tcl “clock scan” command (Google: tcl manual clock scan).
You may find it inconvenient to use the attributes window for anything but simple operations. Another way to control backup and restore operations is available, which will also allow you to access more advanced features of the FILTR.
go to your local store location in the file manager (the folder FILTR in your home directory)
delete new_text_file.txt again
on the menu, choose View>Show hidden files. A file called “.backup_control” will appear. Right-click on this file and choose “Display text”
you will see a configuration file with brief descriptions of advanced options. Scroll down to the bottom of the file.
note that the project and time attribute values you set are present. Under the comment line “Version project tags:” you see the tag deleted, and under the comment line “Version timestamp:” you see the timestamp you pasted in. Delete both values, leaving the lines they're on blank.
Also note that at the bottom, after the comment line “File versions stored for this directory”, there is a listing of all the versions of new_text_file.txt saved in the repository. In this case there are two (the original version, and a “version” showing the file was deleted). Each version is represented after the file name by a group of three values: the first is the timestamp, the second is a more readable date/time string representing the same time as the timestamp, and the third is a list of the project tags attached to the version, if any. When you have many versions of a file saved, you can use this file version information to retrieve the file you want if you need to restore from the repository.
from the group representing the first-saved version of new_text_file.txt, paste either the timestamp or the date/time string onto the blank line below the comment line “Version timestamp:”
scroll to the top of the backup control file, and after “restore this directory now(YES/NO):” change NO to YES.
click on the “Save” button to close the text edit window. Saving the configuration file triggers the restore. A new copy of new_text_file.txt has again been put into your local store directory.
Note that the backup control file will backup and restore entire directories, not single files. It just happens in this case that the local store directory contains only one file. You'll still need to use the attributes window to control backup and restore of single files.
You can still do backups of files and directories that you don't want to or can't move into the local store folder. You can create a link from a directory outside the local store to a subdirectory within the local store. Then all the contents of the outside directory will appear as the contents of the subdirectory. The contents of the linked subdirectory can then be backed up and restored as normal. To create a link:
create a new directory in the local store by choosing File>New Folder... on the menu bar. Give it the name linkdir
go into the linkdir directory by double-clicking on the folder icon
note that a file named “.backup_control” appears in this directory as well. Open it by right-clicking on it and choosing “Display text”
scroll down to the bottom, and note comment section that starts “Current real directory view location.” The line under the comment section shows the pathname of the local store directory plus the “linkdir” subdirectory. Delete this pathname and replace it with the path of another directory on your computer, for example: C:/Program Files
hit the “Save” button to close the edit window, then choose View>Reload on the menu bar
The contents of the directory you linked to now appear as the contents of linkdir. Now you can back up these contents to the repository just like any others in the local store folder. Thus, you don't have to move everything you want to back up into the local store, though the local store is still the place to organize your files and run your backup activities.
To eliminate the link, right-click on the folder icon and choose Delete. The link will disappear, though the folder will remain. Delete again if you want to remove the folder.
You may wish to distribute backed-up files to a secondary location, like a memory stick for extra security or transport, or to a directory read by a web server for public viewing. You can set up a subdirectory of the local store to have one or more mirror locations assigned to it; that is, for every file backed up to the repository, a copy is made to each of the mirrors.
You specify mirror locations in the “.backup_directory” file, similar to how links are specified. Look for the comment section starting with the text “Mirror locations, if any.” Put the pathname of each desired mirror on a separate line below the comment section. Once the mirror locations are set, every time a backup is done, copies of each backed-up file will be made to each mirror.
A mirror location doesn't need to exist at every backup. For example, you may specify the pathname of a memory stick drive as a mirror, but the memory stick may not always be plugged in. In that case the FILTR records the names of the files that should have gone to the mirror. The next time the FILTR program is started, or at the next backup, the FILTR will check to see if the missing mirror location currently exists. If it does, the FILTR will automatically copy over all the files that should have gone to the mirror previously, thus “catching up” the mirror.
You may wish to back up just some of the files in your local store, and keep others out. You can set a quota which applies to any directory in the local store, which will cause the FILTR to ignore selected files when backing up.
create a subdirectory in the local store named “quotadir”
go into the directory “quotadir” by double-clicking on its folder icon
create a new file. Right-click on it and choose Rename... Give it the name “test.txt”
create another new file. Name it “test.doc”
open the file .backup_control in the directory quotadir
scroll midway down to the comment section that starts: “Quota restrictions for this directory”
on the line below the comment section, type the following quota definition: -filename *.doc -quota 0
scroll back to the top and change n to y on the line controlling backup
click the “Save” button to initiate the backup. The file test.txt is copied to the repository, but test.doc is not because the quota set on the folder blocks it.
to verify that the quota has functioned, open .backup_control again. Scroll to the bottom of the file. Note that a line showing file version info exists for test.txt, but none for test.doc.
The default locations of the local store and the backup repository can be changed by settting the values of the environment variables FILTR_LOCAL and FILTR_BACKUP to hold the respective pathnames of the desired locations before the FILTR program is started.
The FILTR will source the .tclshrc file in the home directory on startup, if the tcl interpreter hasn't sourced it already.
Quotas can be set on any quantity returned by "file stat" or "file attributes", plus the attribute "filename", which is the fully normalized pathname of the file.
Two types of quota can be set: an incremented count of files matching a certain criterion, and a running total of a certain quantity. Each quota is defined by a set of switches composing a "quota group," any number of quota groups can be defined. A file must fit within all quotas defined to avoid triggering quota enforcement.
The quotas are enforced as a FIFO stack of files; that is, if a new file is copied to the vfs whose attributes exceed a quota, the file is not rejected, rather, the already present files with the oldest access times that contribute to the quota are deleted until there is room within the quota limit for the addition of the new file.
The exception is if the file's attributes are large enough to violate the quota by itself, it is barred without first deleting all other files contributing to the quota.
When a new file is moved into the virtual directory or an existing file edited, its properties are examined with respect to the defined quotas; if no room can be made for it, the move or edit is rejected.
Quota group definition:
-<quantity> <rule> -[quota|ruletotal] <quota number>
or
-<quantity> -total <quota number>
Options:
-<quantity>
Where <quantity> is any item returned by the "file stat" or "file attributes" commands, with the dash prepended as needed, for example: -archive, -permissions, -size, -mtime etc. The attribute "filename" is assumed to exist as well, defined as the file's full pathname. The quantity need not exist, so the same command line could be used on Unix or Windows, for example. Nonexistent quantities have no effect and are ignored.
<rule>
The rule is the criterion a file must meet to have the quota applied to it. It may take the form of a list of glob patterns as used by the "string match" command: if the quantity value matches all the patterns, the quota is applied. The rule may be Tcl code, to which the quantity value will be appended and then evaluated. The code should return 1 if the file is judged to meet the quota criterion, or 0 if not. If glob patterns are used, each pattern in the list may, in addition to symbols used by "string match", have a "!" prepended to it, which will negate the sense of the match.
-quota
If the quota group contains this switch, then the vfs will keep a running count of all files that satisfy the quota group's rule. It will not allow more than the number of files specified in <quota number> to exist in the virtual file space.
-total
If the quota group contains this switch, then the vfs will track the sum of the values of the specified quantity of all files. It will not allow the sum specified in <quota number> to be exceeded in the virtual file space.
-ruletotal
Like -total, but a rule is defined, and only files satisfying the rule have their values added to the quota sum.
Examples -- to set a 10 MB size limit on your ftp upload directory:
-size -total 10000000 C:/temp/upload C:/vfs/ftp/pub
To allow only PNG or JPEG files in a photo collection:
-filename {!*.png !*.jpg !*.jpeg} -quota 0 /home/shuntley/photos /vfs/photo
To ban GIF files from your web site images subdirectory:
-filename /docroot/images/*.gif -quota 0 {C:/Program Files/Apache/htdocs} /docroot
To disallow creation of subdirectories:
-type directory -quota 0 /ftp/upload /intake
To allow only 1 MB of files greater than 10kB in size:
-size {expr 10000 <} -ruletotal 1000000 /tmp /vfs/dump
To allow only log files and keep only 1 more than one week:
-filename !*.log -quota 0 -mtime {expr [clock scan {7 days ago}] >} -quota 1 /var/log /vfs/history