Smart Cache Loader Features

Major features of Smart Cache Loader:

Design goals

1. Non graphical
2. Non interactive
3. Overkill configurable
4. Portability (uses Java 1.5)

Data storage for downloaded pages

1. null - do not store, used for proxy preloading.
2. local - write pages to filesystem
3. scache - used when working with Smart Cache

Priority based queue

1. URL in queue can have different priority by type, size, …
2. Servers can have different priority

Fetch actions

1. do not download
2. download but do not parse html code
3. send request, but do not wait for any data
4. classic full download
5. fastclose - if data are not parseable, close connection
6. nosave - do not save it

Update actions

1. if object exists, do not re-download it
2. for cache reload of specified object
3. update object if older than XX hours
4. do not reparse saved html on disk
5. forced update

Target categories

1. world - any server
2. known - any server in configuration file
3. file from the same server
4. file from the same location (directory+subdirs)
5. file from subdirectory
6. update object if older than XX hours
7. forced update

Locations

1. You can define multiple location of the same data for avoiding duplicate download
2. Can define short name for server for use from command line.

Masking

1. URL mask with possible striping before comparing
2. File extension mask
3. Source HTML tag mask
4. Target selection mask