The spider is a specialized module designed to follow links and forms in an attempt to discover the application structure and features. You must enable the spider attack method to use this feature. Simply click the "Spider" tab and check the "Enable" option.
There are a number of options to help you configure the spider according to your requirements.
The "Exclude URLs" option defines a list of regular expressions (one per line), which are used to match URLs to be ignored. For example, to force the spider not to follow logout URLs you may use the following regular expression:
logout\.(asp|aspx|php|jsp). You may want to add additional rules to specifically prevent the spider from visiting sensitive application areas.
The "Include URLs" option is used to override other scoping options such as "Exclude URLs". For example, you can define a list of locations which must be excluded but include specific resources within those locations using regular expression rules.
This option defines the maximum number of folders allowed before considering the URL out of scope. For example, "Max Depth" of 2 will correctly spider
/folderA/folderB/file but will not spider
Pre-defined Exclusions Rules
There are a number of pre-defined exclusion rules you can use. For example, binary formats such as audio, videos and image files are not used by the spider as this content is not interactive. These files are excluded by default. There are situations when you may want to spider these files too although it is generally noy recommended or required in the majority of the cases.
Styles, scripts and documents are also excluded by default as in most cases these are static assets, but as discussed there are a number of situations when this may not be desirable. You must define first what you want to achieve in order to optimise the tool to fit the required task.