Theo Todman's Web Page - Notes Pages


Website Documentation

Website Generator Documentation - Spider & Slave Database

(Work In Progress: output at 03/09/2018 13:04:10)

(For earlier versions of this Note, see the table at the end)


This document covers1 the following functions performed by clicking buttons on the front screen:-

  1. Backups + ZoomSearch (cmdBackup_Click)
  2. Run Web Spider (cmdSpider_Click)

To see the Code, click on the procedure names above.

Backup
  1. This compacts & repairs, and then backs up the slave database to the name Web_Generator_Performance_Temp_YYMMDD, using function Compact_Repair.
  2. The slave database is now2 only used to hold Raw_Links table.
  3. Note that there is an entirely separate subsystem (admittedly based on this one) to back-up the file-system on this (or any other) PC – see Backup_Ctrl, etc.

Spider
  1. This process (governed by Spider_Ctrl, and then Spider_Scurry) interrogates the files on my hard drive that form the local copy of my website, and after determining the directory structure, digs out all the hyperlinks by recursively passing through the files as text files using the VBA FileSystemObject.
  2. It is possible to buy, or download free, web-spiders that will check hyperlink integrity across the internet when pointed at a base-URL, but (in my experience) these take forever to run on large sites (if only because the timeout limit has to be set such as to avoid false negatives). Hence, I wrote my own Spider to run quickly on a local drive.
  3. While the Spider will record external links, it does not check them. However, I have a completely separate sub-system that performs this function: see Web Links3.
  4. Despite this being a local checker with no timeout delays, it still takes a long time to run, so there are various parameters that can be set to control what’s done on a particular run.
  5. The three main control tables are opened by cmdSpider_Click, namely:- These can be used to control / limit the run.
  6. Spider_Control: This is a single-row table that contains:-
    • Statistics from the last run, together with
    • A flag “Update_Since_Last_Run” which, if set to “Yes” limits the run to interrogating pages that have a last-changed timestamp subsequent to the last Spider run. This is less useful than it might be, in that I currently run the Spider after I've regenerated the entire site (using Full Website Re-Gen4), so that in practice almost all5 pages are scanned.
    • Another flag “Stop_Spider” which is checked at the start of the recursive Spider_Scurry and which will stop the run if set to “Yes”. The idea is that the control table should be open in another copy of MS Access so that if you want to stop the run without crashing the program, you can do so by setting this flag6.
  7. Directory_Structure: This table contains one row for each sub-directory (including the root) within the website. It is maintained by the Spider itself as far as the number of rows is concerned. However, the user can set a couple of flags against each row:-
    • “Do_Not_Parse”: this can be set to “Yes” to ignore this directory (and any sub-directories). Any corresponding rows in Site_Map are left unchanged.
    • “Updates_Only”: this can be set to over-ride the “Update_Since_Last_Run” flag in Spider_Control for this directory (and any sub-directories).
  8. Site_Map:
    • This table contains one row for each object (HTML page, PDF, GIF, Word document, and the like) in my website.
    • It is maintained by the Spider as far as the number of rows is concerned. The Spider records, amongst other things, how long it took for the object to be parsed (“Time_To_Update”, in milliseconds).
    • The user can set a flag “Block_Update” that will cause the object not to be parsed in this coming run. This is because some pages are very large and take a very long time to parse.
  9. Blocked_Spider_Files
    • This is a query that selects only those rows of Site_Map that have Block_Update = “Yes”, though it is possible to add others from Site_Map by toggling to the table, setting the flag, and bouncing the query.
    • This is a “fine-tuning” way of restricting the operation of the Spider.
  10. During the Spider process, the three main tables:- Are updated via their local analogues, which are maintained during the bulk of the processing, until the Sub Spider_Copy is called at the end. There is a bit of fancy footwork to ensure the various options don’t either leave redundant links in place, or delete live ones.
  11. Spider_Copy also performs the processing to create the “full links” column in the Raw_Links table.
  12. Spider_Copy also segregates out7 the Section-link within a page, and then runs8 the Spider_Missing_Internal_Links query to pick up broken links within the site.
  13. As noted above, the Spider takes a long time to run. In addition, the activity on the Slave Database that contains the Raw_Links table expands the table from its “resting” size of under 600Mb to over the 2Gb limit9 beyond which the database becomes corrupt. Consequently, I have added two diagnostic and repair functions during the processing-
    1. I use debug.print to timestamp the various sub-processes within the run.
    2. I call the function Compact_Repair periodically to Compact & Repair the Slave Database periodically. If this fails, as it sometimes seems to, the process as a whole can fail.
  14. These diagnostics reveal that:-
    1. Spider_Scurry is the longest-running process, as might be expected; it can take up to 5 hours based on the database / website at its current size.
    2. The only other significant process is Full_Link_Up_Levels_Gen, called by Spider_Copy, which takes about 3 hours.
    3. No other process takes more than a few minutes.
  15. Description of the detailed processing now follows:-
    • Spider_Ctrl: as might be expected, this controls the other processing. It asks questions about the run required so that parameters can be set. It then calls the two main processes below, and finally calls the three processes that output the Web-Links Test Webpages.
    • Spider_Scurry: this is self-referential procedure that combs the links recursively until a “leaf” page is found with no further links. Presumably there’s a check for loops!
    • Spider_Copy: this process:-
      • Updates the Directory_Structure, Directory_Fine_Structure and Site_Map tables. The first two tables are in the Backups database, the third is in the main database.
      • Updates the Raw_Links table in the Slave database, using the following procedures:-
        1. Full_Link_Same_Directory_Gen: this process simply uses the query Full_Link_Same_Directory_Updt to set the full link in the simple case where the link is to the same directory. Because so many links are involved (circa 0.5m of this category at the moment), this cannot be an update query, but is run through in code, with the slave database being compacted and repaired every 300,000 records.
        2. Full_Link_Up_Levels_Gen: this process is similar to the above10, but caters for the case where links are not to the same directory. In this case, the raw link will have a variable number of “../” directory-shifters, to get back to the Site root-directory, followed by the address from there. For some reason there’s a query of the Directory_Fine_Structure table.
        3. Full_Link_Sections_Fix
  16. … to be completed …

This Note is awaiting further attention11.



In-Page Footnotes:

Footnote 1: Footnote 2: Footnote 5: Footnote 6: Footnote 7: Footnote 8: Footnote 9: Footnote 10:


Printable Version:



Table of the Previous 2 Versions of this Note:

Date Length Title
13/01/2015 19:07:41 5310 Website Generator Documentation - Spider & Slave Database
02/06/2013 10:58:05 347 Website Generator Documentation - Spider & Slave Database



Note last updated Reference for this Topic Parent Topic
03/09/2018 13:04:13 986 (Website Generator Documentation - Spider & Slave Database) Website Generator Documentation - Control Page

Summary of Note Links from this Page

Awaiting Attention (Documentation) Website Generator Documentation - Full Website Re-Gen Website Generator Documentation - Prune Website Website Generator Documentation - Web Links  

To access information, click on one of the links in the table above.




Summary of Note Links to this Page

Website Generator Documentation - Control Page, 2 Website Generator Documentation - Web Links, 2      

To access information, click on one of the links in the table above.




Text Colour Conventions

  1. Black: Printable Text by me; © Theo Todman, 2018
  2. Blue: Text by me; © Theo Todman, 2018




© Theo Todman, June 2007 - Sept 2018.Please address any comments on this page to theo@theotodman.com.File output:
Website Maintenance Dashboard
Return to Top of this PageReturn to Theo Todman's Philosophy PageReturn to Theo Todman's Home Page