Theo Todman's Web Page - Notes Pages


Status Reports

Website - Progress to Date (2020 - October)

(Work In Progress: output at 22/10/2020 10:18:23)

(For earlier versions of this Note, see the table at the end)


Rationale for this Document



Items Completed In6:-


Items Completed by Category:
  1. Own Website: Completed Items By Category:-
    • Abstracts
      1. 20Q1
        • As revealed by Spider: Link to PaperCitings failing (2 items) - Timing issue
      2. 15Q4
        • Corrected data format on Identity Abstracts
      3. 07Q2
        • Added "Extra Data" parallel to Abstracts; ie. "More …" (Location etc.)
        • Created auto-backup of Abstracts, and more efficient maintenance web-routines
        • Segregated Abstracts into multiple directories
    • Aesthetics
      1. 15Q4
        • Reduced font + changed text in ubiquitous THEO'S PAGE heading
    • Animadversions
      1. 19Q4
        • Create Aeon_Files table & code to generate Aeon Page65 (to simplify processing and get round the 64k character limit).
      2. 07Q3
        • Added a "hot topics" link - things that I'm currently working on and want feedback ("Animadversions")
    • Architecture
      1. 20Q3
      2. 20Q2
        • Enhanced the Functor processing to allow parameters. Previously, I had to create a new Functor ID each time for a very similar requirement. The main problem with this old method was that these Functors (where they call another Functor, or subroutine from a row in Functor) didn't feed through into the Documenter subroutines properly.
        • Truncated Cross_Reference_Changes table (from over 1M rows to under 50k) which should be self-maintaining, but isn't. This seems to have improved the regeneration times of Notes, Archived Notes and some other objects now that insertions into this table are quicker. Needs a permanent solution as part of my Cross-Referencing project.
      3. 18Q4
        • As revealed by Spider: Old links in unregenerated pages. 8 Items.
      4. 18Q3
        • Added 'TT' shortcut icon on all Webpage browser tabs
      5. 18Q2
        • Adjusted MS Access Control Page66 to feature new functions (Development Log & Automatic XRef) and consolidated the categorised Notes Lists into one button.
      6. 17Q4
        • Automated links between Quarterly reports (Task Lists)
      7. 15Q3
        • Made a lot of progress on improving the hyperlinking in my own website – I’ve made a good start on this project. The links now work accurately, but I need to ensure the efficient propagation of linkage changes across the site without requiring a complete regeneration.
      8. 15Q1
        • Vastly improved the “bulleting” options in Notes, etc.
      9. 14Q2
        • Added extra linkages between pages (eg. on Author).
      10. 14Q1
        • Repaired / updated my Website site-map
      11. 13Q3
        • Sorted conflicts between Subject, Topic & Sub-Topic IDs
      12. 13Q2
        • Improved error-recovery where encoding fails for links to Books, Papers and Authors. That is, where one of my bespoke tags (eg. +B67=53) doesn’t match with its corresponding anti-tag (eg. B+).
        • Replaced macros with code.
      13. 09Q2
        • Created Site-map for Website
      14. 07Q4
        • Added password-protection to selected pages
      15. 07Q2
        • Segregated database68=59 into Master and Slaves (code, papers, books, notes, web control, … vs PI)
        • Tidied up Footers throughout Philosophy Papers site (made into tables)
    • Authors
      1. 20Q2
        • Determined why the monthly regeneration process for Authors has ballooned from under 4 hours in January 2020 to over 17 hours in April 2020. Fix on 11/04/20 - time more than halved to 8.25 hours - by re-engineering the 'menu' process that counts the items linked to further down the page.
        • Enabled "alternate names" for identically-named authors (to avoid middle-initial = X):-
          1. Set up and populated Author_Name_Display
          2. Ensured Author_Name_Display appears on the Author page.
        • Further reduced the monthly regeneration process time for Authors from 8.25 hours to 13 minutes! Fixed by materialising and indexing a 'view' to table Authors_Cited_By_All_List.
      2. 20Q1
        • As revealed by Spider: Philosophers_Index_List_OA.htm. Author Names Scrambled. 133 items. Dud page (Ontological Argument) deleted.
        • Restrict jump-table at top of Authors pages to relevant items, and include the item counts.
      3. 19Q4
        • Improve author-citations process:-
          1. Show author-citation counts on Author Summary pages.
          2. Only have an author-citation link on Authors pages if the author is cited!
          3. Import Authors from Aeon_Files table.
      4. 19Q3
        • As revealed by Spider: Author pages not regenerated when name corrected, leading to Sundry broken Links from other pages.
      5. 19Q2
        • Added Jump Table to sections of Authors Pages
      6. 18Q3
        • As revealed by Spider: 'Extra' Links to Notes from Citations List have the wrong directory (clumped in 1,000's like Papers rather than 100's). 7 items. Problem with AddCitations_List, so not just an issue for Authors.
        • Authors Narrative: enable footnotes.
      7. 15Q4
        • Corrected author's names (SN (FN)) for all papers and books
        • Provided easy links to works by the same author; Books, Papers; Joint authors
      8. 15Q1
        • Improved the format and content of “Authors” pages.
      9. 14Q2
        • Created the facility for Author Narratives.
      10. 11Q2
        • Created Author Cross-reference
    • Backups
      1. 15Q4
        • Enhanced my backup system to prune items backed up multiple times (eg. if they moved directory).
      2. 14Q2
        • Used a clone of my Spider routines to create a backup and restore system.
    • Books
      1. 20Q3
        • BookCitings references not sequenced correctly. Corrected.
        • BooksToNotes Link Pages: Re-engineered - used CreatePapersToNotesWebPages to generate pages
      2. 20Q2
        • Determined why the monthly regeneration process for Book Summaries had ballooned from 1.1 hours in January 2020 to 3.5 hours in April 2020. Partly fixed by by materialising and indexing the view (now table) Book_Citings_List_New, so now takes 39 minutes. Further improvement required, given that Paper Summaries only takes 5 minutes.
      3. 18Q3
        • As revealed by Spider: BookCitings. 405 items. Fixed problem with CreateBookCitingsWebPages not checking for change of Book during 'Repeats' processing.
    • Books/Papers
      1. 20Q4
      2. 20Q3
      3. 20Q2
        • Determined why the monthly regeneration process for Book-Paper Abstracts had doubled from 2.4 hours in January 2020 to 4.8 hours in April 2020. Successfully reduced to just upder one hour by materialising and indexing BookPaperAbstracts_List.
      4. 19Q3
        • Print Write-up Notes (not just the links) for appropriate Papers in BookPaperAbstracts
      5. 19Q1
        • Removed Blackberry processing and associated pages as no longer needed with fast broadband.
      6. 18Q3
        • Paper & Book Abstracts - Make 'Disclaimer' more prominent & show 'Paper Statistics' rather than 'Paper Summaries'
      7. 17Q4
        • New "Recent Acquisitions" Book List - Uncategorised
      8. 16Q1
        • Created direct links from Subject, Topic & Sub-topic (in Book and Paper Summaries) to their link pages.
      9. 15Q4
        • Neatened up the linkage headers in Book & Paper Abstracts.
        • Updated booklists to include shelf-identifiers
      10. 13Q3
        • Improved linkages between Book/Paper Lists and Summaries or Abstracts
        • Made enhancements to "Books & Papers Collection Categorised" Pages
      11. 13Q2
        • Developed "Books by Location" Pages
        • Improved the linkage between Papers and Books.
      12. 11Q4
        • Showed Percentage of Paper / Book Read
      13. 11Q2
        • Added “changed dates” to Paper and Book abstracts, and produced “recently changed” pages for them
        • Created Inter-Book-Paper Citings Cross-reference
      14. 09Q3
        • Completed a full 2-way link from Book/Paper Summaries to Notes.
        • Made a lot of changes to the way Books and Papers are categorised; in particular:-
          1. Added 2-level jump tables, to save the browser being crowded with hundreds of cross-categorisations in the same page.
          2. While it’s OK for Books, it seems to have a few potholes for Papers, as some of the links fail.
          3. The problem may be something to do with my economical way of suppressing redundant sub-categorisation - ie. where one or more of Subject, Topic and Sub-topic are the same.
      15. 09Q2
        • Added "quality marker" to Paper & Book Abstracts
        • De-duplicated Books / Papers to Notes links
        • Improved, and corrected errors in, the Blackberry-friendly formatted lists of Books and Papers.
      16. 09Q1
        • Created automatic links from Paper Summary to Book Summary
        • Enabled links from Books to the Papers contained therein
        • Enhanced Books to Papers links to include papers multiply held
        • Improved the Books / Papers to Notes links to include the right Version and Section (and historical Title) of the Note
        • Provided an automated link from Books / Papers to the Notes referencing them
      17. 07Q2
        • Added "read" column in papers / books lists
        • Added Book Comments and Abstracts (as "More…") to Book-lists
        • Added change of author or title in "Abstracts audit"
        • Added Flags for "laptop" and "photocopy"; removed document location from Comments
        • Added multiple Jump-levels for BB Lists (to keep pages small)
        • Created "BB-friendly69=60" lists: Papers (Full) and Books
        • Created "Identity Papers Full by Sub-Topic" list
        • Created "Identity Read Papers by Sub-Topic" list
        • Created "Papers by Title within Topic" list
        • Jump-Tables: added "by topic" option
    • Bridge
      1. 20Q1
        • As revealed by Spider: Bridge_Code_Archive.htm (14 items) & Bridge_Code_Archive_Recent.htm (5 items)
      2. 19Q3
        • Add Mountnessing archive links to Results
        • NGS Refinements
        • Synchronise Documentator with main website generator
      3. 18Q3
        • Reformated + Fixed Hand-dealing pages to CSS standards
      4. 18Q1
        • Corrected format of Links from Bridge Page
      5. 17Q2
        • Created hyperlinks from my consolidated results pages to the actual results
      6. 13Q2
        • Created “Consolidated Bridge Results” webpages (for my own results).
    • Clock
      1. 16Q2
        • Created a "chess clock" application to help time tasks. Not much use in the end.
    • Data
      1. 15Q4
        • Corrected data content on Identity Sub-topics
    • Documenter
      1. 20Q2
      2. 18Q4
        • Outstanding Developments by Category appearing all in Bold, and links to lists failing.
      3. 18Q3
        • Created Functor using Form_Documentation_Links to automatically add control-links to documentation Notes.
        • Fixed Bug where a variable has the same name as a procedure, and is thereby confused with one. Produced a warning query (Variable_Code_Name_Clashes) for manual correction.
      4. 18Q2
        • Investigated the error reports from the Documenter:
          1. Improved the diagostics
          2. Removed unused variables from the Generator code.
      5. 18Q1
        • Created consolidated Development Log, and updated "Progress to Date" and "Outstanding Developments" Reports
        • Documenter Bug-fixes: Objects in Comments, MsgBoxs & Debugs
      6. 17Q4
        • Code Archive: Created Web-pages of Recent Code Changes
        • Code Archive: Hadn't been working - fixed and recovered a selection from backups
      7. 16Q1
        • Created an automatic Forms documenter – with links to the code.
      8. 15Q4
        • Segregated the code-archive, which was bloating71=50 the generator database.
      9. 14Q2
        • Created multiple secure areas on my website, so that Documentation is no longer hiding under my Supervision Notes.
      10. 13Q2
        • Created a code & query archive facility as part of the Code Documenter.
        • Wrote the Code Documenter.
    • Footnotes
      1. 18Q3
      2. 10Q2
        • Added intra-Note footnotes – a real breakthrough (strange I’d not thought of this before). It will enable me to convert to Note form all the old .pdfs from my undergraduate days – which are replete with footnotes – if this is any use.
      3. 07Q3
        • Improved transparency of Footnote referencing in printed Notes
      4. 07Q2
        • Created auto-backup of Footnotes74=61, and more efficient maintenance web-routines
        • Created auto-renumbering75=61 routine for Footnotes
    • General
      1. 19Q2
        • Created 'Chess' Home Page
          1. Created as a main menu item and Note76
          2. Created consolidated results pages
          3. Added games & links to pgn4web
      2. 18Q2
        • Website Tidy-up:-
          1. Wrote a function to delete - with specified exceptions - items not regenerated in the last full website regeneration.
          2. Tidied up Site - deleted non-updated pages (especially old Printed Notes). Automatically on the local site, manually - using the audit log - on the live site.
          3. Investigated setting up an FTP link as a drive so I could automatically delete from the live site, but couldn't get it to work. Easy to set up as a drive, but not to assign a drive-letter so the drive can be used by VBA.
      3. 12Q3
        • Improved Titles for Search Results display optimisation
      4. 07Q2
        • Improved "Disclaimer77=63" Page. Reduce font, add ref to philosophy databases + hyperlinks
    • Infrastructure
      1. 19Q3
        • PDFelement: Purchase & Commission upgrade from Version 5 to Version 7 for $39 - await new Laptop
        • Purchase & Commission new Lenovo Laptop: awaiting replacement following motherboard failure!
      2. 18Q1
        • Migration of Website from UKServers to Namesco. Test out.
    • Notes
      1. 20Q4
        • Add option in Auto_Reference_Notes to automatically ignore words containing certain strings that include the key-word (eg. ignore 'grace' and 'trace' when indexing 'race')
        • Added option in Auto_Reference_Notes to allow an updating run restricted to 'Read' (or at least half-read) Books / Papers only (useful for very long lists)
        • Created a Functor that reports which Notes which Functors (including which options) appear in, in both Functor and Note sequence. For use in Website Generator Documentation - Functors78.
        • Upgrade Auto_Reference_Notes to save Notes_To_Regen to a new table prior to the run, add new rows to this table, and copy it back after the run. Finally, allow the option of regenerating these Notes. In the interim, use Notes_To_Regen to create Note 87479, then clear it before the copy-back.
      2. 20Q3
        • Added Aeon Abstract (ie. Footnote) link to Aeon Webref items in Summary task List reports for 'read' items
        • Added option in Auto_Reference_Notes to - for Live (ie. not TEMP) Notes - update the latest Archived Note to equal the updated Live Note, hence avoiding superfluous archives.
        • Added option in Auto_Reference_Notes to only confirm new items (leaving previously-flagged items untouched)
        • Corrected Issues with Nested Functors: Added Functor_ID to End-tag
        • Determined why the run of Auto_Reference_Notes_Regen took 2.5 hours on 02/09/2020 (and climbing). It was down to the Aeon Note80 being searched; in particular Check_If_In_Container, checking for items in Footnotes and / or Quoted Text. As this Note is regenerated, there's no point auto-linking it. Time reduced to 23 minutes once this Note - and other Notes of inappropriate Note Groups - was removed.
        • Highlighted Archived Notes clearly as not the latest
        • Upgraded Auto_Reference_Notes to allow a parameterised 'termination or continue' for very long update runs
        • Upgraded Auto_Reference_Notes to avoid key-words within +AA+-style Authors names (using Check_If_In_Container).
        • Upgraded Auto_Reference_Notes to log its actions to a new table (Auto_Reference_Notes_Actions) so that any errors can be investigated and improvements made.
      3. 20Q2
        • As revealed by Spider: Sundry uncategorised. Refs failing. 30 items. Fixed manually.
      4. 20Q1
        • As revealed by Spider: #Reference links: fail for Printable Notes. 24 items.
        • As revealed by Spider: Blackberry links from archived Note 1010 (Maintenance Dashboard). 38 items.
        • As revealed by Spider: Blog81 Achived Notes - sundry links to Printable Notes. Live Note OK. 97 items.
        • As revealed by Spider: Old links from Archive Notes to (non-existent) Archived Printable Notes. 42 items.
        • Solve the problem of the size limitation for Notes by allowing their concatenation.
      5. 19Q4
        • Create procedures (manual and via Functor_17) to add Audio Files to Notes
      6. 18Q4
        • As revealed by Spider: Consciousness Status Report82 Achived Notes - links to BookCatalogCategorised_16_115_471.htm failing. Live Note OK. 46 items. 24 items outstanding.
        • As revealed by Spider: Links from Blog in Level 1 or 2 Printed Notes 51283 and 98184 failing. 141 items.
        • As revealed by Spider: Links to sundry 'static' pages failing.
          1. 13 items fixed.
          2. 7 items outstanding: to do with Pete's PhD Test site. Links seem to be OK, so may be a problem with the Spider. Directory "Petes_PhD" was the only one set to "do not parse". The file wasn't in the site map. Reset; hopefully it'll work!
        • As revealed by Spider: Status Reports - Achived Notes - sundry links failing. 50 items in total. Live Notes fixed.
          1. Website: Maintenance Dashboard85. 14 items.
          2. Personal Identity Status Report86. 11 items.
          3. Summary Status Report87. 4 items.
          4. Actual Detail Summary Report88. 6 items.
          5. Chess Status Report89. 12 items.
          6. Etc. 3 items.
        • As revealed by Spider: Sundry links failed for Supervisions. 30 items.
        • Create Priority Task List Report90.
      7. 18Q3
        • As revealed by Spider: Bridge Status Report91 Achived Notes - sundry links failing. Live Note OK. 70 items.
        • As revealed by Spider: Languages Status Report92 Achived Notes - links to Dissertation failing. Live Note OK. 43 items.
        • As revealed by Spider: Religion Status Report93 Achived Notes - sundry links failing. Live Note OK. 59 items.
        • As revealed by Spider: Links to Notes_xxx_Links, from Abstracts (94 items), Notes (25 … corrected manually), NotesPrint (1916). Fixed by +LL+ functor implementation.
        • As revealed by Spider: Notes_List_Control links fail for Supervisions. 58 items.
        • Automatic Key-word Hyperlinking to Notes Project: Phase 2
          1. Added links based on Book & Paper Titles
          2. Added ***Read***-marker to Lists
          3. Allowed links to be ignored (rather than suppressed) during approval process
          4. Skip links to Status Reports, etc.
        • Blog: Created Functor to generate Blog from Blog.
        • Created auto-regen of all Note Link (Note_nnn_Link) Pages
        • Created new '+LL+' Functor for links to Note_Link pages; fixed associated Notes.
      8. 18Q2
        • Added a process to 'fix' - by adding the Note-ID - those note-links added without stating the ID. This will save processing time.
        • Created automatic key-word referencing (ie. to Notes in the Notes-Group "Personal Identity"). This was complicated for a number of reasons:-
          1. There are always false positives, so there needed to be a way of "proofing" what's to be done before doing anything.
          2. There will be times where it is inappropriate (eg. in quoted work by other authors). So, there had to be a way of permanently suppressing such links.
          3. The referencing differs depending on whether the term is in a Footnote or not.
          4. I needed to check for already-existing links, variant grammatical forms, etc.
          5. I created a MsgBox routine - providing context - so that what needed to be done with potential links could easily be decided when the linking process was (re-)run.
          6. Links are always to Notes, but are from Notes, Archived Notes, Authors and Book & Paper Abstracts & Comments.
      9. 18Q1
      10. 17Q4
        • Added %age complete to Notes Reading Lists
        • Enabled deduction of Note link by omitting the Note ID from the mark-up and doing a look up on the key-word and its varients.
      11. 17Q3
        • Added Colour-coded "quality markers" to Notes: included them in lists and jump-tables
      12. 16Q1
        • Allowed Note- & Footnote-links from multiple words101=49. This is such a useful feature!
        • Completed converting “static” pages to Notes.
        • Enabled direct links to Notes from within Footnotes.
        • Stopped (I hope!) “private” Notes showing in Paper Abstracts.
      13. 15Q4
        • Converted the “static” pages in my website to Note format (so they can change and retain history).
        • Fixed problem of images not appearing in "Printed" Notes (wrong directory level) unless the images are repeated
      14. 15Q3
        • Tidied up the archived Notes.
      15. 14Q4
        • Made improvements to the regeneration and referencing of archived Notes.
      16. 14Q3
        • Created a progress-tracker102 for my Thesis.
        • Improved error-recovery where encoding fails for Notes links.
        • Started to split Paper Abstracts from Write-ups.
      17. 14Q2
        • Wrote routines to import old (but missing) archived Notes into the database so they can be (and have been) re-exported in the current format.
      18. 14Q1
        • Made a lot of improvements to my Site generators, in particular the pesky bug that stopped the links to archived Notes working correctly.
      19. 13Q3
        • "Single Image" development for Notes
        • Allowed "Authors Works Lists" to be linked-to from Notes
        • Created Archived Notes Re-Export facility
        • Simplified / Re-engineered Notes processing
      20. 13Q2
        • Improved "Concatenated Notes" production
      21. 10Q3
        • Add “Date Created” to Note_Links (for audit purposes; I’m still not convinced that links disappear when they should).
        • Investigate Note_Links: needs versioning, and section of calling Note
        • Wrote routines to automatically generate the “Printable Notes” footer – and to create the corresponding web pages – from the control table. Very useful.
      22. 09Q4
        • Restricted the size of the “previous versions” tables for Notes to 12.
      23. 09Q3
        • Enabled automatic immediate promotion to Live Website of selected Temp Notes.
        • Enabled Concatenated Notes (by Group). This is only allowed for “Identity” Notes at the moment, which is what I’d wanted it for. It allows all Notes for a Group to be printed off in Title sequence, for ease of review.
        • Ensured that the Notes Reading-list is generated from all 3 sub-topics.
        • Functionally enhanced the Notes-generation processing options; to make niftier production possible. In particular, where individual Notes are selected, I made the re-generation of the cross-references optional (so that if only textual variation is being iterated, the iterations are quicker).
        • Improved the performance of “All Notes” regeneration. The problem was with the generation of the generic reading lists based on Note Title. These only apply to the “Identity” Notes, and it was taking a long time to determine there was no list to produce.
      24. 09Q2
        • Added a table of Notes history at the end of each Note (ie. not just previous version) - to save tedious chaining
      25. 09Q1
        • Added reading-list (of works cited) at the end of the Note, as well as in the Printed Note
        • Created optional "don't print" feature for specific footnotes.
        • Fixed the problem of the overpopulation of the Notes directory
        • Option for Notes-Regen:
          1. Changed-only
          2. Changed, and all impacted by Change
          3. All
      26. 08Q1
        • Allowed regeneration of Temp Notes only
      27. 07Q4
        • Added referenced Books / Papers as footnote jump table to Notes
        • Added referenced reading lists to printable version of Notes
        • Improve "Temp" note processing. Suppress "_T" suffix (just don't copy to Transfer)
        • When a Note is archived, ensured a printable note is also archived103=57 (if this Note has a print)
      28. 07Q3
        • Created "respondent notes" - database identifier & red indicator + text
        • Created104=58 a "quick lookup" table of responses to a note and its sub-notes (recursive function on Notes_Links)
        • Enabled generation of printable version of Notes
        • Made Note-change-history available on-line (added status + publishable flags)
      29. 07Q2
        • Added "footnote" pages105=61 for Carthusians & King's
        • Added "outgoing links" summary footer to Notes pages
        • Allowed two sorts of Note - document-specific ("Write-up") and general
    • Papers
      1. 20Q3
      2. 20Q2
        • Determined why the monthly regeneration process for Paper Abstracts had ballooned from just over 6 hours in January 2020 to over 17 hours in April 2020. Reduced to just under 4 hours as a result of materialising and indexing the view (now table) Paper_Citings_List_New. Further improvements required.
        • Determined why the monthly regeneration process for Paper Summaries had ballooned from just over 3.5 hours in January 2020 to around 11.5 hours in April 2020. Fixed by materialising and indexing the view (now table) Paper_Citings_List_New, so now takes 6 minutes!
      3. 20Q1
        • Make use of Comments for large Abstracts seamless by appending Comments (if > 5k) to Abstracts.
      4. 18Q2
        • Papers associated with broken links:-
          1. Created an area to hold pdfs, and automated links thereto.
          2. Created a pop-up to say I'd remove such pdfs if requested by their authors.
          3. Consequently, haven't password protected the area yet.
      5. 18Q1
        • Correction of narratives on Note time-recording pseudo-papers
    • Photos
      1. 19Q1
        • Created Photo-logging usage via Images table.
        • Created Timeline software
        • Timeline software: Added photos of Coxes Farm
      2. 18Q3
      3. 13Q2
        • Reduced the size of photos (to make the pages quicker to load).
    • Process
      1. 20Q3
        • Determined why Recalculation (cmdRecalculate_Click) took so long: it varied depending on how many Notes were regenerated, but seemed to be 17.5 minutes even if there are none. I think it was down to a problem with nested Functors leading to Notes growing inordinately: in particular the development logs. Now takes around 5 minutes.
      2. 20Q2
        • Full Website Regeneration took 60 hours in April 2020. This has now been substantially fixed, as of end June 2020, in that regeneration now takes just under 11 hours. While further improvements are possible, the best use of time is to avoid the need for regular regeneration altogether by completing my Cross-referencing project.
        • Corrective work was split out into the items requiring improvement, must of which have now been implemented.
        • History and analysis:-
          1. This 'balooning' was on my new laptop which has solid state disks - it was taking 36 hours on my old laptop, then halved when I got the replacement, but then nearly doubled!
          2. I investigated why this was so, and improved performance mainly by putting in a trace (using a timer - GetTickCount - found on the web that allows logging of elapsed time in milliseconds, using StartTimer and EndTimer) and finding the 'pinch points'
          3. I suspected that it was a new release of MS Access that's caused a problem; maybe a lost index or something similar. I couldn't find any evidence of this, but a mumber of queries were taking a second or so to execute, which small times multiplied up drastically when the number of pages on my site - over 100k - are taken into account. Quite why a cliff-edge had been reached, I know not!
          4. Three areas had been particularly impacted: Authors, Paper Abstracts & Paper Summaries, which have all trippled or worse. These are covered as seperate developments under 'Authors' and 'Papers'.
          5. Book Summaries and Book-Paper Abstracts had also at least doubled, but this has had less of an impact as they took much less time in the first place.
          6. Other regeneration processes - in particular 'Notes' - didn't appear to have been impacted.
        • Wrote Check_Database_Size to check that the size of the database isn't at risk of breaching the 2GB limit. Required because of the need to re-materialise various views that have been instituted to speed up the full website regeneration routines, and which need to be run each time cmdRecalculate_Click is run, which is often.
      3. 19Q4
          1. Full Website Regeneration failed in Notes Regen because the first Private Note (133107) couldn't be copied from the Transfer directory as the private note hadn't been created there by then.
          2. It should have been, from looking at the code.
          3. Also, the private note itself (1163108) is not regenerated until later.
          4. In the start-December 2019 run, no such problem occurred. Maybe the problem was caused by copied between laptops. So, I've flagged it as complete for now, pending a recurrence.
        • Fix Functors to enable previous quarter's Status Reports after Time period roll-over
      4. 18Q4
      5. 18Q2
        • Re-engineered Website Maintenance Dashboard110 to show extra topical links (and show them at the top of the page).
      6. 18Q1
        • Improved efficiency of daily changes regeneration (removed msgboxes)
        • Wrote a routine to update a set of .htm files, replacing text in them all
      7. 15Q4
        • Created a consolidated processor to enable overnight regeneration of the whole website.
      8. 14Q1
        • Created a “Website Regeneration Dashboard” to show when the various elements of my website have been refreshed, and how long the routines took to run. Links to this appear in the footer to each webpage.
        • Improved the regeneration routines for the various classes of webpage, and got them to record statistics in support of the “Website Regeneration Dashboard”.
      9. 09Q4
        • Set up my database system to roll forward to the next academic year, and generate web-pages appropriately.
      10. 09Q3
        • Created a “performance stats” database and routines to write to it – though I only implemented them for the Notes generator.
      11. 07Q2
        • Improved ease of FTP process by use of Transfer Directory
    • Search
      1. 17Q2
        • Wrote routines to create a cut-down Website excluding pages with irrelevant or duplicated information. Used this to create the ZoomSearch facility.
      2. 07Q4
        • Created search facility using ZoomSearch
    • Spider
      1. 20Q4
        • System Resources Exceeded - "Run Time Error 3035" - re-emerged in Full_Link_Same_Directory_Gen. Last month I had replaced query Full_Link_Same_Directory_Updt with identical SQL, which had worked OK. This time I removed the "WHERE" statements and put them in the code itself. Seems fine.
      2. 20Q3
        • System Resources Exceeded - "Run Time Error 3035"; query Full_Link_Same_Directory_Updt in Full_Link_Same_Directory_Gen. Fixed by using SQL rather than a query.
        • The size of the main database was bloating to over 1.5Gb during the spider run, so was approaching the 2Gb limit.
          1. Used Check_Database_Size, with a parameter, to monitor the size of the main database and added a message similar to those reporting the compact / repair of the Slave database each time a status message is printed.
          2. Based on the diagnostics produced, it was found that the bloating occured at two stages:-
            → In Spider_Scurry: during the creation of Raw_Links_Temp_Temp.
            → In Spider_Copy: during the creation of Raw_Links_Temp.
          3. Fixes:-
            → Put check in Check_Database_Size to STOP if the database is over 1.5 Gb (parameterised via Max_Database_Size).
            → Moved Raw_Links_Temp_Temp to the Slave database.
      3. 18Q4
        • Determined why Full_Link_Up_Levels_Gen takes so long - approximately 4 hours. Re-architected to reduce database reads.
      4. 16Q4
        • Added “.shtml” and “.shtm” to the list of web-suffices.
      5. 15Q4
        • Finally cracked the Spider for external links, and used it to check the 3,000-odd links from my website. Some tidying up outstanding.
      6. 14Q4
        • Fixed the bug wherein the Spider ran out of system resources on large runs: now compacts & repairs the slave database as it does along (every 200k links; parameterised).
      7. 14Q3
        • Continued working on my Spider and started to use it to clean up my website by diagnosing broken links.
      8. 13Q2
        • De-bugged the Spider.
      9. 10Q2
        • Wrote my own Web Spider.
          1. It’s a little bit of a cheat, in that it doesn’t actually invoke the web-links, but crawls through my directory structure selecting all the .htm files, and rummages through the HTML looking for links which it then copies to a table.
          2. It also maintains a table of the directory structure and files therein, and then works out the real address from the relative addressing.
          3. It then allows cross-checking between the links and the files.
          4. Links external to the site have to be investigated manually (at least I have a list of all the links) – unless I can work out how to use the list to power a browser to check.
          5. There are about 333k links in 55k files – the full run took about 8 hours.
          6. It now runs incrementally based on file last-updated dates.
    • Status
      1. 20Q4
      2. 20Q3
        • Added Weekly Project Plans to Priority Task List
        • Created web-page (using Functor_21) showing oboe practice hours by work played
        • Quarterly Project Reports: Corrected Functor_08. The Project Planned YTD % kept having to be bodged!
      3. 19Q2
        • Corrected Quarterly Reporting for uneven Plan (Holidays)
      4. 18Q4
      5. 18Q3
      6. 18Q2
        • Added extra items to Functors to facilitate Quarterly Reporting:-
          1. Automatically calculated QTD & YTD time statistics in the quarterly reports.
          2. Provided a YTD task list for Status Reports for projects "in abeyance" to make reporting easier.
          3. Modified the statistics - to be just YTD - for YTD Status reports.
          4. Created table for - by Project - "Plans for the near future" tasks, and a functor to add them to the status report.
          5. Took "items competed" and "plans for the near future from the development log for the Web-tools status report.
          6. Created another functor to add these lists to the Summary report.
      7. 18Q1
        • Created web-page generator for Development items (ie. Created the Web-Project "To Do" List and "Outstanding Developments" directly from this list).
      8. 13Q3
        • Re-engineered Monthly Reporting on Change of Year
      9. 13Q2
        • Improved "This Month's Summary" Reporting
        • Improved Quarterly Reporting
      10. 13Q1
        • Re-engineered Monthly Reporting Tables & improved Monthly Reporting
    • Technology
      1. 19Q1
        • Investigated the Hit Stats provided by Webalizer and produced web-pages to show the most active pages.
      2. 13Q3
        • Converted Tables to CSS3 standard
        • Used CSS in all manually-crafted and automatically-generated pages
      3. 13Q1
        • Standardised to HTML5118=56
    • Timesheets
      1. 17Q4
        • Separated Time Recording for Aeon
      2. 15Q1
        • Segregated time-recording so I can easily see how much time is spent developing and supporting my own website, and how much is spent on those of other organisations.
      3. 13Q3
        • Re-engineered Time-Recording, so that actuals are held against Papers (and Books) in a separate table, rather than as columns by year in the Papers / Books tables themselves. Retained the grand totals only.
      4. 08Q4
        • Updated Book totals from totals of the Papers contained therein
    • WebRefs
      1. 20Q3
        • Improved WebRefs checker (Webrefs_Update) further to check for Error 403 "Forbidden". This involved finding way of checking pdfs where the returned page is in fact HTML or XML (see DevLog Ref 379).
        • Reformated WebLinks_Tester.htm, WebLinks_Tester_Map.htm, WebLinks_Tester_Full.htm & WebLinks_Tester_Full_Map.htm: The 'As Above" lines waste space. Consolidate onto single second line.
        • Reformated WebLinks_Tester_Brief, WebLinks_Tester.htm, WebLinks_Tester_Map.htm, WebLinks_Tester_Full.htm & WebLinks_Tester_Full_Map.htm: Allowed more space for 'link returned', 'issue' and 'display text' and added Explanation column
      2. 20Q2
        • Webrefs_Update: Improved the processing of this sub which controls IE to check the URLs corresponding to WeRefs are still valid.
          1. Improved the recovery processing after IE fails or becomes detached. Now seems to work perfectly!
          2. Improved processing for 404 (page not found) errors - detect them where URL returned differs from that requested (so the error is correctly categorised) and also where it is the same (so the error is detected) - by using the GetElementsByTagName method to look through Title and H1 elemnts (where they exist).
          3. Note that there is still an issue for pdfs where the above methods don't work; though the page returned is an HTML or XML page, the changed URL still claims to be a pdf, so I can't check in case it's a real pdf and the process fails.
        • Amend WebRef 'Name' link to #Off-Page_Link_WxxxW style. Also amend Quarterly and Annual task lists to use this reference.
        • Amended WebRef 'Name' links to #Off-Page_Link_WxxxW style so referencing from other pages is possible (useful for Aeon)
        • As revealed by Spider: WebLinks_Tester_Full_Map.htm (etc). Refs failing. 184 items. res://ieframe.dll/ in Returned_URL.
        • WebLinks_Tester: If the URLs of WebRefs are changed between Spider runs, the URL mapping fails, and the WebRefs appear at the end of the last page, wrongly categorised. Fixed to ensure they appear on their own page, added to the jump table as 'WebRef Missing' and added an explanation.
        • WebLinks_Tester_Full & WebLinks_Tester_Full_Map: Reformated Jump Table to 30 columns and multiple rows.
      3. 20Q1
        • As revealed by Spider: WebLinks_Tester_4.htm. Refs failing. 32 items. res://ieframe.dll/ in Returned_URL.
        • As revealed by Spider: WebLinks_Tester_Brief_12.htm (etc). Refs failing. 41 items. res://ieframe.dll/ in Returned_URL.
        • As revealed by Spider: WebLinks_Tester_Map_3.htm (etc). Refs failing. 51 items outstanding. res://ieframe.dll/ in Returned_URL.
        • WebRefs checker failing to detect broken links for Aeon - if a link fails, I get the message 'The page you were looking for could not be found (404)', but since the URL doesn't change, I don't spot it. This may happen for other sites as well. Fixed for Aeon - watch out for others.
      4. 19Q3
        • As revealed by Spider: Hits_Pages_Total_Crosstab_Descending.htm.
      5. 18Q4
        • Added Display text to WebRefs for Links to Stanford, Wikipedia & Aeon
      6. 18Q2
      7. 18Q1
        • Determined & reported usage directly using embedded +WnnnW+ encoding rather than via the Spider. Reconciled the two methods.
        • Documentation & Bug-fixes: Phase 1
          1. Completed "as is" functional documentation so I could see what's going on.
          2. Re-engineered so it made sense and ran efficiently without bugs
          3. Fixed the obvious data errors that confused the statistics - ie. where the returned URL differs from that requested and https issues.
          4. Wrote / improved diagnostic reports (web pages)
      8. 16Q1
        • Some tidying up of the 3,000-odd external links from my website.
      9. 13Q2
        • Some pages render oddly in Chrome (and maybe other browsers) – investigated120=54.
      10. 10Q3
        • Improved the functionality for automatically generating external Web-links in Notes, Book Abstracts and Paper Abstracts. Corrected current data to accommodate the new code.
      11. 07Q2
        • Made Web Refs in Comments / Abstracts into Hyperlinks
  2. Other Websites: Completed Items By Category:-
    • Acts
      1. 16Q3
        • Developed a website for displaying the textual and grammatical analyses and appendices of Pete’s PhD on the Acts of the Apostles. Two versions: Live (Acts: Live Site) and Test (Acts: Test Site).
    • Bridge
      1. 17Q1
        • Added new features (70s Club & Monthly Pairs Winners) to my Bridge mini-websites.
        • Created Master Point Lists & Promotions for the ECBA website.
      2. 16Q3
      3. 16Q1
        • Accommodated Alaric’s new scoring system for St. Edith’s.
        • Created Ladies’ and Gentlemen’s ladders for my mini-websites.
        • Reconciled my X-IMP calculations with a new feature provided by Bridgewebs.
      4. 15Q4
        • Segregated the Bridge website generators from those for my own website. They now live in separate databases and have their own documenters.
      5. 15Q3
        • Created a new website (using Bridgewebs) for the First Class Bridge Academy (First Class Bridge).
        • Ported my “handicap / improvers” routines from Hutton to “Mini Websites”.
      6. 15Q2
        • Development of the “Eve Goblets” (victor ludorum) software for the ECBA, and roll-over to the new season.
      7. 15Q1
        • Further improvements to my Butler-IMPs and Cross-IMPs calculation routines.
        • Started a project to provide small websites for bridge clubs without sites: Essex Bridge Results.
        • Took on the ECBA Website (ECBA Website (Defunct)), at the cost of considerable effort by way of reformatting & tidy-up.
      8. 14Q4
        • Improvements to my Butler-IMPs and Cross-IMPs calculation routines – different formats, including the Jeff Smith format, and extension to Butler-scored Pairs events. This work has been taken up by the ECBA selectors for county-team selection.
      9. 14Q3
        • Created a suite of programs to extract data and calculate Butler-IMPs and Cross-IMPs from the travellers on Bridgewebs Teams events, and output ranking-lists to the Web.
      10. 13Q2
        • Merged121=55 Philosophy & Bridge Databases
    • Enigma
      1. 18Q4
        • Added 'EE' shortcut icon on all Webpage browser tabs
      2. 15Q3
        • Created a small website for a music group Julie and I attend – the Enigma Ensemble.
    • Sophie
      1. 19Q3
      2. 18Q4
        • Added 'SB' shortcut icon on all Webpage browser tabs
      3. 15Q3



In-Page Footnotes:

Footnote 6: Footnotes 32, 72: Footnotes 33, 73: Footnotes 42, 94: Footnotes 49, 101: Footnotes 50, 71: Footnotes 51, 122: Footnotes 53, 67: Footnotes 54, 120: Footnotes 55, 121: Footnotes 56, 118: Footnotes 57, 103: Footnotes 58, 104: Footnotes 59, 68: Footnotes 60, 69: Footnotes 61, 62, 64, 74, 75, 105: Footnotes 63, 77:


Printable Version:



Table of the Previous 12 Versions of this Note: (of 17)

Date Length Title
04/10/2020 00:27:22 95240 Website - Progress to Date (2020 - September)
03/07/2020 22:09:07 86201 Website - Progress to Date (2020 - June)
04/04/2020 00:14:24 72717 Website - Progress to Date (2020 - March)
19/01/2020 23:41:17 69093 Website - Progress to Date (2019 - December)
10/10/2019 23:58:34 66813 Website - Progress to Date (2019 - October)
14/07/2019 20:29:46 65148 Website - Progress to Date (2019 - June)
05/04/2019 10:36:29 70085 Website - Progress to Date (2019 - March)
06/01/2019 23:36:58 68922 Website - Progress to Date (2018 - December)
10/10/2018 16:43:41 65062 Website - Progress to Date (2018 - September)
06/07/2018 18:56:10 59064 Website - Progress to Date (2018 - June)
05/04/2018 10:48:00 51445 Website - Progress to Date (2018 - March)
18/12/2010 19:58:05 8619 Status: Web-Tools (Summary of Progress to Date)



Note last updated Reference for this Topic Parent Topic
22/10/2020 10:18:33 822 (Website - Progress to Date (2020 - October)) Status: Web-Tools (2020 - September)

Summary of Note Links from this Page

Aeon Papers Carthusians - Main Text Copyright Disclaimer L. Messel & Co. Private Note
Status: Actual Detail Summary (2007 - 2021) Status: Bridge (2020 - September) Status: Chess (2020 - September) Status: Consciousness Studies (2020 - September) Status: Languages (2020 - September)
Status: Personal Identity (2020 - September) Status: Philosophy of Religion (2020 - September) Status: Priority Task List (2020 - October) Status: Summary - Actual versus Plan (2020 - October) Status: Summary (2020 - September)
Status: Summary Task List (2020: October) Status: Summary Task List (YTD: 20Q4) Status: Thesis Dashboard (2020: October) Status: Web-Tools (2020 - September) Test Note - Auto-XRef
Theo Todman's Blog Theo Todman's Chess Page Website - Outstanding Developments (2020 - October) Website - Progress to Date (2020 - October) Website Generator Documentation - Control Page
Website Generator Documentation - Cross-Referencing Website Generator Documentation - Functors Website Generator Documentation - Web Links Website: Maintenance Dashboard  

To access information, click on one of the links in the table above.




Summary of Note Links to this Page

Status: Web-Tools (2020 - September) Website - Outstanding Developments (2020 - October) Website - Progress to Date (2020 - October), 2, 3, 4, 5, 6, 7, 8 Website Generator Documentation - Control Page Website Generator Documentation - Functors, 2, 3, 4, 5

To access information, click on one of the links in the table above.




Text Colour Conventions

  1. Black: Printable Text by me; © Theo Todman, 2020
  2. Blue: Text by me; © Theo Todman, 2020




© Theo Todman, June 2007 - Oct 2020.Please address any comments on this page to theo@theotodman.com.File output:
Website Maintenance Dashboard
Return to Top of this PageReturn to Theo Todman's Philosophy PageReturn to Theo Todman's Home Page