Theo Todman's Web Page - Notes Pages

Status Reports

Website - Progress to Date (2020 - September)

(Text as at 04/10/2020 00:27:22)

*** THIS IS NOT THE LATEST VERSION OF THIS NOTE ***


(For the live version and other versions of this Note, see the tables at the end)

Rationale for this Document


Items Completed In6:-

Items Completed by Category:
  1. Own Website: Completed Items By Category:-
    • Abstracts
      1. 20Q1: As revealed by Spider: Link to PaperCitings failing (2 items) - Timing issue
      2. 15Q4: Corrected data format on Identity Abstracts
      3. 07Q2: Added "Extra Data" parallel to Abstracts; ie. "More …" (Location etc.)
      4. 07Q2: Created auto-backup of Abstracts, and more efficient maintenance web-routines
      5. 07Q2: Segregated Abstracts into multiple directories
    • Aesthetics
      1. 15Q4: Reduced font + changed text in ubiquitous THEO'S PAGE heading
    • Animadversions
      1. 19Q4: Create Aeon_Files table & code to generate Aeon Page59 (to simplify processing and get round the 64k character limit).
      2. 07Q3: Added a "hot topics" link - things that I'm currently working on and want feedback ("Animadversions")
    • Architecture
      1. 20Q3: Created Cross_Reference_Changes_Prune to keep Cross_Reference_Changes under control pending completion of the Xref project.
      2. 20Q2: Enhanced the Functor processing to allow parameters. Previously, I had to create a new Functor ID each time for a very similar requirement. The main problem with this old method was that these Functors (where they call another Functor, or subroutine from a row in Functor) didn't feed through into the Documenter subroutines properly.
      3. 20Q2: Truncated Cross_Reference_Changes table (from over 1M rows to under 50k) which should be self-maintaining, but isn't. This seems to have improved the regeneration times of Notes, Archived Notes and some other objects now that insertions into this table are quicker. Needs a permanent solution as part of my Cross-Referencing project.
      4. 18Q4: As revealed by Spider: Old links in unregenerated pages. 8 Items.
      5. 18Q3: Added 'TT' shortcut icon on all Webpage browser tabs
      6. 18Q2: Adjusted MS Access Control Page60 to feature new functions (Development Log & Automatic XRef) and consolidated the categorised Notes Lists into one button.
      7. 17Q4: Automated links between Quarterly reports (Task Lists)
      8. 15Q3: Made a lot of progress on improving the hyperlinking in my own website – I’ve made a good start on this project. The links now work accurately, but I need to ensure the efficient propagation of linkage changes across the site without requiring a complete regeneration.
      9. 15Q1: Vastly improved the “bulleting” options in Notes, etc.
      10. 14Q2: Added extra linkages between pages (eg. on Author).
      11. 14Q1: Repaired / updated my Website site-map
      12. 13Q3: Sorted conflicts between Subject, Topic & Sub-Topic IDs
      13. 13Q2: Improved error-recovery where encoding fails for links to Books, Papers and Authors. That is, where one of my bespoke tags (eg. +B61) doesn’t match with its corresponding anti-tag (eg. B+).
      14. 13Q2: Replaced macros with code.
      15. 09Q2: Created Site-map for Website
      16. 07Q4: Added password-protection to selected pages
      17. 07Q2: Segregated database62 into Master and Slaves (code, papers, books, notes, web control, … vs PI)
      18. 07Q2: Tidied up Footers throughout Philosophy Papers site (made into tables)
    • Authors
      1. 20Q2: Determined why the monthly regeneration process for Authors has ballooned from under 4 hours in January 2020 to over 17 hours in April 2020. Fix on 11/04/20 - time more than halved to 8.25 hours - by re-engineering the 'menu' process that counts the items linked to further down the page.
      2. 20Q2: Enabled "alternate names" for identically-named authors (to avoid middle-initial = X):-
        1. Set up and populated Author_Name_Display
        2. Ensured Author_Name_Display appears on the Author page.
      3. 20Q2: Further reduced the monthly regeneration process time for Authors from 8.25 hours to 13 minutes! Fixed by materialising and indexing a 'view' to table Authors_Cited_By_All_List.
      4. 20Q1: As revealed by Spider: Philosophers_Index_List_OA.htm. Author Names Scrambled. 133 items. Dud page (Ontological Argument) deleted.
      5. 20Q1: Restrict jump-table at top of Authors pages to relevant items, and include the item counts.
      6. 19Q4: Improve author-citations process:-
        1. Show author-citation counts on Author Summary pages.
        2. Only have an author-citation link on Authors pages if the author is cited!
        3. Import Authors from Aeon_Files table.
      7. 19Q3: As revealed by Spider: Author pages not regenerated when name corrected, leading to Sundry broken Links from other pages.
      8. 19Q2: Added Jump Table to sections of Authors Pages
      9. 18Q3: As revealed by Spider: 'Extra' Links to Notes from Citations List have the wrong directory (clumped in 1,000's like Papers rather than 100's). 7 items. Problem with AddCitations_List, so not just an issue for Authors.
      10. 18Q3: Authors Narrative: enable footnotes.
      11. 15Q4: Corrected author's names (SN (FN)) for all papers and books
      12. 15Q4: Provided easy links to works by the same author; Books, Papers; Joint authors
      13. 15Q1: Improved the format and content of “Authors” pages.
      14. 14Q2: Created the facility for Author Narratives.
      15. 11Q2: Created Author Cross-reference
    • Backups
      1. 15Q4: Enhanced my backup system to prune items backed up multiple times (eg. if they moved directory).
      2. 14Q2: Used a clone of my Spider routines to create a backup and restore system.
    • Books
      1. 20Q3: BookCitings references not sequenced correctly. Corrected.
      2. 20Q3: BooksToNotes Link Pages: Re-engineered - used CreatePapersToNotesWebPages to generate pages
      3. 20Q2: Determined why the monthly regeneration process for Book Summaries had ballooned from 1.1 hours in January 2020 to 3.5 hours in April 2020. Partly fixed by by materialising and indexing the view (now table) Book_Citings_List_New, so now takes 39 minutes. Further improvement required, given that Paper Summaries only takes 5 minutes.
      4. 18Q3: As revealed by Spider: BookCitings. 405 items. Fixed problem with CreateBookCitingsWebPages not checking for change of Book during 'Repeats' processing.
    • Books/Papers
      1. 20Q3: CreateBookPaperAbstractsWebPages not translating +RR+ References for Notes Write-ups of associated Papers
      2. 20Q3: As revealed by Spider: BooksToNotes and PapersToNotes links failing (45 items). Pages not being regenerated due to wrong joins in Book_Note_Link_List and Paper_Note_Link_List.
      3. 20Q2: Determined why the monthly regeneration process for Book-Paper Abstracts had doubled from 2.4 hours in January 2020 to 4.8 hours in April 2020. Successfully reduced to just upder one hour by materialising and indexing BookPaperAbstracts_List.
      4. 19Q3: Print Write-up Notes (not just the links) for appropriate Papers in BookPaperAbstracts
      5. 19Q1: Removed Blackberry processing and associated pages as no longer needed with fast broadband.
      6. 18Q3: Paper & Book Abstracts - Make 'Disclaimer' more prominent & show 'Paper Statistics' rather than 'Paper Summaries'
      7. 17Q4: New "Recent Acquisitions" Book List - Uncategorised
      8. 16Q1: Created direct links from Subject, Topic & Sub-topic (in Book and Paper Summaries) to their link pages.
      9. 15Q4: Neatened up the linkage headers in Book & Paper Abstracts.
      10. 15Q4: Updated booklists to include shelf-identifiers
      11. 13Q3: Improved linkages between Book/Paper Lists and Summaries or Abstracts
      12. 13Q3: Made enhancements to "Books & Papers Collection Categorised" Pages
      13. 13Q2: Developed "Books by Location" Pages
      14. 13Q2: Improved the linkage between Papers and Books.
      15. 11Q4: Showed Percentage of Paper / Book Read
      16. 11Q2: Added “changed dates” to Paper and Book abstracts, and produced “recently changed” pages for them
      17. 11Q2: Created Inter-Book-Paper Citings Cross-reference
      18. 09Q3: Completed a full 2-way link from Book/Paper Summaries to Notes.
      19. 09Q3: Made a lot of changes to the way Books and Papers are categorised; in particular:-
        1. Added 2-level jump tables, to save the browser being crowded with hundreds of cross-categorisations in the same page.
        2. While it’s OK for Books, it seems to have a few potholes for Papers, as some of the links fail.
        3. The problem may be something to do with my economical way of suppressing redundant sub-categorisation - ie. where one or more of Subject, Topic and Sub-topic are the same.
      20. 09Q2: Added "quality marker" to Paper & Book Abstracts
      21. 09Q2: De-duplicated Books / Papers to Notes links
      22. 09Q2: Improved, and corrected errors in, the Blackberry-friendly formatted lists of Books and Papers.
      23. 09Q1: Created automatic links from Paper Summary to Book Summary
      24. 09Q1: Enabled links from Books to the Papers contained therein
      25. 09Q1: Enhanced Books to Papers links to include papers multiply held
      26. 09Q1: Improved the Books / Papers to Notes links to include the right Version and Section (and historical Title) of the Note
      27. 09Q1: Provided an automated link from Books / Papers to the Notes referencing them
      28. 07Q2: Added "read" column in papers / books lists
      29. 07Q2: Added Book Comments and Abstracts (as "More…") to Book-lists
      30. 07Q2: Added change of author or title in "Abstracts audit"
      31. 07Q2: Added Flags for "laptop" and "photocopy"; removed document location from Comments
      32. 07Q2: Added multiple Jump-levels for BB Lists (to keep pages small)
      33. 07Q2: Created "BB-friendly63" lists: Papers (Full) and Books
      34. 07Q2: Created "Identity Papers Full by Sub-Topic" list
      35. 07Q2: Created "Identity Read Papers by Sub-Topic" list
      36. 07Q2: Created "Papers by Title within Topic" list
      37. 07Q2: Jump-Tables: added "by topic" option
    • Bridge
      1. 20Q1: As revealed by Spider: Bridge_Code_Archive.htm (14 items) & Bridge_Code_Archive_Recent.htm (5 items)
      2. 19Q3: Add Mountnessing archive links to Results
      3. 19Q3: NGS Refinements
      4. 19Q3: Synchronise Documentator with main website generator
      5. 18Q3: Reformated + Fixed Hand-dealing pages to CSS standards
      6. 18Q1: Corrected format of Links from Bridge Page
      7. 17Q2: Created hyperlinks from my consolidated results pages to the actual results
      8. 13Q2: Created “Consolidated Bridge Results” webpages (for my own results).
    • Clock
      1. 16Q2: Created a "chess clock" application to help time tasks. Not much use in the end.
    • Data
      1. 15Q4: Corrected data content on Identity Sub-topics
    • Documenter
      1. 20Q2: Created various generic Functors to help Document Cross-Referencing by producing cross-tabs and lists from queries. Then the associated stats can be regenerated each time the documentation64 is regenerated (as TEMP): Functor_21, Functor_22, Functor_23, Number_Format.
      2. 18Q4: Outstanding Developments by Category appearing all in Bold, and links to lists failing.
      3. 18Q3: Created Functor using Form_Documentation_Links to automatically add control-links to documentation Notes.
      4. 18Q3: Fixed Bug where a variable has the same name as a procedure, and is thereby confused with one. Produced a warning query (Variable_Code_Name_Clashes) for manual correction.
      5. 18Q2: Investigated the error reports from the Documenter:
        1. Improved the diagostics
        2. Removed unused variables from the Generator code.
      6. 18Q1: Created consolidated Development Log, and updated "Progress to Date" and "Outstanding Developments" Reports
      7. 18Q1: Documenter Bug-fixes: Objects in Comments, MsgBoxs & Debugs
      8. 17Q4: Code Archive: Created Web-pages of Recent Code Changes
      9. 17Q4: Code Archive: Hadn't been working - fixed and recovered a selection from backups
      10. 16Q1: Created an automatic Forms documenter – with links to the code.
      11. 15Q4: Segregated the code-archive, which was bloating65 the generator database.
      12. 14Q2: Created multiple secure areas on my website, so that Documentation is no longer hiding under my Supervision Notes.
      13. 13Q2: Created a code & query archive facility as part of the Code Documenter.
      14. 13Q2: Wrote the Code Documenter.
    • Footnotes
      1. 18Q3: Enabled the re-use of Footnotes in the same Note66. They appeared multiple times, now just once. Fixed for:-
        1. Notes (OutputNotesWebPage)
        2. Printable Notes (NoteForPrinting), and
        3. Archived Notes (OutputNotesWebPage_Archived)
        All of which use Notes_Text_Format and the Note_Footnotes table. Also,
        1. Book & Paper Abstracts & Comments used to use an array in Reference_Footnotes, but changed to use a new table67 Abstract_Footnotes.
      2. 10Q2: Added intra-Note footnotes – a real breakthrough (strange I’d not thought of this before). It will enable me to convert to Note form all the old .pdfs from my undergraduate days – which are replete with footnotes – if this is any use.
      3. 07Q3: Improved transparency of Footnote referencing in printed Notes
      4. 07Q2: Created auto-backup of Footnotes68, and more efficient maintenance web-routines
      5. 07Q2: Created auto-renumbering69 routine for Footnotes
    • General
      1. 19Q2: Created 'Chess' Home Page
        1. Created as a main menu item and Note70
        2. Created consolidated results pages
        3. Added games & links to pgn4web
      2. 18Q2: Website Tidy-up:-
        1. Wrote a function to delete - with specified exceptions - items not regenerated in the last full website regeneration.
        2. Tidied up Site - deleted non-updated pages (especially old Printed Notes). Automatically on the local site, manually - using the audit log - on the live site.
        3. Investigated setting up an FTP link as a drive so I could automatically delete from the live site, but couldn't get it to work. Easy to set up as a drive, but not to assign a drive-letter so the drive can be used by VBA.
      3. 12Q3: Improved Titles for Search Results display optimisation
      4. 07Q2: Improved "Disclaimer71" Page. Reduce font, add ref to philosophy databases + hyperlinks
    • Infrastructure
      1. 19Q3: PDFelement: Purchase & Commission upgrade from Version 5 to Version 7 for $39 - await new Laptop
      2. 19Q3: Purchase & Commission new Lenovo Laptop: awaiting replacement following motherboard failure!
      3. 18Q1: Migration of Website from UKServers to Namesco. Test out.
    • Notes
      1. 20Q3: Added Aeon Abstract (ie. Footnote) link to Aeon Webref items in Summary task List reports for 'read' items
      2. 20Q3: Added option in Auto_Reference_Notes to - for Live (ie. not TEMP) Notes - update the latest Archived Note to equal the updated Live Note, hence avoiding superfluous archives.
      3. 20Q3: Added option in Auto_Reference_Notes to only confirm new items (leaving previously-flagged items untouched)
      4. 20Q3: Corrected Issues with Nested Functors: Added Functor_ID to End-tag
      5. 20Q3: Determined why the run of Auto_Reference_Notes_Regen took 2.5 hours on 02/09/2020 (and climbing). It was down to the Aeon Note72 being searched; in particular Check_If_In_Container, checking for items in Footnotes and / or Quoted Text. As this Note is regenerated, there's no point auto-linking it. Time reduced to 23 minutes once this Note - and other Notes of inappropriate Note Groups - was removed.
      6. 20Q3: Upgraded Auto_Reference_Notes to allow a parameterised 'termination or continue' for very long update runs
      7. 20Q3: Upgraded Auto_Reference_Notes to avoid key-words within +AA+-style Authors names (using Check_If_In_Container).
      8. 20Q2: As revealed by Spider: Sundry uncategorised. Refs failing. 30 items. Fixed manually.
      9. 20Q1: As revealed by Spider: #Reference links: fail for Printable Notes. 24 items.
      10. 20Q1: As revealed by Spider: Blackberry links from archived Note 1010 (Maintenance Dashboard). 38 items.
      11. 20Q1: As revealed by Spider: Blog73 Achived Notes - sundry links to Printable Notes. Live Note OK. 97 items.
      12. 20Q1: As revealed by Spider: Old links from Archive Notes to (non-existent) Archived Printable Notes. 42 items.
      13. 20Q1: Solve the problem of the size limitation for Notes by allowing their concatenation.
      14. 19Q4: Create procedures (manual and via Functor_17) to add Audio Files to Notes
      15. 18Q4: As revealed by Spider: Consciousness Status Report74 Achived Notes - links to BookCatalogCategorised_16_115_471.htm failing. Live Note OK. 46 items. 24 items outstanding.
      16. 18Q4: As revealed by Spider: Links from Blog in Level 1 or 2 Printed Notes 51275 and 98176 failing. 141 items.
      17. 18Q4: As revealed by Spider: Links to sundry 'static' pages failing.
        1. 13 items fixed.
        2. 7 items outstanding: to do with Pete's PhD Test site. Links seem to be OK, so may be a problem with the Spider. Directory "Petes_PhD" was the only one set to "do not parse". The file wasn't in the site map. Reset; hopefully it'll work!
      18. 18Q4: As revealed by Spider: Status Reports - Achived Notes - sundry links failing. 50 items in total. Live Notes fixed.
        1. Website: Maintenance Dashboard77. 14 items.
        2. Personal Identity Status Report78. 11 items.
        3. Summary Status Report79. 4 items.
        4. Actual Detail Summary Report80. 6 items.
        5. Chess Status Report81. 12 items.
        6. Etc. 3 items.
      19. 18Q4: As revealed by Spider: Sundry links failed for Supervisions. 30 items.
      20. 18Q4: Create Priority Task List Report82.
      21. 18Q3: As revealed by Spider: Bridge Status Report83 Achived Notes - sundry links failing. Live Note OK. 70 items.
      22. 18Q3: As revealed by Spider: Languages Status Report84 Achived Notes - links to Dissertation failing. Live Note OK. 43 items.
      23. 18Q3: As revealed by Spider: Religion Status Report85 Achived Notes - sundry links failing. Live Note OK. 59 items.
      24. 18Q3: As revealed by Spider: Links to Notes_xxx_Links, from Abstracts (94 items), Notes (25 … corrected manually), NotesPrint (1916). Fixed by +LL+ functor implementation.
      25. 18Q3: As revealed by Spider: Notes_List_Control links fail for Supervisions. 58 items.
      26. 18Q3: Automatic Key-word Hyperlinking to Notes Project: Phase 2
        1. Added links based on Book & Paper Titles
        2. Added ***Read***-marker to Lists
        3. Allowed links to be ignored (rather than suppressed) during approval process
        4. Skip links to Status Reports, etc.
      27. 18Q3: Blog: Created Functor to generate Blog from Blog.
      28. 18Q3: Created auto-regen of all Note Link (Note_nnn_Link) Pages
      29. 18Q3: Created new '+LL+' Functor for links to Note_Link pages; fixed associated Notes.
      30. 18Q2: Added a process to 'fix' - by adding the Note-ID - those note-links added without stating the ID. This will save processing time.
      31. 18Q2: Created automatic key-word referencing (ie. to Notes in the Notes-Group "Personal Identity"). This was complicated for a number of reasons:-
        1. There are always false positives, so there needed to be a way of "proofing" what's to be done before doing anything.
        2. There will be times where it is inappropriate (eg. in quoted work by other authors). So, there had to be a way of permanently suppressing such links.
        3. The referencing differs depending on whether the term is in a Footnote or not.
        4. I needed to check for already-existing links, variant grammatical forms, etc.
        5. I created a MsgBox routine - providing context - so that what needed to be done with potential links could easily be decided when the linking process was (re-)run.
        6. Links are always to Notes, but are from Notes, Archived Notes, Authors and Book & Paper Abstracts & Comments.
      32. 18Q1: Automated format-change of old tables in Notes and Notes_Archive
      33. 18Q1: Extended deduction of Note links to Papers, and Papers with embedded Notes.
      34. 18Q1: Notes Functor86 Processing: Phase 1
        1. Basic processing to interpret comments in HTML code
        2. 01: Quarterly Report: Activity insert87
        3. 02: Development Log report - Completed Items by Date88
        4. 03: Development Log report - Outstanding Items by Priority89
        5. 04: Development Log report - Outstanding Items by Category90
        6. 05: Development Log report - Completed Items by Category91
        7. 06: Quarterly Report: Priority 1 Outstanding Developments by Category92
      35. 18Q1: Removed old "Printable Notes" links from Archived Notes
      36. 17Q4: Added %age complete to Notes Reading Lists
      37. 17Q4: Enabled deduction of Note link by omitting the Note ID from the mark-up and doing a look up on the key-word and its varients.
      38. 17Q3: Added Colour-coded "quality markers" to Notes: included them in lists and jump-tables
      39. 16Q1: Allowed Note- & Footnote-links from multiple words93. This is such a useful feature!
      40. 16Q1: Completed converting “static” pages to Notes.
      41. 16Q1: Enabled direct links to Notes from within Footnotes.
      42. 16Q1: Stopped (I hope!) “private” Notes showing in Paper Abstracts.
      43. 15Q4: Converted the “static” pages in my website to Note format (so they can change and retain history).
      44. 15Q4: Fixed problem of images not appearing in "Printed" Notes (wrong directory level) unless the images are repeated
      45. 15Q3: Tidied up the archived Notes.
      46. 14Q4: Made improvements to the regeneration and referencing of archived Notes.
      47. 14Q3: Created a progress-tracker94 for my Thesis.
      48. 14Q3: Improved error-recovery where encoding fails for Notes links.
      49. 14Q3: Started to split Paper Abstracts from Write-ups.
      50. 14Q2: Wrote routines to import old (but missing) archived Notes into the database so they can be (and have been) re-exported in the current format.
      51. 14Q1: Made a lot of improvements to my Site generators, in particular the pesky bug that stopped the links to archived Notes working correctly.
      52. 13Q3: "Single Image" development for Notes
      53. 13Q3: Allowed "Authors Works Lists" to be linked-to from Notes
      54. 13Q3: Created Archived Notes Re-Export facility
      55. 13Q3: Simplified / Re-engineered Notes processing
      56. 13Q2: Improved "Concatenated Notes" production
      57. 10Q3: Add “Date Created” to Note_Links (for audit purposes; I’m still not convinced that links disappear when they should).
      58. 10Q3: Investigate Note_Links: needs versioning, and section of calling Note
      59. 10Q3: Wrote routines to automatically generate the “Printable Notes” footer – and to create the corresponding web pages – from the control table. Very useful.
      60. 09Q4: Restricted the size of the “previous versions” tables for Notes to 12.
      61. 09Q3: Enabled automatic immediate promotion to Live Website of selected Temp Notes.
      62. 09Q3: Enabled Concatenated Notes (by Group). This is only allowed for “Identity” Notes at the moment, which is what I’d wanted it for. It allows all Notes for a Group to be printed off in Title sequence, for ease of review.
      63. 09Q3: Ensured that the Notes Reading-list is generated from all 3 sub-topics.
      64. 09Q3: Functionally enhanced the Notes-generation processing options; to make niftier production possible. In particular, where individual Notes are selected, I made the re-generation of the cross-references optional (so that if only textual variation is being iterated, the iterations are quicker).
      65. 09Q3: Improved the performance of “All Notes” regeneration. The problem was with the generation of the generic reading lists based on Note Title. These only apply to the “Identity” Notes, and it was taking a long time to determine there was no list to produce.
      66. 09Q2: Added a table of Notes history at the end of each Note (ie. not just previous version) - to save tedious chaining
      67. 09Q1: Added reading-list (of works cited) at the end of the Note, as well as in the Printed Note
      68. 09Q1: Created optional "don't print" feature for specific footnotes.
      69. 09Q1: Fixed the problem of the overpopulation of the Notes directory
      70. 09Q1: Option for Notes-Regen:
        1. Changed-only
        2. Changed, and all impacted by Change
        3. All
      71. 08Q1: Allowed regeneration of Temp Notes only
      72. 07Q4: Added referenced Books / Papers as footnote jump table to Notes
      73. 07Q4: Added referenced reading lists to printable version of Notes
      74. 07Q4: Improve "Temp" note processing. Suppress "_T" suffix (just don't copy to Transfer)
      75. 07Q4: When a Note is archived, ensured a printable note is also archived95 (if this Note has a print)
      76. 07Q3: Created "respondent notes" - database identifier & red indicator + text
      77. 07Q3: Created96 a "quick lookup" table of responses to a note and its sub-notes (recursive function on Notes_Links)
      78. 07Q3: Enabled generation of printable version of Notes
      79. 07Q3: Made Note-change-history available on-line (added status + publishable flags)
      80. 07Q2: Added "footnote" pages97 for Carthusians & King's
      81. 07Q2: Added "outgoing links" summary footer to Notes pages
      82. 07Q2: Allowed two sorts of Note - document-specific ("Write-up") and general
      83. 03/07/2020: Highlighted Archived Notes clearly as not the latest
    • Papers
      1. 20Q3: PaperstoNotes Link Pages: Re-engineered and corrected CreatePapersToNotesWebPages pages
      2. 20Q2: Determined why the monthly regeneration process for Paper Abstracts had ballooned from just over 6 hours in January 2020 to over 17 hours in April 2020. Reduced to just under 4 hours as a result of materialising and indexing the view (now table) Paper_Citings_List_New. Further improvements required.
      3. 20Q2: Determined why the monthly regeneration process for Paper Summaries had ballooned from just over 3.5 hours in January 2020 to around 11.5 hours in April 2020. Fixed by materialising and indexing the view (now table) Paper_Citings_List_New, so now takes 6 minutes!
      4. 20Q1: Make use of Comments for large Abstracts seamless by appending Comments (if > 5k) to Abstracts.
      5. 18Q2: Papers associated with broken links:-
        1. Created an area to hold pdfs, and automated links thereto.
        2. Created a pop-up to say I'd remove such pdfs if requested by their authors.
        3. Consequently, haven't password protected the area yet.
      6. 18Q1: Correction of narratives on Note time-recording pseudo-papers
    • Photos
      1. 19Q1: Created Photo-logging usage via Images table.
      2. 19Q1: Created Timeline software
      3. 19Q1: Timeline software: Added photos of Coxes Farm
      4. 18Q3: As revealed by Spider: Fixed ImageRef to work for both embedded ('SRC') & linked images on the same page - eg. Carthusians98. 12 items.
      5. 18Q3: Fix Bug: photos in Abstracts with embedded Notes - eg. "Grossman (Lev), Kurzweil (Ray) - 2045: The Year Man Becomes Immortal" - not displaying
      6. 13Q2: Reduced the size of photos (to make the pages quicker to load).
    • Process
      1. 20Q3: Determined why Recalculation (cmdRecalculate_Click) took so long: it varied depending on how many Notes were regenerated, but seemed to be 17.5 minutes even if there are none. I think it was down to a problem with nested Functors leading to Notes growing inordinately: in particular the development logs. Now takes around 5 minutes.
      2. 20Q2: Full Website Regeneration took 60 hours in April 2020. This has now been substantially fixed, as of end June 2020, in that regeneration now takes just under 11 hours. While further improvements are possible, the best use of time is to avoid the need for regular regeneration altogether by completing my Cross-referencing project.
      3. Corrective work was split out into the items requiring improvement, must of which have now been implemented.
      4. History and analysis:-
        1. This 'balooning' was on my new laptop which has solid state disks - it was taking 36 hours on my old laptop, then halved when I got the replacement, but then nearly doubled!
        2. I investigated why this was so, and improved performance mainly by putting in a trace (using a timer - GetTickCount - found on the web that allows logging of elapsed time in milliseconds, using StartTimer and EndTimer) and finding the 'pinch points'
        3. I suspected that it was a new release of MS Access that's caused a problem; maybe a lost index or something similar. I couldn't find any evidence of this, but a mumber of queries were taking a second or so to execute, which small times multiplied up drastically when the number of pages on my site - over 100k - are taken into account. Quite why a cliff-edge had been reached, I know not!
        4. Three areas had been particularly impacted: Authors, Paper Abstracts & Paper Summaries, which have all trippled or worse. These are covered as seperate developments under 'Authors' and 'Papers'.
        5. Book Summaries and Book-Paper Abstracts had also at least doubled, but this has had less of an impact as they took much less time in the first place.
        6. Other regeneration processes - in particular 'Notes' - didn't appear to have been impacted.
      5. 20Q2: Wrote Check_Database_Size to check that the size of the database isn't at risk of breaching the 2GB limit. Required because of the need to re-materialise various views that have been instituted to speed up the full website regeneration routines, and which need to be run each time cmdRecalculate_Click is run, which is often.
      6. 19Q4:
        1. Full Website Regeneration failed in Notes Regen because the first Private Note (13399) couldn't be copied from the Transfer directory as the private note hadn't been created there by then.
        2. It should have been, from looking at the code.
        3. Also, the private note itself (1163100) is not regenerated until later.
        4. In the start-December 2019 run, no such problem occurred. Maybe the problem was caused by copied between laptops. So, I've flagged it as complete for now, pending a recurrence.
      7. 19Q4: Fix Functors to enable previous quarter's Status Reports after Time period roll-over
      8. 18Q4: Added 'time to regenerate' sub-totals & grand-totals to Website Maintenance Dashboard101.
      9. 18Q2: Re-engineered Website Maintenance Dashboard102 to show extra topical links (and show them at the top of the page).
      10. 18Q1: Improved efficiency of daily changes regeneration (removed msgboxes)
      11. 18Q1: Wrote a routine to update a set of .htm files, replacing text in them all
      12. 15Q4: Created a consolidated processor to enable overnight regeneration of the whole website.
      13. 14Q1: Created a “Website Regeneration Dashboard” to show when the various elements of my website have been refreshed, and how long the routines took to run. Links to this appear in the footer to each webpage.
      14. 14Q1: Improved the regeneration routines for the various classes of webpage, and got them to record statistics in support of the “Website Regeneration Dashboard”.
      15. 09Q4: Set up my database system to roll forward to the next academic year, and generate web-pages appropriately.
      16. 09Q3: Created a “performance stats” database and routines to write to it – though I only implemented them for the Notes generator.
      17. 07Q2: Improved ease of FTP process by use of Transfer Directory
    • Search
      1. 17Q2: Wrote routines to create a cut-down Website excluding pages with irrelevant or duplicated information. Used this to create the ZoomSearch facility.
      2. 07Q4: Created search facility using ZoomSearch
    • Spider
      1. 20Q3: System Resources Exceeded - "Run Time Error 3035"; query Full_Link_Same_Directory_Updt in Spider_Copy. Fixed by using SQL rather than a query.
      2. 20Q3: The size of the main database was bloating to over 1.5Gb during the spider run, so was approaching the 2Gb limit.
        1. Used Check_Database_Size, with a parameter, to monitor the size of the main database and added a message similar to those reporting the compact / repair of the Slave database each time a status message is printed.
        2. Based on the diagnostics produced, it was found that the bloating occured at two stages:-
          → In Spider_Scurry: during the creation of Raw_Links_Temp_Temp.
          → In Spider_Copy: during the creation of Raw_Links_Temp.
        3. Fixes:-
          → Put check in Check_Database_Size to STOP if the database is over 1.5 Gb (parameterised via Max_Database_Size).
          → Moved Raw_Links_Temp_Temp to the Slave database.
      3. 18Q4: Determined why Full_Link_Up_Levels_Gen takes so long - approximately 4 hours. Re-architected to reduce database reads.
      4. 16Q4: Added “.shtml” and “.shtm” to the list of web-suffices.
      5. 15Q4: Finally cracked the Spider for external links, and used it to check the 3,000-odd links from my website. Some tidying up outstanding.
      6. 14Q4: Fixed the bug wherein the Spider ran out of system resources on large runs: now compacts & repairs the slave database as it does along (every 200k links; parameterised).
      7. 14Q3: Continued working on my Spider and started to use it to clean up my website by diagnosing broken links.
      8. 13Q2: De-bugged the Spider.
      9. 10Q2: Wrote my own Web Spider.
        1. It’s a little bit of a cheat, in that it doesn’t actually invoke the web-links, but crawls through my directory structure selecting all the .htm files, and rummages through the HTML looking for links which it then copies to a table.
        2. It also maintains a table of the directory structure and files therein, and then works out the real address from the relative addressing.
        3. It then allows cross-checking between the links and the files.
        4. Links external to the site have to be investigated manually (at least I have a list of all the links) – unless I can work out how to use the list to power a browser to check.
        5. There are about 333k links in 55k files – the full run took about 8 hours.
        6. It now runs incrementally based on file last-updated dates.
    • Status
      1. 20Q3: Added Weekly Project Plans to Priority Task List
      2. 20Q3: Created web-page (using Functor_21) showing oboe practice hours by work played
      3. 20Q3: Quarterly Project Reports: Corrected Functor_08. The Project Planned YTD % kept having to be bodged!
      4. 19Q2: Corrected Quarterly Reporting for uneven Plan (Holidays)
      5. 18Q4: Links occasionally missing from Summary Task List103 and Summary Task List (YTD)104.
      6. 18Q3: Status: Summary - Actual versus Plan105 between quarters says "previous quarter"; should be "current quarter"
      7. 18Q2: Added extra items to Functors to facilitate Quarterly Reporting:-
        1. Automatically calculated QTD & YTD time statistics in the quarterly reports.
        2. Provided a YTD task list for Status Reports for projects "in abeyance" to make reporting easier.
        3. Modified the statistics - to be just YTD - for YTD Status reports.
        4. Created table for - by Project - "Plans for the near future" tasks, and a functor to add them to the status report.
        5. Took "items competed" and "plans for the near future from the development log for the Web-tools status report.
        6. Created another functor to add these lists to the Summary report.
      8. 18Q1: Created web-page generator for Development items (ie. Created the Web-Project "To Do" List and "Outstanding Developments" directly from this list).
      9. 13Q3: Re-engineered Monthly Reporting on Change of Year
      10. 13Q2: Improved "This Month's Summary" Reporting
      11. 13Q2: Improved Quarterly Reporting
      12. 13Q1: Re-engineered Monthly Reporting Tables & improved Monthly Reporting
    • Technology
      1. 19Q1: Investigated the Hit Stats provided by Webalizer and produced web-pages to show the most active pages.
      2. 13Q3: Converted Tables to CSS3 standard
      3. 13Q3: Used CSS in all manually-crafted and automatically-generated pages
      4. 13Q1: Standardised to HTML5106
    • Timesheets
      1. 17Q4: Separated Time Recording for Aeon
      2. 15Q1: Segregated time-recording so I can easily see how much time is spent developing and supporting my own website, and how much is spent on those of other organisations.
      3. 13Q3: Re-engineered Time-Recording, so that actuals are held against Papers (and Books) in a separate table, rather than as columns by year in the Papers / Books tables themselves. Retained the grand totals only.
      4. 08Q4: Updated Book totals from totals of the Papers contained therein
    • WebRefs
      1. 20Q3: Improved WebRefs checker (Webrefs_Update) further to check for Error 403 "Forbidden". This involved finding way of checking pdfs where the returned page is in fact HTML or XML (see DevLog Ref 379).
      2. 20Q3: Reformated WebLinks_Tester.htm, WebLinks_Tester_Map.htm, WebLinks_Tester_Full.htm & WebLinks_Tester_Full_Map.htm: The 'As Above" lines waste space. Consolidate onto single second line.
      3. 20Q3: Reformated WebLinks_Tester_Brief, WebLinks_Tester.htm, WebLinks_Tester_Map.htm, WebLinks_Tester_Full.htm & WebLinks_Tester_Full_Map.htm: Allowed more space for 'link returned', 'issue' and 'display text' and added Explanation column
      4. 20Q2: Webrefs_Update: Improved the processing of this sub which controls IE to check the URLs corresponding to WeRefs are still valid.
        1. Improved the recovery processing after IE fails or becomes detached. Now seems to work perfectly!
        2. Improved processing for 404 (page not found) errors - detect them where URL returned differs from that requested (so the error is correctly categorised) and also where it is the same (so the error is detected) - by using the GetElementsByTagName method to look through Title and H1 elemnts (where they exist).
        3. Note that there is still an issue for pdfs where the above methods don't work; though the page returned is an HTML or XML page, the changed URL still claims to be a pdf, so I can't check in case it's a real pdf and the process fails.
      5. 20Q2: Amend WebRef 'Name' link to #Off-Page_Link_WxxxW style. Also amend Quarterly and Annual task lists to use this reference.
      6. 20Q2: Amended WebRef 'Name' links to #Off-Page_Link_WxxxW style so referencing from other pages is possible (useful for Aeon)
      7. 20Q2: As revealed by Spider: WebLinks_Tester_Full_Map.htm (etc). Refs failing. 184 items. res://ieframe.dll/ in Returned_URL.
      8. 20Q2: WebLinks_Tester: If the URLs of WebRefs are changed between Spider runs, the URL mapping fails, and the WebRefs appear at the end of the last page, wrongly categorised. Fixed to ensure they appear on their own page, added to the jump table as 'WebRef Missing' and added an explanation.
      9. 20Q2: WebLinks_Tester_Full & WebLinks_Tester_Full_Map: Reformated Jump Table to 30 columns and multiple rows.
      10. 20Q1: As revealed by Spider: WebLinks_Tester_4.htm. Refs failing. 32 items. res://ieframe.dll/ in Returned_URL.
      11. 20Q1: As revealed by Spider: WebLinks_Tester_Brief_12.htm (etc). Refs failing. 41 items. res://ieframe.dll/ in Returned_URL.
      12. 20Q1: As revealed by Spider: WebLinks_Tester_Map_3.htm (etc). Refs failing. 51 items outstanding. res://ieframe.dll/ in Returned_URL.
      13. 20Q1: WebRefs checker failing to detect broken links for Aeon - if a link fails, I get the message 'The page you were looking for could not be found (404)', but since the URL doesn't change, I don't spot it. This may happen for other sites as well. Fixed for Aeon - watch out for others.
      14. 19Q3: As revealed by Spider: Hits_Pages_Total_Crosstab_Descending.htm.
      15. 18Q4: Added Display text to WebRefs for Links to Stanford, Wikipedia & Aeon
      16. 18Q2: Created Functor to add stats to Website Generator Documentation - Web Links107 page.
      17. 18Q1: Determined & reported usage directly using embedded +WnnnW+ encoding rather than via the Spider. Reconciled the two methods.
      18. 18Q1: Documentation & Bug-fixes: Phase 1
        1. Completed "as is" functional documentation so I could see what's going on.
        2. Re-engineered so it made sense and ran efficiently without bugs
        3. Fixed the obvious data errors that confused the statistics - ie. where the returned URL differs from that requested and https issues.
        4. Wrote / improved diagnostic reports (web pages)
      19. 16Q1: Some tidying up of the 3,000-odd external links from my website.
      20. 13Q2: Some pages render oddly in Chrome (and maybe other browsers) – investigated108.
      21. 10Q3: Improved the functionality for automatically generating external Web-links in Notes, Book Abstracts and Paper Abstracts. Corrected current data to accommodate the new code.
      22. 07Q2: Made Web Refs in Comments / Abstracts into Hyperlinks
  2. Other Websites: Completed Items By Category:-
    • Acts
      1. 16Q3: Developed a website for displaying the textual and grammatical analyses and appendices of Pete’s PhD on the Acts of the Apostles. Two versions: Live (Acts: Live Site (Defunct)) and Test (Acts: Test Site).
    • Bridge
      1. 17Q1: Added new features (70s Club & Monthly Pairs Winners) to my Bridge mini-websites.
      2. 17Q1: Created Master Point Lists & Promotions for the ECBA website.
      3. 16Q3: Revived the archive website for Mountnessing Bridge Club (Mountnessing Bridge Club Archive).
      4. 16Q1: Accommodated Alaric’s new scoring system for St. Edith’s.
      5. 16Q1: Created Ladies’ and Gentlemen’s ladders for my mini-websites.
      6. 16Q1: Reconciled my X-IMP calculations with a new feature provided by Bridgewebs.
      7. 15Q4: Segregated the Bridge website generators from those for my own website. They now live in separate databases and have their own documenters.
      8. 15Q3: Created a new website (using Bridgewebs) for the First Class Bridge Academy (First Class Bridge).
      9. 15Q3: Ported my “handicap / improvers” routines from Hutton to “Mini Websites”.
      10. 15Q2: Development of the “Eve Goblets” (victor ludorum) software for the ECBA, and roll-over to the new season.
      11. 15Q1: Further improvements to my Butler-IMPs and Cross-IMPs calculation routines.
      12. 15Q1: Started a project to provide small websites for bridge clubs without sites: Essex Bridge Results.
      13. 15Q1: Took on the ECBA Website (Essex Contract Bridge Association), at the cost of considerable effort by way of reformatting & tidy-up.
      14. 14Q4: Improvements to my Butler-IMPs and Cross-IMPs calculation routines – different formats, including the Jeff Smith format, and extension to Butler-scored Pairs events. This work has been taken up by the ECBA selectors for county-team selection.
      15. 14Q3: Created a suite of programs to extract data and calculate Butler-IMPs and Cross-IMPs from the travellers on Bridgewebs Teams events, and output ranking-lists to the Web.
      16. 13Q2: Merged109 Philosophy & Bridge Databases
    • Enigma
      1. 18Q4: Added 'EE' shortcut icon on all Webpage browser tabs
      2. 15Q3: Created a small website for a music group Julie and I attend – the Enigma Ensemble (Defunct).
    • Sophie
      1. 19Q3: Dismantle my old version of Sophie Botros.
      2. 18Q4: Added 'SB' shortcut icon on all Webpage browser tabs
      3. 15Q3: Completed the upgrade of Sophie Botros110’s website; see Sophie Botros.


In-Page Footnotes

Footnote 6: Footnotes 26, 66: Footnotes 27, 67: Footnotes 36, 86: Footnotes 43, 93: Footnotes 44, 65: Footnotes 45, 110: Footnotes 47, 61: Footnotes 48, 108: Footnotes 49, 109: Footnotes 50, 106: Footnotes 51, 95: Footnotes 52, 96: Footnotes 53, 62: Footnotes 54, 63: Footnotes 55, 56, 58, 68, 69, 97: Footnotes 57, 71:

Live Version of this Archived Note

Date Length Title
01/11/2023 05:08:10 123556 Website - Progress to Date (2023 - October)

Table of 12 Earlier Versions of this Note (of 16)

Date Length Title
03/07/2020 22:09:07 86201 Website - Progress to Date (2020 - June)
04/04/2020 00:14:24 72717 Website - Progress to Date (2020 - March)
19/01/2020 23:41:17 69093 Website - Progress to Date (2019 - December)
10/10/2019 23:58:34 66813 Website - Progress to Date (2019 - October)
14/07/2019 20:29:46 65148 Website - Progress to Date (2019 - June)
05/04/2019 10:36:29 70085 Website - Progress to Date (2019 - March)
06/01/2019 23:36:58 68922 Website - Progress to Date (2018 - December)
10/10/2018 16:43:41 65062 Website - Progress to Date (2018 - September)
06/07/2018 18:56:10 59064 Website - Progress to Date (2018 - June)
05/04/2018 10:48:00 51445 Website - Progress to Date (2018 - March)
18/12/2010 19:58:05 8619 Status: Web-Tools (Summary of Progress to Date)
03/05/2009 13:07:41 8408 Status: Web-Tools (Summary of Progress to Date)

Table of the 11 Later Versions of this Note

Date Length Title
04/10/2023 21:59:02 124304 Website - Progress to Date (2023 - September)
06/07/2023 00:43:12 123503 Website - Progress to Date (2023 - June)
30/04/2023 01:06:14 122764 Website - Progress to Date (2023 - March)
10/01/2023 23:35:31 119468 Website - Progress to Date (2022 - December)
10/10/2022 22:32:10 116821 Website - Progress to Date (2022 - September)
03/07/2022 01:29:17 116479 Website - Progress to Date (2022 - June)
03/01/2022 23:58:34 113397 Website - Progress to Date (2021 - December)
02/07/2021 20:32:38 111171 Website - Progress to Date (2021 - June)
03/04/2021 19:58:05 109399 Website - Progress to Date (2021 - March)
04/01/2021 19:40:48 101369 Website - Progress to Date (2020 - December)
02/01/2021 14:29:46 100708 Website - Progress to Date (2020 - December)


This version updated Reference for this Topic Parent Topic
04/10/2020 00:27:22 822 (Website - Progress to Date (2020 - September)) Status: Web-Tools (2023 - September)


Summary of Notes Links from this Page

Aeon Papers (4) Carthusians - Main Text (2) L. Messel & Co. (2) Private Note (2) Status: Actual Detail Summary (2007 - 2021) (2)
Status: Bridge (2020 - September) (2) Status: Chess (2020 - September) (2) Status: Consciousness Studies (2020 - September) (2) Status: Languages (2020 - September) (2) Status: Personal Identity (2020 - September) (2)
Status: Philosophy of Religion (2020 - September) (2) Status: Priority Task List (2020 - September) (2) Status: Summary - Actual versus Plan (2020 - September) (2) Status: Summary (2020 - September) (4) Status: Summary Task List (2020: July - September) (2)
Status: Summary Task List (YTD: 19Q4 - 20Q3) (2) Status: Thesis Dashboard (2018: March) (2) Status: Web-Tools (2020 - September) (5) Theo Todman's Blog (2) Theo Todman's Chess Page (2)
Website - Outstanding Developments (2020 - September) (7) Website - Progress to Date (2020 - September) (6) Website Generator Documentation - Control Page (3) Website Generator Documentation - Cross-Referencing (2) Website Generator Documentation - Web Links (2)
Website: Maintenance Dashboard (01 October 2020) (6)        

To access information, click on one of the links in the table above (if any).



Summary of Note Links to this Page

Status: Web-Tools (2020 - September) Website - Outstanding Developments (2020 - September) Website - Progress to Date (2020 - September), 2, 3, 4, 5, 6    

To access information, click on one of the links in the table above (if any).



Text Colour Conventions

  1. Blue: Text by me; © Theo Todman, 2023



© Theo Todman, June 2007 - Nov 2023.Please address any comments on this page to theo@theotodman.com.File output:
Website Maintenance Dashboard
Return to Top of this PageReturn to Theo Todman's Philosophy PageReturn to Theo Todman's Home Page