Theo Todman's Web Page - Notes Pages

Status Reports

Status: Web-Tools (2020 - September)

(Text as at 04/10/2020 00:27:22)

(For earlier versions of this Note, see the table at the end)

Rationale for this Project

Summary of Progress during July - September 2020
  1. I spent 143.75 hours in 20Q3 on this Project, or related work (476.25 hours YTD, where for "YTD" - Year to Date - I mean the (academic) year that commenced in October 2019). That's 110.3% of the planned effort (114.2% YTD). Overall, 20.6% of my Project effort in the Quarter was directed towards this project (making 17.1% YTD) - as against 18.5% planned (14.1% YTD).
  2. In 20Q3 I made quite a lot of further progress over a wide area of highly relevant items, as listed below, with a particular emphasis on Auto_Reference_Notes.
  3. Consequently, I overspent my time budget by 10%. Time well spent, though.
  4. Completed items included:-
    1. Own Website:
      • Architecture
        1. Created Cross_Reference_Changes_Prune to keep Cross_Reference_Changes under control pending completion of the Xref project.
      • Books
        1. BookCitings references not sequenced correctly. Corrected.
        2. BooksToNotes Link Pages: Re-engineered - used CreatePapersToNotesWebPages to generate pages
      • Books/Papers
        1. As revealed by Spider: BooksToNotes and PapersToNotes links failing (45 items). Pages not being regenerated due to wrong joins in Book_Note_Link_List and Paper_Note_Link_List.
        2. CreateBookPaperAbstractsWebPages not translating +RR+ References for Notes Write-ups of associated Papers
      • Notes
        1. Added option in Auto_Reference_Notes to only confirm new items (leaving previously-flagged items untouched)
        2. Added Aeon Abstract (ie. Footnote) link to Aeon Webref items in Summary task List reports for 'read' items
        3. Determined why the run of Auto_Reference_Notes_Regen took 2.5 hours on 02/09/2020 (and climbing). It was down to the Aeon Note13 being searched; in particular Check_If_In_Container, checking for items in Footnotes and / or Quoted Text. As this Note is regenerated, there's no point auto-linking it. Time reduced to 23 minutes once this Note - and other Notes of inappropriate Note Groups - was removed.
        4. Corrected Issues with Nested Functors: Added Functor_ID to End-tag
        5. Added option in Auto_Reference_Notes to - for Live (ie. not TEMP) Notes - update the latest Archived Note to equal the updated Live Note, hence avoiding superfluous archives.
        6. Upgraded Auto_Reference_Notes to avoid key-words within +AA+-style Authors names (using Check_If_In_Container).
        7. Upgraded Auto_Reference_Notes to allow a parameterised 'termination or continue' for very long update runs
      • Papers
        1. PaperstoNotes Link Pages: Re-engineered and corrected CreatePapersToNotesWebPages pages
      • Process
        1. Determined why Recalculation (cmdRecalculate_Click) took so long: it varied depending on how many Notes were regenerated, but seemed to be 17.5 minutes even if there are none. I think it was down to a problem with nested Functors leading to Notes growing inordinately: in particular the development logs. Now takes around 5 minutes.
      • Spider
        1. The size of the main database was bloating to over 1.5Gb during the spider run, so was approaching the 2Gb limit.
          1. Used Check_Database_Size, with a parameter, to monitor the size of the main database and added a message similar to those reporting the compact / repair of the Slave database each time a status message is printed.
          2. Based on the diagnostics produced, it was found that the bloating occured at two stages:-
            → In Spider_Scurry: during the creation of Raw_Links_Temp_Temp.
            → In Spider_Copy: during the creation of Raw_Links_Temp.
          3. Fixes:-
            → Put check in Check_Database_Size to STOP if the database is over 1.5 Gb (parameterised via Max_Database_Size).
            → Moved Raw_Links_Temp_Temp to the Slave database.
        2. System Resources Exceeded - "Run Time Error 3035"; query Full_Link_Same_Directory_Updt in Spider_Copy. Fixed by using SQL rather than a query.
      • Status
        1. Quarterly Project Reports: Corrected Functor_08. The Project Planned YTD % kept having to be bodged!
        2. Added Weekly Project Plans to Priority Task List
        3. Created web-page (using Functor_21) showing oboe practice hours by work played
      • WebRefs
        1. Reformated WebLinks_Tester.htm, WebLinks_Tester_Map.htm, WebLinks_Tester_Full.htm & WebLinks_Tester_Full_Map.htm: The 'As Above" lines waste space. Consolidate onto single second line.
        2. Reformated WebLinks_Tester_Brief, WebLinks_Tester.htm, WebLinks_Tester_Map.htm, WebLinks_Tester_Full.htm & WebLinks_Tester_Full_Map.htm: Allowed more space for 'link returned', 'issue' and 'display text' and added Explanation column
        3. Improved WebRefs checker (Webrefs_Update) further to check for Error 403 "Forbidden". This involved finding way of checking pdfs where the returned page is in fact HTML or XML (see DevLog Ref 379).
    2. Other Websites:
      • Nothing to Report.
  5. Full details for 20Q3 are given below14:-
Website (Total Hours = 140.75)
  1. Website - Development (Total Hours = 108.75)
    • Todman (Theo) - Tottering Towers & Listing Buildings: Incorporate RICS involvement (1.25 hours)
      → See "Todman (Theo) - Tottering Towers & Listing Buildings" (1.25 hours)
    • Website - Generator - Page for Oboe practice15 details (2.75 hours)
    • Website - Generator - CreateBookPaperAbstractsWebPages not translating +RR+ References for Notes Write-ups of associated Papers (0.25 hours)
    • Website - Generator - Regen_Note_Links failing for Note 117016 (1.25 hours)
    • Website - Generator - Webrefs_Update failing because IE loops with Aeon / Psyche pages (3.25 hours)
    • Website - Generator - Add Aeon Abstract link to Aeon Webref items in Summary task List reports (1 hour)
    • Website - Generator - Add Weekly Project Plans to Priority Task List (0.75 hours)
    • Website - Generator - As revealed by Spider: BooksToNotes and PapersToNotes links failing (1 hour)
    • Website - Generator - Auto-Reference Notes - Review & Update Documentation & Processing (28.5 hours)
    • Website - Generator - BookCitings references not sequenced correctly (1 hour)
    • Website - Generator - Correct Functor_08. The Project Planned YTD % keeps having to be bodged! (1 hour)
    • Website - Generator - Correct Paper_Note_Link_List_New for CreatePapersToNotesWebPages pages (12.5 hours)
    • Website - Generator - Correct WebRef page links from WebLinks_Test pages (add 'Off-Page_Link_') (0.5 hours)
    • Website - Generator - Document Auto-Referencing Notes (7.5 hours)
    • Website - Generator - Document Cross-Referencing (4 hours)
    • Website - Generator - Document Cross-Referencing: Create Cross_Reference_Changes_Prune (1 hour)
    • Website - Generator - Document Spider (0.25 hours)
    • Website - Generator - Highlight Archived Notes as not the latest (0.5 hours)
    • Website - Generator - Improve WebLinks Tester suite of pages (17.5 hours)
    • Website - Generator - Investigate & Fix WebRefs_Update checker for 404 check not working (6 hours)
    • Website - Generator - Issues with Nested Functors (3.5 hours)
    • Website - Generator - Revise Cross-Referencing Documentation (1.25 hours)
    • Website - Generator - Site Map: Document17 and update (0.25 hours)
    • Website - Generator - Spider - Monitor performance & Main database size (4.75 hours)
    • Website - Generator - Spider - System Resources Exceeded - "Run Time Error 3035"; query "Full_Link_Same_Directory_Updt" (1.5 hours)
    • Website - Generator - Update Development Log in the light of recent activities (3.25 hours)
    • Website - Generator - Wrote ZapFiles - to clear out WebLinks Tester, Documentation and other page-sets prior to regeneration (2.5 hours)
      → See "Software Development - Website - Development" (107.5 hours)
  2. Website - Education (Total Hours = 3)
  3. Website - Infrastructure (Total Hours = 9.25)
    • Buy & Commission replacement iPhone protective case (0.25 hours)
    • Buy & Commission replacement Mouse (0.75 hours)
    • EE Broadband - Router & Line Issues (4 hours)
    • Microsoft Windows 10 / MS Office - Releases, Bugs & Periodic Re-boots (2.25 hours)
    • PC Backups / OneDrive (0.75 hours)
    • Printer - Problems with USB Drivers (0.25 hours)
    • Try out old TV as Monitor (1 hour)
      → See "Admin - Website - Admin & Maintenance" (9.25 hours)
  4. Website - Maintenance (Total Hours = 19.75)
    • 20Q2 Status Reports (2.75 hours)
    • Website - Generator - WebRefs - Manual / Automatic URL Checks & Fixes (11 hours)
    • Website - Periodic Full Regeneration (3.5 hours)
    • Website - Run Web Spider (2.5 hours)
      → See "Admin - Website - Admin & Maintenance" (19.75 hours)
Website Others (Total Hours = 3)
  1. Website Others - Hutton DBC Maintenance
  2. Website Others - Mountnessing DBC Maintenance

Plans for the Near Future

The Plan below is taken automatically from the Priority 1 items on my Development Log, as published in my Outstanding Developments18 Report. I’ve maintained the weekly allocation at 10 hours. This is to allow further work on my Cross-Referencing project.
  1. Own Website: Priority 1 Items By Category:-
    • Architecture
      1. Compact and Repair Problems
        1. On compacting and repairing my main database I sometimes get the error "The query cannot be completed. Either the size of the query result is larger than the maximum size of a database (2 GB), or there is not enough temporary storage space on the disk to store the query result".
        2. It happens 3 times while the database is re-opening.
        3. There is lots of space, and the database is only 600Mb (and the error started when it was under 500Mb).
        4. This mostly happens after I've run long processes, so I usually close the database, re-open it and then try the compact and repair. Usually this works, but not always. But I then try again and the message disappears.
        5. I strongly suspect that this is MS Access itself re-indexing tables, and blowing up a temporary database, but I can’t find any evidence for this on-line. Or help, other than suggestions to split databases and do other sensible things. Given that the error occurs when the database is re-opening, with no temporary file visible, is very strange.
        6. 17/04/20 - set MaxLocksPerFile to 1,000,000 (from the default 9,500). Sadly, it doesn't seem to have made a difference.
      2. Complete XRef-re-engineering project:-
        1. Ensure all links and link-pages use the new XRef table, and pension off the old tables.
        2. Look into writing out specific object-identifiers, and linking thereto for Citations, rather than paragraph references. An issue is multiple instances of the same object in a document.
        3. Check all link-types still work and fix any errors.
        4. Complete the auto-triggering of regeneration of “associated” link pages.
        5. Fix update bug in Convert_Webrefs.
        6. Fix Bug whereby PaperSummary pages seem to have “Works-” and “Books/Papers-” Citings that refer to the same link-pages.
        7. Document the process!
      3. Document19, Repair & update my Website site-map
      4. Review effectiveness of hyperlinking method in the light of PhD and Philosophy of Religion experience.
      5. Where possible, use ID rather than NAME for in-page hyperlinks
    • Backups
      1. Investigate Record-count discrepancies:-
        1. How do website files work as far as counts are concerned?
        2. Why aren't they recorded in Backup_History, nor the fact that the website was backed up?
        3. Different counts depending on whether new or old laptop is backed up. Investigate 63k discrepancy - lower on new laptop.
      2. Review architecture to improve performance; Need to document first
    • Books
      1. Further improve the time to regenerate Book Summaries. Now takes about 27 minutes, but should be under 5 minutes!
    • Books/Papers
      1. Change CreateBookPaperAbstractsWebPages so that - while a full re-gen uses the new method - re-gen for a particular book uses the old method (without the materialised view). This is so cmdRecalculate_Click doesn't need to be run beforehand.
      2. Investigate whether multiple Subject/Topic/Subtopic usage leads anywhere (ie. are just the first (of 3) actually used). Fix anything amiss.
      3. Reformat the BookCitings and PaperCitings pages:-
        1. Detail PaperCitings Pages: Include only useful information on the detail pages; but if there are multiple links from the same object, include them on the same line as 'extra links' as in BookCitings (copy the code: or, better, combine the two subs).
        2. Summary (Author Letter) pages: Include counts (as in Authors' Citations).
        3. Ensure uses the Cross_Reference table.
        4. Document!
    • Bridge
      1. Develop auto-reconciliation routines vs EBU results download
    • Documenter
      1. Investigate the error reports from the Documenter, especially unused variables & queries.
      2. Provide Functional Documentation for Website Generator (using Notes)
    • Education
      1. "Sitepoint (Learnable) - Sitepoint Learnable Web Development Courses": Membership cancelled, but plan what to do with the eBooks in my possession.
      2. Read "PC Pro - Computing in the Real World".
      3. Read "White (Ron) & Downs (Timothy Edward) - How Computers Work: The Evolution of Technology".
    • Infrastructure
      1. iCloud for Windows: Re-install & solve 'The upload folder for iCloud Photos is missing' problem. Try on new Laptop.
    • Notes
      1. Add "Note Alternates" to Note pages.
      2. Add option in Auto_Reference_Notes to allow an updating run restricted to 'Read' Books / Papers only (useful for very long lists)
      3. Add option in Auto_Reference_Notes to automatically ignore words containing certain strings that include the key-word (eg. ignore 'grace' and 'trace' when indexing 'race')
      4. Allow the option to concatenate Notes in the Printed version (ie. linearly embed them essay-style), rather than treating the hyperlinks as footnotes – but still keep the hyperlink & cross-referencing in place.
        1. For use as "disclaimers" - eg. for "Plug Notes".
        2. For Thesis / essays: the difficulty here is the need for linking passages to make the text run smoothly.
      5. As part of the Cross-Referencing project, check out the consistent treatment of Note 87520, which should be universally ignored. Recently, links to it appeared on Book-Summaries, Book_Paper_Abstracts and Note_Book_Links, as a Note referencing a Book. The critical item was a row on the Note_Book_Links table.
      6. Determine why very long printable notes (eg. Level 3+ for Note 17021) are being truncated. Probably suppress them in any case, as they take far too long to load.
      7. Enhance Functors to work for selected non-temp Notes so that up-to-date stats can be incorporated. This is complex as I want to avoid production of an Archived Note each time a non-Temp Note is regenerated. I also want to ensure that Notes whose variable text consists entirely of Functors get archived correctly (which they might not if I removed Functor-generated text from Notes before saving them to the Notes table).
      8. Fix bugs in multi-level footnoting in Printable Notes – the referencing is going wrong.
      9. Investigate Note_Links: Section references seem to be incorrect
      10. Printable Notes: fix the bug whereby the “private” flag is round the wrong way.
      11. Split Aeon Page22 into multiple sub-pages (either by topic or by priority)
      12. Suppress the publication of the Printable versions of Temp Notes
      13. Upgrade Auto_Reference_Notes to reference Sub-Notes: Currently only affects one note - Somerset Maugham - so not yet urgent
      14. Upgrade Auto_Reference_Notes to save Notes_To_Regen to a new table prior to the run, add new rows to this table, and copy it back after the run. Finally, allow the option of regenerating these Notes. In the interim, use Notes_To_Regen to create Note 87423, then clear it before the copy-back.
      15. Upgraded Auto_Reference_Notes to log its actions to a new table (Auto_Reference_Notes_Actions) so that any errors can be investigated and improvements made.
    • Papers
      1. The monthly regeneration process for Paper Abstracts was still takeing just over 5 hours. Problem is with Cross_Reference_Deletions and Cross_Reference_Additions. I thought it could not be fixed until the cross-referencing project is fully complete and documented. However, it spontaneously improved to 1.6 hours in the August 2020 run. Monitor!
    • Photos
      1. Develop software & procedure to make adding more content to the photos pages easier to undertake.
      2. Timeline software: Add photos for Holidays & Family History
    • Process
      1. Determine why Recalculation & Changed Book/Papers produce unneeded regeneration.
    • Spider
      1. Analyse the results of the data collection exercise and design a plan of campaign to fix broken Internal links and prevent recurrence.
        1. Correct the code so the problems discovered by the Spider don’t recur.
        2. Delete 'orphan pages' that are never linked to, ie. Use the Spider to prune redundant pages24 automatically where possible.
        3. Fix the historical data where errors are uncovered by the Spider. An easier task now the site has a full-regen function.
    • Technology
      1. Look into Sistrix Smart25. Errors and warnings itemised are:-
        1. Duplicate content: seems to be variants on
        2. Title Tags: Empty, too long, identical
        3. Page Not Found
        4. Filesize in excess of 1Mb
        5. Meta-Description: Empty
        6. Few words on Page
        7. H1: Not used, used multiple times per page, identical across pages
        8. Pictures: Alt attribute missing
    • WebRefs
      1. Webrefs_Update failing because IE loops with Aeon / Psyche pages. Currently doing manual checking - try to find an automated solution.
      2. Documentation & Bug-fixes: Phase 2
        1. Re-document the procedures in the light of recent changes.
        2. Resolve issues generated / revealed by the spider.
        3. Investigate - and fix where possible - broken links.
      3. Find a way of recording Missing Webrefs other than debug.print: create table, then suppress message for known problems
      4. Investigate items flagged as defunct. Populate Defunct_Explanation in WebRefs_Table. Consider use of FairUse (Link (Fair Use)) for documents no longer available that I'd downloaded.
      5. Investigate WebRefs with Issue = 'URL Translated OK': does the translation really work? How?
  2. Other Websites: Priority 1 Items By Category:-

Summary of Progress to Date

This is hived off to various separate documents, which have now been harmonising and / or consolidated:-
  1. Summary of Progress to Date26.
  2. Outstanding Developments27,
  3. Functional Documentation28,
  4. A summary of time expended across the years developing my website29 is at "Software Development - Website - Development".

In-Page Footnotes:

Footnote 4: Footnote 5: Footnote 7: Footnote 9: Footnote 10: Footnote 11: Footnote 12: Footnote 14: Footnote 24: Footnote 25: Footnote 29:

Printable Version:

Table of the Previous 12 Versions of this Note: (of 78)

Date Length Title
03/07/2020 22:09:07 28426 Status: Web-Tools (2020 - June)
04/04/2020 00:14:24 19563 Status: Web-Tools (2020 - March)
19/01/2020 23:41:17 19103 Status: Web-Tools (2019 - December)
10/10/2019 23:58:34 18052 Status: Web-Tools (2019 - September)
14/07/2019 20:29:46 16642 Status: Web-Tools (2019 - June)
05/04/2019 10:36:29 16128 Status: Web-Tools (2019 - March)
06/01/2019 23:36:58 18445 Status: Web-Tools (2018 - December)
10/10/2018 16:43:41 22079 Status: Web-Tools (2018 - September)
06/07/2018 18:56:10 16773 Status: Web-Tools (2018 - June)
05/04/2018 10:48:00 16588 Status: Web-Tools (2018 - March)
05/01/2018 00:11:31 13295 Status: Web-Tools (2017 - December)
09/10/2017 23:25:26 11848 Status: Web-Tools (2017 - September)

Note last updated Reference for this Topic Parent Topic
04/10/2020 00:27:22 520 (Status: Web-Tools (2020 - September)) Status: Summary (2020 - September)

Summary of Note Links from this Page

Aeon Papers CT Introduction Internet Technology and Philosophy Problem of the Many Research - Internet Technology
Research - Proposal Status: Oboe Practice (2020: October) Test Note - Auto-XRef Website - Outstanding Developments (2020 - October) Website - Progress to Date (2020 - October)
Website Generator Documentation - Control Page Website Generator Documentation - Site Map Websites Maintained by Theo Todman    

To access information, click on one of the links in the table above.

Summary of Note Links to this Page

Internet Technology and Philosophy King's Maths Questions Simon - T1S1T1 Status: Consciousness Studies (2020 - September) Status: Priority Task List (2020 - October)
Status: Summary (2020 - September), 2 Status: Summary Task List (2020: October) Status: Summary Task List (YTD: 20Q4) Theo Todman's Philosophy Page Theo Todman's Website Maintainance History
Website - Progress to Date (2020 - October), 2, 3, 4, 5 Website Generator Documentation - Control Page Website Generator Documentation - Functors, 2, 3, 4, 5    

To access information, click on one of the links in the table above.

Authors, Books & Papers Citing this Note

Author Title Medium Extra Links Read?
Software Development Website - Development Paper Low Quality Abstract    

Text Colour Conventions

  1. Black: Printable Text by me; © Theo Todman, 2020
  2. Blue: Text by me; © Theo Todman, 2020

© Theo Todman, June 2007 - Oct 2020.Please address any comments on this page to output:
Website Maintenance Dashboard
Return to Top of this PageReturn to Theo Todman's Philosophy PageReturn to Theo Todman's Home Page