Theo Todman's Web Page - Notes Pages

Status Reports

Status: Web-Tools (2021 - December)

(Text as at 03/01/2022 23:58:34)

(For earlier versions of this Note, see the table at the end)

Rationale for this Project

Summary of Progress during October - December 2021
  1. I spent 66.25 hours in 21Q4 on this Project, or related work (66.25 hours YTD, where for "YTD" - Year to Date - I mean the (academic) year that commenced in October 2021). That's 73% of the planned effort (73% YTD). Overall, 9.6% of my Project effort in the Quarter was directed towards this project (making 9.6% YTD) - as against 10.9% planned (10.9% YTD).
  2. I was under budget, mainly on account of illness in December. Also, I only undertook bits and pieces, so nothing features in the “items completed” lists.
  3. Quite a lot of time was spent on checking out and fixing broken WebRefs. Also, a couple of days were spent on the Mountnessing and Mayflower DBC websites.
  4. Completed items included:-
    1. Own Website:
      • Nothing to Report.
    2. Other Websites:
      • Nothing to Report.
  5. Full details for 21Q4 are given below13:-
    Website (Total Hours = 52)
    1. Website - Bridge Maintenance
    2. Website - Development (Total Hours = 7.75)
      • Website - Generator - Bible Reading Status Reporting (0.5 hours)
      • Website - Generator - Compact / Repair 2Gb problem (0.75 hours)
      • Website - Generator - Enable easy access to PDFs off-line, but suppress on-line: Improvements re missing data (0.5 hours)
      • Website - Generator - Enhance comparative Ling database (0.5 hours)
      • Website - Generator - Fix Year-End roll-over issues (0.5 hours)
      • Website - Generator - Improve documentation on PID Note, Book & Paper Usage14 (0.5 hours)
      • Website - Generator - Review & Enhance Xref Methodology (0.25 hours)
      • Website - Generator - Set up comparative Ling database: Data Entry (2.5 hours)
      • Website - Generator - Thesis Reading Pages Functor: Improvements re missing data (0.5 hours)
      • Website - Generator - WebRefs - 'Translated OK' (1.25 hours)
        → See "Software Development - Website - Development" (7.75 hours)
    3. Website - Education
    4. Website - Infrastructure (Total Hours = 15.25)
    5. Website - Maintenance (Total Hours = 25.75)
      • 21Q3 Status Reports (2.25 hours)
      • Website - Generator - WebRefs - Manual / Automatic URL Checks & Fixes (18.25 hours)
      • Website - Periodic Full Regeneration (3.75 hours)
      • Website - Run Web Spider (1.5 hours)
    Website Others (Total Hours = 14.25)
    1. Website Others - Enigma Ensemble
    2. Website Others - Mayflower DBC Maintenance
    3. Website Others - Mountnessing DBC Maintenance
Plans for the Near Future

The Plan below is taken automatically from the Priority 1 items on my Development Log, as published in my Outstanding Developments15 Report. I’ve had to maintain the reduced weekly allocation at 7 hours. This is to allow for increased effort on other projects.
  1. Own Website: Priority 1 Items By Category:-
    • Architecture
      1. Compact and Repair Problems
        1. On compacting and repairing my main database I sometimes get the error "The query cannot be completed. Either the size of the query result is larger than the maximum size of a database (2 GB), or there is not enough temporary storage space on the disk to store the query result".
        2. It happens 3 times while the database is re-opening.
        3. There is lots of space, and the database is only 650Mb (and the error started when it was under 500Mb).
        4. This mostly happens after I've run long processes, so I usually close the database, re-open it and then try the compact and repair. Usually this works, but not always. But I then try again and the message disappears.
        5. I strongly suspect that this is MS Access itself re-indexing tables, and blowing up a temporary database, but I can’t find any evidence for this on-line. Or help, other than suggestions to split databases and do other sensible things. Given that the error occurs when the database is re-opening, with no temporary file visible, this is all very strange.
        6. 17/04/20 - set MaxLocksPerFile to 1,000,000 (from the default 9,500). Sadly, it doesn't seem to have made a difference.
      2. Complete XRef-re-engineering project:-
        1. Ensure all links and link-pages use the new XRef table, and pension off the old tables.
        2. Look into writing out specific object-identifiers, and linking thereto for Citations, rather than paragraph references. An issue is multiple instances of the same object in a document.
        3. Check all link-types still work and fix any errors.
        4. Complete the auto-triggering of regeneration of “associated” link pages.
        5. Fix update bug in Convert_Webrefs.
        6. Fix Bug whereby PaperSummary pages seem to have “Works-” and “Books/Papers-” Citings that refer to the same link-pages.
        7. Document16 the process!
      3. Document17, repair & update my Website site-map
      4. Review effectiveness of hyperlinking method in the light of PhD and Philosophy of Religion experience.
      5. Where possible, use ID rather than NAME for in-page hyperlinks
    • Backups
      1. Investigate Record-count discrepancies:-
        1. How do website files work as far as counts are concerned?
        2. Why aren't they recorded in Backup_History, nor the fact that the website was backed up?
        3. Different counts depending on whether new or old laptop is backed up. Investigate 63k discrepancy - lower on new laptop.
      2. Review architecture to improve performance; Need to document18 first
    • Books
      1. Further improve the time to regenerate Book Summaries. Took about 21 minutes on 18/12/2021, but should be under 5 minutes!
    • Books/Papers
      1. Investigate whether multiple Subject/Topic/Subtopic usage leads anywhere (ie. are just the first (of 3) actually used). Fix anything amiss.
      2. Reformat the BookCitings and PaperCitings pages:-
        1. Detail PaperCitings Pages: Include only useful information on the detail pages; but if there are multiple links from the same object, include them on the same line as 'extra links' as in BookCitings (copy the code: or, better, combine the two subs).
        2. Summary (Author Letter) pages: Include counts (as in Authors' Citations).
        3. Ensure uses the Cross_Reference table.
        4. Document19!
    • Documenter
      1. Investigate the error reports from the Documenter, especially unused variables & queries.
      2. Provide Functional Documentation for Website Generator (using Notes)
    • Education
      1. Read "PC Pro - Computing in the Real World".
      2. Read "White (Ron) & Downs (Timothy Edward) - How Computers Work: The Evolution of Technology".
    • Infrastructure
      1. iCloud for Windows: Re-install & solve 'The upload folder for iCloud Photos is missing' problem. Try on new Laptop.
    • Notes
      1. Add "Note Alternates" to Note pages.
      2. Allow the option to concatenate Notes in the Printed version (ie. linearly embed them essay-style), rather than treating the hyperlinks as footnotes – but still keep the hyperlink & cross-referencing in place.
        1. For use as "disclaimers" - eg. for "Plug Notes".
        2. For Thesis / essays: the difficulty here is the need for linking passages to make the text run smoothly.
      3. As part of the Cross-Referencing project, check out the consistent treatment of Note 87520, which should be universally ignored. Recently, links to it appeared on web pages Book_Summary, BookPaperAbstracts and BooksToNotes, as a Note referencing a Book. The critical item was a row on the Note_Book_Links table.
      4. Develop a method to create Papers for rows in Aeon_Files that have an associated Identity_Note_ID but no Paper_ID.
      5. Investigate Note_Links: Section references seem to be incorrect
      6. Printable Notes: fix the bug whereby the “private” flag is round the wrong way.
      7. Split Aeon Page21 into multiple sub-pages (either by topic or by priority)
    • Papers
      1. Split PDFs Available on Local Website22 page into alphabetical sub-pages
      2. The monthly regeneration process for Paper Abstracts is still taking too long. Monitor! Last run: 1.57 hours on 18/12/2021.
    • Photos
      1. Timeline software23: Add photos for Bakers' Coxes Farm Development Phase III
      2. Timeline software24: Add photos for Holidays & Family History
      3. Develop software & procedure to make adding more content to the photos pages easier to undertake.
    • Process
      1. Determine why Recalculation & Changed Book/Papers produce unneeded regeneration.
    • Spider
      1. Analyse the results of the data collection exercise and design a plan of campaign to fix broken Internal links and prevent recurrence.
        1. Correct the code so the problems discovered by the Spider don’t recur.
        2. Delete 'orphan pages' that are never linked to, ie. Use the Spider to prune redundant pages25 automatically where possible.
        3. Fix the historical data where errors are uncovered by the Spider. An easier task now the site has a full-regen function.
      2. Investigate reduction in size of Web_Generator_Performance database that holds Raw_Links.
        1. Suppress MsgBox for 'Near to 2Gb' database size check (just use Debug).
        2. One-off deletion of rather useless links: EMail, TheosStyle.css, TT_ICO.png, #Top, ...
        3. Remove code that generates these links, and check them.
        4. Investigate removal of website 'root' (C:\Theo's Files\Websites\Theo's Website\) from Raw_Link and Full_Link. This would save 200Mb, but needs careful checking - especially against the Directory_Structure and related tables and code.
        5. If the above 'root' change is to go ahead - one-off updates + code changes required.
    • Technology
      1. Look into Sistrix Smart26. Errors and warnings itemised are:-
        1. Duplicate content: seems to be variants on
        2. Title Tags: Empty, too long, identical
        3. Page Not Found
        4. Filesize in excess of 1Mb
        5. Meta-Description: Empty
        6. Few words on Page
        7. H1: Not used, used multiple times per page, identical across pages
        8. Pictures: Alt attribute missing
    • WebRefs
      1. Documentation & Bug-fixes: Phase 2
        1. Re-document the procedures in the light of recent changes.
        2. Resolve issues generated / revealed by the spider.
        3. Investigate - and fix where possible - broken links.
      2. Find a way of recording Missing Webrefs other than debug.print: create table, then suppress message for known problems
      3. Investigate items flagged as defunct. Populate Defunct_Explanation in WebRefs_Table. Consider use of FairUse (Link (Fair Use)) for documents no longer available that I'd downloaded.
      4. Investigate WebRefs with Issue = 'URL Translated OK': does the translation really work? How?
  2. Other Websites: Priority 1 Items By Category:-

Summary of Progress to Date

This is hived off to various separate documents, which have now been harmonising and / or consolidated:-
  1. Summary of Progress to Date27.
  2. Outstanding Developments28,
  3. Functional Documentation29,
  4. A summary of time expended across the years developing my website30 is at "Software Development - Website - Development".

In-Page Footnotes:

Footnote 4: Footnote 5: Footnote 7: Footnote 9: Footnote 10: Footnote 11: Footnote 12: Footnote 13: Footnote 25: Footnote 26: Footnote 30:

Table of the Previous 12 Versions of this Note: (of 82)

Date Length Title
02/07/2021 20:32:38 20350 Status: Web-Tools (2021 - June)
03/04/2021 19:58:05 23112 Status: Web-Tools (2021 - March)
04/01/2021 19:40:48 23135 Status: Web-Tools (2020 - December)
04/10/2020 00:27:22 25418 Status: Web-Tools (2020 - September)
03/07/2020 22:09:07 28426 Status: Web-Tools (2020 - June)
04/04/2020 00:14:24 19563 Status: Web-Tools (2020 - March)
19/01/2020 23:41:17 19103 Status: Web-Tools (2019 - December)
10/10/2019 23:58:34 18052 Status: Web-Tools (2019 - September)
14/07/2019 20:29:46 16642 Status: Web-Tools (2019 - June)
05/04/2019 10:36:29 16128 Status: Web-Tools (2019 - March)
06/01/2019 23:36:58 18445 Status: Web-Tools (2018 - December)
10/10/2018 16:43:41 22079 Status: Web-Tools (2018 - September)

Note last updated Reference for this Topic Parent Topic
03/01/2022 23:58:34 520 (Status: Web-Tools (2021 - December)) Status: Summary (2021 - December)

Summary of Note Links from this Page

Aeon Papers Internet Technology and Philosophy PDFs Available on Local Website PID Note, Book & Paper Usage Research - Internet Technology
Research - Proposal Test Note - Auto-XRef Website - Outstanding Developments (2022 - January) Website - Progress to Date (2022 - January) Website Generator Documentation - Citations
Website Generator Documentation - Control Page Website Generator Documentation - Cross-Referencing Website Generator Documentation - Site Map Website Generator Documentation - Spider & Slave Database Website Generator Documentation - Timelines
Websites Maintained by Theo Todman        

To access information, click on one of the links in the table above.

Summary of Note Links to this Page

Internet Technology and Philosophy King's Maths Questions Simon - T1S1T1 Status: Consciousness Studies (2021 - December) Status: Priority Task List (2022: January)
Status: Summary (2021 - December), 2 Status: Summary Task List (2022: January) Status: Summary Task List (YTD: 21Q4 - 22Q1) Theo Todman's Philosophy Page Theo Todman's Website Maintainance History
Website - Progress to Date (2022 - January), 2, 3, 4, 5 Website Generator Documentation - Control Page Website Generator Documentation - Functors, 2, 3, 4, 5, 6, 7, 8    

To access information, click on one of the links in the table above.

Authors, Books & Papers Citing this Note

Author Title Medium Extra Links Read?
Software Development Website - Development Paper Low Quality Abstract    

Text Colour Conventions

  1. Blue: Text by me; © Theo Todman, 2022

© Theo Todman, June 2007 - Jan 2022.Please address any comments on this page to output:
Website Maintenance Dashboard
Return to Top of this PageReturn to Theo Todman's Philosophy PageReturn to Theo Todman's Home Page