Theo Todman's Web Page - Notes Pages
Status Reports
Status: Web-Tools (2021 - June)
(Text as at 02/07/2021 20:32:38)
*** THIS IS NOT THE LATEST VERSION OF THIS NOTE ***
(For the live version and other versions of this Note, see the tables at the end)
Rationale for this Project
- This Project was alluded to briefly in a footnote on research methodology in my original Research Proposal1 under the head Research - Internet Technology2. When last at Birkbeck, I wrote a more extensive paper3 defending the Project and describing its rationale. Now that my PhD is in suspense, I have decided to take this Project further. There’s a lot to do: still quite a few items on the “wish list”. It is fairly critical as an enabler for my research, so I need to get a move on as I want it all out of the way before I re-start4 formal research.
- For documentation on my website (currently password protected) follow the links below:-
- Functional5 Documentation6.
- Technical7 Documentation.
- Other Websites8
- Active
- I’ve created and continue to maintain a small website for a music group Julie and I attend – the Enigma Ensemble.
- I established the Hutton Bridge Club (Defunct) Website in 11Q4 using the standard Bridgewebs service, but with a couple of competitions using my own routines. This was handed over in 15Q3, but I’ve taken it on again as of March 2020, not that there’s currently much to do. It needs a spring clean, but I’m waiting to see whether the club (and its members) survive the coronavirus pandemic.
- In 16Q3 I created the Mountnessing Bridge Club Archive website, using the vast bulk of the pages from their legacy site, as the club had moved to Bridgewebs and lost its historical data. As of March 2020 I’ve taken over the aforementioned Mountnessing Bridge Club (Defunct) website itself.
- Inactive
- Sometime around 2005, I created a website for Dr. Sophie Botros, one of my supervisors at Birkbeck, but we then lost touch and it got maintained (very badly) by some desktop support outfit. In 15Q2 I took it back on again and spruced it up a bit, and maintain it periodically until 19Q3, when it was taken on by a professional outfit, Bookswarm. The “Sophie Botros” link in this bullet is now to their version of the site.
- I created and / or ran a multitude of other bridge websites, but as of January 2018 I have either handed them over or mothballed them9:-
- In 15Q1, I took over the support and development of the Essex Contract Bridge Association (ECBA) website, which also uses Bridgewebs, but is very much larger. I wrote a lot of code10 to make this job less tedious. The site was handed over in 17Q4.
- For several years, I collected data11 on bridge activity in the Billericay/Brentwood area (initially needed for a project to set up a new consolidated club) by “scraping” data off web pages, consolidating it into a database and modelling it in various ways.
- I used this data to generate websites with a multitude of ladders for small clubs (Essex Bridge Results). These are now mothballed.
- I created and maintained a new website for the First Class Bridge Academy, giving it “small clubs” ladders (Bernie's Ladders Archive) as these were easy to maintain with little intervention.
- In 16Q3 I created the Mountnessing Bridge Club Archive website, using the vast bulk of the pages from their legacy site, as the club had moved to Bridgewebs and lost its historical data.
- I created a website12 for displaying the textual and grammatical analyses and appendices of Pete’s PhD on the Acts of the Apostles: Acts: Test Site.
Summary of Progress during April - June 2021
- I spent 145.25 hours in 21Q2 on this Project, or related work (416.5 hours YTD, where for "YTD" - Year to Date - I mean the (academic) year that commenced in October 2020). That's 113.2% of the planned effort (107% YTD). Overall, 21% of my Project effort in the Quarter was directed towards this project (making 20.5% YTD) - as against 18.5% planned (18.4% YTD).
- I was comfortably over budget in 21Q2 Time well spent, too, though most of it was spent enhancing functions related to my PID Note, Book & Paper Usage13 page, so doesn’t feature in the “items completed” lists.
- Completed items included:-
- Own Website:
- Architecture
- Created Check_Delimiters to protect against unpaired delimiters in lists, which squiff the formatting of documents that contain them. Output diagnostics to MsgBox, if not a re-gen, and to debug window to enable correction.
- Papers
- As revealed by Spider: PapersToNotes links pages not being generated (3 items: Papers "Hirsch (Eli) - Essence and Identity", "Lowe (E.J.) - Instantiation, Identity and Constitution" & "Hume (David) - On the Immortality of the Soul") leading to links from the PapersToNotes summary pages, as well as from Paper Abstracts and Paper Summaries (making 9 broken links). The links were present on Cross_Reference, but not on Cross_Reference_Prelims, which is central to Paper_Note_Link_List. It looked like PapersToNotes_Prelims_GEN would have worked, so I regenerated the PapersToNotes pages, and the links all worked. It may have been a timing issue of some sort, so I'll look out for recurrence.
- Other Websites:
- Full details for 21Q2 are given below14:-
Website (Total Hours = 143.25)
- Website - Development (Total Hours = 118.5)
- Website - Generator - Automatically create Papers from relevant Aeon Files (7 hours)
- Website - Generator - Consolidated Site-Hits: Review Webalizer Methodology & Consolidate Stats (0.75 hours)
- Website - Generator - Create Check_Delimiters to check lists (3.5 hours)
- Website - Generator - Enable easy access to PDFs off-line, but suppress on-line (14.75 hours)
- Website - Generator - Enhance comparative Ling database (5.5 hours)
- Website - Generator - Facilitate WebRefs checking for PID Papers (1.25 hours)
- Website - Generator - Fixes re Broken Links revealed by Spider (4.5 hours)
- Website - Generator - Investigate Spider Run-times & Database size (10.5 hours)
- Website - Generator - Investigating potential web-crawler using MS Edge (see Daniel Pineault: VBA – Exploring Microsoft Edge) (0.5 hours)
- Website - Generator - Move various large tables to new Audit database (1 hour)
- Website - Generator - Remodelling and improvement of Thesis Dashboard15 (3.5 hours)
- Website - Generator - Report & Manage Links between PID Notes, Papers and Thesis (24.5 hours)
- Website - Generator - Review Xref Methodology (1.25 hours)
- Website - Generator - Set up comparative Ling database (31.5 hours)
- Website - Generator - Set up comparative Ling database: Data Entry (7 hours)
- Website - Generator - Update Development Log in the light of recent activities (1.5 hours)
→ See "Software Development - Website - Development" (118.5 hours)
- Website - Education
- Website - Infrastructure (Total Hours = 4)
- Kaspersky password issue (0.25 hours)
- Microsoft Windows 10 / MS Office - Releases, Bugs & Periodic Re-boots (1.75 hours)
- PC Backups / OneDrive (0.75 hours)
- Replacement Mouse (0.25 hours)
- Set up Kindle Paperwhite (1 hour)
→ See "Admin - Website - Admin & Maintenance" (4 hours)
- Website - Maintenance (Total Hours = 19.25)
- 21Q1 Status Reports (0.5 hours)
- Website - Generator - WebRefs - Manual / Automatic URL Checks & Fixes (8.75 hours)
- Website - Periodic Full Regeneration (5.5 hours)
- Website - Run Web Spider (4.5 hours)
→ See "Admin - Website - Admin & Maintenance" (19.25 hours)
Website Others
Plans for the Near Future
The Plan below is taken automatically from the Priority 1 items on my Development Log, as published in my Outstanding Developments16 Report. I’ve had to reduce the weekly allocation to 7 hours. This is to allow for increased effort on other projects.
- Own Website: Priority 1 Items By Category:-
- Architecture
- Compact and Repair Problems
- On compacting and repairing my main database I sometimes get the error "The query cannot be completed. Either the size of the query result is larger than the maximum size of a database (2 GB), or there is not enough temporary storage space on the disk to store the query result".
- It happens 3 times while the database is re-opening.
- There is lots of space, and the database is only 650Mb (and the error started when it was under 500Mb).
- This mostly happens after I've run long processes, so I usually close the database, re-open it and then try the compact and repair. Usually this works, but not always. But I then try again and the message disappears.
- I strongly suspect that this is MS Access itself re-indexing tables, and blowing up a temporary database, but I can’t find any evidence for this on-line. Or help, other than suggestions to split databases and do other sensible things. Given that the error occurs when the database is re-opening, with no temporary file visible, this is all very strange.
- 17/04/20 - set MaxLocksPerFile to 1,000,000 (from the default 9,500). Sadly, it doesn't seem to have made a difference.
- Complete XRef-re-engineering project:-
- Ensure all links and link-pages use the new XRef table, and pension off the old tables.
- Look into writing out specific object-identifiers, and linking thereto for Citations, rather than paragraph references. An issue is multiple instances of the same object in a document.
- Check all link-types still work and fix any errors.
- Complete the auto-triggering of regeneration of “associated” link pages.
- Fix update bug in Convert_Webrefs.
- Fix Bug whereby PaperSummary pages seem to have “Works-” and “Books/Papers-” Citings that refer to the same link-pages.
- Document17 the process!
- Document18, repair & update my Website site-map
- Review effectiveness of hyperlinking method in the light of PhD and Philosophy of Religion experience.
- Where possible, use ID rather than NAME for in-page hyperlinks
Backups
- Investigate Record-count discrepancies:-
- How do website files work as far as counts are concerned?
- Why aren't they recorded in Backup_History, nor the fact that the website was backed up?
- Different counts depending on whether new or old laptop is backed up. Investigate 63k discrepancy - lower on new laptop.
- Review architecture to improve performance; Need to document19 first
Books
- Further improve the time to regenerate Book Summaries. Took about 21 minutes on 06/06/2021, but should be under 5 minutes!
Books/Papers
- Determine why the indentation of "Various - Papers in Desk Drawer" squiffs after "Nakamoto (Satoshi) - Bitcoin - A Peer-to-Peer Electronic Cash System". Maybe add it to the "suppressed" list, though "Various - Papers on Desktop" is OK.
- Investigate whether multiple Subject/Topic/Subtopic usage leads anywhere (ie. are just the first (of 3) actually used). Fix anything amiss.
- Reformat the BookCitings and PaperCitings pages:-
- Detail PaperCitings Pages: Include only useful information on the detail pages; but if there are multiple links from the same object, include them on the same line as 'extra links' as in BookCitings (copy the code: or, better, combine the two subs).
- Summary (Author Letter) pages: Include counts (as in Authors' Citations).
- Ensure uses the Cross_Reference table.
- Document20!
Bridge
- Develop auto-reconciliation routines vs EBU results download
Documenter
- Investigate the error reports from the Documenter, especially unused variables & queries.
- Provide Functional Documentation for Website Generator (using Notes)
Education
- "Sitepoint (Learnable) - Sitepoint Learnable Web Development Courses": Membership cancelled, but plan what to do with the eBooks in my possession.
- Read "PC Pro - Computing in the Real World".
- Read "White (Ron) & Downs (Timothy Edward) - How Computers Work: The Evolution of Technology".
Infrastructure
- iCloud for Windows: Re-install & solve 'The upload folder for iCloud Photos is missing' problem. Try on new Laptop.
Notes
- Add "Note Alternates" to Note pages.
- Allow the option to concatenate Notes in the Printed version (ie. linearly embed them essay-style), rather than treating the hyperlinks as footnotes – but still keep the hyperlink & cross-referencing in place.
- For use as "disclaimers" - eg. for "Plug Notes".
- For Thesis / essays: the difficulty here is the need for linking passages to make the text run smoothly.
- As part of the Cross-Referencing project, check out the consistent treatment of Note 87521, which should be universally ignored. Recently, links to it appeared on web pages Book_Summary, BookPaperAbstracts and BooksToNotes, as a Note referencing a Book. The critical item was a row on the Note_Book_Links table.
- As revealed by Spider: NotesPrint from Secure_Jen fails for Notes_Jump links. Need to fix the processing of "../"-style links to go back to root and back. Example: Note 51322.
- Create a page to compare the vocabulary across related languages on Ling
- Determine why very long printable notes (eg. Level 3+ for Note 17023) are being truncated. Probably suppress them in any case, as they take far too long to load.
- Develop a method to create Papers for rows in Aeon_Files that have an associated Identity_Note_ID but no Paper_ID.
- Fix bugs in multi-level footnoting in Printable Notes – the referencing is going wrong.
- Investigate Note_Links: Section references seem to be incorrect
- Printable Notes: fix the bug whereby the “private” flag is round the wrong way.
- Split Aeon Page24 into multiple sub-pages (either by topic or by priority)
- Suppress the publication of the Printable versions of Temp Notes
- Upgrade Auto_Reference_Notes to reference Sub-Notes: Currently only affects one note - Somerset Maugham - so not yet urgent
- Very large printable Notes: eg. for Status: Summary25. Too large to be useful, and to transfer. Suppress for now.
Papers
- The monthly regeneration process for Paper Abstracts was still taking just over 5 hours. I thought it could not be fixed until the cross-referencing project is fully complete and documented. However, it spontaneously improved to 1.6 hours in the August 2020 run. Monitor! Last run: 1.6 hours on 06/06/2021.
Photos
- Timeline software26: Add photos for Bakers' Coxes Farm Development Phase III
- Timeline software27: Add photos for Holidays & Family History
- Develop software & procedure to make adding more content to the photos pages easier to undertake.
Process
- Determine why Recalculation & Changed Book/Papers produce unneeded regeneration.
Spider
- Analyse the results of the data collection exercise and design a plan of campaign to fix broken Internal links and prevent recurrence.
- Correct the code so the problems discovered by the Spider don’t recur.
- Delete 'orphan pages' that are never linked to, ie. Use the Spider to prune redundant pages28 automatically where possible.
- Fix the historical data where errors are uncovered by the Spider. An easier task now the site has a full-regen function.
- Investigate reduction in size of Web_Generator_Performance database that holds Raw_Links.
|a|Suppressed MsgBox for 'Near to 2Gb' database size check (just use Debug).
|a|One-off deletion of rather useless links: EMail, TheosStyle.css, TT_ICO.png, #Top, ...
|a|Remove code that generates these links, and check them.
|a|Investigate removal of website 'root' (C:\Theo's Files\Websites\Theo's Website\) from Raw_Link and Full_Link. This would save 200Mb, but needs careful checking - especially agains the Directory_Structure and related tables and code.
|a|If the above 'root' change is to go ahead - one-off updates + code changes required.
Technology
- Look into Sistrix Smart29. Errors and warnings itemised are:-
- Duplicate content: seems to be variants on theotodman.com
- Title Tags: Empty, too long, identical
- Page Not Found
- Filesize in excess of 1Mb
- Meta-Description: Empty
- Few words on Page
- H1: Not used, used multiple times per page, identical across pages
- Pictures: Alt attribute missing
WebRefs
- Documentation & Bug-fixes: Phase 2
- Re-document the procedures in the light of recent changes.
- Resolve issues generated / revealed by the spider.
- Investigate - and fix where possible - broken links.
- Find a way of recording Missing Webrefs other than debug.print: create table, then suppress message for known problems
- Investigate items flagged as defunct. Populate Defunct_Explanation in WebRefs_Table. Consider use of FairUse (Link (Fair Use)) for documents no longer available that I'd downloaded.
- Investigate WebRefs with Issue = 'URL Translated OK': does the translation really work? How?
Other Websites: Priority 1 Items By Category:-
Summary of Progress to Date
This is hived off to various separate documents, which have now been harmonising and / or consolidated:-
- Summary of Progress to Date30.
- Outstanding Developments31,
- Functional Documentation32,
- A summary of time expended across the years developing my website33 is at "Software Development - Website - Development".
In-Page Footnotes
Footnote 4:
- Well, in a sense, I’ve missed the boat as I’m now putting effort into my research, though in an informal basis, so will need to continue with both projects in parallel.
- This was always likely to be necessary, as new features will always arise in use. It’s a prototype methodology, after all.
Footnote 5:
- This is very tedious to produce and consequently is both incomplete and out of date.
Footnote 7:
- This is much more fun, as it’s a purely technical task.
- I’ve written a vastly-improved general-purpose technical documenter for MS Access.
Footnote 9:
- It’s a shame to abandon the “mini websites” with all their ladders, as it’s rather well done.
- However, I couldn’t waste time on these after I’d abandoned bridge.
Footnote 10:
- In particular, for the ECBA “Victor Ludorum” competition.
- I cannot hand any of this code over, so the tedium will return, though not to me!
Footnote 11:
- I had agreed to share this data sometime early in 2018
- But will wait until asked again, as I doubt it’ll be of any real use to anyone.
Footnote 12:
- It used to exist in two versions, live and test.
- Pete decided not to renew the license for the live site, now it has achieved its purpose, so only the test site remains.
Footnote 14:
- Note that where fixes or small enhancements are made to a previously “completed” development, I don’t announce it again against the list of “completed” items above, though the work appears in the full list for the quarter.
Footnote 28:
- Note that Backup_Prune_Ctrl deletes (relevant) pages that weren't regenerated in the last full site-regen, but this isn't the same thing.
Footnote 29:
- See Sistrix
- This used to be called Optimizr, see Sistrix (which now auto-forwards to Sistrix).
- A quick look doesn’t show it to be an obvious scam, but I need to double-check.
- An unsolicited analysis of my site turned up monthly from Optimizr from January 2015 to October 2017, listing a large number of “problems” that I think I know about, but which are in the queue to address.
- It restarted in February 2018, under the Sistrix name (this seems to have been associated with Optimizr since November 2015).
- The free version of this software is restricted to 1,000 pages, which is a very small proportion of my Site, though I may be able to point it to difference base-URLs.
- But I do need to address the problems validly itemised, and a sub-set is still useful.
Footnote 33:
- As distinct from developing other peoples’ websites – time which is also recorded against this project, but not against this task.
Live Version of this Archived Note
Table of 12 Earlier Versions of this Note (of 81)
Table of 12 Later Versions of this Note (of 18)
Summary of Notes Links from this Page
To access information, click on one of the links in the table above (if any).
Summary of Note Links to this Page
| Status: Consciousness Studies (2021 - June) |
Status: Priority Task List (2021 - June) |
Status: Priority Task List (2021 - September) |
Status: Summary (2021 - June), 2 |
Status: Summary Task List (2021: April - June) |
| Status: Summary Task List (2021: July - September) |
Status: Summary Task List (YTD: 20Q4 - 21Q2) |
Status: Summary Task List (YTD: 20Q4 - 21Q3) |
Theo Todman's Philosophy Page |
Website - Progress to Date (2021 - June), 2, 3, 4, 5 |
| Website Generator Documentation - Functors, 2, 3, 4, 5, 6, 7, 8 |
|
|
|
|
To access information, click on one of the links in the table above (if any).
Text Colour Conventions
- Blue: Text by me; © Theo Todman, 2026