Theo Todman's Web Page - Notes Pages
Status Reports
Status: Web-Tools (2024 - September)
(Text as at 10/10/2024 00:47:31)
Rationale for this Project
- This Project was alluded to briefly in a footnote on research methodology in my original Research Proposal1 under the head Research - Internet Technology2. When last at Birkbeck, I wrote a more extensive paper3 defending the Project and describing its rationale. I continue to take this Project further. There’s still a lot to do: still quite a few items on the “wish list”. It is fairly critical as an enabler for my research, so I need to get a move on as I had wanted it all out of the way before I re-started4 formal research.
- For documentation on my website follow the links below:-
- Functional5 Documentation6.
- Technical7 Documentation.
- Other Websites8
- Active
- Billericay Mayflower Bridge Club: since November 2021, on and off
- Old House Friday Bridge Club: a website that I’d created (amongst the other ‘small sites’) back in 2013. Revitalised in 2023.
- Enigma Ensemble: I created and maintained a small website for a music group Julie and I attend and which I now run. I lapsed the bespoke URL in favour of this sub-site of my own website but now intend to reinstate it. I’ve updated the site in the light of the retirement of our Director and to reference our new location.
- Inactive
- Sometime around 2005, I created a website for Dr. Sophie Botros, one of my supervisors at Birkbeck, but we then lost touch and it got maintained (very badly) by some desktop support outfit. In 15Q2 I took it back on again and spruced it up a bit, and maintain it periodically until 19Q3, when it was taken on by a professional outfit, Bookswarm. The “Sophie Botros” link in this bullet is now to their version of the site. My site looked like this.
- I created and / or ran a multitude of other bridge websites, but as of January 2018 I have either handed them over or mothballed them9:-
- I established the Hutton Bridge Club Website in 11Q4 using the standard Bridgewebs service, but with a couple of competitions using my own routines. This was handed over in 15Q3. I briefly took it on again in March 2020. It needed a spring clean, but I was waiting to see whether the club (and its members) survived the coronavirus pandemic, and I handed it back on that account. The club went ‘on-line only’ but has now closed for lack of support.
- In 15Q1, I took over the support and development of the Essex Contract Bridge Association (ECBA) website, which also uses Bridgewebs, but is very much larger. I wrote a lot of code10 to make this job less tedious. The site was handed over in 17Q4.
- For several years, I collected data11 on bridge activity in the Billericay/Brentwood area (initially needed for a project to set up a new consolidated club) by “scraping” data off web pages, consolidating it into a database and modelling it in various ways.
- I used this data to generate websites with a multitude of ladders for small clubs (8 small Essex Bridge websites). These are now mothballed, with the exception of the Old House one noted above.
- I created and maintained a new website for the First Class Bridge Academy, giving it “small clubs” ladders (8 further “small sites” for Bernie) as these were easy to maintain with little intervention.
- As of March 2020 I took over the Mountnessing Bridge Club website. Earlier, in 16Q3, I created the Mountnessing Bridge Club Archive website, using the vast bulk of the pages from their legacy site, as the club had moved to Bridgewebs and lost its historical data. Sadly, the Club has now been wound up, but I intend to keep the historical record.
- I created a website12 for displaying the textual and grammatical analyses and appendices of Pete’s PhD on the Acts of the Apostles: Acts: Test Site.
Summary of Progress during July - September 2024
- I spent 100.25 hours in 24Q3 on this Project, or related work (338.75 hours YTD, where for "YTD" - Year to Date - I mean the (academic) year that commenced in October 2023). That's 109.9% of the planned effort (83.8% YTD). Overall, 14.9% of my Project effort in the Quarter was directed towards this project (making 11.5% YTD) - as against 13.5% planned (13.5% YTD).
- As can be seen, I spent about 10% more than my time allocation on this project last Quarter, though fell somewhat short YTD.
- Time was spent in two main areas: Bridge and Documentation.
- Bridge: The developments on my Bridge database were unplanned and arose on account of a bug introduced into MS Access that means that the garbage collector no longer works. Temporary datasets are left hanging, even if set to Nothing, and the Access database soon runs out available recordsets. This affected both my ‘Small Sites’ and my own Bridge statistics. I put in temporary fixes that mean the symptoms – ‘cannot open any more databases’, hanging processes and inability to compact / repair – no longer occur. I’ve raised CRs for permanent fixes.
- Documentation: I made considerable progress on documenting Timelines13 & Backups14, but didn’t complete either. The bulk of the time on backups was taken up with documenting what is backed up – with the intention of sorting out the mess in due course.
- Completed items included:-
- Own Website:
- Bridge
- Small Systems: Make code changes to work around the ‘Cannot open any more databases: Run-time error 3048’ bug.
- Bridge Results Export: Make code changes to work around the ‘Cannot open any more databases: Run-time error 3048’ bug.
- Infrastructure
- New Colour Inkjet Printer: Investigation, Purchase & Installation of Epson EcoTank
- Migrate from Outlook to New Outlook (3 PCs)
- Other Websites:
- Full details for 24Q3 are given below15:-
Website (Total Hours = 94.25)
- Website - Bridge Development
- Website - Bridge Maintenance
- Website - Development (Total Hours = 61)
- Reinstate the Enigma Ensemble Website (5.5 hours)
- Website - Bridge Generator - Fatal Database Errors: ‘Cannot open any more databases' - Fixes for Bridge Statistics Processing (5 hours)
- Website - Bridge Generator - Fatal Database Errors: ‘Cannot open any more databases' - Fixes for Small Systems Processing (5 hours)
- Website - Bridge Generator - Fatal Database Errors: ‘Cannot open any more databases: Run-time error 3048' (1.25 hours)
- Website - Generator - Timeline software16: Complete Documentation (5.25 hours)
- Website - Generator - Timeline software17: Data Entry - Coxes Farm - Bakers Phase 3 (0.75 hours)
- Website - Generator - Amend Functor_21 (Option 31) to allow for gaps in Reading of NT/OT in Greek + Hebrew (0.5 hours)
- Website - Generator - Document Backup Systems18 (31.5 hours)
- Website - Generator - Document Spider19 (0.5 hours)
- Website - Generator - Documention Review & Filing (4.25 hours)
- Website - Generator - Improve Daily and Top 10 Task Reporting in Status: Priority Task List20 (0.5 hours)
- Website - Generator - Ling - Output pages for individual languages (0.25 hours)
- Website - Generator - Ling - Output pages for individual languages - Bug Fixes (0.75 hours)
→ See "Software Development - Website - Development" (61 hours)
- Website - Education
- Website - Infrastructure (Total Hours = 20.25)
- EE - New Contract + Hub (Outbuildings): Purchase & Installation (4 hours)
- FreeSat TV Box: Investigation, Purchase & Installation (& Sky Cancellation) (1.5 hours)
- Inkjet Printer: Investigation, Purchase & Installation of Epson EcoTank (6.25 hours)
- Install BibleWorks on Desktop (0.75 hours)
- Microsoft Windows / MS Office / Norton - Releases, Bugs & Periodic Re-boots (1.5 hours)
- Microsoft Windows: External CD Drive (0.75 hours)
- Microsoft Windows: New Outlook (2.75 hours)
- PC Backups / OneDrive (2.75 hours)
→ See "Admin - Website - Admin & Maintenance" (20.25 hours)
- Website - Maintenance (Total Hours = 4.75)
- 24Q2 Status Reports (0.5 hours)
- Website - Generator - Consolidated Site-Hits: Review Webalizer Methodology & Consolidate Stats (2 hours)
- Website - Generator - WebRefs - Manual / Automatic URL Checks & Fixes (1.25 hours)
- Website - Periodic Full Regeneration (1 hour)
→ See "Admin - Website - Admin & Maintenance" (4.75 hours)
Website Others (Total Hours = 6)
- Website Others - Mayflower DBC Maintenance
- Website Others - Small Bridge Sites
Plans for the Near Future
- I’ve more than doubled my weekly time allocation to 15 hours because otherwise there just isn’t enough time to accommodate all the things I want to do. I hope this push will be enough to make progress on important items.
- The Plan below is taken automatically from the Priority 1 items on my Development Log – as published in my Outstanding Developments21 Report.
- Own Website: Priority 1 Items By Category:-
- Architecture
- Complete XRef-re-engineering project:-
- Ensure all links and link-pages use the new XRef table, and pension off the old tables where appropriate.
- Check all link-types still work and fix any errors.
- Complete the auto-triggering of regeneration of “associated” link pages.
- Fix Bug whereby PaperSummary pages seem to have “Works-” and “Books/Papers-” Citings that refer to the same link-pages.
- Complete Documenting22 the process!
- Backups
- Determine why flag on directories "not to be backed up" gets reset
- Document and enable regular running of the de-duplication function ('pruning') of the backup disk. Keep a full record of deleted duplicates.
- Enable Backups to be undertaken on both my PC & Laptop and write reconciliation software.
- Incorporate new 5Tb WD Backup Drive
- Investigate historic error messages
- Investigate Record-count discrepancies:-
- How do website files work as far as counts are concerned?
- Why aren't they recorded in Backup_History, nor the fact that the website was backed up?
- Different counts depending on whether new or old laptop is backed up. Investigate 63k discrepancy - lower on new laptop.
- Review architecture to improve performance; Need to document23 first
- Sort out the contents of my C:Drives, OneDrive and Emails as documented in my Backup Documentation
- Books/Papers
- For Books and Papers - add link pages - analogous to:-
→ BooksToNotes_
→ BookCitings_
→ PapersToNotes_
→ PaperCitings_
linked to from:-
→ BookSummaries_
→ PaperSummary_
and (maybe) from:-
→ BookPaperAbstracts_
→ Abstract_
But, the links are FROM the Book / Paper, rather than TO
For Books, ensure that all the links from all the Book's Papers are also included
- Sub-Topic ID has reached 996: What happens after 999? Research and Fix
- Bridge
- Bridge Results Export: Correct residual problems following code changes to work around the ‘Cannot open any more databases: Run-time error 3048’ bug.
- After run, while the database closes OK, MS Access remains active and has to be crashed via Task Manager.
- If the Export is run a second time, as is required for Partner's calculated NGS to show correctly, without closing the database first, we get 'permision denied' when trying to create Bridge_Results_Theo.htm.
- Small Systems: Correct residual problems following code changes to work around the ‘Cannot open any more databases: Run-time error 3048’ bug.
- After importing results via 'Joint Project Conversion', while the database closes OK, MS Access remains active and has to be crashed via Task Manager.
- After running the Mini-Wedsite Generator, while the database closes OK, MS Access remains active as a ghost Background Process and has to be crashed via Task Manager. Until this is done, MS Access won't open.
- Longer term - correct the fudge that simply re-creates Small Systems for the current year (hard coded to 2024).
- Documentation
- Website Generator Documentation - Audio24: Review & Complete Documentation
- Website Generator Documentation - Author Narratives25: Complete Documentation
- Website Generator Documentation - Citations26: Complete Documentation
- Website Generator Documentation - Code Documenter27: Complete Documentation
- Website Generator Documentation - Create Notes Web Pages28: Complete Documentation
- Website Generator Documentation - Cross-Referencing29: Complete Documentation
- Website Generator Documentation - Functional Overview30: Review & Complete Documentation
- Website Generator Documentation - Language / Ling Pages31: Complete Documentation
- Website Generator Documentation - Links & Link-Pages32: Complete Documentation
- Website Generator Documentation - Monthly Reporting33: Complete Documentation
- Website Generator Documentation - New Items34: Complete Documentation
- Website Generator Documentation - Note References & Reading List35: Complete Documentation
- Website Generator Documentation - PID Note, Book & Paper Usage36: Complete Documentation
- Website Generator Documentation - Precis Export37: Complete Documentation
- Website Generator Documentation - Prune Website38: Complete Documentation
- Website Generator Documentation - Spider & System Backups39: Complete Documentation
- Website Generator Documentation - Thesis Dashboard40: Complete Documentation
- Website Generator Documentation - Thesis Technology41: Complete Documentation
- Website Generator Documentation - Time Recording42: Complete Documentation
- Website Generator Documentation - Timelines (Photos & Blog)43: Complete Documentation
- Education
- Read "PC Pro - Computing in the Real World".
- Read "White (Ron) & Downs (Timothy Edward) - How Computers Work: The Evolution of Technology".
- Infrastructure
- Fatal Database Errors: ‘Cannot open any more databases: Run-time error 3048’.
- When I then try to compact and repair the database - the first port of call when there's a strange problem - it complains that it is locked by the Administrator. Then the database won’t close and has to be zapped by Task Manager. Only then can I delete the lock file, but this doesn't resolve anything as the problem recurs. I think the 'too many databases' error message is misleading.
- This started on Tuesday 27th August 2024 with Bridge_Web_Generator.accdb on Desktop. I stopped running it thereon and it was fine for a day on my Laptop, then the problem recurred.
- Looking on-line, this seems to be a known Microsoft bug for which a fix is awaited. It looks like the garbage dispoal routines (after set db = nothing) aren't working. It's not just where you're sloppy and don't do this and rely on the system to do garbage collection: I did changes to 'Small Systems' processing, but it made no difference.
- My main 'Thesis' database is working fine - I managed a full Regeneration starting late on 31st August. It may be that the bug was fixed under Windows 11 but not Windows 10. However, I'm not taking the risk of running my Bridge database on my Desktop in case it mucks up my Thesis database. The failure of Compact / Repair of my Bridge database is particularly worrying, but my Thesis databse C/Rs fine on my Laptop (and Desktop). This is a relief, as this database becomes unusable after a few days otherwise as it reaches the 2Gb limit
- Currently I'm waiting and seeing. Re-importing all the elements into a new Bridge database made no difference.
- However, so that I can continue running my Bridge processing, I've raised a couple of Development Items to enable
→ Small Systems
→ Bridge Statistics
- Migrate 2 x Laptop + Desktop from Norton AV to Microsoft Defender: Norton non-renews 13/11/24
- Migrate from Chrome to Bitwarden for password management
- Notes
- Active Note Groups with Notes Updated in the last 7, 14, 28 days: seem to be producing random results. Important as there are links from my Home Page.
- Allow the option to embed blocks of re-usable text in Notes (ie. linearly embed them essay-style) rather than using hyperlinking. Slightly analogous to Note_Snippets or Sub_Notes, but more flexible.
- For use as "disclaimers".
- The identical block of text in the Dashboard44 to that in the Note Thesis - Method & Form45 that describes the Thesis.
- For Thesis / essays: the difficulty here is the need for linking passages to make the text run smoothly.
- As a more controllable alternative to 'Printable Notes'.
- Correct processing for 'References & Reading List' section in Thesis - References46. There are over 7,000 but only 7 items appear!
- Determine why all my Status Reports from September 2021 are missing from Notes_Archive. Seek to recover them from backups.
- Develop method for (manually) deleting otiose Archived Notes: ie. ensure links work for remaining Notes!
- Enhance PID Note, Book & Paper Usage47 to include a section listing works cited in my Thesis but not read
- Investigate Note_Links: Section references seem to be incorrect
- Notes_Links: Segment & Provide Jump Table (Eg. For Thought48)
- PID Thesis References - De-duplicate / suppress and / or change sort order / aggregation for non-anthology book chapters
- Update Personal Page49 & Home Page
- Upgrade Auto_Reference_Notes to reference Sub-Notes: Used to only affect one note - Somerset Maugham - but now may be urgent
- When using the '#' reference to Notes, allow the omission of the Note ID for links within the same Note.
- Papers
- Split PDFs Available on Local Website50 page into alphabetical sub-pages
- Photos
- Timeline software51: Add photos for Bakers' Coxes Farm Development Phase III
- Timeline software52: Add photos for Henry's Biography
- Process
- Determine why Recalculation & Changed Book/Papers produce unneeded regeneration.
- Spider
- Investigate reduction in size of Web_Generator_Performance database that holds Raw_Links.
- Suppress MsgBox for 'Near to 2Gb' database size check (just use Debug).
- One-off deletion of rather useless links: EMail, TheosStyle.css, TT_ICO.png, #Top, ...
- Remove code that generates these links, and check them.
- Investigate removal of website 'root' (C:\Theo's Files\Websites\Theo's Website\) from Raw_Link and Full_Link. This would save 200Mb, but needs careful checking - especially against the Directory_Structure and related tables and code.
- If the above 'root' change is to go ahead - one-off updates + code changes required.
- Use Edge rather than Internet Explorer for External Links Spider (use Bing AI)
- Status
- Reorganise Status Reporting:-
- Merge Consciousness Project53 with Thesis54 (as a sub-project).
→ Merge historic plan & actuals
→ Adjust current Thesis plan, actuals & task list.
- Create a new Project: Culture & Politics
→ Move 'Thesis background reading / writing and 'Aeon' to this new project.
→ Create project plan and move historic actuals to it.
→ Adjust 'Thesis' historic actuals and task list: move relevant items to new project.
- Make sure the Status reports - both manual and automatic - still work
- WebRefs
- Create a page showing the use of WebRefs more clearly. The pages accessed via Theo Todman's Web-Links Test Webpages are of little use other than as an audit. Also, there are many broken links not picked up by the Spider. Need pages that use meaningful names.
- Documentation & Bug-fixes: Phase 2
- Re-document the procedures in the light of recent changes.
- Resolve issues generated / revealed by the spider.
- Investigate - and fix where possible - broken links.
- Find a way of recording Missing / Unknown Webrefs (eg. Webrefs 8395 and 8342 in Archived Notes 1292 and 1317) other than debug.print: create table, then suppress message for known problems
- Investigate items flagged as defunct. Populate Defunct_Explanation in WebRefs_Table. Consider use of FairUse (Link (Fair Use)) for documents no longer available that I'd downloaded.
- Investigate WebRefs with Issue = 'URL Translated OK': does the translation really work? How?
- Visitor Stats - cmdVisitorStats_Click (and following) - loops and requires 'DAO.DBEngine.SetOption dbmaxlocksperfile,1000000' to be typed in the Immediate window. Re-engineer.
- Other Websites: Priority 1 Items By Category:-
Summary of Progress to Date
This is hived off to various separate documents, which have now been harmonising and / or consolidated:-
- Summary of Progress to Date55.
- Outstanding Developments56,
- Functional Documentation57,
- A summary of time expended across the years developing my website58 is at "Software Development - Website - Development".
In-Page Footnotes:
Footnote 4:
- Well, in a sense, I’ve missed the boat as I’m now putting effort into my research, though in an informal basis, so will need to continue with both projects in parallel.
- This was always likely to be necessary, as new features will always arise in use. It’s a prototype methodology, after all.
Footnote 5:
- This is very tedious to produce and consequently is both incomplete and out of date.
Footnote 7:
- This is much more fun, as it’s a purely technical task.
- I’ve written a vastly-improved general-purpose technical documenter for MS Access.
Footnote 9:
- It’s a shame to abandon the “mini websites” with all their ladders, as it’s rather well done.
- However, I couldn’t waste time on these after I’d abandoned bridge.
Footnote 10:
- In particular, for the ECBA “Victor Ludorum” competition.
- I cannot hand any of this code over, so the tedium will return, though not to me!
Footnote 11:
- I had agreed to share this data sometime early in 2018
- But will wait until asked again, as I doubt it’ll be of any real use to anyone.
Footnote 12:
- It used to exist in two versions, live and test.
- Pete decided not to renew the license for the live site, now it has achieved its purpose, so only the test site remains.
Footnote 15:
- Note that where fixes or small enhancements are made to a previously “completed” development, I don’t announce it again against the list of “completed” items above, though the work appears in the full list for the quarter.
Footnote 58:
- As distinct from developing other peoples’ websites – time which is also recorded against this project, but not against this task.
Table of the Previous 12 Versions of this Note: (of 93)
Summary of Notes Referenced by This Note
To access information, click on one of the links in the table above.
Summary of Notes Citing This Note
Internet Technology and Philosophy |
King's Maths Questions |
Simon - T1S1T1 |
Status: Consciousness Studies (2023 - September) |
Status: Priority Task List (2024 - September) |
Status: Summary (2024 - September), 2, 3 |
Status: Summary Task List (2024: October) |
Status: Summary Task List (YTD: 24Q4) |
Theo Todman's Philosophy Page |
Theo Todman's Website Maintainance History |
Website - Progress to Date (2024 - October), 2, 3, 4, 5 |
Website Generator Documentation - Control Page |
Website Generator Documentation - Functors, 2, 3, 4, 5, 6, 7 |
|
|
To access information, click on one of the links in the table above.
Authors, Books & Papers Citing this Note
Author |
Title |
Medium |
Extra Links |
Read? |
Software Development |
Website - Development |
Paper |
|
|
Text Colour Conventions
- Blue: Text by me; © Theo Todman, 2024