Theo Todman's Web Page - Notes Pages
Status: Web-Tools (2023 - June)
(Text as at 06/07/2023 00:43:12)
Rationale for this Project
- This Project was alluded to briefly in a footnote on research methodology in my original Research Proposal1 under the head Research - Internet Technology2. When last at Birkbeck, I wrote a more extensive paper3 defending the Project and describing its rationale. I continue to take this Project further. There’s still a lot to do: still quite a few items on the “wish list”. It is fairly critical as an enabler for my research, so I need to get a move on as I want it all out of the way before I re-start4 formal research.
- For documentation on my website follow the links below:-
- Functional5 Documentation6.
- Technical7 Documentation.
- Other Websites8
- I’ve created and continue to maintain a small website for a music group Julie and I attend – the Enigma Ensemble. I’ve lapsed the bespoke URL in favour of a sub-site of my own. I intend to reinstate it.
- As of March 2020 I’ve taken over the Mountnessing Bridge Club website. Earlier, in 16Q3, I created the Mountnessing Bridge Club Archive website, using the vast bulk of the pages from their legacy site, as the club had moved to Bridgewebs and lost its historical data.
- From November 2021 to April 2022 I supported the Billericay Mayflower Bridge Club website. I’ve now taken it back on!
- I’ve revitalised Old House Friday Bridge Club.
- Sometime around 2005, I created a website for Dr. Sophie Botros, one of my supervisors at Birkbeck, but we then lost touch and it got maintained (very badly) by some desktop support outfit. In 15Q2 I took it back on again and spruced it up a bit, and maintain it periodically until 19Q3, when it was taken on by a professional outfit, Bookswarm. The “Sophie Botros” link in this bullet is now to their version of the site. My site looked like this.
- I created and / or ran a multitude of other bridge websites, but as of January 2018 I have either handed them over or mothballed them9:-
- I established the Hutton Bridge Club Website in 11Q4 using the standard Bridgewebs service, but with a couple of competitions using my own routines. This was handed over in 15Q3. I briefly took it on again in March 2020. It needed a spring clean, but I was waiting to see whether the club (and its members) survived the coronavirus pandemic, and I handed it back on that account. The club went on-line only, but has now closed for lack of support.
- In 15Q1, I took over the support and development of the Essex Contract Bridge Association (ECBA) website, which also uses Bridgewebs, but is very much larger. I wrote a lot of code10 to make this job less tedious. The site was handed over in 17Q4.
- For several years, I collected data11 on bridge activity in the Billericay/Brentwood area (initially needed for a project to set up a new consolidated club) by “scraping” data off web pages, consolidating it into a database and modelling it in various ways.
- I used this data to generate websites with a multitude of ladders for small clubs (8 small Essex Bridge websites). These are now mothballed, with the exception of the Old House one noted above.
- I created and maintained a new website for the First Class Bridge Academy, giving it “small clubs” ladders (8 further “small sites” for Bernie) as these were easy to maintain with little intervention.
- I created a website12 for displaying the textual and grammatical analyses and appendices of Pete’s PhD on the Acts of the Apostles: Acts: Test Site.
Summary of Progress during April - June 2023
Plans for the Near Future
- I spent 91.25 hours in 23Q2 on this Project, or related work (267 hours YTD, where for "YTD" - Year to Date - I mean the (academic) year that commenced in October 2022). That's 101.6% of the planned effort (108.4% YTD). Overall, 12% of my Project effort in the Quarter was directed towards this project (making 13.7% YTD) - as against 11.9% planned (12% YTD).
- As can be seen, I spent round about my time allocation on this project last Quarter. Unfortunately, almost all of this was bridge-related: either Club sites or bridge-related elements of my own site.
- I’ve still not completed an important project to reduce the of my Web_Generator_Performance database, which holds all the links between the various objects on my website. It’s now in a mess, and the Spider function no longer works. I intend to sort this out in the coming quarter.
- I continued to develop and run the function within my Bridge database to record the NGS rankings of all players in Essex and Suffolk (using screen-scraping from the EBU website) on a day-by day basis. This is for two reasons:
- To reconcile the NGS calculations, knowing all the (non-hidden) NGS ratings of the participants.
- To get an indication of which days of the week – and how often – people play NGS-rated bridge in these difficult post-pandemic times.
- I started a project to translate numerals in my Ling databases. Hopefully I’ll complete this next Quarter.
- Completed items included:-
- Own Website:
- Toshiba Laptop - Coffee over keyboard! Fixed - disabled inbuilt keboard + installed USB keyboard.
- Enhanced comparative Ling database - Added Polish to Ukrainian vs Russian
- Ling database: Reformated webpages for iPhone
- Other Websites:
- Revived Small Bridge Sites
- Full details for 23Q2 are given below13:-
Website (Total Hours = 60.5)
Website Others (Total Hours = 30.75)
- Website - Bridge Development (Total Hours = 8)
- Bridge - Website - Automate NGS Data Gathering & Recalculation (1.25 hours)
- Bridge - Website - Determine Playing Days (5.25 hours)
- Bridge - Website - Results Database - Develop NGS Par Report (1.5 hours)
→ See "Software Development - Website - Development" (8 hours)
- Website - Bridge Maintenance (Total Hours = 22.25)
- Bridge - Website - NGS Data Gathering & Recalculation (15.75 hours)
- Bridge - Website - Results Database Maintenance (6.5 hours)
→ See "Admin - Bridge - Admin" (22.25 hours)
- Website - Development (Total Hours = 13.25)
- Website - Generator - Enhance comparative Ling database - Ukrainian vs Russian - Add Polish (0.75 hours)
- Website - Generator - Ling database: Reformat webpages for iPhone (2 hours)
- Website - Generator - Ling database: Translate Numerals (6.25 hours)
- Website - Generator - Set up comparative Ling database: Data Entry (4 hours)
- Website - Generator - Use Edge for Spider (0.25 hours)
→ See "Software Development - Website - Development" (13.25 hours)
- Website - Education
- Website - Infrastructure (Total Hours = 7.25)
- Investigate & Install Bitwarden Password Manager (0.5 hours)
- iPhone Photos: Move from My Photo Stream to iCloud (0.25 hours)
- Kaspersky replacement by Norton on own & Julie's laptops + Desktop (0.25 hours)
- Microsoft Windows / MS Office / Norton - Releases, Bugs & Periodic Re-boots (1.75 hours)
- PC Backups / OneDrive (1 hour)
- Researching, Purchasing & Installing Keyboard & Speakers for Desktop PC (0.75 hours)
- Toshiba Laptop - Coffee over keyboard! Fix - disable inbuilt keboard + install USB keyboard (2.75 hours)
→ See "Admin - Website - Admin & Maintenance" (7.25 hours)
- Website - Maintenance (Total Hours = 7.25)
- 23Q1 Status Reports (1.75 hours)
- Website - Generator - WebRefs - Manual / Automatic URL Checks & Fixes (0.75 hours)
- Website - Home Page, Etc, Maintenance (3.75 hours)
- Website - Periodic Full Regeneration (1 hour)
→ See "Admin - Website - Admin & Maintenance" (7.25 hours)
- Website Others - ECBA Maintenance
- Website Others - Mayflower DBC Maintenance
- Website Others - Mountnessing DBC Maintenance
- Website Others - Small Bridge Sites
The Plan below is taken automatically from the Priority 1 items on my Development Log, as published in my Outstanding Developments14 Report. I’ve had to reduce my weekly time allocation to 5 hours.
- Own Website: Priority 1 Items By Category:-
- Complete XRef-re-engineering project:-
- Ensure all links and link-pages use the new XRef table, and pension off the old tables where appropriate.
- Check all link-types still work and fix any errors.
- Complete the auto-triggering of regeneration of “associated” link pages.
- Fix Bug whereby PaperSummary pages seem to have “Works-” and “Books/Papers-” Citings that refer to the same link-pages.
- Complete Documenting15 the process!
- Document16, repair & update my Website site-map
- Enable use of Snippets for long Notes and Abstracts
- Review effectiveness of hyperlinking method in the light of PhD and Philosophy of Religion experience.
- Where possible, use ID rather than NAME for in-page hyperlinks
- Import Author Narratives from Aeon (and other sources) into the Authors table
- Determine why flag on directories "not to be backed up" gets reset
- Document and enable regular running of the de-duplication function ('pruning') of the backup disk. Keep a full record of deleted duplicates.
- Investigate historic error messages
- Investigate Record-count discrepancies:-
- How do website files work as far as counts are concerned?
- Why aren't they recorded in Backup_History, nor the fact that the website was backed up?
- Different counts depending on whether new or old laptop is backed up. Investigate 63k discrepancy - lower on new laptop.
- Review architecture to improve performance; Need to document17 first
- Investigate whether multiple Subject/Topic/Subtopic usage leads anywhere (ie. are just the first (of 3) actually used). Fix anything amiss.
- Reformat the BookCitings and PaperCitings pages:-
- Detail PaperCitings Pages: Include only useful information on the detail pages; but if there are multiple links from the same object, include them on the same line as 'extra links' as in BookCitings (copy the code: or, better, combine the two subs).
- Summary (Author Letter) pages: Include counts (as in Authors' Citations).
- Ensure uses the Cross_Reference table.
- Correct NGS Recalculation: Incorporate 3-year limit
- Convert Chess Consolidated Results page to ELO Grading
- Investigate the error reports from the Documenter, especially unused variables & queries.
- Provide Functional Documentation for Website Generator (using Notes)
- Read "PC Pro - Computing in the Real World".
- Read "White (Ron) & Downs (Timothy Edward) - How Computers Work: The Evolution of Technology".
- iCloud for Windows: Re-install & solve 'The upload folder for iCloud Photos is missing' problem. Try on new Laptop.
- Investigate Visual Studio Code
- Migrate from Chrome to Bitwarden for password management
- Move from My Photo Stream to iCloud by 26th July 2023
- Add "Note Alternates" to Note pages.
- Allow the option to concatenate Notes in the Printed version (ie. linearly embed them essay-style), rather than treating the hyperlinks as footnotes – but still keep the hyperlink & cross-referencing in place.
- For use as "disclaimers" - eg. for "Plug Notes".
- For Thesis / essays: the difficulty here is the need for linking passages to make the text run smoothly.
- As part of the Cross-Referencing project, check out the consistent treatment of Note 87519, which should be universally ignored. Recently, links to it appeared on web pages Book_Summary, BookPaperAbstracts and BooksToNotes, as a Note referencing a Book. The critical item was a row on the Note_Book_Links table.
- Bug: "++ NP ++" links to previously linked note or Note 0 and causes type mismatch in Reference_Notes for embedded Notes in Papers. Example20!
- Determine why all my Status Reports from September 2021 are missing from Notes_Archive. Seek to recover them from backups.
- Develop a method to create Papers for rows in Aeon_Files that have an associated Identity_Note_ID but no Paper_ID.
- Develop method for (manually) deleting otiose Archived Notes: ie. ensure links work for remaining Notes!
- Document and correct processing for Note References & Reading List21.
- Enhance PID Note, Book & Paper Usage22 to include a section listing works cited in my Thesis but not read
- Enhance comparative Ling database - Replace numerals by words in Vocabulary and Dialogues
- Investigate Note_Links: Section references seem to be incorrect
- PID Thesis References - De-duplicate / suppress and / or change sort order / aggregation for non-anthology book chapters
- Printable Notes: fix the bug whereby the “private” flag is round the wrong way.
- Split PDFs Available on Local Website23 page into alphabetical sub-pages
- The monthly regeneration process for Paper Abstracts is still taking too long. Monitor! Last run: 1.13 hours on 01/06/2023.
- Timeline software24: Add photos for Bakers' Coxes Farm Development Phase III
- Timeline software25: Add photos for Henry's Biography
- Timeline software26: Add photos for Holidays & Family History
- Develop software & procedure to make adding more content to the photos pages easier to undertake.
- Determine why Recalculation & Changed Book/Papers produce unneeded regeneration.
- Provide 'Quick' option for Recalculation (cmdRecalculate_Click) - exclude unnecessary items on request
- Analyse the results of the data collection exercise and design a plan of campaign to fix broken Internal links and prevent recurrence.
- Correct the code so the problems discovered by the Spider don’t recur.
- Delete 'orphan pages' that are never linked to, ie. Use the Spider to prune redundant pages28 automatically where possible.
- Fix the historical data where errors are uncovered by the Spider. An easier task now the site has a full-regen function.
- As revealed by Spider: Consolidated Site-Hits Test Webpage. Mountnessing\index.htm Link. Should be .shtml.
- Failed 'Printable Note' links not picked up by Spider: Especially in Blog
- Investigate reduction in size of Web_Generator_Performance database that holds Raw_Links.
- Suppress MsgBox for 'Near to 2Gb' database size check (just use Debug).
- One-off deletion of rather useless links: EMail, TheosStyle.css, TT_ICO.png, #Top, ...
- Remove code that generates these links, and check them.
- Investigate removal of website 'root' (C:\Theo's Files\Websites\Theo's Website\) from Raw_Link and Full_Link. This would save 200Mb, but needs careful checking - especially against the Directory_Structure and related tables and code.
- If the above 'root' change is to go ahead - one-off updates + code changes required.
- Use Edge rather than Internet Explorer for External Links Spider
- Change URL for Page names to something meaningful: For Notes, Papers and Books. Needs to be short!
- Convert site to HTTPS
- Look into Sistrix Smart29. Errors and warnings itemised are:-
- Duplicate content: seems to be variants on theotodman.com
- Title Tags: Empty, too long, identical
- Page Not Found
- Filesize in excess of 1Mb
- Meta-Description: Empty
- Few words on Page
- H1: Not used, used multiple times per page, identical across pages
- Pictures: Alt attribute missing
- Documentation & Bug-fixes: Phase 2
- Re-document the procedures in the light of recent changes.
- Resolve issues generated / revealed by the spider.
- Investigate - and fix where possible - broken links.
- Find a way of recording Missing / Unknown Webrefs (eg. Webrefs 8395 and 8342 in Archived Notes 1292 and 1317) other than debug.print: create table, then suppress message for known problems
- Investigate items flagged as defunct. Populate Defunct_Explanation in WebRefs_Table. Consider use of FairUse (Link (Fair Use)) for documents no longer available that I'd downloaded.
- Investigate WebRefs with Issue = 'URL Translated OK': does the translation really work? How?
- Other Websites: Priority 1 Items By Category:-
Summary of Progress to Date
This is hived off to various separate documents, which have now been harmonising and / or consolidated:-
- Summary of Progress to Date27.
- Outstanding Developments30,
- Functional Documentation31,
- A summary of time expended across the years developing my website33 is at "Software Development - Website - Development".
In-Page Footnotes:Footnote 4:
- Well, in a sense, I’ve missed the boat as I’m now putting effort into my research, though in an informal basis, so will need to continue with both projects in parallel.
- This was always likely to be necessary, as new features will always arise in use. It’s a prototype methodology, after all.
- This is very tedious to produce and consequently is both incomplete and out of date.
- This is much more fun, as it’s a purely technical task.
- I’ve written a vastly-improved general-purpose technical documenter for MS Access.
- It’s a shame to abandon the “mini websites” with all their ladders, as it’s rather well done.
- However, I couldn’t waste time on these after I’d abandoned bridge.
- In particular, for the ECBA “Victor Ludorum” competition.
- I cannot hand any of this code over, so the tedium will return, though not to me!
- I had agreed to share this data sometime early in 2018
- But will wait until asked again, as I doubt it’ll be of any real use to anyone.
- It used to exist in two versions, live and test.
- Pete decided not to renew the license for the live site, now it has achieved its purpose, so only the test site remains.
- Note that where fixes or small enhancements are made to a previously “completed” development, I don’t announce it again against the list of “completed” items above, though the work appears in the full list for the quarter.
- Note that Backup_Prune_Ctrl deletes (relevant) pages that weren't regenerated in the last full site-regen, but this isn't the same thing.
- See Sistrix
- This used to be called Optimizr, see Optimizr (Defunct) (which now auto-forwards to Sistrix).
- A quick look doesn’t show it to be an obvious scam, but I need to double-check.
- An unsolicited analysis of my site turned up monthly from Optimizr from January 2015 to October 2017, listing a large number of “problems” that I think I know about, but which are in the queue to address.
- It restarted in February 2018, under the Sistrix name (this seems to have been associated with Optimizr since November 2015).
- The free version of this software is restricted to 1,000 pages, which is a very small proportion of my Site, though I may be able to point it to difference base-URLs.
- But I do need to address the problems validly itemised, and a sub-set is still useful.
- As distinct from developing other peoples’ websites – time which is also recorded against this project, but not against this task.
Table of the Previous 12 Versions of this Note: (of 88)
Summary of Notes Referenced by This Note
To access information, click on one of the links in the table above.
Summary of Notes Citing This Note
|Internet Technology and Philosophy
||King's Maths Questions
||Simon - T1S1T1
||Status: Consciousness Studies (2023 - June)
||Status: Priority Task List (2023 - September)
|Status: Summary (2023 - June), 2
||Status: Summary Task List (2023: July - September)
||Status: Summary Task List (YTD: 22Q4 - 23Q3)
||Theo Todman's Philosophy Page
||Theo Todman's Website Maintainance History
|Website - Progress to Date (2023 - September), 2, 3, 4, 5
||Website Generator Documentation - Control Page
||Website Generator Documentation - Functors, 2, 3, 4, 5, 6, 7
To access information, click on one of the links in the table above.
Authors, Books & Papers Citing this Note
||Website - Development
Text Colour Conventions
- Blue: Text by me; © Theo Todman, 2023