Theo Todman's Web Page - Notes Pages
Status: Web-Tools (2020 - March)
(Text as at 04/04/2020 00:14:24)
(For earlier versions of this Note, see the table at the end)
Rationale for this Project
- This Project was alluded to briefly in a footnote on research methodology in my original Research Proposal1 under the head Research - Internet Technology2. When last at Birkbeck, I wrote a more extensive paper3 defending the Project and describing its rationale. Now that my PhD is in suspense, I have decided to take this Project further. There’s a lot to do: still quite a few items on the “wish list”. It is fairly critical as an enabler for my research, so I need to get a move on as I want it all out of the way before I re-start4 formal research.
- For documentation on my website (currently password protected) follow the links below:-
- Functional5 Documentation6.
- Technical7 Documentation.
- Other Websites8
- I’ve created and continue to maintain a small website for a music group Julie and I attend – the Enigma Ensemble.
- I established the Hutton Bridge Club Website in 11Q4 using the standard Bridgewebs service, but with a couple of competitions using my own routines. This was handed over in 15Q3, but I’ve taken it on again as of March 2020, not that there’s currently much to do. It needs a spring clean, but I’m waiting to see whether the club (and its members) survive the coronavirus pandemic.
- In 16Q3 I created the Mountnessing Bridge Club Archive website, using the vast bulk of the pages from their legacy site, as the club had moved to Bridgewebs and lost its historical data. As of March 2020 I’ve taken over the aforementioned Mountnessing Bridge Club website itself.
- Sometime around 2005, I created a website for Dr. Sophie Botros, one of my supervisors at Birkbeck, but we then lost touch and it got maintained (very badly) by some desktop support outfit. In 15Q2 I took it back on again and spruced it up a bit, and maintain it periodically until 19Q3, when it was taken on by a professional outfit, Bookswarm. The “Sophie Botros” link in this bullet is now to their version of the site.
- I created and / or ran a multitude of other bridge websites, but as of January 2018 I have either handed them over or mothballed them9:-
- In 15Q1, I took over the support and development of the Essex Contract Bridge Association (ECBA) website, which also uses Bridgewebs, but is very much larger. I wrote a lot of code10 to make this job less tedious. The site was handed over in 17Q4.
- For several years, I collected data11 on bridge activity in the Billericay/Brentwood area (initially needed for a project to set up a new consolidated club) by “scraping” data off web pages, consolidating it into a database and modelling it in various ways.
- I used this data to generate websites with a multitude of ladders for small clubs (Essex Bridge Results). These are now mothballed.
- I created and maintained a new website for the First Class Bridge Academy, giving it “small clubs” ladders (Bernie's Ladders Archive) as these were easy to maintain with little intervention.
- In 16Q3 I created the Mountnessing Bridge Club Archive website, using the vast bulk of the pages from their legacy site, as the club had moved to Bridgewebs and lost its historical data.
- I created a website for displaying the textual and grammatical analyses and appendices of Pete’s PhD on the Acts of the Apostles. It exists in two versions: Acts: Live Site and Acts: Test Site.
Summary of Progress during January - March 2020
Website (Total Hours = 76.5)
- I spent 99.5 hours in 20Q1 on this Project, or related work (194.25 hours YTD, where for "YTD" - Year to Date - I mean the (academic) year that commenced in October 2019). That's 111.1% of the planned effort (107.4% YTD). Overall, 14.6% of my Project effort in the Quarter was directed towards this project (making 14.3% YTD) - as against 11.9% planned (11.8% YTD).
- Completed items included:-
- Own Website:
- As revealed by Spider: Link to PaperCitings failing (2 items) - Timing issue
- As revealed by Spider: Philosophers_Index_List_OA.htm. Author Names Scrambled. 133 items. Dud page (Ontological Argument) deleted.
- Restrict jump-table at top of Authors pages to relevant items, and include the item counts.
- As revealed by Spider: Bridge_Code_Archive.htm (14 items) & Bridge_Code_Archive_Recent.htm (5 items)
- Solve the problem of the size limitation for Notes by allowing their concatenation.
- As revealed by Spider: Old links from Archive Notes to (non-existent) Archived Printable Notes. 42 items.
- As revealed by Spider: Blog12 Achived Notes - sundry links to Printable Notes. Live Note OK. 97 items outstanding.
- As revealed by Spider: #Reference links: fail for Printable Notes. 24 items.
- As revealed by Spider: Blackberry links from archived Note 1010 (Maintenance Dashboard). 38 items.
- Make use of Comments for large Abstracts seamless by appending Comments (if > 5k) to Abstracts.
- As revealed by Spider: WebLinks_Tester_Map_3.htm (etc). Refs failing. 51 items outstanding. res://ieframe.dll/ in Returned_URL.
- As revealed by Spider: WebLinks_Tester_Brief_12.htm (etc). Refs failing. 41 items. res://ieframe.dll/ in Returned_URL.
- WebRefs checker failing for Aeon - if a link fails, I get the message 'The page you were looking for could not be found (404)', but since the URL doesn't change, I don't spot it. This may happen for other sites as well. Fixed for Aeon - watch out for others.
- As revealed by Spider: WebLinks_Tester_4.htm. Refs failing. 32 items. res://ieframe.dll/ in Returned_URL.
- Other Websites:
- Full details for 20Q1 are given below13:-
Website Others (Total Hours = 23)
- Website - Bridge Development (Total Hours = 4.25)
- Bridge - Website - Development (0.25 hours)
- Bridge - Website - Development - Fix 'Code Changes" reports (1.25 hours)
- Bridge - Website - Development - Query from Tony Heyes re Bridgewebs leads scraping (1.75 hours)
- Bridge - Website - Development - Update Documentation (1 hour)
→ See "Admin - Bridge - Admin" (4.25 hours)
- Website - Bridge Maintenance
- Website - Development (Total Hours = 47.25)
- Website - Chess Results & Games Pages - Create & Maintain (0.5 hours)
- Website - Development - Remove ZoomSearch from Search Page and Code (0.5 hours)
- Website - Generator - Add Aeon Abstract link to Aeon Webref items in Summary task List reports (1.25 hours)
- Website - Generator - Append long Comments to Abstract (3.25 hours)
- Website - Generator - Author Citation list improvements (3.5 hours)
- Website - Generator - Automate Aeon Page output (0.75 hours)
- Website - Generator - Automate Aeon Page output - Documentation (3 hours)
- Website - Generator - Correct Functor_16 to remove hyperlinks from Title (0.75 hours)
- Website - Generator - Correct sequence of Aeon Webref items in Summary task List reports (0.5 hours)
- Website - Generator - Development Planning (1 hour)
- Website - Generator - Fixes re Broken Links revealed by Spider (12.5 hours)
- Website - Generator - Investigate & Fix WebRefs checker for Aeon, etc - 404 check not working (6.5 hours)
- Website - Generator - Investigate anomalies in YTD Task List (Functor_01) in Quarterly Reports (0.5 hours)
- Website - Generator - Investigate anomalies in YTD Task List (Functor_08) in Quarterly Reports (0.75 hours)
- Website - Generator - Notes Concatenation (3.25 hours)
- Website - Generator - Summary Task List items error (0.75 hours)
- Website - Generator - WebRefs checker - Improve efficiency (0.75 hours)
- Website - Periodic Full Regeneration - Error copying Private Note (1.25 hours)
- Website - Update 'Photos' Page to link to Coxes Farm Photo Pages (5.75 hours)
- Website - ZoomSearch database refresh - Failed, so dismantle (0.25 hours)
→ See "Software Development - Website - Development" (47.25 hours)
- Website - Education
- Website - Infrastructure (Total Hours = 5.75)
- FileZilla FTP Failing with 530 Critical Error (excessive connection rate) (1.5 hours)
- Microsoft Windows 10 / MS Office - Kaspersky complaints (0.25 hours)
- Microsoft Windows 10 / MS Office - Releases, Bugs & Periodic Re-boots (1.25 hours)
- Old routers, etc, clear-out (0.75 hours)
- PC Backups / OneDrive (0.75 hours)
- Re-installing PdfElement (0.25 hours)
- Renew Kaspersky on own & Julie's laptops (0.25 hours)
- Sky Q Order & Installation + new TV (0.75 hours)
→ See "Admin - Website - Admin & Maintenance" (5.75 hours)
- Website - Maintenance (Total Hours = 10.75)
- 19Q4 Status Reports (1.25 hours)
- Updated my Home page (1.25 hours)
- Updated my 'Websites maintained' page (0.25 hours)
- Website - Generator - WebRefs - Manual / Automatic URL Checks & Fixes (3.25 hours)
- Website - Periodic Full Regeneration (3.25 hours)
- Website - Run Web Spider (1.5 hours)
→ See "Admin - Website - Admin & Maintenance" (10.75 hours)
- Website Others - Enigma Ensemble
- Website Others - Hutton DBC Maintenance
- Website Others - Mountnessing DBC Maintenance (Total Hours = 18.25)
Plans for the Near Future
The Plan below is taken automatically from the Priority 1 items on my Development Log, as published in my Outstanding Developments14 Report. I’ve increased the weekly allocation marginally to 8 hours.
- Own Website: Priority 1 Items By Category:-
- Complete XRef-re-engineering project:-
- Ensure all links and link-pages use the new XRef table, and pension off the old tables.
- Check all link-types still work and fix any errors.
- Complete the auto-triggering of regeneration of “associated” link pages.
- Fix update bug in Convert_Webrefs.
- Fix Bug whereby PaperSummary pages seem to have “Works-” and “Books/Papers-” Citings that refer to the same link-pages.
- Review effectiveness of hyperlinking method in the light of PhD and Philosophy of Religion experience.
- Where possible, use ID rather than NAME for in-page hyperlinks
- Completely re-engineer Authors processing to use IDs rather than Names:-
- Authors' pages to be identified by Author_nnnnn, where nnnnn is the (numeric) Author_ID.
- Directory-structure in thousands: Author_mm.
- Use Author_ID in all Tables.
- Need to document at the same time!
- Needed because of difficulties transferring special characters in file names.
- Enable "alternate names" for identically-named authors (to avoid middle-initial = X).
- Investigate Record-count discrepancies:-
- How do website files work as far as counts are concerned?
- Why aren't they recorded in Backup_History, nor the fact that the website was backed up?
- Different counts depending on whether new or old laptop is backed up. Investigate 63k discrepancy - lower on new laptop.
- Review architecture to improve performance; Need to document first
- Investigate whether multiple Subject/Topic/Subtopic usage leads anywhere (ie. are just the first (of 3) actually used). Fix anything amiss.
- Develop auto-reconciliation routines vs EBU results download
- Investigate the error reports from the Documenter, especially unused variables & queries.
- Provide Functional Documentation for Website Generator (using Notes)
- "Sitepoint (Learnable) - Sitepoint Learnable Web Development Courses": Plan what to do (and with the eBooks in my possession).
- Read "PC Pro - Computing in the Real World".
- iCloud for Windows: Re-install & solve 'The upload folder for iCloud Photos is missing' problem. Try on new Laptop.
- Add "Note Alternates" to Note pages.
- Allow the option to concatenate Notes in the Printed version (ie. linearly embed them essay-style), rather than treating the hyperlinks as footnotes – but still keep the hyperlink & cross-referencing in place.
- For use as "disclaimers" - eg. for "Plug Notes".
- For Thesis / essays: the difficulty here is the need for linking passages to make the text run smoothly.
- As revealed by Spider: Sundry uncategorised. Refs failing. 30 items.
- Investigate Note_Links: Section references seem to be incorrect
- Printable Notes: fix the bug whereby the “private” flag is round the wrong way.
- Suppress the publication of the Printable versions of Temp Notes
- Develop software & procedure to make adding more content to the photos pages easier to undertake.
- Timeline software: Add photos for Holidays
- Determine why Recalculation & Changed Book/Papers produce unneeded regeneration.
- Full Website Regeneration is now taking 18.5 hours:-
- This is on my new laptop - it was taking 36 hours on my old laptop. Investigate why so, and improve performance!
- The end-to-end time on the new laptop is no better because the process pauses mid-way when the system goes to sleep overnight. I've set the power mode to 'Presentation', so we'll see if this fixes things.
- I got a copy error for private Note for Note 133. This contributed to extended run-times on the new laptop. Hopefully just a set-up problem.
- Analyse the results of the data collection exercise and design a plan of campaign to fix broken Internal links and prevent recurrence.
- Correct the code so the problems discovered by the Spider don’t recur.
- Delete 'orphan pages' that are never linked to, ie. Use the Spider to prune redundant pages15 automatically where possible.
- Fix the historical data where errors are uncovered by the Spider. An easier task now the site has a full-regen function.
- The Spider was generating WebRefs. Procedurally, this ought not to have been possible.
I've re-opened the case!
- The major problem turned out to be because unprocessed16 URLs got added to the end of the last WebLinks_Tester_Brief page, which then got Spidered. I've stopped this happening, so hopefully the problem will not recur. The fix was made in 18Q2.
- However, 4 other creations appeared - dated 18/05/18 - from the run of 10/07/18. The creation date was from the previous spider run, but the IDs show that they were produced in the latest run.
- Look into Sistrix Smart17. Errors and warnings itemised are:-
- Duplicate content: seems to be variants on theotodman.com
- Title Tags: Empty, too long, identical
- Page Not Found
- Filesize in excess of 1Mb
- Meta-Description: Empty
- Few words on Page
- H1: Not used, used multiple times per page, identical across pages
- Pictures: Alt attribute missing
- As revealed by Spider: WebLinks_Tester_Full_Map.htm (etc). Refs failing. 184 items. res://ieframe.dll/ in Returned_URL. 4 Items remaining.
- Documentation & Bug-fixes: Phase 2
- Re-document the procedures in the light of recent changes.
- Resolve issues generated / revealed by the spider.
- Investigate - and fix where possible - broken links.
- Reformat WebLinks_Tester.htm, WebLinks_Tester_Map.htm, WebLinks_Tester_Full.htm & WebLinks_Tester_Full_Map.htm
- Clarify 'truncated': Display, not link
- Allow more space for 'link returned', 'issue' and 'display text'
- The 'As Above" lines waste space. Only for Notes Archive? Consolidate onto single second line.
- Reformat WebLinks_Tester_Brief: Allow more space for 'link returned', 'issue' and 'display text'
- Other Websites: Priority 1 Items By Category:-
Summary of Progress to Date
This is hived off to various separate documents, which have now been harmonising and / or consolidated:-
- Summary of Progress to Date18.
- Outstanding Developments19,
- Functional Documentation20,
- A summary of time expended across the years developing my website21 is at "Software Development - Website - Development".
In-Page Footnotes:Footnote 4:
- Well, in a sense, I’ve missed the boat as I’m now putting effort into my research, though in an informal basis, so will need to continue with both projects in parallel.
- This was always likely to be necessary, as new features will always arise in use. It’s a prototype methodology, after all.
- This is very tedious to produce and consequently is both incomplete and out of date.
- This is much more fun, as it’s a purely technical task.
- I’ve written a vastly-improved general-purpose technical documenter for MS Access.
- It’s a shame to abandon the “mini websites” with all their ladders, as it’s rather well done.
- However, I couldn’t waste time on these after I’d abandoned bridge.
- In particular, for the ECBA “Victor Ludorum” competition.
- I cannot hand any of this code over, so the tedium will return, though not to me!
- I had agreed to share this data sometime early in 2018
- But will wait until asked again, as I doubt it’ll be of any real use to anyone.
- Note that where fixes or small enhancements are made to a previously “completed” development, I don’t announce it again against the list of “completed” items above, though the work appears in the full list for the quarter.
- Note that Backup_Prune_Ctrl deletes (relevant) pages that weren't regenerated in the last full site-regen, but this isn't the same thing.
- These are URLs that were used in web pages but hadn't yet been converted to the +WnnnW+ format, so appeared at the end with no WebRef ID.
- See Sistrix
- This used to be called Optimizr, see Optimizr (which now auto-forwards to Sistrix).
- A quick look doesn’t show it to be an obvious scam, but I need to double-check.
- An unsolicited analysis of my site turned up monthly from Optimizr from January 2015 to October 2017, listing a large number of “problems” that I think I know about, but which are in the queue to address.
- It restarted in February 2018, under the Sistrix name (this seems to have been associated with Optimizr since November 2015).
- The free version of this software is restricted to 1,000 pages, which is a very small proportion of my Site, though I may be able to point it to difference base-URLs.
- But I do need to address the problems validly itemised, and a sub-set is still useful.
- As distinct from developing other peoples’ websites – time which is also recorded against this project, but not against this task.
Table of the Previous 12 Versions of this Note: (of 76)
Summary of Note Links from this Page
To access information, click on one of the links in the table above.
Summary of Note Links to this Page
To access information, click on one of the links in the table above.
Authors, Books & Papers Citing this Note
||Website - Development
Text Colour Conventions
- Black: Printable Text by me; © Theo Todman, 2020
- Blue: Text by me; © Theo Todman, 2020