User:Inductiveload








 * works

Ask me to do things
I will need information:


 * Batch uploads
 * Downloads from Hathi Trust
 * Bulk moves to subpages
 * Move pages or indexes

Tools and scripts
User preferences and custom javascripts:


 * /common.js/
 * /monobook.css/
 * /Regexp toolbar.js/
 * /Running header.js/ - completes a rh template by copying the header from the last-but-one page and incrementing the number.
 * /Custom toolbar buttons.js/
 * /InlinePagenums.js/ - toggle display of inline pagenums
 * /ColourBackground.js/ - shade text boxes and background in edit mode to avoid eye strain.
 * /Visibility.js/ - custom visibility switching
 * /index preview.js/ - preview index page thumbnails with alt-click on the page-list

Commons scripts

 * commons:User:Inductiveload/basic upload templates.js: add buttons for preloading book templates on the Basic Upload Form

Popups Reloaded
Popups, but way better. /popups_reloaded/

/popups_reloaded.js/

/popups_reloaded.css/

Quick Access
Keyboard-driven tool access Documentation

/quick_access.js/

Cleanup (Alpha)
This is a much more in-depth version of User:Samwilson/PageCleanUp.js that includes hundreds of regexes for scannos that have unambiguous or almost-certain corrections. For example, no work in English ends in : this is almost certainly

This is an alpha-level tool. Configuration, especially, is likely to change.

Documentation

/cleanup.js/

This tool will work with default configurations, but it is more reliable with configurations.

Page Carousel
Quick buttons to load the next/previous pages of a book.

Documentation

/page carousel.js/

Save/Load Actions
Run Javascript actions on page save or page load. Can be used to implement custom text transforms, for example from  to.

Documentation

/save load actions.js/

This tool probably needs configuration in your JS. Just adding it won't do anything use. Consult the documentation for more information.

Preview markup
/show_markup.js/

/show_markup.css/

Maintenance Wizard and Replacer
Perform maintenance without going into edit mode. /maintain.js/

Jump to file
Add a button to go to the book file at Commons from the Index or Page namespace, and to the transcluding page from the Page namespace Documentation

/jump to file.js/

/jump to file/load.js/

MiniPane
Add a mini image page in the Page namespace for keeping the proofread text nearer the input box. Documentation

/MiniPane.js/

IaUploadPopup
Add a small popup to assist uploading works at the IA via IA-Upload Documentation

/IaUploadPopup.js/

Scan Transcludes
Show, in the pagelist grid, which pages are transcluded and which are not, colouring the pages according to expected transclusion status.

/scan transcludes.js/

ActivePageAlert
Display an icon if a page has been edited recently, with the ability to vary the definition of "recently" on a per-other-user basis. Documentation

/ActivePageAlert.js/

Show an indicator when a script loads
I use this to check my local script is loading

Functions
Some extra functions that might be handy for other scripts:


 * /Roman numerals.js/ - couple of functions that might be handy

The rant zone
What does a $130 million budget not get you?


 * T95878 (filed 2015) There's still no mobile editing for Wikidata, after 9 years. Seriously.
 * T259183 The totally-not-beta UI is also still trash on "narrow" screens.
 * Fixed after 7 months.
 * T288980 Codex might one day get usable by general folk?
 * OOUI is now not only barely documented and horrific to actually use (OK actually it's pretty good from the server side), but also no-one cares because it's going to get "one day, promise" replaced by WVUI
 * VisualEditor (aka the 2017 editor!) is still completely useless, and not just at Wikisource
 * Literally every data model is still work in progress for both WD and SDC and no one cares.
 * T121646 (filed 2016) There's still no API for evicting data from local storage.
 * There's still no guidance for best practices for gadgets. None.
 * The chat ecosystem is completed fragmented into siloed commercial apps and trying to bridge them together is, at best, shunted to "Volunteer time". Just pay for a darn Matrix homeserver like every other FOSS project and stop pretending like everyone will use IRC in 2021 instead of leaving for Discord and Telegram. Even the WMF doesn't really use IRC, it hides its machinations on Slack so no-one can see what the hell is going on.
 * Speaking of, why does Community Tech not work in the open?

But they still need your cash for...something? And the begging banner will use every dark pattern they can fit in there.

And a textbook example of the Mrs Micawber principle : Tasks created in (2021-12): 1617 Tasks closed in (2021-12): 1325 Open and stalled tasks in total: 49056

Median age in days of open tasks by priority:

Unbreak now: 24 Needs Triage: 726 High: 1033 Normal: 1601 Low: 2210 Lowest: 2290

The counter-rant zone (aka why maybe it's not all bad)

 * The Special:APISandbox is brilliant
 * The Wikipedia Library is brilliant
 * Commons providing basically infinite storage is brilliant
 * Toolforge and WMCS is brilliant
 * Site Reliability knows where their towels are
 * An API that allows Pywikibot can't be all that bad

Maintenance and reports
Below are lists of pages in Wikisource which are useful for various purposes. All of these could be out of date. If you really need up-to-date reports, just leave me a note, and I will do it as soon as I can.


 * /templates/ A list of all templates in use on enWS, along with links and usage counts.
 * /wikisource pages/ A list of all Wikisource namespace pages.
 * /portals/ A list of all Portal pages.
 * /ws-portal redirects/ A list of all Wikisource pages which redirect to Portal pages. No pages should link to these.
 * /ws-wp no backlink/ A list of Wikisource pages linking to Wikipedia pages which do not link back here.
 * /false root pages/ A list of pages that should be subpages but aren't.


 * /site-css-js/: in-progress CSS tidying-up - look here for CSS and/or JS moved out of MediaWiki namespace (rather than being deleted)


 * /SPARQL/: Useful SPARQL queries.
 * /HTML processing/: Useful HTML transforms for extracting data


 * Live thumbnail generation times


 * 19363685: Proofread indexes not transcluded
 * 19363689: Validated indexes not transcluded

Bot activities
I operate a bot, InductiveBot, which performs minor maintenance tasks. It is based on pywikipedia and is quite flexible. If you have a specific request, please let me know on my talk page, and I'll see what I can do!


 * /InductiveBot information/, containing information about custom scripts runnig over pywikipedia, etc.

Run extension linter
docker-compose exec mediawiki bash -c "cd extensions/ProofreadPage && composer install" docker-compose exec mediawiki bash -c "cd extensions/ProofreadPage && composer test" docker-compose exec mediawiki bash -c "cd extensions/ProofreadPage && composer phan"

Run extension parser tests
docker-compose exec mediawiki php tests/parser/parserTests.php --file=extensions/ProofreadPage/tests/parser/proofreadpage_pages_pagelist.txt Or to run all the tests in a directory:

docker-compose exec mediawiki sh -c 'find extensions/ProofreadPage/tests -name "*.txt" -exec php tests/parser/parserTests.php --file={} \;'

Run extension unit tests
docker-compose exec mediawiki php tests/phpunit/phpunit.php extensions/ProofreadPage/tests/phpunit

Run linter
npm run-script test

PDF munging
For meanings of /P, /S, /St see §8.3.1 Page Labels in the PDF Spec qpdf --json --object-streams=disable ss.pdf | jq '[ .pages[] | .label ]'
 * Extract page labels from a PDF → JSON

Pywikibot
./pwb.py imagetransfer -site:commons:commons -tosite:wikisource:en -keepname -force_if_shared "File:Foobar.djvu"
 * Transfer an image from Commons to enWS

Random links

 * mw.hook fire points
 * https://codesearch.wmcloud.org/search/?q=mw%5C.hook(.*)%5C.fire&i=nope&files=&excludeFiles=&repos=


 * Current lag
 * https://en.wikisource.org/w/api.php?action=query&titles=MediaWiki&format=json&maxlag=-1


 * All wikis edit firehose
 * https://event-streams.toolforge.org/


 * Wikisource Image Uploader uploads at Commons:
 * https://commons.wikimedia.org/wiki/Special:RecentChanges?tagfilter=OAuth+CID%3A+2348


 * Zuul Status (Gerrit → Jenkins pipeline)
 * https://integration.wikimedia.org/zuul/


 * Thumbor thumbnails - Grafana dashboard:
 * https://grafana.wikimedia.org/d/Pukjw6cWk/thumbor


 * Upload files to Phabricator
 * https://phabricator.wikimedia.org/file/upload/

Useful things to share
These are some useful scripts I have hacked together. I guarantee nothing! They are certainly not always neatly coded or structured, but they work for quick and dirty jobs.


 * /Script development/: general notes on JS script development
 * /Wikisource in Docker/: setting up a dev Wikisource in Docker


 * Universal batch image to DJVU converter. This script takes JPG, GIF, PNG, TIFF and anything else that Imagemagick can convert to PPM.
 * DJVU OCRing script which uses Tesseract to OCR and insert a text layer into a DJVU
 * Pagewise DJVU OCR extractor
 * Index page tabulator, creates tables of individual files for use in collecting files into an index page. See for example Index:The Complete Collection of Pictures & Songs by Randolph Caldecott.jpg.
 * Template usage tabulator. This script generates a table of all templates, along with the number of uses. Results can be found at User:Inductiveload/templates. Ask me if you want it regenerating, but bear in mind that it is a lot of requests to the server.
 * /Page namespace editor/ A simple script to decompose Page: namespace pages, perform operations on the header, footer, and body separately, and reupload.
 * Page shifter A script to shift a set of Page: pages within the same index, or move to a different index.
 * PDF page converter A shell script to convert a PDF to images (threaded and doesn't run out of memory and die after a few hundred pages like convert can do)
 * Image splitter A Python script to split images in two. This is useful if you have books scanned at two-page spreads.
 * Page concatenator A Python/Pywikipedia script to grab a bunch of pages and string them together. Good for assembling a complete text out of many chapter subpages prior to match and split.
 * Archive.org API How to use the Internet Archive S3-like API to upload large files, instead of the flaky web-client.
 * Move to subpage Python script to bulk-move pages to subpages. Documentation on how to drive it is at /Requests/Moves to subpages.
 * /Scripts/upload_image.py Uploads images for works based on a YAML data file

Tesseract retraining project
Very slowly, I am working on a retraining of some models for Tesseract: /Tesseract/

General Python scripts

 * /Integer to Roman numerals converter/ (eg. 11 -> XI)

GIMP scripts

 * The best one:
 * /Whiten-background.scm/ Gimp script to remove the background of an image. An adaptation of a script by Leonid Koninin. This is fairly harsh on some images, so use with care.


 * Less good ones:
 * /Remove-paper-texture.scm/ Gimp script to remove the paper background from a scan of a black and white image by a pretty brutal adjustment of the levels
 * /Remove-background-colour.scm/ Gimp script to remove a flat background colour from an image. This is a fairly brutal algorithm, use with care on delicate images. Essentially, this is just a "bundle" of "select colour, erase the colour, and desaturate. It'll likely be better to use one of the above scripts, but this one is fairly instructive from a script-writing point of view.

Bibliographic junk

 * http://www.rdaregistry.info/termList/

Data module ecosystem
None of these work fully. Yet.


 * Core modules
 * Module:Work data: Provides data about a work in general.
 * Module:Work data/properties: Core property ID maps
 * Module:Work license: provides information about the license of a work


 * Client modules
 * Module:Work link

One touch template wrapping with Autohotkey
If you use Autohotkey (and you should be), the following is a useful function that lets you wrap the current mouse selection in a template, which saves you having to paste in the contents.

F2 & s :: wrapTemplate("sc") ; small caps

wrapTemplate( name ) {   front :="" wrapTags( front, back) return }

wrapTags( front, back ) {   AutoTrim Off               ; Retain any leading and trailing whitespace on the clipboard. ClipSaved := ClipboardAll ; Save the entire clipboard so we can restore it when we're done clipboard =               ; clear the clipboard SendInput ^x              ; cut the selection to the clipboard ClipWait                  ; wait for the clipboard to contain something SendInput %front%%clipboard%%back% ; Output what was selected, surrounded by front and back Clipboard := ClipSaved    ; Restore the original clipboard ClipSaved =               ; Free the memory in case the clipboard was very large. return }

My requests
Works I'm keeping an eye out for.

Periodicals

 * Scientific Machinist (only have v. 10)
 *  (only have v. 62)
 * Cornell Studies in Classical Philology (lots on the IA)

Technical wishlist
Some things I'd like to see done (that isn't actual proofreading). Some of it is unimportant, some of it may be controversial and un-discussed, but would be nice to address and tighten up.


 * Get dynamic layouts to work for non scan-backed works and scrap prose and other hard-coded formatting.
 * Fix poem tags - only by having all lines as p or span-tags can we have hanging-indented continuation lines like 95% of all printed poems are. Stanzas should be divs . Might need a whole new tag in the poem extension, but might not be that hard?(???)
 * Train Tesseract specifically for 1700s-style printing esp. with long-s
 * Allow match-and-split to match to PDFs (since there are now ~1m PDFs on Commons)
 * Get para breaks working in OCR loading: T230415
 * Add common fonts:
 * Cursive: T166138
 * JUnicode: T173573
 * Sans Outline (and remove hacks like ℕ𝔼𝕎 𝕐𝕆ℝ𝕂)
 * Serif Outline
 * Maybe a better Polytonic greek?
 * Move MediaWiki:Proofreadpage_index_template to a module
 * Move Template:Header to module

Half-done

 * Ebook review process leading to categorisation, then... Category:Ready for export
 * Improve index autofill to fetch author links from Commons creator templates: Getting there: Mediawiki:Gadget-Fill Index.js
 * Fix FI which is invoking full-size images every single time. Merge with and/or deprecate large image.
 * Fix headers on mobile, the tabular structure is unfriendly on narrow screens (main header done, other namespaces pending main header module-ificaton). See header/main block

Done

 * Tool to convert import IA page list JSON: Mediawiki:Gadget-ImportPagelist.js
 * Fix the print CSS: centre is broken Fixed:
 * Fix page numbers in TOC begin c.f. T232477
 * ODPS catalogue of "exportable" ebooks for integration into e-readers: T270387. See Category:Ready for export for link