Commit graph

53 commits

Author SHA1 Message Date
sbailey 6aa4cdeba9 Linter Tag and Template search feature, UI and report code
* Tag and Template search is enabled using config variable
   'LinterUserInterfaceTagAndTemplateStage' and also checks for
   the linter table column 'linter_tag' to exist to protect the
   report code from error if the column is absent. As the linter
   table alter maintenance added both the linter_tag and
   linter_template at the same time, there is no reason to check
   both. The user interface code does not check for the field
   presence only the config variable.

 * This code depends on the recordLintJob code writing the tag
   and template data which is enabled by the config variable
   'LinterWriteTagAndTemplateColumnsStage' and also assumes the
   data migration maintenance script migrateTagTemplate.php has
   been run to populate linter error records created prior to
   the table alter and the write code being enabled.

Bug: T175177
Change-Id: I2f951dfcd34e3dc6ca17e8754cfaeba8baa3e835
2023-02-27 06:55:06 -08:00
sbailey 499a1c3796 Use new linter_namespace column and index in the Linter table
* This performance improvement patch uses the namespace from the
   new field 'linter_namespace' in the linter table instead of
   the 'page_namespace' in the page table. It checks for and
   requires the presence of the linter_namespace field in the
   linter table, as well as the config variable
   'LinterUseNamaspaceColumnStage' being set true.

 * If the linter_namespace field is present and aforementioned
   config variable is true, the code assumes that the config
   variable 'LinterWriteNamespaceColumnStage' is set true and
   recording the linter_namespace for new lint errors is
   active and the migrateNamespace.php migrate code has been
   run to migrate the page_namespace data into existing linter
   records that were created before the linter_namespace column
   existed and were left NULL during the table alter maintenance
   operation.

 * A follow on patch should remove the configuration variables and
   conditional code producing the final, refactored code dependent
   on the new namespace column index.

Bug: T299612
Change-Id: I4a1497d9e4dcd6a9a7befdaccf3e34c61694365d
2023-02-24 06:54:08 -08:00
Kunal Mehta eaf2d26db8 Remove API action=record-lint
The action=record-lint was a hack that allowed Parsoid/JS to send data
to MediaWiki to be stored in the linter database. Thankfully we no
longer need it in the glorious Parsoid/PHP world because it can write
directly to the database.

The API module, i18n messages and $wgLinterSubmitterWhitelist are all
removed.

Bug: T329992
Change-Id: Iba70e05a2e28f4ecd02101cff51993ebe65f19d0
2023-02-20 20:44:00 -05:00
sbailey d12bf639f6 Change linter maintenance scripts to use existing config varaibles
* Having separate config variables to enable the maintenance
   migrateNamespace and migrateTagTemplate scripts is duplicitous
   and should be shared with the write enable config variables.

Bug: T329342
Change-Id: I4cb453fc0678b065cb42a2ca59863da1ab9cdbe4
2023-02-14 09:43:54 -08:00
Kunal Mehta cb84f3a872 Link to detailed Special:LintErrors from action=info
action=info has a summary table of number of lint errors by category,
but we have richer information available via Special:LintErrors. If
there is a "Lint errors" section, provide a link below the table to
Special:LintErrors for the errors on this page.

Update ApiRecordLint for the new Hooks constructor and leave a FIXME
to eliminate the coupling.

Bug: T301374
Change-Id: Ic1fcf42b50d1392ac53201ceb256691133cf62ff
2023-02-06 20:41:45 -05:00
Kunal Mehta b5a7aaf7e0 Move LoadExtensionSchemaUpdatesHook handler to separate class
This hook is not allowed to have a service, so before we can add
services to the main Hooks class, it needs to be split out.

Change-Id: Ia7b4b8bf7c91ebb851c5de9f0f54f56b0993bf83
2023-02-06 20:36:45 -05:00
Umherirrender eb31c6c4dc Replace deprecated HTMLForm::addHeaderText
Bug: T325474
Change-Id: If1a4b14b6cec480f600065c913cb1f27260fbc5f
2022-12-27 12:48:42 +01:00
sbailey 350d677c5b Phase 3 of T175177: Migrate linter_params into new fields
* The migrate code is designed to perform a one-time update of
   linter_params JSON encoded template and tag information into
   the new discrete template and tag text fields for use as
   additional search criteria. The function can be restarted if
   it is interrupted.
 * It now uses configurable batching and sleep times between
   batches to allow the database to do other work and replication
   to occur without stressing infrastructure.
 * The migrate code is only called by test code and needs to be
   called one-time from a maintenance script.

Bug: T175177
Change-Id: Idc4ca88d4762bc7a3bcbc4e66c0f275562083867
2022-12-09 12:01:06 -08:00
sbailey 702ce215d0 Phase 3 migrate code for namespace column add to Linter table
* Migrates namespace info from the page tables page_namespace field
   to the new linter table field linter_namespace. This duplication
   of the namespace value was requested to greatly reduce the amount
   of database activity required by the linter search and reporting
   code.

 * This patch has been prepared as a dark launch patch enabled with
   config value LinterMigrateNamespaceStage and assumes that the
   Linter table has had the linter_namespace column added to it,
   and recording of the namespace field is already enabled and is
   populating the namespace column.

 * The migrate code now runnable from Linter/maintenance directory,
   using migrateNamespace.php, which will be deployed in a separate
   patch. The maintenance code creates an appropriate environment
   to call migrateNamespace( in Database.php.

Bug: T299612
Change-Id: I73cb80729d6a5a8716fe93164ad1e42e6958d672
2022-11-28 08:07:54 -08:00
Reedy 1c526b7443 Convert to HookHandler
Change-Id: Ib8ad9a58e781f7dad4b047d170a8f689802c4bad
2022-09-07 02:58:04 +01:00
sbailey b358b20dca Second phase of T175177: Adds template and tag to RecordLintJob
Bug: T175177
Change-Id: I59be7cabb80ace98da3c7f6f36a0d3d4f6b17d23
2022-08-22 12:47:01 -07:00
Arlo Breault 0f46ed3dbc Get config from services, not globals
If908c4dc99c966cde2981f9a03be38a577406a4e introduced the
wgLinterWriteNamespaceColumnStage global but didn't add it to the
extension.json

Change-Id: I8ee5849f67ddfd894a25425582b59404eb52aef2
2022-08-03 13:01:42 -04:00
sbailey 0e56c22277 Delete lint error records when content model changes from wikitext
* Determines if new content type is not wikitext and if so
   deletes all existing lint error records for that pageID.

Bug: T298343
Change-Id: I20fac9a0c901f3e7a5cc898566a4487fbe70798f
2022-02-25 13:22:10 -08:00
Arlo Breault 6898415d8d Disable "inline-media-caption" category
Note that this only removes the category from the special page and the
lints will still show up by page.  Those need purging separately or more
code to filter out disabled categories from the results.

Bug: T297443
Change-Id: Idfcf5ee5d69f5dc65100229aacb22137be475866
2022-01-16 12:25:28 -05:00
Arlo Breault e8a1b8218c Add linter category for "inline-media-caption"
Bug: T297443
Depends-On: I4cdbfe46bce7556d2187a5f1c908b3538d1185fb
Change-Id: I83d51c40b3c4c1f3e5de327aba6f05e65d9814c7
2021-12-15 17:27:52 -05:00
Arlo Breault 2fa7a30f14 Remove hardcoded list of categories with no parameters
Change-Id: Ic8b9ced613c873cada0a9909ed0d3799160504a1
2021-12-15 17:27:36 -05:00
DannyS712 830f879e22 Convert LintErrorTest to pure unit tests
No integration needed.

Requires bumping minimum version of mediawiki to when
MediaWikiUnitTestCase was introduced in 1.34.

Change-Id: Ibc0a1028cc61a7bdc149081aeaa1109de18ee119
2021-03-27 02:28:25 +00:00
Reedy b1c9ff6ee9 Convert to manifest_version 2
Bug: T252470
Change-Id: I1926592da8774feddbe44d541b4fe68beb080e44
2020-05-12 01:53:32 +01:00
Arlo Breault 40f0b3cef9 Implement ParserLogLinterData hook
Bug: T238456
Change-Id: I3e05d1c113b18c83db02c7e500d5834e931afb7f
2019-11-27 13:51:01 -05:00
Amir Aharoni 243604ffc6 Split apihelp messages to a separate file
Bug: T189982
Change-Id: Ie06eb176da9257816057d2a9af2a793dfe169689
2019-07-04 15:38:00 +03:00
SFarida 0f37222bbb Add new Linter category wikilink-in-extlink
Add the category in extension.json and descriptions
of the category in qqq.json and en.json

Bug: T202905
Change-Id: Ie2e76c653ac9aa80bc7c32f3227323225a0cd8e5
2019-02-19 10:53:43 -08:00
C. Scott Ananian b1b4cc8ca7 Add IPv6 localhost to default whitelist along with IPv4 localhost
Change-Id: I970b91539606305da9afa6d3e40d66885385615e
2018-05-01 20:18:06 +00:00
Kunal Mehta 34b160621c Use SPDX 3.0 license identifier
SPDX released version 3 of their license list (<https://spdx.org/licenses/>),
which changed the FSF licenses to explicitly end in -only or -or-later
instead of relying on an easy to miss + symbol.

Bug: T183858
Change-Id: Iecbc1b73bf7a4f3e52f8a4ed4aba9442707516ab
2018-01-04 01:32:59 +00:00
Umherirrender f3166dee38 Require MediaWiki 1.31 in extension.json for AutoloadNamespaces
AutoloadNamespaces is a new feature in 1.31

Change-Id: If877db9d628598f3d91a55b767be421cfa8a50b1
2017-12-31 14:05:31 +01:00
jenkins-bot 2e740a8312 Merge "Move category IDs into the extension.json information" 2017-12-21 18:40:24 +00:00
Kunal Mehta 33dcc7211e Switch to using a PSR-4 autoloader
Change-Id: Iea8f1348f7911e0e094a40db2e9dbe294e8dbe7f
2017-12-11 17:20:02 -08:00
C. Scott Ananian 2e49c745d7 Move category IDs into the extension.json information
This reduces the number of different files that need to be modified
in order to add a new linter category.

Change-Id: Id095317d6d761c57e2ce632d34ebd962bf85e785
2017-12-08 11:28:21 -05:00
C. Scott Ananian 60562e74fc Move $hasNameCats into the information in extension.json
Change-Id: Id4f01a2929ce43ac9d6c33655eed25bfd2cf37e3
2017-12-08 06:42:43 +00:00
C. Scott Ananian c0ad96898d Add 4 new high priority linter categories
* multiple-unclosed-formatting-tags: this is a subset of the unclosed
  formatting tag lint, but is higher priority because unclosed tags
  like <small> and <big> compound their effects

* unclosed-quotes-in-heading: unclosed wikitext i/b tag with a heading
  ancestor (this causes breakage which leaks out of the table of
  content to affect the rest of the page)

* multiline-html-table-in-list: html table with newline breaks nested
  in a list

* misc-tidy-replacement-issues: this is a catchall category for
  infrequent long-tail issues, used to enable speedy deployment of new
  linter categories during tidy replacement as wikis get RemexHtml
  enabled.  These will have a subtype property to identify the
  specific issue.

Change-Id: Ic2c965132f7a09679574489865bdc81df9e43845
2017-12-08 06:39:16 +00:00
Subramanya Sastry 95f4038db8 Add tidy-font-bug linter high-priority category
* <font> tags with color attribute that wrap links and images
  will have different behavior and hence rendering compared to Tidy.

Change-Id: I7a551ef9b7e8f57d7a43c823832f0e3add6b1367
2017-10-23 17:08:57 -05:00
Subramanya Sastry d13bb1aa6c Add html5-misnesting high-priority category
Change-Id: I840c4dfc14308dffd02dd8a89ab4dcac13f6141a
2017-09-26 12:31:43 -05:00
Arlo Breault 875e0f84a0 Add multi-colon-escape linter medium-priority category
Change-Id: I9f21f3128b78e7b2459c18a9121605dffd5a1bc4
2017-09-18 14:04:04 -04:00
Subramanya Sastry 11f4a97ba6 Provide parsermigration-edit edit link option for categories
* Since the high priority categories are being used to effect
  Tidy migration, we need editors to be able to compare their fixes
  and ensure that the pages look identical after the fix.

* This patch adds a new property for linter categories and displays
  the requested edit link in the pager.

* This patch enables the property for high priority categories.

Change-Id: Ia9b23d79da686f0a6c0203e2dba58a876a4a3d4a
2017-08-22 10:07:39 +00:00
Subramanya Sastry ce8b7c873c Add tidy-whitespace-bug linter high-priority category
Change-Id: I9c068c2c45b4cf9bb5612f41dfa63a606a7a679a
2017-08-10 12:20:09 -04:00
Kunal Mehta 472ac093aa Add other primary authors
Change-Id: I8a96898f979c68912188f42e5b9a968656c1a2bf
2017-06-20 14:21:46 -07:00
Kunal Mehta 35a2ba9a6c Add API meta=linterstats module
This module just exposes the number of lint errors in each category for
tools that want to collect statistics and stuff. It's currently under a
'totals' key to give us more flexibility if we want to add other
information in the future.

Change-Id: Iad5136a6a5989ce5bcb1a00a4f82f49a397e0170
2017-06-06 10:02:49 -07:00
Kunal Mehta eebd04aa00 Add caching to looking up totals
The query itself is too expensive to be run on large Wikimedia wikis. So
put it behind WAN cache and touch the check keys for each category
whenever those have errors added or deleted from them.

If this happens to get out of sync, it will get fully refreshed
regularly when the totals are sent to statsd.

WANObjectCache's 'lockTSE' feature will help avoid cache stampedes that
made this query expensive in the past.

Change-Id: I3774103a29fa0f29d36283950f136259fa71bffe
2017-05-29 07:33:41 -07:00
Subramanya Sastry 0f3c66418a Organize lint categories by priority
Displaying categories by priority provides editors with better
guidance about what to spend time on. The Linter help page provides
more information about why the specific priorities have been chosen.

Change-Id: If6f28570189e24a67b4380f666f4cd64a2296989
2017-04-28 10:42:47 -05:00
Kunal Mehta 9c65d786e2 Support new wikitext editor for edit helper
Register a hook for when VE is finished initializing to select the error
section, just like the textbox-based editor.

Use the BeforePageDisplay hook so it runs on VE page loads too.

Bug: T160102
Change-Id: I59a7e0a3e8be32e4689cbf41c4904970902c4dff
2017-04-25 15:21:13 +01:00
Subramanya Sastry 0ac10e1f20 Change severity levels for stripped-tag & misnested-tag
* stripped-tag is harmless and should be a warning.
* misnested-tag could be changing intent of markup and should
  be an error.

Change-Id: I42fdab1c0e2f0793ca3845d28880388244b65527
2017-04-19 15:25:06 -05:00
Subramanya Sastry a866fa7c76 Add pwrap-bug-workaround linter category
Change-Id: If1ec022e5bb741f4dc825428b5a689c2f0b0c445
2017-04-19 15:19:13 -05:00
Subramanya Sastry 0d230e722f Add misnested-tag linter category
Change-Id: I3508c68272940d21be6a5c9b78ee34c845debc74
2017-04-10 17:28:36 -05:00
Subramanya Sastry 619ddc8677 Add deletable-table-tag linter category
Change-Id: I48000170a74363872edbce8f78780bccf160600c
2017-04-06 12:47:51 -05:00
Kunal Mehta 5c606fca03 Display count of lint errors on ?action=info
Change-Id: Ifcdfcb365e5ff6106b521d58a06df8c006772473
2017-01-20 11:26:44 -08:00
Kunal Mehta 8b8c50e9d8 Add Linter categories to action=query&meta=siteinfo
Bug: T153056
Change-Id: Ieed01741d31c9e268a8769a5bc5d566d8020e760
2017-01-20 11:26:21 -08:00
Kunal Mehta f3778856f2 Add severity for error categories
Categories can now have a severity level of "error" or "warning"
designated, which places them in a different heading on
Special:LintErrors.

Bug: T152822
Change-Id: I1276b9502d90765e88dcb8ea78569dee910c5d88
2016-12-14 17:07:40 -08:00
Kunal Mehta 9ba2839e7d Record totals in categories in statsd
For historical purposes so we can see new errors being added/fixed.

Bug: T152664
Change-Id: Icec62d236351e43ec397ac448805ab16131c943c
2016-12-08 16:52:52 -08:00
Kunal Mehta 9d5759329f Add alias file
So people can localize the special page name

Change-Id: Icd331d02f400dfca465236bb95c1357c58480260
2016-12-02 07:55:13 -08:00
Kunal Mehta 70cc3ac190 Delete linter rows when pages are deleted
Bug: T151281
Change-Id: I3b113d0729b288683d9bc6c2f552b99ed2890f5c
2016-11-22 18:31:21 -08:00
Arlo Breault 958623f0fb Display self-closed tags
Change-Id: I6c084a6c72114b57a21cd367d398c673c5c4d991
2016-11-17 12:32:46 -08:00