Commit graph

84 commits

Author SHA1 Message Date
Umherirrender e47e471799 Move sql file into own folder
Change-Id: I12ed78d755426644011fca7595dd82192825859f
2019-03-08 21:41:35 +01:00
Umherirrender e649f3029b Fix type of LintError->templateInfo
Change-Id: I2be1d59463ec42ab8955235c83516b2f6a1be276
2019-03-08 20:06:48 +01:00
Kunal Mehta fcb9eba542 DWIM in CategoryManager
If `"has-name": false` is set, then CategoryManager should interpret that
as false, and not treat the existence of the key as making it true.

Same with "parser-migration".

Change-Id: I6b8d5b23fb33707b7596d9172da26fd84d652da8
2019-03-02 22:57:42 +00:00
Kunal Mehta e41e8e32a4 Drop excess events at the API layer
Originally we were dropping excess events inside the job queue, but that means
all of the events need to be passed into the job queue...which can cause
problems.

So drop them in the API module. The only other place we construct
RecordLintJob is when an article has been deleted, and those jobs have
no errors since they're all being deleted.

Bug: T202179
Change-Id: I61940280e0dfb99398d9f047d0e66007d91a0241
2018-08-17 20:36:06 -07:00
Subramanya Sastry 2d115ecc5e Fix broken test for enabling parsermigration-edit links
Bug: T183746
Change-Id: I29b09d1cd608cb02065319e9de72ea51fbf1cc9c
2018-04-30 23:00:38 -05:00
Umherirrender 0b86ae2cee Add use for InvalidArgumentException
Change-Id: I67735aa40f45d75e6ed6bd0b205bb718f29ddfb8
2018-04-04 15:51:21 +02:00
jenkins-bot 13dfbf4ee9 Merge "Use normal lint error updating logic for page deletions" 2018-02-19 12:33:29 +00:00
Kunal Mehta db5e5e9003 Use estimateRowCount() instead of actually counting everything
On large wikis with lots of lint errors, counting the entire table can
be problematic from a performance perspective, sometimes taking minutes.

Instead, use Database::estimateRowCount(), which uses EXPLAIN SELECT
COUNT(*) to get an approximate value for the number of rows. We do make
sure that if the category actually has no rows, that it will return 0.

This should be considered a temporary solution, and we should look into
doing something like the SiteStats incremental updates table in the long
run.

Bug: T184280
Change-Id: I2d4dcc615477fd60e41dfed4a3d1a3ad52a9f4af
2018-02-01 13:15:06 -08:00
Reedy ecd8769508 Update IPSet use statement
Bug: T182273
Change-Id: I4f04462d13a37efd10e1e530a9b79e45cfb90309
2018-01-21 04:45:55 +00:00
C. Scott Ananian 2e49c745d7 Move category IDs into the extension.json information
This reduces the number of different files that need to be modified
in order to add a new linter category.

Change-Id: Id095317d6d761c57e2ce632d34ebd962bf85e785
2017-12-08 11:28:21 -05:00
C. Scott Ananian 60562e74fc Move $hasNameCats into the information in extension.json
Change-Id: Id4f01a2929ce43ac9d6c33655eed25bfd2cf37e3
2017-12-08 06:42:43 +00:00
C. Scott Ananian c0ad96898d Add 4 new high priority linter categories
* multiple-unclosed-formatting-tags: this is a subset of the unclosed
  formatting tag lint, but is higher priority because unclosed tags
  like <small> and <big> compound their effects

* unclosed-quotes-in-heading: unclosed wikitext i/b tag with a heading
  ancestor (this causes breakage which leaks out of the table of
  content to affect the rest of the page)

* multiline-html-table-in-list: html table with newline breaks nested
  in a list

* misc-tidy-replacement-issues: this is a catchall category for
  infrequent long-tail issues, used to enable speedy deployment of new
  linter categories during tidy replacement as wikis get RemexHtml
  enabled.  These will have a subtype property to identify the
  specific issue.

Change-Id: Ic2c965132f7a09679574489865bdc81df9e43845
2017-12-08 06:39:16 +00:00
Haikal Izzuddin 94fa73ac1b Replace deprecated usage of ApiBase::dieUsage()
Linter calls the deprecated ApiBase::dieUsage() function in
ApiRecordLint.php, replacing it with dieWithError().

Bug: T181758
Change-Id: I0cbf784f591b86b206b032fccbc0e32564a3e9e8
2017-12-03 15:26:39 -08:00
jenkins-bot ad6af1133b Merge "Rename typo "linker-" keys to proper "linter-"" 2017-11-25 16:41:25 +00:00
jenkins-bot 44aa608aae Merge "Show edit and history links even if page is protected" 2017-11-25 13:56:52 +00:00
xyx_is a262956017 API: Allow querying linterrors by pageid
The pageid parameter limits lint errors to specific pages. Users can get
detailed lint error information of a certain page.

Bug: T181303
Change-Id: I164449254649caff29fcffa3bc7c923c20b8e837
2017-11-24 21:03:05 -08:00
Kunal Mehta 35cfda325b Rename typo "linker-" keys to proper "linter-"
I must have made a typo and prefixed the keys with "linker-" instead of
"linter-".

Change-Id: I041aa3341aaa2efa51d20c848cb19efb648209ac
2017-11-24 20:38:25 -08:00
Kunal Mehta e0ac4c6e6b Show edit and history links even if page is protected
If the user does not have permission to edit the page, still show a
"view source" and history link so they are able to figure out what the
error is.

Bug: T177289
Change-Id: I049d27d37073e452dc0c11128dab5204d110d81f
2017-11-24 20:35:15 -08:00
Kunal Mehta 2c08143ed0 Improve logging for non-existent categories in the database
Suggested by Chad in the review for 3a8d3b9e0.

Bug: T179423
Change-Id: I9286ae33bdb3b0b50aa6f1619402caa5486682e3
2017-11-22 23:05:04 -08:00
jenkins-bot c78e1c8d75 Merge "Handle non-existent categories in the database better" 2017-11-23 00:53:45 +00:00
jenkins-bot 674efd33e6 Merge "Add test case to verify all the i18n messages for a category exists" 2017-10-31 20:09:34 +00:00
Kunal Mehta 3a8d3b9e03 Handle non-existent categories in the database better
If a newer version of MediaWiki gets rolled back, it's possible for
there to be lint entries in the database that don't exist according to
CategoryManager.

Instead of showing an error to the user, just silently hide those rows.

All callers to CategoryManager::getCategoryId() already check the
category exists. The callers for CategoryManager::getCategoryName() will
catch the MissingCategoryException, and log it if necessary. Notably
LinterError::makeLintError() will return false on invalid rows, and all
callers have been updated to handle that.

Bug: T179423
Change-Id: Ia5f56f18a51fa871511b02410222a6079efbfff6
2017-10-31 10:42:07 -07:00
Subramanya Sastry 95f4038db8 Add tidy-font-bug linter high-priority category
* <font> tags with color attribute that wrap links and images
  will have different behavior and hence rendering compared to Tidy.

Change-Id: I7a551ef9b7e8f57d7a43c823832f0e3add6b1367
2017-10-23 17:08:57 -05:00
Kunal Mehta 6e4ae8aced Add test case to verify all the i18n messages for a category exists
Change-Id: If079206dd1f1c7c4a1b92f0ee05452bcc28b2323
2017-10-16 17:28:02 -07:00
Kunal Mehta a1421792e9 Use normal lint error updating logic for page deletions
Page deletions were bypassing the logic in RecordLintJob that
ensured the right category totals cache was cleared and the
statsd updates. Fix that by just using RecordLintJob directly.

Bug: T170313
Change-Id: I06b821b65f65609ddac8ed4e7c662336082d8266
2017-10-05 21:03:30 -07:00
Kunal Mehta 5416525735 Parse linter-category-{name}-desc messages
Bug: T177429
Change-Id: I3f36b27fee877122d3df3c68b3aa9515412f6869
2017-10-04 20:17:22 -07:00
Subramanya Sastry d13bb1aa6c Add html5-misnesting high-priority category
Change-Id: I840c4dfc14308dffd02dd8a89ab4dcac13f6141a
2017-09-26 12:31:43 -05:00
libraryupgrader 773cc2e8b2 build: Updating mediawiki/mediawiki-codesniffer to 13.0.0
Change-Id: I09c3533b611b98169868f724ac929b20dbe7e43a
2017-09-24 03:51:18 +00:00
Arlo Breault 875e0f84a0 Add multi-colon-escape linter medium-priority category
Change-Id: I9f21f3128b78e7b2459c18a9121605dffd5a1bc4
2017-09-18 14:04:04 -04:00
Kunal Mehta 139a8aafea Fix up 11f4a97ba6
CategoryManager is a member variable. Also explicitly define the
haveParserMigrationExt member variable too.

Change-Id: I2aa77e5c1819a80ba18f67c0f65b2d47dbaa0303
2017-08-22 16:08:50 -07:00
Subramanya Sastry 11f4a97ba6 Provide parsermigration-edit edit link option for categories
* Since the high priority categories are being used to effect
  Tidy migration, we need editors to be able to compare their fixes
  and ensure that the pages look identical after the fix.

* This patch adds a new property for linter categories and displays
  the requested edit link in the pager.

* This patch enables the property for high priority categories.

Change-Id: Ia9b23d79da686f0a6c0203e2dba58a876a4a3d4a
2017-08-22 10:07:39 +00:00
Subramanya Sastry ce8b7c873c Add tidy-whitespace-bug linter high-priority category
Change-Id: I9c068c2c45b4cf9bb5612f41dfa63a606a7a679a
2017-08-10 12:20:09 -04:00
Subramanya Sastry f85184df54 Parsoid provides ns-prefixed template titles => don't use NS_TEMPLATE
* This prevents resolving main namespace titles as template titles

Bug: T169472
Change-Id: I3c10227767257bcd62f95e0284d44f5e44c33057
2017-07-05 15:17:18 -05:00
Subramanya Sastry eb99fd05e3 Display table tag for deletable-table-tag pages to prevent UI confusion
I remember reading in some talk page / phab task about what the tag
was that had to be deleted.

Change-Id: I74bc6f3207177239f7173ce2963ed55350e26f6f
2017-07-05 14:39:53 -05:00
Arlo Breault c08469e87d Drop lints w/o dsr info
Bug: T168900
Change-Id: Ia29629b4ad790f53cb99c6a6d3b20e20b7e35e54
2017-06-29 20:30:04 +00:00
Kunal Mehta 60e608d47f Add debug logging for missing 'dsr'
Bug: T168900
Change-Id: I496a001291a806f46eca7fd8bc362847cc98c421
2017-06-26 15:22:45 -07:00
Kunal Mehta f2a516582d build: Updating mediawiki/mediawiki-codesniffer to 0.9.0
The following sniffs are failing and were disabled:
* MediaWiki.FunctionComment.Missing.Protected
* MediaWiki.FunctionComment.Missing.Public

Change-Id: I96e32df48d13040893bfd1be6d90d0db4f7c7d0a
2017-06-20 00:10:41 -07:00
jenkins-bot 2a242f9fce Merge "Add API meta=linterstats module" 2017-06-06 17:26:01 +00:00
Kunal Mehta 35a2ba9a6c Add API meta=linterstats module
This module just exposes the number of lint errors in each category for
tools that want to collect statistics and stuff. It's currently under a
'totals' key to give us more flexibility if we want to add other
information in the future.

Change-Id: Iad5136a6a5989ce5bcb1a00a4f82f49a397e0170
2017-06-06 10:02:49 -07:00
jenkins-bot ee2f0efdcf Merge "Add caching to looking up totals" 2017-06-06 09:53:27 +00:00
Kunal Mehta eebd04aa00 Add caching to looking up totals
The query itself is too expensive to be run on large Wikimedia wikis. So
put it behind WAN cache and touch the check keys for each category
whenever those have errors added or deleted from them.

If this happens to get out of sync, it will get fully refreshed
regularly when the totals are sent to statsd.

WANObjectCache's 'lockTSE' feature will help avoid cache stampedes that
made this query expensive in the past.

Change-Id: I3774103a29fa0f29d36283950f136259fa71bffe
2017-05-29 07:33:41 -07:00
Subramanya Sastry 3eea8ed824 Handle multi-part-template-block output from Parsoid's linter
The messaging is a bit tricky. In some cases, the flag is set
because the output involves a single template and some top-level
content -- not multiple templates always.

Ex: '{{1x|<div>}}foo</span>'

In this case, the entire content is considered template-affected
because of DOM structuring reasons. This wikitext will generate
lint errors with multi-part-template-block template info.

In production usage, however, the probability is quite high that
multiple templates will be involved. This is usually because of
tables that are constructed with multiple templates, for example.

So, if we want to fudge and hand-wave a bit, the i18n message can
say that the output came from multiple templates.

Bug: T162920
Change-Id: I35cee6787800b03724856775fdf53991ae8e8125
2017-05-08 23:01:32 -05:00
Subramanya Sastry 0f3c66418a Organize lint categories by priority
Displaying categories by priority provides editors with better
guidance about what to spend time on. The Linter help page provides
more information about why the specific priorities have been chosen.

Change-Id: If6f28570189e24a67b4380f666f4cd64a2296989
2017-04-28 10:42:47 -05:00
Kunal Mehta 9c65d786e2 Support new wikitext editor for edit helper
Register a hook for when VE is finished initializing to select the error
section, just like the textbox-based editor.

Use the BeforePageDisplay hook so it runs on VE page loads too.

Bug: T160102
Change-Id: I59a7e0a3e8be32e4689cbf41c4904970902c4dff
2017-04-25 15:21:13 +01:00
Subramanya Sastry a866fa7c76 Add pwrap-bug-workaround linter category
Change-Id: If1ec022e5bb741f4dc825428b5a689c2f0b0c445
2017-04-19 15:19:13 -05:00
Subramanya Sastry a5e2f822f3 Show tag name for the stripped-tag category
Change-Id: I53546c11391fb2388dbcf3f70c0a21b2b1af9b09
2017-04-19 08:51:21 -04:00
Kunal Mehta f5e43ddb1a API: Add pageid/ns to list=linterrors output
Change-Id: Ia6a092e3e59412d4635b1d3fa232653c7d3a0752
2017-04-17 10:25:36 -07:00
jenkins-bot c9f45520f5 Merge "Revert "Expose category totals in API response"" 2017-04-16 12:17:18 +00:00
jenkins-bot 49ecefa75e Merge "Separate linter-numerrors bidi from rest of the line" 2017-04-16 07:37:40 +00:00
Legoktm 379137df59 Revert "Expose category totals in API response"
API meta=siteinfo is a frequently called API query
and the category totals database query is not
sufficiently fast enough to be part of this by default.

This reverts commit 45b4bf6382.

Change-Id: Ia5bf97855d48955ce53c3679c4d138fabafae8b7
2017-04-16 07:36:32 +00:00