Commit graph

58 commits

Author SHA1 Message Date
Subramanya Sastry d13bb1aa6c Add html5-misnesting high-priority category
Change-Id: I840c4dfc14308dffd02dd8a89ab4dcac13f6141a
2017-09-26 12:31:43 -05:00
libraryupgrader 773cc2e8b2 build: Updating mediawiki/mediawiki-codesniffer to 13.0.0
Change-Id: I09c3533b611b98169868f724ac929b20dbe7e43a
2017-09-24 03:51:18 +00:00
Arlo Breault 875e0f84a0 Add multi-colon-escape linter medium-priority category
Change-Id: I9f21f3128b78e7b2459c18a9121605dffd5a1bc4
2017-09-18 14:04:04 -04:00
Kunal Mehta 139a8aafea Fix up 11f4a97ba6
CategoryManager is a member variable. Also explicitly define the
haveParserMigrationExt member variable too.

Change-Id: I2aa77e5c1819a80ba18f67c0f65b2d47dbaa0303
2017-08-22 16:08:50 -07:00
Subramanya Sastry 11f4a97ba6 Provide parsermigration-edit edit link option for categories
* Since the high priority categories are being used to effect
  Tidy migration, we need editors to be able to compare their fixes
  and ensure that the pages look identical after the fix.

* This patch adds a new property for linter categories and displays
  the requested edit link in the pager.

* This patch enables the property for high priority categories.

Change-Id: Ia9b23d79da686f0a6c0203e2dba58a876a4a3d4a
2017-08-22 10:07:39 +00:00
Subramanya Sastry ce8b7c873c Add tidy-whitespace-bug linter high-priority category
Change-Id: I9c068c2c45b4cf9bb5612f41dfa63a606a7a679a
2017-08-10 12:20:09 -04:00
Subramanya Sastry f85184df54 Parsoid provides ns-prefixed template titles => don't use NS_TEMPLATE
* This prevents resolving main namespace titles as template titles

Bug: T169472
Change-Id: I3c10227767257bcd62f95e0284d44f5e44c33057
2017-07-05 15:17:18 -05:00
Subramanya Sastry eb99fd05e3 Display table tag for deletable-table-tag pages to prevent UI confusion
I remember reading in some talk page / phab task about what the tag
was that had to be deleted.

Change-Id: I74bc6f3207177239f7173ce2963ed55350e26f6f
2017-07-05 14:39:53 -05:00
Arlo Breault c08469e87d Drop lints w/o dsr info
Bug: T168900
Change-Id: Ia29629b4ad790f53cb99c6a6d3b20e20b7e35e54
2017-06-29 20:30:04 +00:00
Kunal Mehta 60e608d47f Add debug logging for missing 'dsr'
Bug: T168900
Change-Id: I496a001291a806f46eca7fd8bc362847cc98c421
2017-06-26 15:22:45 -07:00
Kunal Mehta f2a516582d build: Updating mediawiki/mediawiki-codesniffer to 0.9.0
The following sniffs are failing and were disabled:
* MediaWiki.FunctionComment.Missing.Protected
* MediaWiki.FunctionComment.Missing.Public

Change-Id: I96e32df48d13040893bfd1be6d90d0db4f7c7d0a
2017-06-20 00:10:41 -07:00
jenkins-bot 2a242f9fce Merge "Add API meta=linterstats module" 2017-06-06 17:26:01 +00:00
Kunal Mehta 35a2ba9a6c Add API meta=linterstats module
This module just exposes the number of lint errors in each category for
tools that want to collect statistics and stuff. It's currently under a
'totals' key to give us more flexibility if we want to add other
information in the future.

Change-Id: Iad5136a6a5989ce5bcb1a00a4f82f49a397e0170
2017-06-06 10:02:49 -07:00
jenkins-bot ee2f0efdcf Merge "Add caching to looking up totals" 2017-06-06 09:53:27 +00:00
Kunal Mehta eebd04aa00 Add caching to looking up totals
The query itself is too expensive to be run on large Wikimedia wikis. So
put it behind WAN cache and touch the check keys for each category
whenever those have errors added or deleted from them.

If this happens to get out of sync, it will get fully refreshed
regularly when the totals are sent to statsd.

WANObjectCache's 'lockTSE' feature will help avoid cache stampedes that
made this query expensive in the past.

Change-Id: I3774103a29fa0f29d36283950f136259fa71bffe
2017-05-29 07:33:41 -07:00
Subramanya Sastry 3eea8ed824 Handle multi-part-template-block output from Parsoid's linter
The messaging is a bit tricky. In some cases, the flag is set
because the output involves a single template and some top-level
content -- not multiple templates always.

Ex: '{{1x|<div>}}foo</span>'

In this case, the entire content is considered template-affected
because of DOM structuring reasons. This wikitext will generate
lint errors with multi-part-template-block template info.

In production usage, however, the probability is quite high that
multiple templates will be involved. This is usually because of
tables that are constructed with multiple templates, for example.

So, if we want to fudge and hand-wave a bit, the i18n message can
say that the output came from multiple templates.

Bug: T162920
Change-Id: I35cee6787800b03724856775fdf53991ae8e8125
2017-05-08 23:01:32 -05:00
Subramanya Sastry 0f3c66418a Organize lint categories by priority
Displaying categories by priority provides editors with better
guidance about what to spend time on. The Linter help page provides
more information about why the specific priorities have been chosen.

Change-Id: If6f28570189e24a67b4380f666f4cd64a2296989
2017-04-28 10:42:47 -05:00
Kunal Mehta 9c65d786e2 Support new wikitext editor for edit helper
Register a hook for when VE is finished initializing to select the error
section, just like the textbox-based editor.

Use the BeforePageDisplay hook so it runs on VE page loads too.

Bug: T160102
Change-Id: I59a7e0a3e8be32e4689cbf41c4904970902c4dff
2017-04-25 15:21:13 +01:00
Subramanya Sastry a866fa7c76 Add pwrap-bug-workaround linter category
Change-Id: If1ec022e5bb741f4dc825428b5a689c2f0b0c445
2017-04-19 15:19:13 -05:00
Subramanya Sastry a5e2f822f3 Show tag name for the stripped-tag category
Change-Id: I53546c11391fb2388dbcf3f70c0a21b2b1af9b09
2017-04-19 08:51:21 -04:00
Kunal Mehta f5e43ddb1a API: Add pageid/ns to list=linterrors output
Change-Id: Ia6a092e3e59412d4635b1d3fa232653c7d3a0752
2017-04-17 10:25:36 -07:00
jenkins-bot c9f45520f5 Merge "Revert "Expose category totals in API response"" 2017-04-16 12:17:18 +00:00
jenkins-bot 49ecefa75e Merge "Separate linter-numerrors bidi from rest of the line" 2017-04-16 07:37:40 +00:00
Legoktm 379137df59 Revert "Expose category totals in API response"
API meta=siteinfo is a frequently called API query
and the category totals database query is not
sufficiently fast enough to be part of this by default.

This reverts commit 45b4bf6382.

Change-Id: Ia5bf97855d48955ce53c3679c4d138fabafae8b7
2017-04-16 07:36:32 +00:00
Ebrahim Byagowi 8302db893f Separate linter-numerrors bidi from rest of the line
Change-Id: I63a06cdd051f47b6c43ccc4f5f8ae13521b994f6
2017-04-16 07:35:29 +00:00
Kunal Mehta fbff1cffff Add namespace filter to Special:LintErrors
Bug: T162918
Change-Id: Id41faeafb9af161d32d5c273094fb2e6d8a8a64b
2017-04-13 20:49:21 -07:00
Subramanya Sastry 0d230e722f Add misnested-tag linter category
Change-Id: I3508c68272940d21be6a5c9b78ee34c845debc74
2017-04-10 17:28:36 -05:00
Kunal Mehta eacb3c0d0d Split statsd metrics by wiki
This way we can track the progress of individual wikis in cleaning up
errors. The wiki name is at the end of the key so we can still use
e.g. "linter.category.$name.*" to see across all wikis at once.

Change-Id: I62463b9256e125d32d97396bd939334d71b46027
2017-04-06 11:21:41 -07:00
Subramanya Sastry 619ddc8677 Add deletable-table-tag linter category
Change-Id: I48000170a74363872edbce8f78780bccf160600c
2017-04-06 12:47:51 -05:00
Kunal Mehta 7f17abe393 Add link to history on Special:LintErrors
Bug: T160607
Change-Id: Ifd2ef33dd7477722691c8f2e570b7ac1af206135
2017-03-22 13:28:33 -07:00
Kunal Mehta 45b4bf6382 Expose category totals in API response
Instead of just including the error category names, include the number
of errors each category has, to make it easier to collect aggregate
stats.

This changes the structure from an array to an object, in JSON, but I'm
not aware of any clients using these specific fields yet.

Change-Id: Iaf942b923a0f4047721055ad9cb48aacc5aa6784
2017-03-20 14:14:22 -07:00
Kunal Mehta 5c606fca03 Display count of lint errors on ?action=info
Change-Id: Ifcdfcb365e5ff6106b521d58a06df8c006772473
2017-01-20 11:26:44 -08:00
Kunal Mehta 8b8c50e9d8 Add Linter categories to action=query&meta=siteinfo
Bug: T153056
Change-Id: Ieed01741d31c9e268a8769a5bc5d566d8020e760
2017-01-20 11:26:21 -08:00
Kunal Mehta b201a3122c Add __METHOD__ to MWCallableUpdate
For stats and debugging purposes.

Change-Id: I5acd7fad0a9f09519595c2fa1f6232bb2624a713
2016-12-14 19:46:18 -08:00
Kunal Mehta f3778856f2 Add severity for error categories
Categories can now have a severity level of "error" or "warning"
designated, which places them in a different heading on
Special:LintErrors.

Bug: T152822
Change-Id: I1276b9502d90765e88dcb8ea78569dee910c5d88
2016-12-14 17:07:40 -08:00
jenkins-bot 69fd50c811 Merge "Record totals in categories in statsd" 2016-12-15 00:03:42 +00:00
Kunal Mehta 651d8f174b Add missing semicolon to &mdash
Change-Id: If669c89cd06ca173c0dcf6a01a793ddf8ce31793
2016-12-14 01:01:57 -08:00
Kunal Mehta 9ba2839e7d Record totals in categories in statsd
For historical purposes so we can see new errors being added/fixed.

Bug: T152664
Change-Id: Icec62d236351e43ec397ac448805ab16131c943c
2016-12-08 16:52:52 -08:00
Kunal Mehta 782313088d Show error counts on Special:LintErrors
Change-Id: Ib49f2e391ca4b9d1eef5443c011d54a42921ce4e
2016-12-08 16:52:31 -08:00
Kunal Mehta 81ef6878af Include location in LintError::id()
Since it's no longer in $this->params, we need to explicitly include it.

Change-Id: I45f26db061f02066aa03a501dc022bf33e2c096e
2016-12-02 16:15:17 -08:00
Kunal Mehta 8e2d4e42ee Use INSERT IGNORE when putting new lint errors in the database
The most likely scenario of duplicate key errors is that it's the exact
same lint error and there's just a race condition when calculating which
new errors need to be inserted, so just ignore them.

Follows-up 419610bcdb.

Change-Id: I84749ab221bbd517b474be8875bb6a59e4f3258e
2016-12-02 15:54:02 -08:00
Kunal Mehta a50398d4ce Remove unused LintError::equals()
Change-Id: I0d0d743119fee8a95627c365e1b2f45b8fcf67e7
2016-12-02 07:56:19 -08:00
Kunal Mehta 14b53d6281 Add integration tests for Database class
These tests insert variations of fake lint errors into the database, and
then read out of the database to check they round-trip properly.

And while we're at it, improve the setForPage() return value.

These tests can be run with something like:
 php tests/phpunit/phpunit.php extensions/Linter/tests/phpunit/

Change-Id: Ifdba8a8a104d218a822f909bc5d7b3512aca499d
2016-11-30 21:17:51 -08:00
Kunal Mehta 419610bcdb Enforce category/page/position uniqueness constraint in the database
Move location to two separate columns in the database: linter_start and
linter_end. This allows us to have the database enforce the uniqueness
of those fields, instead of just relying upon the PHP code to do so,
which could be bypassed since we have multiple servers and concurrent
processes.

Change-Id: I3e67ce1b7cb3c93866a388ec3248af4cff2a81e0
2016-11-30 18:55:19 -08:00
Kunal Mehta 9550db450d Fix inserting errors if none exist for that page
If no errors existed for the page, inserting new ones would fail with a
database error since $errors was key-ed by unique id, which the database
wrapper interpreted as field names, causing issues. Using array_values()
gets rid of the keys, fixing the issue.

Change-Id: I7645de4e5d3ac4462d7980374c8ef8be6280442b
2016-11-30 18:17:36 -08:00
Kunal Mehta 29379edb0b De-duplicate errors and trim excessive errors in the same category
It's possible to have duplicate, identical lint errors if the same exact
error is repeated in a template transclusion (e.g. {{1x|<b/> <b/>}})
since the position via dsr is the same. In this case, just de-duplicate
the errors since we can't differentiate them.

At the same time, trim excessive errors on the same page in the same
category. It's most likely that if a page has that many of the same
errors, the editor or bot will just fix all of them at the same time, so
we don't need to include all of them in the database. 20 is kind of a
low value, but we can always increase it later on as necessary.

Change-Id: I9cded720169870d0eea574e1a930ce4e9b190ac0
2016-11-23 19:47:04 -08:00
Kunal Mehta 70cc3ac190 Delete linter rows when pages are deleted
Bug: T151281
Change-Id: I3b113d0729b288683d9bc6c2f552b99ed2890f5c
2016-11-22 18:31:21 -08:00
Kunal Mehta 06671d2034 API: Fix lntcategories parameter
* Treat it as an array in all cases
* Make the default value all categories
* Rename from category to categories

Bug: T151288
Change-Id: I5c0c341112894c5a7ec3aaebb6ac9085353f55bd
2016-11-22 18:31:21 -08:00
Kunal Mehta 2f39ab7fff API: Fix up action=record-lint's getAllowedParams()
Since this module is internal, it doesn't really matter, but it cleans
up the output on api.php?modules=record-lint.

Bug: T151285
Change-Id: I859a2780d6ed1918cc81101e9e2c2cd348a2390a
2016-11-22 18:31:21 -08:00
Kunal Mehta d8437fc9c3 LintErrorsPager: Set explicit output format on some messages
Bug: T151283
Change-Id: Ic52bc11a6aacf7f5bb23d9273fa0bf2a27e0cfc4
2016-11-22 18:31:21 -08:00