Commit graph

31 commits

Author SHA1 Message Date
Kunal Mehta eacb3c0d0d Split statsd metrics by wiki
This way we can track the progress of individual wikis in cleaning up
errors. The wiki name is at the end of the key so we can still use
e.g. "linter.category.$name.*" to see across all wikis at once.

Change-Id: I62463b9256e125d32d97396bd939334d71b46027
2017-04-06 11:21:41 -07:00
Subramanya Sastry 619ddc8677 Add deletable-table-tag linter category
Change-Id: I48000170a74363872edbce8f78780bccf160600c
2017-04-06 12:47:51 -05:00
Kunal Mehta 7f17abe393 Add link to history on Special:LintErrors
Bug: T160607
Change-Id: Ifd2ef33dd7477722691c8f2e570b7ac1af206135
2017-03-22 13:28:33 -07:00
Kunal Mehta 45b4bf6382 Expose category totals in API response
Instead of just including the error category names, include the number
of errors each category has, to make it easier to collect aggregate
stats.

This changes the structure from an array to an object, in JSON, but I'm
not aware of any clients using these specific fields yet.

Change-Id: Iaf942b923a0f4047721055ad9cb48aacc5aa6784
2017-03-20 14:14:22 -07:00
Kunal Mehta 5c606fca03 Display count of lint errors on ?action=info
Change-Id: Ifcdfcb365e5ff6106b521d58a06df8c006772473
2017-01-20 11:26:44 -08:00
Kunal Mehta 8b8c50e9d8 Add Linter categories to action=query&meta=siteinfo
Bug: T153056
Change-Id: Ieed01741d31c9e268a8769a5bc5d566d8020e760
2017-01-20 11:26:21 -08:00
Kunal Mehta b201a3122c Add __METHOD__ to MWCallableUpdate
For stats and debugging purposes.

Change-Id: I5acd7fad0a9f09519595c2fa1f6232bb2624a713
2016-12-14 19:46:18 -08:00
Kunal Mehta f3778856f2 Add severity for error categories
Categories can now have a severity level of "error" or "warning"
designated, which places them in a different heading on
Special:LintErrors.

Bug: T152822
Change-Id: I1276b9502d90765e88dcb8ea78569dee910c5d88
2016-12-14 17:07:40 -08:00
jenkins-bot 69fd50c811 Merge "Record totals in categories in statsd" 2016-12-15 00:03:42 +00:00
Kunal Mehta 651d8f174b Add missing semicolon to &mdash
Change-Id: If669c89cd06ca173c0dcf6a01a793ddf8ce31793
2016-12-14 01:01:57 -08:00
Kunal Mehta 9ba2839e7d Record totals in categories in statsd
For historical purposes so we can see new errors being added/fixed.

Bug: T152664
Change-Id: Icec62d236351e43ec397ac448805ab16131c943c
2016-12-08 16:52:52 -08:00
Kunal Mehta 782313088d Show error counts on Special:LintErrors
Change-Id: Ib49f2e391ca4b9d1eef5443c011d54a42921ce4e
2016-12-08 16:52:31 -08:00
Kunal Mehta 81ef6878af Include location in LintError::id()
Since it's no longer in $this->params, we need to explicitly include it.

Change-Id: I45f26db061f02066aa03a501dc022bf33e2c096e
2016-12-02 16:15:17 -08:00
Kunal Mehta 8e2d4e42ee Use INSERT IGNORE when putting new lint errors in the database
The most likely scenario of duplicate key errors is that it's the exact
same lint error and there's just a race condition when calculating which
new errors need to be inserted, so just ignore them.

Follows-up 419610bcdb.

Change-Id: I84749ab221bbd517b474be8875bb6a59e4f3258e
2016-12-02 15:54:02 -08:00
Kunal Mehta a50398d4ce Remove unused LintError::equals()
Change-Id: I0d0d743119fee8a95627c365e1b2f45b8fcf67e7
2016-12-02 07:56:19 -08:00
Kunal Mehta 14b53d6281 Add integration tests for Database class
These tests insert variations of fake lint errors into the database, and
then read out of the database to check they round-trip properly.

And while we're at it, improve the setForPage() return value.

These tests can be run with something like:
 php tests/phpunit/phpunit.php extensions/Linter/tests/phpunit/

Change-Id: Ifdba8a8a104d218a822f909bc5d7b3512aca499d
2016-11-30 21:17:51 -08:00
Kunal Mehta 419610bcdb Enforce category/page/position uniqueness constraint in the database
Move location to two separate columns in the database: linter_start and
linter_end. This allows us to have the database enforce the uniqueness
of those fields, instead of just relying upon the PHP code to do so,
which could be bypassed since we have multiple servers and concurrent
processes.

Change-Id: I3e67ce1b7cb3c93866a388ec3248af4cff2a81e0
2016-11-30 18:55:19 -08:00
Kunal Mehta 9550db450d Fix inserting errors if none exist for that page
If no errors existed for the page, inserting new ones would fail with a
database error since $errors was key-ed by unique id, which the database
wrapper interpreted as field names, causing issues. Using array_values()
gets rid of the keys, fixing the issue.

Change-Id: I7645de4e5d3ac4462d7980374c8ef8be6280442b
2016-11-30 18:17:36 -08:00
Kunal Mehta 29379edb0b De-duplicate errors and trim excessive errors in the same category
It's possible to have duplicate, identical lint errors if the same exact
error is repeated in a template transclusion (e.g. {{1x|<b/> <b/>}})
since the position via dsr is the same. In this case, just de-duplicate
the errors since we can't differentiate them.

At the same time, trim excessive errors on the same page in the same
category. It's most likely that if a page has that many of the same
errors, the editor or bot will just fix all of them at the same time, so
we don't need to include all of them in the database. 20 is kind of a
low value, but we can always increase it later on as necessary.

Change-Id: I9cded720169870d0eea574e1a930ce4e9b190ac0
2016-11-23 19:47:04 -08:00
Kunal Mehta 70cc3ac190 Delete linter rows when pages are deleted
Bug: T151281
Change-Id: I3b113d0729b288683d9bc6c2f552b99ed2890f5c
2016-11-22 18:31:21 -08:00
Kunal Mehta 06671d2034 API: Fix lntcategories parameter
* Treat it as an array in all cases
* Make the default value all categories
* Rename from category to categories

Bug: T151288
Change-Id: I5c0c341112894c5a7ec3aaebb6ac9085353f55bd
2016-11-22 18:31:21 -08:00
Kunal Mehta 2f39ab7fff API: Fix up action=record-lint's getAllowedParams()
Since this module is internal, it doesn't really matter, but it cleans
up the output on api.php?modules=record-lint.

Bug: T151285
Change-Id: I859a2780d6ed1918cc81101e9e2c2cd348a2390a
2016-11-22 18:31:21 -08:00
Kunal Mehta d8437fc9c3 LintErrorsPager: Set explicit output format on some messages
Bug: T151283
Change-Id: Ic52bc11a6aacf7f5bb23d9273fa0bf2a27e0cfc4
2016-11-22 18:31:21 -08:00
Kunal Mehta 96c5e091f1 SpecialLintErrors: Implement getSubpagesForPrefixSearch()
Test plan:
Type "Special:LintErrors/" into the search box, and see that the
autocomplete dropdowns include the subpages for individual error
categories.

Bug: T151289
Change-Id: I919e0e51a3b956f275f9a372b2f2844002972ea7
2016-11-22 18:31:21 -08:00
Kunal Mehta b41e74ce8b Hardcode category ids
Instead of having a complex auto-increment category manager that had
additional caching, just hardcode the (currently 6) ids for each
category, which allows us to simplify a lot of code.

If Parsoid sends a lint error in a category that we don't know about, it
is silently dropped.

Bug: T151287
Change-Id: Ice6edf1b7985390aa0c1c410d357bc565bb69108
2016-11-22 18:31:17 -08:00
Justin Du 9970699910 API: Fix example URL for list=linterrors
The prefix should be lnt, not le.

Bug: T151284
Change-Id: Ib8ffbc9ee181e93da6b990ef8c788da7e49767cd
2016-11-21 20:25:29 -08:00
Arlo Breault 958623f0fb Display self-closed tags
Change-Id: I6c084a6c72114b57a21cd367d398c673c5c4d991
2016-11-17 12:32:46 -08:00
Kunal Mehta 9ecf62ead3 Update lint errors via the job queue
The job queue will allow us to have better flood control and rate
limiting instead of trying to do all the database writes as soon as
parsoid contacts MediaWiki.

On the downside, this means it may take longer for changes to be
reflected in the database and to users, but we already have no promise
for that, so it seems okay.

Note that if you don't have a job queue runner set up, you'll need to
run the runJobs.php script every time to have the jobs execute.

Change-Id: I25fd54734aca4dab09711e7f6aee027654931300
2016-11-15 11:31:45 -08:00
Kunal Mehta ec6f4722aa Store linter_cat names in a separate table
linter_cat is now an int ID that points to a row in the new
lint_categories table.

The mapping between category names and ids is all handled in PHP, and
cached in APC.

Note that you will need to drop the `linter` table manually and re-run
update.php for this to take effect.

Change-Id: I369d9b4d8d08289b4a20d1cd29a2e327bad28ef8
2016-11-03 14:51:10 -07:00
Kunal Mehta c33361e980 Show tag name for missing-end-tag too
Added in Parsoid 6d432a61f.

Change-Id: Ia716949f4fadacc3c8aab25337897c9b039f1e54
2016-10-20 14:03:25 -07:00
Kunal Mehta bce5b31616 Initial commit
This configures a MediaWiki extension to recieve Parsoid's lint errors
and expose them to users.

Change-Id: Ie0776aecf145eb1c87c2a539ddf3ea8d35a899f5
2016-10-17 16:02:53 -07:00