Commit graph

5 commits

Author SHA1 Message Date
Kunal Mehta eacb3c0d0d Split statsd metrics by wiki
This way we can track the progress of individual wikis in cleaning up
errors. The wiki name is at the end of the key so we can still use
e.g. "linter.category.$name.*" to see across all wikis at once.

Change-Id: I62463b9256e125d32d97396bd939334d71b46027
2017-04-06 11:21:41 -07:00
Kunal Mehta 9ba2839e7d Record totals in categories in statsd
For historical purposes so we can see new errors being added/fixed.

Bug: T152664
Change-Id: Icec62d236351e43ec397ac448805ab16131c943c
2016-12-08 16:52:52 -08:00
Kunal Mehta 419610bcdb Enforce category/page/position uniqueness constraint in the database
Move location to two separate columns in the database: linter_start and
linter_end. This allows us to have the database enforce the uniqueness
of those fields, instead of just relying upon the PHP code to do so,
which could be bypassed since we have multiple servers and concurrent
processes.

Change-Id: I3e67ce1b7cb3c93866a388ec3248af4cff2a81e0
2016-11-30 18:55:19 -08:00
Kunal Mehta 29379edb0b De-duplicate errors and trim excessive errors in the same category
It's possible to have duplicate, identical lint errors if the same exact
error is repeated in a template transclusion (e.g. {{1x|<b/> <b/>}})
since the position via dsr is the same. In this case, just de-duplicate
the errors since we can't differentiate them.

At the same time, trim excessive errors on the same page in the same
category. It's most likely that if a page has that many of the same
errors, the editor or bot will just fix all of them at the same time, so
we don't need to include all of them in the database. 20 is kind of a
low value, but we can always increase it later on as necessary.

Change-Id: I9cded720169870d0eea574e1a930ce4e9b190ac0
2016-11-23 19:47:04 -08:00
Kunal Mehta 9ecf62ead3 Update lint errors via the job queue
The job queue will allow us to have better flood control and rate
limiting instead of trying to do all the database writes as soon as
parsoid contacts MediaWiki.

On the downside, this means it may take longer for changes to be
reflected in the database and to users, but we already have no promise
for that, so it seems okay.

Note that if you don't have a job queue runner set up, you'll need to
run the runJobs.php script every time to have the jobs execute.

Change-Id: I25fd54734aca4dab09711e7f6aee027654931300
2016-11-15 11:31:45 -08:00