Commit graph

111 commits

Author SHA1 Message Date
Daimona Eaytoy 97afa97403 Exclude old, single-use scripts from coverage reports
These scripts were already included in the updater (and hence executed)
several MW versions ago. There's no need to write tests for them right
now, so exclude these from coverage.

Change-Id: I43e46f06b98bb3b9b9d61a45baaf232e2a99c308
2020-09-30 12:42:34 +02:00
Daimona Eaytoy 55ba083b13 Introduce a KeywordsManager service
This will decouple a bit the huge and chaotic tangle of AF classes. Some
boilerplate code for AbuseFilter services is also added with this patch.

Note that this requires injecting a KeywordsManager in
AbuseFilterVariableHolder, or unit tests would fail. This is still
incomplete, and the Manager is only injected in tests, because
VariableHolder still has to be refactored.

The test for the UpdateVarDumps script had to be updated, because
serializing VHs in there was a bad choice. As pointed out in a comment,
the test is likely going to break again once we remove the BC code, but
I hope that we'll be able to remove the test at that point.

Change-Id: I12a656a310adb8c5f75cab63f6db9e121e109717
2020-09-28 23:03:52 +00:00
Huji Lee 43e471d056 Introduce searchFilters.php
A maintenance script that makes it easier for those with shell
access to search for all filters matching a regular expression
pattern on any of the wikis in a wiki farm.

Bug: T262052
Change-Id: Iea9e87a9055c0b1cedd06e8211fc99e3cef53c3a
2020-09-16 09:03:43 -04:00
Daimona Eaytoy d3b21901a2 updateVarDumps: Add more options, aesthetic changes
This fixes a few minor issues noted while running in prod. Notably:

 - Don't print "Printing orphaned records" in dry-run
 - Print progress markers every 10 batches, not every batch
 - Change the option for printing progress markers to take a file, and
 recommend against stdout for big databases.
 - Add an option to sleep between batches.

Bug: T252696
Bug: T246539
Change-Id: I970e7649472625ade003c259f98b611d9d3d69d2
2020-05-18 15:42:00 +02:00
Daimona Eaytoy 120d1500a4 updateVarDumps: wait for replication after each batch
Otherwise the script could destroy the wikis...

Change-Id: If5f86952cb4927d612d6e2243df0823025ec5bd5
2020-05-07 17:51:36 +02:00
Daimona Eaytoy 8d25609290 updateVarDumps: move MediaWikiServices away from constructor
Services are not available when the constructor is called. Since it was
used in a single method, replace the property with a local variable.
Pass it as a param, though, so that we have to instantiate the service
only once.

Bug: T213006
Change-Id: I4df27f3d02432c201c04d9fa118f0129b0a79778
2020-04-30 18:01:36 +00:00
Daimona Eaytoy 11f5790eb7 updateVarDumps: Print orphaned ES records, don't try updating ES records
Following discussion at T246938, just append data in the ES. Add a flag
to print what records will be orphaned (for manual cleanup, if needed).

Bug: T213006
Bug: T246539
Change-Id: I39bca2f07905cbf89e1906d60a568252b4729c98
2020-03-12 15:09:59 +01:00
Daimona Eaytoy cd1a8efb90 Minor fixes for the updateVarDumps script
- Increase batch size to 500
- Add an option to print progress markers
- Fix some bad logic which caused some JSONified data to be stored in
the text table without checking (and respecting) old_flags. This caused
some errors on the beta cluster.

Additionally, add a return typehint to AbuseFilter::loadVarDump to make
sure that errors are caught asap. Not only there's no apparent way that
loadVarDump can return an array, but most code is already using the
result as a VariableHolder, unconditionally. This is probably another
leftover from the past.

Bug: T213006
Bug: T246539
Change-Id: Iaebd28badb70d27693fa809cad4db956881e3e5e
2020-03-03 18:31:52 +00:00
Daimona Eaytoy 2c03c77d9f Add a maintenance script to clean afl_var_dump
This script aims to fix every problem reported in T213006. Subsequent
patches will add new code and drop the back-compat one.

Bug: T213006
Bug: T187153
Bug: T204236
Bug: T187731
Bug: T204235
Bug: T214193
Bug: T214196
Bug: T34478
Depends-On: I5b29ff556eca45fe59d15e2e3df4d06f1f6b3934
Change-Id: I22cf698c5be77506727cbd227c67e037a5d89b5c
2020-02-28 19:41:30 +00:00
Daimona Eaytoy 9d8e004972 fixOldLogEntries: don't pass empty lists to Database::makeList
This made the script fail at a later stage, although it didn't cause any
harm.

Bug: T228655
Change-Id: I0c983cb4b5a262e81372fbd23f893ffb176eb9b5
2020-02-10 17:24:18 +01:00
Daimona Eaytoy d79970ef38 fixOldLogEntries: Add missing pairs of parentheses
Bug: T228655
Change-Id: I3bdb7e9efc8e12b055ba37440f1197fe6ed3b69d
2020-02-08 15:18:52 +01:00
Daimona Eaytoy 54512dd124 Fix more inconsistencies in scripts for logging entries
This includes a bunch of improvements.

In addMissingLoggingEntries:
 - Don't access mDescription directly
 - Build a ManualLogEntry instead of stuffing data in the DB

In fixOldLogEntries:
 - Fix entries having log_page = NULL instead of 0 due to
 addMissingLoggingEntries skipping that field
 - Fix entries having log_deleted = afh_deleted caused by
 addMissingLoggingEntries -- those are completely unrelated
 - Add batching, controlled by log_id, with default size of 500
 - Use Database::strreplace to have a single UPDATE per batch, instead
 of one per row.
 - In dry run, when checking rows to update, exclude the rows that
 would've been deleted in the first phase.

Bug: T228655
Change-Id: I885dba3f0772633d843b8a55e483047b169dc9ba
2020-02-03 15:30:07 +00:00
Daimona Eaytoy 395bb9c7e7 Fix silly bug in fixOldLogEntries
If there are no faulty entries, exit early instead of running a query
with 'log_params' => [], which would throw an exception.

Bug: T228655
Change-Id: I6a0f0439d1f3e18f90075a36894f02090cac9202
2019-12-16 17:26:15 +00:00
Daimona Eaytoy 2755685b5c Update fixOldLogEntries
Make it also delete duplicated rows created by addMissingLoggingEntries.

Bug: T228655
Bug: T208931
Change-Id: I0e294d75e06592440773f224ef6aef172b8b9eba
2019-12-03 17:57:33 +00:00
Daimona Eaytoy 302c967ce7 Fix addMissingLogEntriesScript
This script was plagued by several problems:
 - it used SUBSTRING_INDEX, thus breaking support for Postgres and
 SQLite
 - it didn't recognize non-legacy rows, thus creating duplicates
 - it didn't extend LoggedUpdateMaintenance, but we only want it to be
 executed once
 - it didn't have a dry-run option

And most importantly: it inserted new rows using '\n' as separator,
instead of "\n" (note single quotes), thus creating broken entries.

Bug: T228655
Bug: T208931
Change-Id: I3a7b0fe32f1516ba21fa0ef380a9f54062e9c680
2019-12-03 17:45:00 +00:00
Daimona Eaytoy 173bd089b3 Remove script for blockautopromote entries
It was executed on WMF wikis, and since they were the only affected
wikis we can remove the script.
Also remove a temporary back-compat check in the log formatter.

Bug: T231131
Change-Id: I534acd9c86894eb1bdd96331e9fa85afc7502f88
2019-09-09 13:56:56 +02:00
Daimona Eaytoy 7f554734e6 Don't hardcode blockautopromote duration
As explained on phab, and add a script to fix broken entries.

Bug: T231131
Change-Id: I95d70acb936b5ca987af8f237d236fe47b663919
2019-08-24 11:40:11 +02:00
Umherirrender cd4e0a5968 Use Maintenance::addDescription
It is better to use a setter function than setting a property

Change-Id: I64c133f00ea12cc5ba787c83ad3dee5f996baf15
2019-08-19 18:53:57 +02:00
Daimona Eaytoy 1860b4ca5a fixOldLogEntries: Don't print an extra line
Bug: T228655
Change-Id: Idce1f121e468ce0ac0720951f28613e6cc564113
2019-07-24 11:23:28 +00:00
Daimona Eaytoy 40ebf9ad6d Aesthetic changes for fixOldLogEntries script
Bug: T228655
Change-Id: Ic94a32ec906eb49db85c96212718bb3cf03f86af
2019-07-23 13:21:00 +00:00
Daimona Eaytoy 07e5168eb9 Add a maintenance script to fix broken log entries
A very simple fix for some old entries imploded with '\n' instead of
"\n". As for other scripts, this extends LoggedUpdateMaintenance but it
includes a dry run option to be tested, and a subsequent patch (to be
merged after testing) will add it to update.php.

Bug: T208931
Change-Id: I16d629c05c3b9c9055c68dafd261b0a5c0db9c25
2019-07-15 11:59:45 +00:00
Daimona Eaytoy f0e428d4c3 Fix query in normalizeThrottleParameters
afh_actions was removed from the fields list in
I48f83f64d406c553a55ac4bbee700d59002e6a18, but we actually need it to
build the new actions.

Bug: T209565
Change-Id: I21acced670c568307f87e90179ac3a9d944b0bb1
2019-07-05 18:46:36 +02:00
Thiemo Kreuz aba489a1f4 Add missing type hint to SpecialPage::execute()
[Also make use of the list() feature in one case I forgot before in
If2b6c95.]
 -> Changed to use direct array access by Daimona per inline comment.

Change-Id: I708dff30b6e00ccab3257b2e6fa5995eb9e30e0f
2019-05-16 14:31:54 +00:00
Thiemo Kreuz c6f20a64dd Add missing limits to explode() calls
This is fixing potential bugs where invalid strings with more than one
comma have silently been accepted.

Change-Id: Ib1e7d0c99973f243ef6faad6389bab688187c1cf
2019-05-15 16:14:12 +02:00
jenkins-bot 903f3db8fe Merge "Beautify old, broken abuse_filter_history rows" 2019-04-10 05:11:38 +00:00
Daimona Eaytoy 25ed009518 Beautify old, broken abuse_filter_history rows
And right when the throttle script seemed complete... Here is another
function! So, this change splits the logic in new functions called
sequentially, and the only actual change is adding the beautifyHistory
function. Its purpose is to search ANY row in abuse_filter_history with
empty/missing parameters and normalize it. More specifically, missing
period and count are inserted as 0, and for missing groups we add
"none", used by a newly introduced message. This way, messages shown on
Special:AbuseFilter/history will be clearer and won't have gaps.

Bug:T209565
Bug:T215787
Change-Id: I38395f4df9d83badfd26cdf584ffba743b6417a9
2019-04-10 04:51:58 +00:00
Daimona Eaytoy b4346678c6 Replace deprecated wfWaitForSlaves
Deprecated since 1.27.

Change-Id: I16787520dd4939e6d1579261ac35a4a1f6288b70
2019-03-29 11:01:37 +01:00
Max Semenik 9afac9d6ff Use setDescription() instead of accessing guts directly
Change-Id: I369b88419afc6550922caf625059ad9570428cac
2019-03-25 19:51:20 -07:00
Daimona Eaytoy f2c1beec44 Replace double-equals with triple-equals
Since double-equals are evil. I left some of them in place where I
wasn't sure, but I may be changed some which were intended to be
doubles. It could be a good idea to delay merging this patch until we'll
have more code coverage.

Change-Id: I1721a3ba532d481e3ecf35f51099c1438b6b73b2
2019-03-22 16:12:13 +01:00
jenkins-bot b4780723e4 Merge "Fix faulty query in normalizeThrottleParameters" 2019-03-17 09:50:39 +00:00
Umherirrender da682f78ac Use ::class for class name resolution
Available since php5.5

Change-Id: Ie60465484858867d507081dfe5a305c6ab0599de
2019-03-03 01:41:40 +01:00
Daimona Eaytoy 1244960435 Fix faulty query in normalizeThrottleParameters
The DESC must be specified in the ORDER BY clause, or it will return the
first row instead of the last. Plus select only needed fields and remove
deprecated Database::nextSequenceValue.

Bug: T209565
Change-Id: I48f83f64d406c553a55ac4bbee700d59002e6a18
2019-03-02 12:57:41 +00:00
Daimona Eaytoy 15019ed34d Update the throttle script
Another batch of changes for the throttle script, fixing bugs discovered
on its last run and improving performance.
For a list of fixes, see T209565#4903044.
After merging, we need backports (REL1_32 and wmf branches) and another
dry-run.

Bug: T209565
Change-Id: I530a22d57971f8b22892e43faae5d1c6fa1e14ed
2019-01-24 00:13:09 +01:00
Daimona Eaytoy af9c7ee852 Re-fix the throttle script
This include a technical improvement (use local variables instead of
class members), and prevents displaying duplicates in the list of broken
filters.
It also covers other two quite common cases: the one where groups aren't
lowercase (for instance 'Page' instead of 'page') and extra spaces (e.g.
'user, ip' instead of 'user,ip'). The former is now fixed automatically,
while the second is a correct syntax which we don't need to fix, but now
it's effectively recognized as correct.

Bug: T209565
Change-Id: Idbfa114048bfb1127b1240c787cffa8973a47220
2019-01-09 15:58:49 +01:00
Daimona Eaytoy ea89dd4ca3 Report all filters with wrong throttle parameters
Instead of only the first one. This is quite hacky, but I don't have
much time right now, and this script is written in the spirit of JFDI.

Bug: T209565
Change-Id: Ic12ff21dd41b619bea9c71001555fd4d6299e58b
2018-12-17 20:19:20 +01:00
Daimona Eaytoy aa280998c0 Fix big problems with normalizeThrottleParameters
My final testing unveiled 4 problems, see T209565#4780868. Testing again
after this patch yields the expected outcome.

Update: A fifth problem is that we cannot disable throttling if throttle
groups are empty or fully invalid: that case is similar to the one with
invalid rate, the throttle limit is never reached and thus throttle just
doesn't work. Instead, ask users to fix it by hand.

Bug: T203336
Bug: T209565
Change-Id: Id03c9880f60764efc596ac40b8662087fdb30550
2018-12-15 18:36:16 +01:00
Daimona Eaytoy d3a8491c3f Change throttle selector to restore old functionality, overall improvement
Long (sigh) explanation in T203587#4569698. Also, simplified the way
TagMultiselect are generated, this one and the one for change tags.
This new selector is back-compat both with the old textarea and the OOUI
checkboxMultiselect; actually, this one is //fully// compatible with the
old textarea.
Add validation for throttle parameters and unit tests for validation
(split from I976c95658cddb2585910b6f8a5f047aadc4e4d47).
Added a trim when retrieving throttle identifier to allow syntax like
'ip, user'.
Improved the message shown on history.
Re-added the maintenance script to clean DB.

As I wrote in the task, a review by two other people would be great, at
least for the maintenance script (it could potentially break the DB).

Bug: T203587
Bug: T203336
Bug: T203584
Bug: T203585
Depends-On: I3b2e763bd8835207dc5df1db43d3e1881e6961c3
Change-Id: I7831dbb0bab55807392ac1f7915d6cb0cb713593
2018-11-14 12:51:36 +01:00
Matěj Suchánek 45d1d71def Reduce use of globals in favor of Config
I'd like to have this reviewed by more than one user before merging, to avoid regressions of annoying typos.

Change-Id: I91a9c5cca55e540a6c95b750579c1c369a760b15
2018-05-02 02:27:26 +00:00
Daimona Eaytoy 3c3a521fec Fix coding conventions exclusion rules
This should fix every error with excluded rules, leaving only the one
for $wgTitle. A double check would be nice in order to avoid regressions
due to stupid mistakes.

Bug: T178007
Change-Id: I22c179f3a01d652640304b59e43fcb5b5a9abac3
2018-04-20 08:40:18 +00:00
Brad Jorsch 3014871cb5 Update for the actor table change
Core change I8d825eb0 begins the process of changing core database
tables from using xx_user and xx_user_text fields to using xx_actor.
This updates the extension to continue to function during and after the
transition.

Bug: T167246
Change-Id: I4065716022aa60c0fa1a258659db22be2b7f43de
2018-03-09 16:34:24 -05:00
Brad Jorsch 7399cd4348 Update CommentStore usage
CommentStore's calling conventions changed in I3abb62a5c.

Change-Id: I80012f82b39e5054ee40a44b5a8e92dec46c2962
2018-03-09 16:22:32 -05:00
Huji Lee 8ca391c8e0 Correctly reference the "Abuse Filter" extension in maintenance scripts
Bug: T186928
Change-Id: I415f39aa618a6148ad29a3fb4b059aadbfd63da5
2018-02-10 16:13:41 +00:00
Max Semenik 58d29d04a1 Use getBatchSize() instead of direct field access
Change-Id: If2dd3a6b80a228605138ae7b6c8da718c0f07d62
2017-11-07 19:39:34 -08:00
jenkins-bot 689b7abaff Merge "Remove back-compat code paths" 2017-10-31 16:12:40 +00:00
Brad Jorsch 6071e7a43f Remove back-compat code paths
Per T178092, AbuseFilter now maintains compatibility with older versions
of MediaWiki using release branches. Thus, various back-compat code
paths may be removed from the master branch.

Change-Id: Ia1b5eade30d7486e3b1b386b15a7db4e5c8cfead
2017-10-31 09:37:54 -04:00
Simeon Dahl 4c0636bc0a Cleanup, added spaces
Added spaces in some classes so it follow the same style as the others.
Edit is pure cleanup.

Change-Id: If5d5e6e4e99eed83aa69dfb4a224fbcc7c077d43
2017-10-19 09:40:22 +02:00
Brad Jorsch cfa98448c3 Use CommentStore to access core comment fields when available
See core change Ic3a434c0.

Bug: T166732
Change-Id: I130510381ef97d8a2d29686843e2710cb5f72195
2017-08-30 13:27:46 -04:00
Max Semenik 0686f99653 DB_SLAVE -> DB_REPLICA
Change-Id: I371e092b3cd7f0af3770bf4e64b01a630e23ff92
2017-08-29 19:51:39 -07:00
Umherirrender a063e33ee8 Use short array syntax
Done by phpcbf over composer fix

Change-Id: I53fd1fc8d056b9b60194d2d630852cfca37aadea
2017-06-15 17:02:57 +02:00
pppery 66f52694a2 Make maintenance scripts use $this->requireExtension()
Bug: T152139
Change-Id: I9c3a6a186275b6554a761de8548bc16f8938d488
2016-12-12 22:42:00 +00:00
Paladox b3081e1798 Update mediawiki/mediawiki-codesniffer to 0.5.1
Change-Id: I4b2055a76db4362a8136e3fd595228cf07d083a9
2016-08-23 16:18:27 +02:00
Siebrand Mazeland ce1396aea7 Add support for PHP CodeSniffer checks
Also fix any remaining errors and warnings.

Change-Id: Ie49c6172e6bbf8040e5524d33d2f719e96784745
2016-01-06 09:59:47 -08:00
jenkins-bot 5f1447d698 Merge "No longer call removed SpecialPage::getLocalPageForName" 2015-08-22 23:29:39 +00:00
Alex Monk d010f3c0c9 No longer call removed SpecialPage::getLocalPageForName
Removed in I5122f0f2, which was after this script was written but long before
it was merged.

Bug: T54919
Change-Id: If67a5f137590414ba8c6c958d819e87099a3b010
2015-08-23 00:01:18 +01:00
jenkins-bot 01a024ffdc Merge "Fix addMissingLoggingEntries maintenance script handling of newlines" 2015-08-22 22:45:22 +00:00
Alex Monk e3db4db83e Fix addMissingLoggingEntries maintenance script handling of newlines
Bug: T54919
Change-Id: I0a979d9022ce3371813e214a7e4e1432e34c823b
2015-08-22 23:28:18 +01:00
jenkins-bot 75cb547f27 Merge "Add maintenance script to restore missing log entries for each filter history entry" 2015-08-11 19:46:45 +00:00
Krenair 1b9c71fabe Add maintenance script to restore missing log entries for each filter history entry
Quite a few of these occured on wikis between r48501 (2009-03-17) and r52734 (2009-07-03)

Bug: 52919
Change-Id: Ib33d0a1814fe131a5291ed7408057515482aa61f
2015-08-08 01:32:37 +01:00
Aaron Schulz 4219fa141c Fixed broken batching in –PurgeOldLogIPData
Bug: T95382
Change-Id: I133642e5dd8e75e98e6189ad10f54dbbea648c8d
2015-04-07 19:28:26 -07:00
Reedy b7fa8d5d20 $res is a bool, not a resultwrapper..
Display running count. Silent scripts are annoying

Change-Id: I4dcebbf1cd49dce6332ed1bdc0e2ba7aced4b8f1
2014-01-03 18:06:49 +00:00
Reedy cac808a35c Write maintenance script for AbuseFilter to purge old IP data from logs
Bug: 51573
Change-Id: I93392d77fb6172d55353d2ba7fb1be3a19f13b11
2013-11-19 21:44:14 +00:00