This will decouple a bit the huge and chaotic tangle of AF classes. Some
boilerplate code for AbuseFilter services is also added with this patch.
Note that this requires injecting a KeywordsManager in
AbuseFilterVariableHolder, or unit tests would fail. This is still
incomplete, and the Manager is only injected in tests, because
VariableHolder still has to be refactored.
The test for the UpdateVarDumps script had to be updated, because
serializing VHs in there was a bad choice. As pointed out in a comment,
the test is likely going to break again once we remove the BC code, but
I hope that we'll be able to remove the test at that point.
Change-Id: I12a656a310adb8c5f75cab63f6db9e121e109717
A maintenance script that makes it easier for those with shell
access to search for all filters matching a regular expression
pattern on any of the wikis in a wiki farm.
Bug: T262052
Change-Id: Iea9e87a9055c0b1cedd06e8211fc99e3cef53c3a
This fixes a few minor issues noted while running in prod. Notably:
- Don't print "Printing orphaned records" in dry-run
- Print progress markers every 10 batches, not every batch
- Change the option for printing progress markers to take a file, and
recommend against stdout for big databases.
- Add an option to sleep between batches.
Bug: T252696
Bug: T246539
Change-Id: I970e7649472625ade003c259f98b611d9d3d69d2
Services are not available when the constructor is called. Since it was
used in a single method, replace the property with a local variable.
Pass it as a param, though, so that we have to instantiate the service
only once.
Bug: T213006
Change-Id: I4df27f3d02432c201c04d9fa118f0129b0a79778
Following discussion at T246938, just append data in the ES. Add a flag
to print what records will be orphaned (for manual cleanup, if needed).
Bug: T213006
Bug: T246539
Change-Id: I39bca2f07905cbf89e1906d60a568252b4729c98
- Increase batch size to 500
- Add an option to print progress markers
- Fix some bad logic which caused some JSONified data to be stored in
the text table without checking (and respecting) old_flags. This caused
some errors on the beta cluster.
Additionally, add a return typehint to AbuseFilter::loadVarDump to make
sure that errors are caught asap. Not only there's no apparent way that
loadVarDump can return an array, but most code is already using the
result as a VariableHolder, unconditionally. This is probably another
leftover from the past.
Bug: T213006
Bug: T246539
Change-Id: Iaebd28badb70d27693fa809cad4db956881e3e5e
This script aims to fix every problem reported in T213006. Subsequent
patches will add new code and drop the back-compat one.
Bug: T213006
Bug: T187153
Bug: T204236
Bug: T187731
Bug: T204235
Bug: T214193
Bug: T214196
Bug: T34478
Depends-On: I5b29ff556eca45fe59d15e2e3df4d06f1f6b3934
Change-Id: I22cf698c5be77506727cbd227c67e037a5d89b5c
This includes a bunch of improvements.
In addMissingLoggingEntries:
- Don't access mDescription directly
- Build a ManualLogEntry instead of stuffing data in the DB
In fixOldLogEntries:
- Fix entries having log_page = NULL instead of 0 due to
addMissingLoggingEntries skipping that field
- Fix entries having log_deleted = afh_deleted caused by
addMissingLoggingEntries -- those are completely unrelated
- Add batching, controlled by log_id, with default size of 500
- Use Database::strreplace to have a single UPDATE per batch, instead
of one per row.
- In dry run, when checking rows to update, exclude the rows that
would've been deleted in the first phase.
Bug: T228655
Change-Id: I885dba3f0772633d843b8a55e483047b169dc9ba
If there are no faulty entries, exit early instead of running a query
with 'log_params' => [], which would throw an exception.
Bug: T228655
Change-Id: I6a0f0439d1f3e18f90075a36894f02090cac9202
Make it also delete duplicated rows created by addMissingLoggingEntries.
Bug: T228655
Bug: T208931
Change-Id: I0e294d75e06592440773f224ef6aef172b8b9eba
This script was plagued by several problems:
- it used SUBSTRING_INDEX, thus breaking support for Postgres and
SQLite
- it didn't recognize non-legacy rows, thus creating duplicates
- it didn't extend LoggedUpdateMaintenance, but we only want it to be
executed once
- it didn't have a dry-run option
And most importantly: it inserted new rows using '\n' as separator,
instead of "\n" (note single quotes), thus creating broken entries.
Bug: T228655
Bug: T208931
Change-Id: I3a7b0fe32f1516ba21fa0ef380a9f54062e9c680
It was executed on WMF wikis, and since they were the only affected
wikis we can remove the script.
Also remove a temporary back-compat check in the log formatter.
Bug: T231131
Change-Id: I534acd9c86894eb1bdd96331e9fa85afc7502f88
A very simple fix for some old entries imploded with '\n' instead of
"\n". As for other scripts, this extends LoggedUpdateMaintenance but it
includes a dry run option to be tested, and a subsequent patch (to be
merged after testing) will add it to update.php.
Bug: T208931
Change-Id: I16d629c05c3b9c9055c68dafd261b0a5c0db9c25
afh_actions was removed from the fields list in
I48f83f64d406c553a55ac4bbee700d59002e6a18, but we actually need it to
build the new actions.
Bug: T209565
Change-Id: I21acced670c568307f87e90179ac3a9d944b0bb1
[Also make use of the list() feature in one case I forgot before in
If2b6c95.]
-> Changed to use direct array access by Daimona per inline comment.
Change-Id: I708dff30b6e00ccab3257b2e6fa5995eb9e30e0f
This is fixing potential bugs where invalid strings with more than one
comma have silently been accepted.
Change-Id: Ib1e7d0c99973f243ef6faad6389bab688187c1cf
And right when the throttle script seemed complete... Here is another
function! So, this change splits the logic in new functions called
sequentially, and the only actual change is adding the beautifyHistory
function. Its purpose is to search ANY row in abuse_filter_history with
empty/missing parameters and normalize it. More specifically, missing
period and count are inserted as 0, and for missing groups we add
"none", used by a newly introduced message. This way, messages shown on
Special:AbuseFilter/history will be clearer and won't have gaps.
Bug:T209565
Bug:T215787
Change-Id: I38395f4df9d83badfd26cdf584ffba743b6417a9
Since double-equals are evil. I left some of them in place where I
wasn't sure, but I may be changed some which were intended to be
doubles. It could be a good idea to delay merging this patch until we'll
have more code coverage.
Change-Id: I1721a3ba532d481e3ecf35f51099c1438b6b73b2
The DESC must be specified in the ORDER BY clause, or it will return the
first row instead of the last. Plus select only needed fields and remove
deprecated Database::nextSequenceValue.
Bug: T209565
Change-Id: I48f83f64d406c553a55ac4bbee700d59002e6a18
Another batch of changes for the throttle script, fixing bugs discovered
on its last run and improving performance.
For a list of fixes, see T209565#4903044.
After merging, we need backports (REL1_32 and wmf branches) and another
dry-run.
Bug: T209565
Change-Id: I530a22d57971f8b22892e43faae5d1c6fa1e14ed
This include a technical improvement (use local variables instead of
class members), and prevents displaying duplicates in the list of broken
filters.
It also covers other two quite common cases: the one where groups aren't
lowercase (for instance 'Page' instead of 'page') and extra spaces (e.g.
'user, ip' instead of 'user,ip'). The former is now fixed automatically,
while the second is a correct syntax which we don't need to fix, but now
it's effectively recognized as correct.
Bug: T209565
Change-Id: Idbfa114048bfb1127b1240c787cffa8973a47220
Instead of only the first one. This is quite hacky, but I don't have
much time right now, and this script is written in the spirit of JFDI.
Bug: T209565
Change-Id: Ic12ff21dd41b619bea9c71001555fd4d6299e58b
My final testing unveiled 4 problems, see T209565#4780868. Testing again
after this patch yields the expected outcome.
Update: A fifth problem is that we cannot disable throttling if throttle
groups are empty or fully invalid: that case is similar to the one with
invalid rate, the throttle limit is never reached and thus throttle just
doesn't work. Instead, ask users to fix it by hand.
Bug: T203336
Bug: T209565
Change-Id: Id03c9880f60764efc596ac40b8662087fdb30550
Long (sigh) explanation in T203587#4569698. Also, simplified the way
TagMultiselect are generated, this one and the one for change tags.
This new selector is back-compat both with the old textarea and the OOUI
checkboxMultiselect; actually, this one is //fully// compatible with the
old textarea.
Add validation for throttle parameters and unit tests for validation
(split from I976c95658cddb2585910b6f8a5f047aadc4e4d47).
Added a trim when retrieving throttle identifier to allow syntax like
'ip, user'.
Improved the message shown on history.
Re-added the maintenance script to clean DB.
As I wrote in the task, a review by two other people would be great, at
least for the maintenance script (it could potentially break the DB).
Bug: T203587
Bug: T203336
Bug: T203584
Bug: T203585
Depends-On: I3b2e763bd8835207dc5df1db43d3e1881e6961c3
Change-Id: I7831dbb0bab55807392ac1f7915d6cb0cb713593
I'd like to have this reviewed by more than one user before merging, to avoid regressions of annoying typos.
Change-Id: I91a9c5cca55e540a6c95b750579c1c369a760b15
This should fix every error with excluded rules, leaving only the one
for $wgTitle. A double check would be nice in order to avoid regressions
due to stupid mistakes.
Bug: T178007
Change-Id: I22c179f3a01d652640304b59e43fcb5b5a9abac3
Core change I8d825eb0 begins the process of changing core database
tables from using xx_user and xx_user_text fields to using xx_actor.
This updates the extension to continue to function during and after the
transition.
Bug: T167246
Change-Id: I4065716022aa60c0fa1a258659db22be2b7f43de
Per T178092, AbuseFilter now maintains compatibility with older versions
of MediaWiki using release branches. Thus, various back-compat code
paths may be removed from the master branch.
Change-Id: Ia1b5eade30d7486e3b1b386b15a7db4e5c8cfead
