AbuseFilter emulates the storage mechanism also used for page content.
Instead of duplicating the relevant code, AbuseFilter should use the
same BlobStore service also used by RevisionStore.
Note that this change is not strictly needed to resolve T198341, but is
needed to unblock T183490
Bug: T261889
Bug: T198341
Bug: T183490
Change-Id: I3fc8475dd8d50d73d705b706ff597a130267e990
This is just a temporary location for these two methods. Since they're
used a lot, having them in the AbuseFilter class means that the
dependency graph is unnecessarily complicated. Thus, since these methods
aren't doing much, they were moved to a dedicated class. Future todo is
finding an appropriate location, that might be either as part of another
service, or keep them in a Utilities class, perhaps a single class with
all util methods, rather than a specific class.
Change-Id: I52cc47a6b9a387cd1e68c5127f6598a4c43ca428
Add a script to migrate the columns (which can also
be executed in dry run), and a config option with the migration stage
(defaults to SCHEMA_COMPAT_OLD).
Some of the script-related code is stolen from
Ic755526d5f989c4a66b1d37527cda235f61cb437.
Bug: T220791
Change-Id: I7460a2d63f60c2933b36f8383a8abdbba8649e12
The following sniffs are failing and were disabled:
* MediaWiki.Commenting.FunctionComment.ObjectTypeHintParam
* MediaWiki.Commenting.FunctionComment.ObjectTypeHintReturn
* MediaWiki.Commenting.PropertyDocumentation.MissingDocumentationPublic
* PSR2.ControlStructures.SwitchDeclaration.TerminatingComment
* PSR2.ControlStructures.SwitchDeclaration.WrongOpenercase
Change-Id: Ie2ce904037ef34282f0180ef6f4d8426f5a3fb6a
Unfortunately, this isn't using DI completely, because of the
User::newSystemUser call. I'm not even sure if we really need to call it
or we can just stick to new UserIdentityValue, but leaving like this for
now.
Also, the types were weakened to UserIdentity, so the transition is
going to be easy anyway.
Change-Id: I08f8fae0fcc622ff0ac3f86771476d06d1c18549
- Depend on a generic IContextSource rather than SpecialAbuseFilter
(lower coupling);
- Inject a LinkRenderer (IContextSource doesn't have a ::getLinkRenderer
method)
- Add a helper method in SpecialAbuseFilter to get the page title, that
can also be used elsewhere (and the name constant can be made private
now)
- Pull down the mFilter property (and rename it to just 'filter') to
classes that actually need it. Some classes didn't need this at all
and the types were different among subclasses
Now the only cause of coupling between the View classes and
SpecialAbuseFilter is the static call in getTitle.
Change-Id: I3df0c3a7621f0cc9a64a16b0a402a15aae2d5d73
This is I4df27f3d02432c201c04d9fa118f0129b0a79778 striking again. Fool
me once, shame on thee, fool me twice...
Change-Id: Icea025a2c81e3b413b7bd9ece52866aeaf42937d
- Include an attempt to restore the dump in case the text table
contains a truncated dump (not 100% sure that this can really
happen, nor do I know the cause, but it shouldn't hurt)
- Remove a check for 'action'. The variable might be missing in case of
a corrupted dump. Having an array at that point can only mean "new
format".
- Don't assume that old_wikitext and new_wikitext are set when showing
past filter hits (again, might be unset due to data corruption).
Bug: T264513
Change-Id: I7510d28fc3f43f985a1283e23b413f07adfe7921
These scripts were already included in the updater (and hence executed)
several MW versions ago. There's no need to write tests for them right
now, so exclude these from coverage.
Change-Id: I43e46f06b98bb3b9b9d61a45baaf232e2a99c308
This will decouple a bit the huge and chaotic tangle of AF classes. Some
boilerplate code for AbuseFilter services is also added with this patch.
Note that this requires injecting a KeywordsManager in
AbuseFilterVariableHolder, or unit tests would fail. This is still
incomplete, and the Manager is only injected in tests, because
VariableHolder still has to be refactored.
The test for the UpdateVarDumps script had to be updated, because
serializing VHs in there was a bad choice. As pointed out in a comment,
the test is likely going to break again once we remove the BC code, but
I hope that we'll be able to remove the test at that point.
Change-Id: I12a656a310adb8c5f75cab63f6db9e121e109717
A maintenance script that makes it easier for those with shell
access to search for all filters matching a regular expression
pattern on any of the wikis in a wiki farm.
Bug: T262052
Change-Id: Iea9e87a9055c0b1cedd06e8211fc99e3cef53c3a
This fixes a few minor issues noted while running in prod. Notably:
- Don't print "Printing orphaned records" in dry-run
- Print progress markers every 10 batches, not every batch
- Change the option for printing progress markers to take a file, and
recommend against stdout for big databases.
- Add an option to sleep between batches.
Bug: T252696
Bug: T246539
Change-Id: I970e7649472625ade003c259f98b611d9d3d69d2
Services are not available when the constructor is called. Since it was
used in a single method, replace the property with a local variable.
Pass it as a param, though, so that we have to instantiate the service
only once.
Bug: T213006
Change-Id: I4df27f3d02432c201c04d9fa118f0129b0a79778
Following discussion at T246938, just append data in the ES. Add a flag
to print what records will be orphaned (for manual cleanup, if needed).
Bug: T213006
Bug: T246539
Change-Id: I39bca2f07905cbf89e1906d60a568252b4729c98
- Increase batch size to 500
- Add an option to print progress markers
- Fix some bad logic which caused some JSONified data to be stored in
the text table without checking (and respecting) old_flags. This caused
some errors on the beta cluster.
Additionally, add a return typehint to AbuseFilter::loadVarDump to make
sure that errors are caught asap. Not only there's no apparent way that
loadVarDump can return an array, but most code is already using the
result as a VariableHolder, unconditionally. This is probably another
leftover from the past.
Bug: T213006
Bug: T246539
Change-Id: Iaebd28badb70d27693fa809cad4db956881e3e5e
This script aims to fix every problem reported in T213006. Subsequent
patches will add new code and drop the back-compat one.
Bug: T213006
Bug: T187153
Bug: T204236
Bug: T187731
Bug: T204235
Bug: T214193
Bug: T214196
Bug: T34478
Depends-On: I5b29ff556eca45fe59d15e2e3df4d06f1f6b3934
Change-Id: I22cf698c5be77506727cbd227c67e037a5d89b5c
This includes a bunch of improvements.
In addMissingLoggingEntries:
- Don't access mDescription directly
- Build a ManualLogEntry instead of stuffing data in the DB
In fixOldLogEntries:
- Fix entries having log_page = NULL instead of 0 due to
addMissingLoggingEntries skipping that field
- Fix entries having log_deleted = afh_deleted caused by
addMissingLoggingEntries -- those are completely unrelated
- Add batching, controlled by log_id, with default size of 500
- Use Database::strreplace to have a single UPDATE per batch, instead
of one per row.
- In dry run, when checking rows to update, exclude the rows that
would've been deleted in the first phase.
Bug: T228655
Change-Id: I885dba3f0772633d843b8a55e483047b169dc9ba
If there are no faulty entries, exit early instead of running a query
with 'log_params' => [], which would throw an exception.
Bug: T228655
Change-Id: I6a0f0439d1f3e18f90075a36894f02090cac9202
Make it also delete duplicated rows created by addMissingLoggingEntries.
Bug: T228655
Bug: T208931
Change-Id: I0e294d75e06592440773f224ef6aef172b8b9eba
This script was plagued by several problems:
- it used SUBSTRING_INDEX, thus breaking support for Postgres and
SQLite
- it didn't recognize non-legacy rows, thus creating duplicates
- it didn't extend LoggedUpdateMaintenance, but we only want it to be
executed once
- it didn't have a dry-run option
And most importantly: it inserted new rows using '\n' as separator,
instead of "\n" (note single quotes), thus creating broken entries.
Bug: T228655
Bug: T208931
Change-Id: I3a7b0fe32f1516ba21fa0ef380a9f54062e9c680
It was executed on WMF wikis, and since they were the only affected
wikis we can remove the script.
Also remove a temporary back-compat check in the log formatter.
Bug: T231131
Change-Id: I534acd9c86894eb1bdd96331e9fa85afc7502f88
A very simple fix for some old entries imploded with '\n' instead of
"\n". As for other scripts, this extends LoggedUpdateMaintenance but it
includes a dry run option to be tested, and a subsequent patch (to be
merged after testing) will add it to update.php.
Bug: T208931
Change-Id: I16d629c05c3b9c9055c68dafd261b0a5c0db9c25
afh_actions was removed from the fields list in
I48f83f64d406c553a55ac4bbee700d59002e6a18, but we actually need it to
build the new actions.
Bug: T209565
Change-Id: I21acced670c568307f87e90179ac3a9d944b0bb1
[Also make use of the list() feature in one case I forgot before in
If2b6c95.]
-> Changed to use direct array access by Daimona per inline comment.
Change-Id: I708dff30b6e00ccab3257b2e6fa5995eb9e30e0f
This is fixing potential bugs where invalid strings with more than one
comma have silently been accepted.
Change-Id: Ib1e7d0c99973f243ef6faad6389bab688187c1cf
And right when the throttle script seemed complete... Here is another
function! So, this change splits the logic in new functions called
sequentially, and the only actual change is adding the beautifyHistory
function. Its purpose is to search ANY row in abuse_filter_history with
empty/missing parameters and normalize it. More specifically, missing
period and count are inserted as 0, and for missing groups we add
"none", used by a newly introduced message. This way, messages shown on
Special:AbuseFilter/history will be clearer and won't have gaps.
Bug:T209565
Bug:T215787
Change-Id: I38395f4df9d83badfd26cdf584ffba743b6417a9
Since double-equals are evil. I left some of them in place where I
wasn't sure, but I may be changed some which were intended to be
doubles. It could be a good idea to delay merging this patch until we'll
have more code coverage.
Change-Id: I1721a3ba532d481e3ecf35f51099c1438b6b73b2
The DESC must be specified in the ORDER BY clause, or it will return the
first row instead of the last. Plus select only needed fields and remove
deprecated Database::nextSequenceValue.
Bug: T209565
Change-Id: I48f83f64d406c553a55ac4bbee700d59002e6a18
Another batch of changes for the throttle script, fixing bugs discovered
on its last run and improving performance.
For a list of fixes, see T209565#4903044.
After merging, we need backports (REL1_32 and wmf branches) and another
dry-run.
Bug: T209565
Change-Id: I530a22d57971f8b22892e43faae5d1c6fa1e14ed
This include a technical improvement (use local variables instead of
class members), and prevents displaying duplicates in the list of broken
filters.
It also covers other two quite common cases: the one where groups aren't
lowercase (for instance 'Page' instead of 'page') and extra spaces (e.g.
'user, ip' instead of 'user,ip'). The former is now fixed automatically,
while the second is a correct syntax which we don't need to fix, but now
it's effectively recognized as correct.
Bug: T209565
Change-Id: Idbfa114048bfb1127b1240c787cffa8973a47220
Instead of only the first one. This is quite hacky, but I don't have
much time right now, and this script is written in the spirit of JFDI.
Bug: T209565
Change-Id: Ic12ff21dd41b619bea9c71001555fd4d6299e58b
My final testing unveiled 4 problems, see T209565#4780868. Testing again
after this patch yields the expected outcome.
Update: A fifth problem is that we cannot disable throttling if throttle
groups are empty or fully invalid: that case is similar to the one with
invalid rate, the throttle limit is never reached and thus throttle just
doesn't work. Instead, ask users to fix it by hand.
Bug: T203336
Bug: T209565
Change-Id: Id03c9880f60764efc596ac40b8662087fdb30550
Long (sigh) explanation in T203587#4569698. Also, simplified the way
TagMultiselect are generated, this one and the one for change tags.
This new selector is back-compat both with the old textarea and the OOUI
checkboxMultiselect; actually, this one is //fully// compatible with the
old textarea.
Add validation for throttle parameters and unit tests for validation
(split from I976c95658cddb2585910b6f8a5f047aadc4e4d47).
Added a trim when retrieving throttle identifier to allow syntax like
'ip, user'.
Improved the message shown on history.
Re-added the maintenance script to clean DB.
As I wrote in the task, a review by two other people would be great, at
least for the maintenance script (it could potentially break the DB).
Bug: T203587
Bug: T203336
Bug: T203584
Bug: T203585
Depends-On: I3b2e763bd8835207dc5df1db43d3e1881e6961c3
Change-Id: I7831dbb0bab55807392ac1f7915d6cb0cb713593
I'd like to have this reviewed by more than one user before merging, to avoid regressions of annoying typos.
Change-Id: I91a9c5cca55e540a6c95b750579c1c369a760b15
This should fix every error with excluded rules, leaving only the one
for $wgTitle. A double check would be nice in order to avoid regressions
due to stupid mistakes.
Bug: T178007
Change-Id: I22c179f3a01d652640304b59e43fcb5b5a9abac3
Core change I8d825eb0 begins the process of changing core database
tables from using xx_user and xx_user_text fields to using xx_actor.
This updates the extension to continue to function during and after the
transition.
Bug: T167246
Change-Id: I4065716022aa60c0fa1a258659db22be2b7f43de
Per T178092, AbuseFilter now maintains compatibility with older versions
of MediaWiki using release branches. Thus, various back-compat code
paths may be removed from the master branch.
Change-Id: Ia1b5eade30d7486e3b1b386b15a7db4e5c8cfead
Removed in I5122f0f2, which was after this script was written but long before
it was merged.
Bug: T54919
Change-Id: If67a5f137590414ba8c6c958d819e87099a3b010
Quite a few of these occured on wikis between r48501 (2009-03-17) and r52734 (2009-07-03)
Bug: 52919
Change-Id: Ib33d0a1814fe131a5291ed7408057515482aa61f
