AbuseFilter emulates the storage mechanism also used for page content.
Instead of duplicating the relevant code, AbuseFilter should use the
same BlobStore service also used by RevisionStore.
Note that this change is not strictly needed to resolve T198341, but is
needed to unblock T183490
Bug: T261889
Bug: T198341
Bug: T183490
Change-Id: I3fc8475dd8d50d73d705b706ff597a130267e990
This is just a temporary location for these two methods. Since they're
used a lot, having them in the AbuseFilter class means that the
dependency graph is unnecessarily complicated. Thus, since these methods
aren't doing much, they were moved to a dedicated class. Future todo is
finding an appropriate location, that might be either as part of another
service, or keep them in a Utilities class, perhaps a single class with
all util methods, rather than a specific class.
Change-Id: I52cc47a6b9a387cd1e68c5127f6598a4c43ca428
Add a script to migrate the columns (which can also
be executed in dry run), and a config option with the migration stage
(defaults to SCHEMA_COMPAT_OLD).
Some of the script-related code is stolen from
Ic755526d5f989c4a66b1d37527cda235f61cb437.
Bug: T220791
Change-Id: I7460a2d63f60c2933b36f8383a8abdbba8649e12
The following sniffs are failing and were disabled:
* MediaWiki.Commenting.FunctionComment.ObjectTypeHintParam
* MediaWiki.Commenting.FunctionComment.ObjectTypeHintReturn
* MediaWiki.Commenting.PropertyDocumentation.MissingDocumentationPublic
* PSR2.ControlStructures.SwitchDeclaration.TerminatingComment
* PSR2.ControlStructures.SwitchDeclaration.WrongOpenercase
Change-Id: Ie2ce904037ef34282f0180ef6f4d8426f5a3fb6a
Unfortunately, this isn't using DI completely, because of the
User::newSystemUser call. I'm not even sure if we really need to call it
or we can just stick to new UserIdentityValue, but leaving like this for
now.
Also, the types were weakened to UserIdentity, so the transition is
going to be easy anyway.
Change-Id: I08f8fae0fcc622ff0ac3f86771476d06d1c18549
- Depend on a generic IContextSource rather than SpecialAbuseFilter
(lower coupling);
- Inject a LinkRenderer (IContextSource doesn't have a ::getLinkRenderer
method)
- Add a helper method in SpecialAbuseFilter to get the page title, that
can also be used elsewhere (and the name constant can be made private
now)
- Pull down the mFilter property (and rename it to just 'filter') to
classes that actually need it. Some classes didn't need this at all
and the types were different among subclasses
Now the only cause of coupling between the View classes and
SpecialAbuseFilter is the static call in getTitle.
Change-Id: I3df0c3a7621f0cc9a64a16b0a402a15aae2d5d73
This is I4df27f3d02432c201c04d9fa118f0129b0a79778 striking again. Fool
me once, shame on thee, fool me twice...
Change-Id: Icea025a2c81e3b413b7bd9ece52866aeaf42937d
- Include an attempt to restore the dump in case the text table
contains a truncated dump (not 100% sure that this can really
happen, nor do I know the cause, but it shouldn't hurt)
- Remove a check for 'action'. The variable might be missing in case of
a corrupted dump. Having an array at that point can only mean "new
format".
- Don't assume that old_wikitext and new_wikitext are set when showing
past filter hits (again, might be unset due to data corruption).
Bug: T264513
Change-Id: I7510d28fc3f43f985a1283e23b413f07adfe7921
These scripts were already included in the updater (and hence executed)
several MW versions ago. There's no need to write tests for them right
now, so exclude these from coverage.
Change-Id: I43e46f06b98bb3b9b9d61a45baaf232e2a99c308
This will decouple a bit the huge and chaotic tangle of AF classes. Some
boilerplate code for AbuseFilter services is also added with this patch.
Note that this requires injecting a KeywordsManager in
AbuseFilterVariableHolder, or unit tests would fail. This is still
incomplete, and the Manager is only injected in tests, because
VariableHolder still has to be refactored.
The test for the UpdateVarDumps script had to be updated, because
serializing VHs in there was a bad choice. As pointed out in a comment,
the test is likely going to break again once we remove the BC code, but
I hope that we'll be able to remove the test at that point.
Change-Id: I12a656a310adb8c5f75cab63f6db9e121e109717
A maintenance script that makes it easier for those with shell
access to search for all filters matching a regular expression
pattern on any of the wikis in a wiki farm.
Bug: T262052
Change-Id: Iea9e87a9055c0b1cedd06e8211fc99e3cef53c3a
This fixes a few minor issues noted while running in prod. Notably:
- Don't print "Printing orphaned records" in dry-run
- Print progress markers every 10 batches, not every batch
- Change the option for printing progress markers to take a file, and
recommend against stdout for big databases.
- Add an option to sleep between batches.
Bug: T252696
Bug: T246539
Change-Id: I970e7649472625ade003c259f98b611d9d3d69d2
Services are not available when the constructor is called. Since it was
used in a single method, replace the property with a local variable.
Pass it as a param, though, so that we have to instantiate the service
only once.
Bug: T213006
Change-Id: I4df27f3d02432c201c04d9fa118f0129b0a79778
Following discussion at T246938, just append data in the ES. Add a flag
to print what records will be orphaned (for manual cleanup, if needed).
Bug: T213006
Bug: T246539
Change-Id: I39bca2f07905cbf89e1906d60a568252b4729c98
- Increase batch size to 500
- Add an option to print progress markers
- Fix some bad logic which caused some JSONified data to be stored in
the text table without checking (and respecting) old_flags. This caused
some errors on the beta cluster.
Additionally, add a return typehint to AbuseFilter::loadVarDump to make
sure that errors are caught asap. Not only there's no apparent way that
loadVarDump can return an array, but most code is already using the
result as a VariableHolder, unconditionally. This is probably another
leftover from the past.
Bug: T213006
Bug: T246539
Change-Id: Iaebd28badb70d27693fa809cad4db956881e3e5e
This script aims to fix every problem reported in T213006. Subsequent
patches will add new code and drop the back-compat one.
Bug: T213006
Bug: T187153
Bug: T204236
Bug: T187731
Bug: T204235
Bug: T214193
Bug: T214196
Bug: T34478
Depends-On: I5b29ff556eca45fe59d15e2e3df4d06f1f6b3934
Change-Id: I22cf698c5be77506727cbd227c67e037a5d89b5c
This includes a bunch of improvements.
In addMissingLoggingEntries:
- Don't access mDescription directly
- Build a ManualLogEntry instead of stuffing data in the DB
In fixOldLogEntries:
- Fix entries having log_page = NULL instead of 0 due to
addMissingLoggingEntries skipping that field
- Fix entries having log_deleted = afh_deleted caused by
addMissingLoggingEntries -- those are completely unrelated
- Add batching, controlled by log_id, with default size of 500
- Use Database::strreplace to have a single UPDATE per batch, instead
of one per row.
- In dry run, when checking rows to update, exclude the rows that
would've been deleted in the first phase.
Bug: T228655
Change-Id: I885dba3f0772633d843b8a55e483047b169dc9ba
If there are no faulty entries, exit early instead of running a query
with 'log_params' => [], which would throw an exception.
Bug: T228655
Change-Id: I6a0f0439d1f3e18f90075a36894f02090cac9202
Make it also delete duplicated rows created by addMissingLoggingEntries.
Bug: T228655
Bug: T208931
Change-Id: I0e294d75e06592440773f224ef6aef172b8b9eba
This script was plagued by several problems:
- it used SUBSTRING_INDEX, thus breaking support for Postgres and
SQLite
- it didn't recognize non-legacy rows, thus creating duplicates
- it didn't extend LoggedUpdateMaintenance, but we only want it to be
executed once
- it didn't have a dry-run option
And most importantly: it inserted new rows using '\n' as separator,
instead of "\n" (note single quotes), thus creating broken entries.
Bug: T228655
Bug: T208931
Change-Id: I3a7b0fe32f1516ba21fa0ef380a9f54062e9c680
It was executed on WMF wikis, and since they were the only affected
wikis we can remove the script.
Also remove a temporary back-compat check in the log formatter.
Bug: T231131
Change-Id: I534acd9c86894eb1bdd96331e9fa85afc7502f88
A very simple fix for some old entries imploded with '\n' instead of
"\n". As for other scripts, this extends LoggedUpdateMaintenance but it
includes a dry run option to be tested, and a subsequent patch (to be
merged after testing) will add it to update.php.
Bug: T208931
Change-Id: I16d629c05c3b9c9055c68dafd261b0a5c0db9c25
afh_actions was removed from the fields list in
I48f83f64d406c553a55ac4bbee700d59002e6a18, but we actually need it to
build the new actions.
Bug: T209565
Change-Id: I21acced670c568307f87e90179ac3a9d944b0bb1
[Also make use of the list() feature in one case I forgot before in
If2b6c95.]
-> Changed to use direct array access by Daimona per inline comment.
Change-Id: I708dff30b6e00ccab3257b2e6fa5995eb9e30e0f
This is fixing potential bugs where invalid strings with more than one
comma have silently been accepted.
Change-Id: Ib1e7d0c99973f243ef6faad6389bab688187c1cf
And right when the throttle script seemed complete... Here is another
function! So, this change splits the logic in new functions called
sequentially, and the only actual change is adding the beautifyHistory
function. Its purpose is to search ANY row in abuse_filter_history with
empty/missing parameters and normalize it. More specifically, missing
period and count are inserted as 0, and for missing groups we add
"none", used by a newly introduced message. This way, messages shown on
Special:AbuseFilter/history will be clearer and won't have gaps.
Bug:T209565
Bug:T215787
Change-Id: I38395f4df9d83badfd26cdf584ffba743b6417a9
Since double-equals are evil. I left some of them in place where I
wasn't sure, but I may be changed some which were intended to be
doubles. It could be a good idea to delay merging this patch until we'll
have more code coverage.
Change-Id: I1721a3ba532d481e3ecf35f51099c1438b6b73b2
The DESC must be specified in the ORDER BY clause, or it will return the
first row instead of the last. Plus select only needed fields and remove
deprecated Database::nextSequenceValue.
Bug: T209565
Change-Id: I48f83f64d406c553a55ac4bbee700d59002e6a18
Another batch of changes for the throttle script, fixing bugs discovered
on its last run and improving performance.
For a list of fixes, see T209565#4903044.
After merging, we need backports (REL1_32 and wmf branches) and another
dry-run.
Bug: T209565
Change-Id: I530a22d57971f8b22892e43faae5d1c6fa1e14ed
This include a technical improvement (use local variables instead of
class members), and prevents displaying duplicates in the list of broken
filters.
It also covers other two quite common cases: the one where groups aren't
lowercase (for instance 'Page' instead of 'page') and extra spaces (e.g.
'user, ip' instead of 'user,ip'). The former is now fixed automatically,
while the second is a correct syntax which we don't need to fix, but now
it's effectively recognized as correct.
Bug: T209565
Change-Id: Idbfa114048bfb1127b1240c787cffa8973a47220
Instead of only the first one. This is quite hacky, but I don't have
much time right now, and this script is written in the spirit of JFDI.
Bug: T209565
Change-Id: Ic12ff21dd41b619bea9c71001555fd4d6299e58b
