Commit graph

335 commits

Author SHA1 Message Date
Daimona Eaytoy 71a61c2089 Add value objects to represent filters
This is just a start; next step is adding a factory/store method to
get/store these objects. And then use these value objects whenever
applicable.

Note: the actions-related code is still not fully implemented. This is
going to happen as part of the FilterLookup.

Change-Id: I5f33227887c035e301313bbe24d1c1fefb75bc6a
2020-11-04 12:56:14 +01:00
Daimona Eaytoy 1f8df50cb3 Add a service to retrieve the central DB
This is a thin wrapper around LBFactory and the global variable, that
can be injected in classes requiring it (no real class right now, but
that's going to change soon).

Also, remove some DWIM-style returns which made the code harder to
understand.

Change-Id: I1d28ad4a67f914103f3a17cda5f61b28070c7f1c
2020-10-31 12:32:46 +00:00
jenkins-bot ec5b9bef44 Merge "Add a service to retrieve the filter user" 2020-10-29 09:52:56 +00:00
Matěj Suchánek 77f6ecce13 Cleanup FilterProfiler API
Make FilterProfiler::getFilterProfile return stats unchanged,
in a structured way. Move computations to AbuseFilterViewEdit,
as they are only useful there. Don't return false on cache
misses, return arrays with zero values instead.

Bug: T266531
Change-Id: I8718cc31a5004340bf742315c7075e10a61fcbfd
2020-10-28 12:48:30 +00:00
Daimona Eaytoy 916234598d Simplify ViewEdit, last round
This deals with data inconsistencies in buildFilterEditor. Every
property of $row was tested in all 5 scenarios (also using Selenium) to
check when it's set. The result is in the normalizeRow method, which
aims to remove any inconsistencies, so that buildFilterEditor always
receives a "complete" row with all defaults set.

The code in buildFilterEditor is now cleaner (because there are no
isset() checks), and it gives us a unique place where we can set
defaults (rather than partly doing that in
loadRequest/loadFilterData/loadImport, and partly relying on isset).

This will be especially useful when introducing value objects to
represent filters, because now you just have to look at normalizeRow()
to tell which properties are allowed to be missing, and thus what "kind"
of filter object you need (see
I5f33227887c035e301313bbe24d1c1fefb75bc6a).

Additionally, reduce the properties that get passed around during
export/import, and make the selenium test try a roundtrip, rather than
relying on hardcoded data that may get outdated. A future patch will
refactor the import/export code.

Change-Id: Id52c466baaf6da18e2981f27a81ffdad3a509e78
2020-10-26 13:07:29 +00:00
Daimona Eaytoy cbea88f818 Add a service to retrieve the filter user
Unfortunately, this isn't using DI completely, because of the
User::newSystemUser call. I'm not even sure if we really need to call it
or we can just stick to new UserIdentityValue, but leaving like this for
now.
Also, the types were weakened to UserIdentity, so the transition is
going to be easy anyway.

Change-Id: I08f8fae0fcc622ff0ac3f86771476d06d1c18549
2020-10-26 14:06:53 +01:00
jenkins-bot 50ae561641 Merge "Simplify ViewEdit, round 2" 2020-10-25 09:10:11 +00:00
Daimona Eaytoy 18ade98339 tests: Move any profiling-related test to FilterProfilerTest
This commit removes several tests from AbuseFilterConsequences, thus
speeding it up a lot (especially because these tests were very slow,
with each test *case* taking up to 30s in the coverage job).

Everything is now covered by the new AbuseFilterFilterProfilerTest
which, although not being a pure unit test, is much much faster than
*Consequences.

Change-Id: Ic6b16d23ec99abee287f36093b8573505f9c613a
2020-10-24 18:09:26 +02:00
Matěj Suchánek 6d81fca76b Improve FilterProfiler coverage
Also improve documentation of some FilterProfiler methods.

Change-Id: I08198c643a7d2dac10e928914e8a5c7413f2543d
2020-10-24 16:23:47 +02:00
Matěj Suchánek 1445d5962a Introduce BlockAutopromoteStore service
This service is responsible for the blockautopromote feature:
(un)block autopromotion and check status.

The patch mostly moves code from static methods to the new class
and relaxes type hints (e.g. from User to UserIdentity).

Change-Id: I79a72377881cf06717931cd09af12f3b8e5f3e3f
2020-10-24 12:31:44 +00:00
Daimona Eaytoy 416dcd9ba3 Simplify ViewEdit, round 2
- Add a helper method to output an unrecoverable error, comprising a
   button to go back to the filters list;
- Move the token check to attemptSave, so to make the conditionals
  easier to read, and group errors together
- Make buildFilterEditor take an HTML parameter for the error, so the
  caller can specify whether it's error or warning
- Move the check for non-existing filters out of buildFilterEditor
- Add a bunch of typehints
- Don't set af_throttled and af_hit_count in the empty row template, but
  set af_deleted (these are only used in buildFilterEditor)
- Make AbuseFilter::translateFromHistory consistently include the af_global
  property (previously it would only be set for global filters; this error
  was introduced when first implementing global filters)
- The only user-facing change is that, when trying to use a custom
  warning/disallow message on a global filter, this is now considered a
  non-fatal error, so we now show the editing interface (and not just an
  unrecoverable error).

The next step is resolving the @todo in buildFilterEditor about null
checks.

Change-Id: I9d217dcac3f4cc0b26e53eca735cc327d5efc76d
2020-10-23 13:00:43 +00:00
Daimona Eaytoy 4c06dd52c8 Replace $wgAbuseFilterRestrictions with more specific variables
So that sysadmins can further customize the extension. It was also wrong
to use the same variable for many different things.

Note that there's no associated patch in wmf-config because we use the
defaults. However, before merging this patch, please recheck that
AbuseFilterRestrictions and AbuseFilterDisallowGlobalLocalBlocks aren't
used there (https://codesearch.wmflabs.org/operations/?q=AbuseFilterDisallowGlobalLocalBlocks%7CAbuseFilterRestrictions&i=nope&files=&repos=)

Bug: T175221
Change-Id: I7581b3ee6d9d11a6cf1599b8ff874e8c3d54adf4
2020-10-22 13:38:59 +00:00
Matěj Suchánek 93556284a0 Inject ChangeTagsManager to ChangeTagger
We decided to have the tag name provided by ChangeTagsManager,
so make ChangeTagger depend on it.

Change-Id: If3cbfd992f45651f47477031befffc0fd30f4a28
2020-10-21 16:30:43 +02:00
Daimona Eaytoy 9bc885b6b3 Add a ChangeTagger class
The logic about action IDs and the persistent buffer is now encapsulated
inside a single service, which is a step towards getting rid of global
state in the AbuseFilter class, and reducing the responsibilities of the
Runner.

An important change made here is that we now require a LinkTarget rather
than a Title. This removes a dependency on the Title class (a monster
object), makes tests simpler, and denies the need to inject a
TitleFactory. This means living without some bits of context (e.g. we're
no longer using makeTitleSafe to ensure a valid title, and we have to
build a "prefixedtext" manually), but this shouldn't be a problem, given
that the titles are only used to create a cache key: invalid titles are
not a problem, and concatenating namespace + title should always be
sufficient.

Bug: T265370
Change-Id: Iff59cd3d889454a482a89c16691bfefcc5ec0a12
2020-10-21 13:19:30 +02:00
Matěj Suchánek d91ddd2169 Cleanup filter id handling on Special:AbuseFilter/history
In particular, the interface shouldn't generate links to
"Special:AbuseFilter/history/0" (AbuseFilterHistoryPager::getTitle,
can be seen when visiting "Special:AbuseFilter/history").

Change-Id: Id3dc1bb4fc3c5e853603bf0ec04a6b1751f7d862
2020-10-10 11:40:46 +00:00
Daimona Eaytoy f0539e0c1e Represent new filters with null instead of 'new'
PHP is not strongly typed, so it's not a good idea to use scalars of
different types (here it's an integer vs the string 'new') to represent
different possibilities. This can have bad effects when type juggling
occurs, and it's also harder to figure out what the type of the
parameter can be (because a numeric ID might have been passed as a
string). Using integer vs null avoids all of this, and also allows us to
use nullable typehints.

These changes were partly copied from
If981cb35bf19a8469aa6c43c907e107cf8c65bc2 and should help with the
migration to the Filter value objects.

Change-Id: I8837d46c3c33761fea53f67b530b721dc7bd49b0
2020-10-10 12:23:50 +02:00
jenkins-bot c0defc1055 Merge "Add a new FilterProfiler service" 2020-10-10 10:08:58 +00:00
Daimona Eaytoy 9f2906e34b Reduce dependencies of AbuseFilter::saveFilter
This patch removes the dependency of saveFilter on the ContextSource
kitchen sink. It also removes some unneded dependency, and adds
$originalRow/$originalActions as parameter, rather than hacky properties
in $newRow that are easy to forget. The related test can also be greatly
simplified.

This also introduces a behaviour change: checking $newRow instead of the Request allows us
to account for values normalization done in
AbuseFilterViewEdit::loadRequest, and to also work correctly for imports
(and generally speaking, it makes the method suitable for an
AbuseFilterEdit API module, too).

Next step is moving this method to a service. Some signatures,
indenting, name choices etc. are subpar, but this is just because these
methods are temporary anyway.

Bug: T213037
Change-Id: I235b928d7b9c2ef1c46ea0bf3e3ed212500b4161
2020-10-09 11:52:02 +00:00
jenkins-bot c6ab7b8e1f Merge "Use triple equals in abuse filter parser tests" 2020-10-06 17:42:29 +00:00
Huji Lee d07717dd0d Use triple equals in abuse filter parser tests
The only exception is mwexamples-comparisons.t which intentionally
includes examples with = and == to test the "weak" version of the
comparison operator.

Bug: T262063
Change-Id: I6f92aadc69489da481a606bfda89617b8efbb261
2020-10-06 12:29:47 -04:00
Daimona Eaytoy bc9898f1a1 Add a new FilterProfiler service
Change-Id: Ib66c42ac220731f4e1da9ee6cfb5290759dd6494
2020-10-04 22:00:57 +00:00
Daimona Eaytoy 6c8a29698b Add test traits for uploads and account creation
Ideally, this might live in MediaWikiIntegrationTestCase. For the
createaccount one, AuthManager should also provide a method to log the
creation, because currently we are forced to copypaste that code here.

 - Add the missing tests for 'upload' in RCVariableGenerator, and adjust
the existing ones (delete file afterwards, more tablesUsed, use the
right extension).

 - Exclude from the coverage report a couple of lines which should
theoretically be unreachable. Escalate logging to WARN level, where it's
more likely to be spotted.

 - Remove an unused method (RCVariableGenerator::newFromID). This denies
   the need to maintain and cover it. We also don't want this generator
   to act as a factory.

Overall, this change brings the coverage for RCVariableGenerator to 100%

Bug: T201193
Change-Id: I425c3d9f6800f74eb6e4eda483b90cfb3bbbcb51
2020-10-04 13:16:58 +00:00
Daimona Eaytoy 2e13d58c74 Add tests for retrieving RC variables
This was also long overdue. Also fix a bug that caused page creations to
not be shown when examining past edits (using rc_last_oldid doesn't work
for page creations).

Bug: T201193
Bug: T262903
Change-Id: I5f7a994add12332c950904146248c5de7c2beee5
2020-10-04 12:43:04 +00:00
jenkins-bot a9654c3ab3 Merge "Refactor AbuseFilterView instantiation" 2020-10-04 12:28:24 +00:00
Matěj Suchánek eb81b92c06 Refactor AbuseFilterView instantiation
- Make a separate method which determines the view
  to be shown from subpage syntax and test it.
- Reduce circular dependency between SpecialAbuseFilter
  and AbuseFilterView. Use params to transfer information
  to views.

Change-Id: Ib9442ea5f9990a5c48f9b9e04055aa22bf7e456e
2020-10-04 13:15:04 +02:00
Daimona Eaytoy 60519e1886 updateVarDumps: avoid using services in the constructor
This is I4df27f3d02432c201c04d9fa118f0129b0a79778 striking again. Fool
me once, shame on thee, fool me twice...

Change-Id: Icea025a2c81e3b413b7bd9ece52866aeaf42937d
2020-10-03 23:23:37 +00:00
Daimona Eaytoy 7bc5248ed7 Selenium tests: wait for clickable button
It's possible that we try clicking this button before it's ready. This
theory is hard to verify because I get no problem locally, but this
shouldn't hurt.

Also specialize input selectors for a couple elements that might not be
univocally determined within the page.

Change-Id: Ida65c3c5fd4d8b3b35ecbee7e99977c71c7c4b96
2020-10-01 00:51:35 +02:00
jenkins-bot fa412f4e7e Merge "Rewrite the VariableHolder code to translate deprecated variables" 2020-09-30 09:10:37 +00:00
Daimona Eaytoy 1bdf4e5351 Rewrite the VariableHolder code to translate deprecated variables
The current code was more of a subpar, temporary solution. However, we
need a stable solution in case more variables will be deprecated in the
future (T213006 fixes the problem for the past deprecation round). So,
instead of setting a hacky property, directly translate all variables
when loading the var dump. This is not only stable, but has a couple
micro-performance advantages:
 - Calling getDeprecatedVariables happens only once when loading the
   dump, and not every time a variable is accessed
 - No checks are needed when retrieving a variable,
   because names can always assumed to be new

Some simple benchmarks reveals a runtime reduction of 8-15% compared to
the old code (8% when it had varsVersion = 2, 15% for varsVersion = 1),
which comes at no cost together with increased readability and
stability. It ain't much, but it's honest work.

Change-Id: Ib32a92c4ad939790633aa63eb3ef8d4629488bea
2020-09-29 15:06:14 +00:00
Daimona Eaytoy 62adeb3ce5 Add a lot of selenium tests for the editing view
The editing view is currently full of tech debt, brittle and surprising
code and whatnot. It's basically a miracle if it works without problem,
and it'd be an even bigger miracle if you could change something there
without breaking anything.

For these reasons, and because that class must be refactored as part of
the upcoming overhaul, this patch adds a bunch of selenium tests to test
the main functionality of that page.

In particular, these tests cover all possible cases (each corresponding
to a data source) for which buildFilterEditor can be called, which FTR are:
1 - View the result of importing a filter
2 - Create a new filter
3 - Load the current version of an existing filter
4 - Load an old version of an existing filter
5 - Show the user input again if saving fails after one of the steps
  above

Having automated tests to cover these cases means that we don't have to
manually test all the scenarios manually each time the class is touched.

Bug: T201193
Change-Id: I408e0a132905416effe0d6d6dc0921991edd66bd
2020-09-29 14:22:53 +00:00
jenkins-bot 7a684c487c Merge "Move some misplaced AbuseFilterParser entry points" 2020-09-29 13:51:17 +00:00
Daimona Eaytoy 55ba083b13 Introduce a KeywordsManager service
This will decouple a bit the huge and chaotic tangle of AF classes. Some
boilerplate code for AbuseFilter services is also added with this patch.

Note that this requires injecting a KeywordsManager in
AbuseFilterVariableHolder, or unit tests would fail. This is still
incomplete, and the Manager is only injected in tests, because
VariableHolder still has to be refactored.

The test for the UpdateVarDumps script had to be updated, because
serializing VHs in there was a bad choice. As pointed out in a comment,
the test is likely going to break again once we remove the BC code, but
I hope that we'll be able to remove the test at that point.

Change-Id: I12a656a310adb8c5f75cab63f6db9e121e109717
2020-09-28 23:03:52 +00:00
Daimona Eaytoy a1626a0d7f Move some misplaced AbuseFilterParser entry points
These methods had no reals reason to be static and belong to the
AbuseFilter class. Most of them were moved to Parser class as common
variations of the existing entry points. One was specific to the
EvalExpression API module and was moved there.

This change comes at no cost, and will make it possible to inject a
parser where needed.

Change-Id: Ifd169cfc99df8a5eb4ca94ac330f301ca28a2442
2020-09-29 00:36:08 +02:00
Daimona Eaytoy 8fa9e6625a Add tests for 'upload' action
This adds some coverage for the *VariableGenerator classes. It's still
not perfect, but something to start with in sight of future
refactorings.

Bug: T201193
Change-Id: Iafa85fb8623ea278ce6e42118df72751806382c2
2020-09-28 11:53:53 +00:00
Daimona Eaytoy 241a5db604 tests: Create dedicated classes for VariableGenerators
Also fix the test for _first_contributor vars to cover all variables,
use builtin methods to compute the var (rather than calling
setLazyLoadVar), and to be an integration test.

Change-Id: I2594439acc786e31bce1cd4373d3cbf434204eda
2020-09-28 11:53:47 +00:00
Reedy 41a403ebd8 Give AbuseFilterSaveTest::testSaveFilter() a return value for selectRow
Change-Id: Ifdf6d2155ec3d51600bceaa63832bb55a71599d3
2020-09-24 18:25:26 +01:00
Daimona Eaytoy e5746bbb0e parser: Add a BC option to get DNULL for unset variables
While checking a filter, if a variable is not set (e.g. added_lines for
an account creation), the VariableHolder will return a DNULL, rather
than a DUNDEFINED. This means that some filters will resume working, and
the WMF servers will stop getting AF warnings at a rate of 4 millions per
day. This also requires adjusting some tests to reflect the new
behaviour (which is actually the OLD behaviour, that filters had until
last year when we introduced the DUNDEFINED data type). It also requires
adjusting a check in the old parser, but that's not really relevant
because the plan is to remove the old parser before 1.36 is released
(see I0e75f334c7e0dfc1239f2e5f5f7d7452b0bbf29e).

Bug: T230256
Change-Id: I4d06303047397674c1edbfc32628f1bc83ac3340
2020-09-18 15:05:58 +02:00
jenkins-bot 92c3c8dbe1 Merge "Revert "Fix a test which will be broken by Hooks::run() migration"" 2020-09-16 08:10:07 +00:00
Daimona Eaytoy 5d2eaa052c Revert "Fix a test which will be broken by Hooks::run() migration"
This reverts commit be3fbdf347.

Reason for revert: The commit mentioned here was merged a few months ago, so this temporary workaround can now be reverted.

Change-Id: I1e8b4a23b700a2b566c087b8a3ea2229c95bcc3f
2020-09-16 07:06:06 +00:00
proc a31f4e46af
Strict type comparison
Bug: T248806
Change-Id: I039ab7f103bb37052987b815412b71f70643a6d2
2020-06-27 15:55:57 +01:00
jaredblumer 12f9be5e69 eslint: Update to eslint-config-wikimedia 0.16.0
* Update ESLint config with Selenium WebdriverIO test suite
* Update modules and Selenium pageobjects and specs per ESLint
requirements
* Update grunt-eslint package to 23.0.0 as required by
eslint-config-wikimedia 0.16.0

Bug: T254495
Change-Id: Ibfcf9115adedf9f2c3e7dac1ac626b41fc97b7c4
2020-06-08 21:17:50 -04:00
vidhi-mody 76025137a3 Selenium: Update to WebdriverIO v6
Update NPM packages: @wdio/* and webdriverio.

Bug: T253167
Change-Id: I11edb5dca8d1d45ffc336c805a40c502b815ce1f
2020-05-28 19:56:43 +05:30
vidhi-mody f2759a17c9 Selenium: Update to WebdriverIO v5
Update NPM packages: webdriverio, wdio-mediawiki.
Update ESlint configuration.

Replace NPM packages:
- wdio-mocha-framework with @wdio/mocha-framework.
- wdio-spec-reporter with @wdio/spec-reporter.

New NPM packages: @wdio/cli, @wdio/local-runner, @wdio/sync.

Replace:
- `browser.element` with `$`.
- `chromeOptions` with `'goog:chromeOptions'`.
- `password` with `mwPwd`.
- `username` with `mwUser`.

Bug: T253167
Change-Id: Ia26da6d0f56398a908bd4e99ae77db603647f533
2020-05-21 12:36:32 +00:00
Tim Starling be3fbdf347 Fix a test which will be broken by Hooks::run() migration
Temporarily mark the test skipped on earlier versions of core, to avoid
a circular dependency which blocks the merge.

Change-Id: I5db9937f249edaf7c070b2436c0caea369dac8ef
2020-05-05 11:11:54 +10:00
jenkins-bot b118fd50dc Merge "Improve var dumping in /details, /examine and /tools" 2020-04-29 20:00:54 +00:00
Ed Sanders 2af7bc67c7 eslint: Remove unused rules
Change-Id: I306c25c3f604893dea24b542619832a5cd3d1759
2020-04-24 22:21:15 +01:00
DannyS712 1b65bd1862 Remove a remaining use of Revision objects
Remove use of Title::getFirstRevision and Revision::getUserText

Bug: T249393
Bug: T250579
Change-Id: I0f77b124a0c7de1dec6baf4c997e0997ecdd55f8
2020-04-23 18:39:20 +00:00
Daimona Eaytoy 1d6b9f6617 Add new methods for checking DUNDEFINED recursively, use them
The problem is explained at T250570#6068702; basically, the previous
check didn't account for DUNDEFINED nested deep inside arrays.

Bug: T250570
Change-Id: Iacee2db54ca00108de6339bb3dae70af7e2eeb56
2020-04-19 13:58:14 +02:00
Aaron Schulz 904da3d080 Remove references to "abuse_filter_actions" from tests
Change-Id: Idfe61e8f2654ee9953d37ce84a9e10f622b23ca4
2020-04-06 16:06:15 -07:00
Daimona Eaytoy 4c98aecf4d Improve var dumping in /details, /examine and /tools
Using var_export for better visual effect, especially for arrays.
The result from /tools is much clearer and the 'wrong syntax' message is
a bit more explicative than before.

Bug: T190653
Bug: T239972
Change-Id: I79a17305c7f19f7900f896f895e9365bb5f2fd58
2020-03-28 17:35:43 +01:00