This commit adds a class AFPSyntaxChecker which can statically analyze
a filter code to detect the following errors:
- unbound variables (which comes in two modes: conservative and liberal,
default to conservative)
- unused variables (disabled by default for compatibilty)
- assignment on built-in identifiers
- function application's arity mismatch
- function application's invalid function name
- non-string literal in the first argument of set / set_var
The existing parser and evaluator are modified as follows:
- The new (caching) evaluator no longer needs to perform variable
hoisting at runtime.
- Note that for array assignment, this changes the semantics.
- The new parser is more lenient, reducing parsing errors.
The static analyzer will catch these errors instead, allowing us
to give a much better error message and reduces the complexity of
the parser.
* The parser now allows function name to be any identifier.
* The parser now allows arity mismatch to occur.
* The parser now allows the first argument of set to be any expression.
Concretely, obvious changes that users will see are:
1. a := [1]; false & (a[] := 2); a[0] === 1
would evaluate to true, while it used to evaluate to the undefined value
due to hoisting
2. f(1)
will now error with 'f is not a valid function' as opposed to
'Unexpected "T_BRACE"'
3. length
will now error with 'Illegal use of built-in identifier "length"'
as opposed to 'Expected a ('
Appendix: conservative and liberal mode
The conservative mode is completely compatible with the current evaluator.
That is,
false & (a := 1); a
will not deem `a` as unbound, though this is actually undesirable because
`a` would then be bound to the troublesome undefined value.
The liberal mode rejects the above pattern by deeming `a` as unbound.
However, it also rejects
true & (a := 1); a
even though (a := 1) is always executed. Since there are several filters
in Wikimedia projects that rely on this behavior, we default the mode
to conservative for now.
Note that even the liberal mode doesn't really respect lexical scope
appeared in some other programming languages (see also T234690).
For instance:
(if true then (a := 1) else (a := 2) end); a
would be accepted by the liberal checker, even though under lexical scope,
`a` would be unbound. However, it is unlikely that lexical scope
will be suitable for the filter language, as most filters in
Wikimedia projects that have user-defined variable do violate lexical scope.
Bug: T260903
Bug: T238709
Bug: T237610
Bug: T234690
Bug: T231536
Change-Id: Ic6d030503e554933f8d220c6f87b680505918ae2
Create a dedicated "Exception" sub-namespace and remove the "AFP"
prefix, a leftover from the pre-namespace era.
Change-Id: I7e5fded9316d8b7d1628bc1a6ba8b1879ac901e1
Regression tests to make sure T286140 does not
happen again.
In the process, discovered what caused that bug
with afl_rev_id not being set: EditRevUpdater::updateRev()
compares the WikiPage given in the PageSaveComplete hook
to the one given to it by AbuseFilterHooks from
onEditFilterMergedContent, and compares the two using
`===`, meaning that they must refer to the same underlying
object. That bug was caused because AbuseFilterHooks
changed to providing a different object, despite still
referring to the same underlying page.
We should probably change that behavior in EditRevUpdater,
but for now updated AbuseFilterConsequencesTest to pass
the same object around by using RequestContext::setWikiPage()
and providing the WikiPage object to
MediaWikiIntegrationTestCase::editPage().
Bug: T286140
Change-Id: I6562f513c463538af6b59b12a64564b254024613
These are part of legacy styles and aren't provided by all skins.
Using Html::successbox abstracts the classes away.
Internally that uses div class="successbox" instead.
Bug: T280766
Change-Id: I0cca59e2f391510095c2c6fb187ace5e91fdde8b
All methods were moved to the new parser. Tests and other pieces were
adjusted to expect just a single parser. There are still some TODOs
(remove AFPTransitionBase, remove $this->mCur), but these are left for
another commit.
Note that the new parser was not renamed: this is because the names are
wrong anyway (CachingParser is more of an Evaluator than a Parser, and
AFPTreeParser is the real parser, and should be renamed as well).
NOTE to reviewers: this patch looks quite big, but if you diff the old
parser with the new version of the CachingParser, you'll notice that the
diff is actually small, since everything was basically copied verbatim.
Bug: T239990
Change-Id: Ie914ef64c70503a201b4d2dec698ca2fa8e69b10
The actorId parameter to the UserIdentityValue constructor has been
deprecated.
Change-Id: I4a22e761276a9fefa15c7b1554a0d03980d0c663
Needed-By: I9925906d11e47efaec3c1f48d5cb3f9896a982c1
Since WebdriverIO v5, Puppeteer is available as a Chromedriver
alternative.
Puppeteer is bundled with WebdriverIO. Chromedriver needs to be installed
(and started/stopped) separately. Getting rid of Chromedriver simplifies
our documentation, among other things.
The commit updates tests/selenium/wdio.conf.js to use Puppeteer.
Bug: T269566
Change-Id: Ib2a547792a34e6d40137432f7800b5f71c254c36
The hook names contain a dash, which is mapped to an underscore by the
hook runner (see Ie8c8fb603b33ff95c8f8d52f392227f147c528d8), and the
previous method names weren't matching this.
Follow-up: Ic5c82a367e34135bbc0f00ece5aeef4f2d92881b
Change-Id: Ie80b62c49b2f4aaea49d5a1883f513348689d16a
Requires injecting a temporary block factory, and excluding
ManualLogEntry::insert from the test, but it's now much cleaner and
quicker.
It still cannot be a unit test due to the usage of User.
Change-Id: Iba9732d6d79733b31b45eb4d0187b1c8a82499dc
It is currently possible to save a filter with an invalid group, if you
manually change the form data. So prevent this by validating the group
before saving.
Change-Id: I03f80b8c6ab583a357273f7b2679a424ac784db7
The actor ID is being removed from UserIdentityValue. Non-zero values
are triggering a deprecation warning now.
Needed-By: I9925906d11e47efaec3c1f48d5cb3f9896a982c1
Change-Id: Id60e56e70f6e4b44f49887d9e5ae5a23b1fd19a2
This commit doesn't change any permissions for anybody.
It's the first step to achieve what the task asks for.
Bug: T242821
Change-Id: I8060ca926e6769b11d470fe4037854cda496000d
1 - Change the structure of if/elseif for readability
2 - In the old parser, if there's an empty argument, never add it (the
new parser was already doing that).
Bug: T156095
Bug: T156096
Change-Id: I4237b1a0ba01e7ce04dcc945f7daf34612fcf07d
Introduce a clear distinction between internal exceptions and
user-visible exceptions, leaving AFPException as base abstract class.
Later, it should be possible to narrow some types around, e.g. in
ParserStatus (that might work with user-visible exceptions only).
Also a future TODO is putting all the exceptions in their own namespace
(probably ...\Parser\Exception).
Change-Id: I4e33a45117f0a3e73af03cc1e3f2734beaf2b5e1
Thanks to this, we will be able to provide more information
to consequences and watchers, which will open door for new
features and possibly cleaner code.
Change-Id: I7135509823ea84b2a2923d2c1831ce293b98a9f9
Avoid strtotime and compare TS_MW timestamps
Set a fake time to get the same block expiry for relative times
Bug: T272236
Depends-On: I1357d3a78538b8bcb2a3507d86f35371e3f26d47
Change-Id: I5447953c5a0d7ecf4534f4ac4bc2260fa9f42117
This patch adds a transparent HTMLForm field that can be used to insert
the edit box inside an HTMLForm, and updates /test and /tools to use
that. The field class, together with the other editbox-related classes,
is now in a dedicated namespace. A future TODO is making it a real
HTMLForm field.
Also improve a bit the form in /test: add section labels and
avoid reusing the same label message used on Special:AbuseFilter.
Bug: T261584
Change-Id: Ib74bb5fdba4f8476169b754030fce6d4f72ce65a
This service allows linking the EditFilterMergedContent and
PageSaveComplete hooks for the same edit, so we can update rev IDs in
the abuse_filter_log table. Having such a services also avoids two hacky
static props, and should allow separating the hook handlers easily.
Change-Id: I622d15225ee3af202cb5730a7112652aef8ca71a
Also add a bunch of tests for this function.
REMINDER: Change the docs on mw.org when this will be merged.
Bug: T218074
Depends-On: I155024341e8e6b13240e37b30c31b95dc83a47e0
Change-Id: I979e45110bc0e76b499679184993085062ffcac5
In I63d9807264d7e2295afef51fc9d982447f92fcbd we are
changing how the permission checks are applied for revision,
so it uses passed User instance as Authority. However, when
user is mocked, the tests are breaking since the new user methods
are not mocked. Pass a real user for now to fix the test. Once
Authority reaches maturity and is ok to use in extensions, the
test should be rewritten to use authority directly.
Bug: T271458
Change-Id: Iacab813b253cc6e1439007e573e8ace06645860f
In particular, this brings stronger typing for getID(), and we can get
rid of many phan suppressions.
Change-Id: Icbf3a6f7db8105082646ec227f62c09449fb165d
With explicit calls it's easier to see what method is being used,
whether it's deprecated, etc. Some methods here are in fact deprecated
or already have a proper replacement, but this is left for a follow-up.
Change-Id: Iee3154855f86c76aab98e7c14250c14e8b9ee939
- Exclude a couple of classes from coverage reports
- Add tests for all handlers
- Add tests for the runner, copied from core
- Make AbuseFilterRunner a real service
Change-Id: I7a0fe3cd8300faef5ef72d7f986b1734c324d8d1
This is using core methods, so it can be unit tested. The same isn't
true for load-recent-authors, which performs a custom DB query and whose
test is probably the slowest AbuseFilter test. Simplify it for now,
until the method is moved to MW core.
Change-Id: Ifbdae1a06aabca996eeac151a6d029fd991ad64d
Additionally, avoid building Title objects in LazyVariableComputer, it
just adds a dependency on TitleFactory and creating mocks is more
complicated, but it's pointless because the caller already has a Title
object.
And also stop using Title::getEarliestRevTime(), since the replacement
is easy (we already have a RevisionLookup).
Note for reviewers about renames:
- Code VariableGeneratorDBTest was moved to LazyVariableComputerDBTest,
RCVariableGeneratorTest, and AbuseFilterVariableGeneratorTest
- AbuseFilterVariableGenerator test was moved into a dedicated
directory, methods were changed not to test the var values
Change-Id: I3dff8739a9b79f33321d836449b082c3ce63f277
Mainly constructor and conditions limit, which can be removed from
ConsequencesTest (where it was very slow).
Additionally, inject globals into FilterRunner.
Change-Id: I56ca67de6878dbc2185038faae3eb2b04fb56be9
Code change: in buildVarDumpTable remove special-cased null value. This
was used to avoid passing null to Html::element, but is no longer
necessary, since we now pretty-print the value.
Change-Id: I6180f6c53448d2a8c8c6066f222e9fd9df577554
So everything can be loaded using PSR-4. These classes weren't renamed,
nor the alias for the AbuseFilter class was deprecated, because they
should be refactored first.
Change-Id: Ia328db58eb326968edf5591daac9bacf8c2f75da
So we can use DI in all generators. Some improvements were deliberately
omitted, e.g. injecting more services and relaxing User/Title to
UserIdentity/LinkTarget, and they'll be included in a subsequent commit.
Depends-On: I1f351071ef2b0b7c80e91407a9c3bb17be293044
Depends-On: Ie71740fac35a86f8fe03023080ae8ca08671243d
Depends-On: I589a0e1c2c5891070ab82cd5adfd9cedec19e67d
Change-Id: I92ef0abd5e45b672e6f297a71b3c2c345d56f136
This makes VariableHolder a true value object, and introduces a
stateless service, VariableManager, to operate on it.
Note, in theory, this new service is still cyclically coupled with
LazyVariableComputed. However, it's now two stateless service being
coupled, not two smart/god value objects, so we've still earned
something. For now, the dependency is hidden by using a callback. Some
alternatives for that are mentioned in a code comment.
Bug: T261069
Change-Id: I2f2c84c8e91472ba36084a8bbb4a923f6e04354b
This is an important step towards removing the AbuseFilter class. Note:
proposals for the name of the new service are welcome.
Change-Id: Ib4632173f728b1bdafadef96e01645a833bfceaa
Move to 'integration' all tests that are meant to stay there. Move
SaveTest outside because, while we might want to finalize it as an
integration test, some parts can still be moved to a unit test.
Change-Id: Id4b6deaac6875fdd85eebbebf0c5fb952d1fbb06
Moves more methods away from the AbuseFilter class. Testing
buildVarDumpTable is not easy because we'd have to parse the generated HTML.
Change-Id: I073a537201de150ba9dd7bf15a99f3a009dc6ba1
Add namespaces, shorten class names.
Non-unit tests and AbuseFilterTest are untouched because those should be
refactored first.
Change-Id: Ie46ef18d6ba1017e25c76b1762f678e5452264d9
Skip a test that fails with
Wikimedia\Rdbms\DBQueryError: Error 5: database is locked
Function: Wikimedia\Rdbms\Database::beginIfImplied (MediaWiki\Extension\AbuseFilter\FilterLookup::getAllActiveFiltersInGroupFromDB)
Probably due to some concurrency issue caused by the duplicate connection, and also with
Wikimedia\Rdbms\DBQueryError: Error 1: no such table: unittest_external_abuse_filter
Function: MediaWiki\Extension\AbuseFilter\FilterLookup::getAllActiveFiltersInGroupFromDB
for unknown reasons.
Move the mwGlobals override inside the test to avoid the same "database is locked" error
on every other test in that class.
Bug: T251967
Change-Id: I552a8d1fa532941f630fd734e590993e7462aeb0
Introduce ReversibleConsequence interface for Consequence classes
whose potentially destructive actions can be reverted using
Special:AbuseFilter/revert. This allows moving reverting logic from
AbuseFilterViewRevert to individual Consequence classes and testing.
Unfortunately, the code is definitely not very clean now.
Change-Id: I558da711f1645ccf64792c6102cf743827171320
See task for a description of the plan. Also note that
AFComputedVariable should be renamed and its properties made private.
This commit includes some adjustments for taint-check in
AbuseFilter::buildVarDumpTable and ::revisionToString.
There's some space for improvement in the new LazyVariableComputer, but
that's left for another commit.
Bug: T261069
Change-Id: Ia44f6e079d39f44cf0122dec5ddb5513ab54f0c6
This requires a MessageLocalizer, which currently means providing the
main RequestContext. This is the only alternative right now, until core
provides a proper MessageLocalizer service (see T247127).
Change-Id: I8c93e2ae7e7bd4fc561c5e8490ed2feb1ef0edc2
Use Echo for delivering the notification to the last
user who edited the filter.
Much boilerplate.
Change-Id: I7a46a03b4f15de20902ec70c62fb4fe750096842
Depends-On: If585b14a6dd6fb8c7d2c3bee1f20d9d08eaac706
This commit introduces some boilerplate for emitting warnings from the
AbuseFilter parser, and also code for showing these warnings in the ace
editor. Adding new warnings should be as simple as appending to
AbuseFilterParser::warnings (and adding the relevant i18n).
Bug: T264768
Bug: T269770
Change-Id: Ic11021b379f997a89f59c8c0572338d957e089a6
This is the last big step towards moving Consequences-related things away from
AbuseFilterRunner. There's still some cleanup to do (+ write proper tests), but
this should really be the last important code change.
Change-Id: I347795fe93ba496c43b1d5cfc9ba6e1326842c06
AbuseFilter emulates the storage mechanism also used for page content.
Instead of duplicating the relevant code, AbuseFilter should use the
same BlobStore service also used by RevisionStore.
Note that this change is not strictly needed to resolve T198341, but is
needed to unblock T183490
Bug: T261889
Bug: T198341
Bug: T183490
Change-Id: I3fc8475dd8d50d73d705b706ff597a130267e990
This is just a temporary location for these two methods. Since they're
used a lot, having them in the AbuseFilter class means that the
dependency graph is unnecessarily complicated. Thus, since these methods
aren't doing much, they were moved to a dedicated class. Future todo is
finding an appropriate location, that might be either as part of another
service, or keep them in a Utilities class, perhaps a single class with
all util methods, rather than a specific class.
Change-Id: I52cc47a6b9a387cd1e68c5127f6598a4c43ca428
This is the last use, and it was a bit harder to remove because it was
buried inside AFComputedVariable. Starting with
I4444cada720ab62d187f2dd0c4760697e465f2ff, we can freely change the
parameters to AFComputedVariable without breaking old log entries.
Note, we still need a fallback for other extensions calling this
method...
Bug: T246733
Depends-On: I4444cada720ab62d187f2dd0c4760697e465f2ff
Change-Id: I5d786a518ef88fad9c8d9c25ef4553a0bf30b2b2
Add a script to migrate the columns (which can also
be executed in dry run), and a config option with the migration stage
(defaults to SCHEMA_COMPAT_OLD).
Some of the script-related code is stolen from
Ic755526d5f989c4a66b1d37527cda235f61cb437.
Bug: T220791
Change-Id: I7460a2d63f60c2933b36f8383a8abdbba8649e12
$wgAbuseFilterActions shouldn't be used normally, as it excludes actions
registered by other extensions.
Note: mw:Extension:AbuseFilter#Integration_with_other_extensions should
be updated after merging.
Bug: T239348
Change-Id: I89b3f0228eacdf145e8f2dd2a5602d0c7ce75a86
Also fix a bug in FilterProfiler. It would attempt to reset
stats for global filters but we do not record them (yet?).
Change-Id: I0228d8c85dab146deb877dfce506f1e8e7711a9f
Just moving code around. Without a unit test because DI
coverage of change tags in core isn't available yet.
Change-Id: Iac861e1e24dae13581b8d9173357a1d6c94be88a
It makes sense to look at this and Iedd7a5dca24 together,
as this patch itself doesn't really fix anything.
Change-Id: Ifef5266b1803d1a96489789b08d9beed044d908f
The consequence-taking logic is moved away from AbuseFilterRunner, to
dedicated classes. There's now one class per consequence, encapsulating
everything it needs to take the consequence.
Several interfaces allow customizing different types of consequences.
Every "special check" in AbuseFilter was generalized to use these
interfaces, rather than knowing how to handle each consequence.
Adding more consequences from other extensions will also be easier, and
it should happen via a hook (not a global), returning a class that
implements Consequence. The BCConsequence class was temporarily added
for legacy custom consequences.
A ConsequenceFactory class is added to instantiate consequences; this
would possibly benefit from using ObjectFactory, but it doesn't because
it would also reduce readability (although we might do that in the
future).
These classes are still not covered by unit tests, and this is left to
do for later. The new unit tests should mostly replace
AbuseFilterConsequencesTest. @covers tag were added to keep the status
quo (i.e. code that was considered covered while in AbuseFilterRunner
will still be considered covered), although we'll have to adjust them.
Change-Id: Ia1a9a8bbf55ddd875dfd5bbc55fcd612cff568ef
This will ease adding new watchers, for instance to send Echo
notifications (see T179495 and T100892).
For now, this is just boilerplate, and converting EmergencyWatcher to
the new interface.
Change-Id: I18d62aba53471202b709cdb19033b1729c5c25b4
-Exclude methods and classes that cannot be meaningfully covered
-Add a simple test for AbuseFilterServices
-Exclude ServiceWiring because there's no way to tell PHPUnit it's
covered
Change-Id: I4c67b0d3fea68c7a3b3cbe01b5608f87e1b492db
The behaviour is:
- When assigning to an undefined offset, delete the whole array and turn
it into another DUNDEFINED
- When retrieving from an undefined offset, just return DUNDEFINED.
Bug: T237214
Change-Id: I621ee7a16c90bb86a57be04e7ce0a748ecdbfcc7
The main benefit of having a dedicated interface is that we can easily
change the output format. So we're now using a custom array without
references to the DB schema, thus making the import/export process
completely independent from the schema.
Change-Id: I4c0de41d914baf1e9a0e588bd31f95b3524a424b
This is a no-op, moving code around, introducing another distinction re
"filtering actions", which now happens in 2 steps:
- The first step only uses "generic" information available by looking
at enabled actions as a "group". This includes keeping only the
longest block, and removing 'disallow' if other blocking actions are
enabled.
- The second step uses information that is only available after having
"partly executed" (named "pre-checked") a consequence. For instance,
we need to pre-check 'throttle' to see if the throttle was hit, and
remove any other actions if not.
Change-Id: I7be5cfaa61e942a06f97ed52f50e9c8c70a120e8
This way we don't have special cases in executeFilterActions, and instead, we execute
all actions in the same place. In turn, this is going to ease the
transition to a new consequences system: next step is refactoring this
code into a service with proper DI etc.
Bug: T204447
Change-Id: I8134ecc41fbecdbed99faf406e9e3ca91b6123b9
The scope is still quite limited, but as noted in a todo, we might want
to make this completely independent from the database, and add the use
case of ViewDiff.
Change-Id: Ie980fff0983b3e86037265e85da04444c809a6e8
This moves a lot of things away from the AbuseFilter class. There's a
nasty static dependency on ChangeTags, but it's very limited anyway, and
it's going to be fixed once T245964 is resolved.
Change-Id: Ia7df4b4d3289c2722323f59ceecf3fdd38277785
Some pieces of code were updated to use Filter objects, while other
places are still to be updated. We also need to change the history part
to exclude actions somehow, cleanup the ViewEdit, reduce direct DB
access or anything mentioning DB fields outside of FilterLookup, etc.
Change-Id: I42b7ded685db76eddd45e4b1336f9828cba811ce
This requires adjusting some methods to work with Filter objects. Some
methods and tests are left in an inconsistent/suboptimal state, plus some todos
were added, but all of this is going to be remediated in another commit.
Change-Id: Id063ee73d97c7aef56323e1457d99704f77ab943
This is just a start; next step is adding a factory/store method to
get/store these objects. And then use these value objects whenever
applicable.
Note: the actions-related code is still not fully implemented. This is
going to happen as part of the FilterLookup.
Change-Id: I5f33227887c035e301313bbe24d1c1fefb75bc6a
This is a thin wrapper around LBFactory and the global variable, that
can be injected in classes requiring it (no real class right now, but
that's going to change soon).
Also, remove some DWIM-style returns which made the code harder to
understand.
Change-Id: I1d28ad4a67f914103f3a17cda5f61b28070c7f1c
Make FilterProfiler::getFilterProfile return stats unchanged,
in a structured way. Move computations to AbuseFilterViewEdit,
as they are only useful there. Don't return false on cache
misses, return arrays with zero values instead.
Bug: T266531
Change-Id: I8718cc31a5004340bf742315c7075e10a61fcbfd
This deals with data inconsistencies in buildFilterEditor. Every
property of $row was tested in all 5 scenarios (also using Selenium) to
check when it's set. The result is in the normalizeRow method, which
aims to remove any inconsistencies, so that buildFilterEditor always
receives a "complete" row with all defaults set.
The code in buildFilterEditor is now cleaner (because there are no
isset() checks), and it gives us a unique place where we can set
defaults (rather than partly doing that in
loadRequest/loadFilterData/loadImport, and partly relying on isset).
This will be especially useful when introducing value objects to
represent filters, because now you just have to look at normalizeRow()
to tell which properties are allowed to be missing, and thus what "kind"
of filter object you need (see
I5f33227887c035e301313bbe24d1c1fefb75bc6a).
Additionally, reduce the properties that get passed around during
export/import, and make the selenium test try a roundtrip, rather than
relying on hardcoded data that may get outdated. A future patch will
refactor the import/export code.
Change-Id: Id52c466baaf6da18e2981f27a81ffdad3a509e78
Unfortunately, this isn't using DI completely, because of the
User::newSystemUser call. I'm not even sure if we really need to call it
or we can just stick to new UserIdentityValue, but leaving like this for
now.
Also, the types were weakened to UserIdentity, so the transition is
going to be easy anyway.
Change-Id: I08f8fae0fcc622ff0ac3f86771476d06d1c18549
