Commit graph

258 commits

Author SHA1 Message Date
Daimona Eaytoy 44ea3aa7f4 Fix generation of HTML vars, simplify tests
-new_html: also strip the "Transclusion limit" comment if present, and
anyway take it into account (as well as a "</div>"), which right now
prevent the PP limit report from being stripped as well.
-new_text: trim extra whitespace on the right, which is created when
stripping the aforementioned comments.

Also simplify the test for getEditVars, make it not blindly copy what
AFComputedVariable does.

Extra: kill a temporary variable.

These changes are partly taken from
I96785c6c5fdf381c21d5f8930ee12e706abb7f3f.

Change-Id: I2b4c84a3d9d0d17ce229088197b75781d5181b4f
2020-01-12 17:44:02 +00:00
jenkins-bot 8fea62529b Merge "Fix AbuseFilterCachingParser violating return type constraint" 2019-12-27 10:04:57 +00:00
jenkins-bot 5c9fe8bd9b Merge "Always evaluate the offset when retrieving array elements" 2019-12-27 09:58:50 +00:00
Daimona Eaytoy 8ad4ecd31d Always evaluate the offset when retrieving array elements
Even if the array is DUNDEFINED, we need to check the offset to ensure
that it's valid.

Bug: T237351
Change-Id: Ibfa360c4ae1d80abe14d9fdf66991b76cb5954df
2019-12-23 16:04:45 +00:00
jenkins-bot ce85c215f4 Merge "Ensure that a min/max arg count is available for all built-in functions" 2019-12-23 12:11:57 +00:00
Daimona Eaytoy b3e0529d55 Log deprecated vars in the cached phase in the new parser
For the new parser, xhgui shows that AbuseFilterParser::getVarValue is
taking up a lot of time; in turn, most of the time spent inside
getVarValue is used to log the use of deprecated variables. Hence, given
that:
 - We should keep the new parser performant
 - There are tons of deprecated variables out there and they likely
 won't be replaced
 - Having gazillions of debugLog entries doesn't help

log them only in the cached phase.

Bug: T234427
Change-Id: I2bfc692c829c3cbe889e5076f5205e2c99097087
2019-12-16 13:54:58 +01:00
Daimona Eaytoy f382304aae Add a base class for parser transition
Change-Id: I31282b8632c332b6d46a6bb4a42f57ac0d005b5f
2019-12-15 13:29:56 +00:00
Daimona Eaytoy d5ab147dcf Fix AbuseFilterCachingParser violating return type constraint
This is identical to I8a3c31e7385283d95b4712d457784016239a0b3b, except
for the array append case.

Bug: T236870
Change-Id: Iac033ba467232f6ff110d575920e968759ce0e15
2019-12-04 18:27:46 +00:00
Daimona Eaytoy 07572da2fe Really throw for too many params
Bug: T230803
Change-Id: I4e68bb7220f1151bb32b2be859f6cffc55888a30
2019-11-30 10:57:16 +00:00
Daimona Eaytoy 2ddd79fd98 Forbid assignments where the LHS is a built-in identifier
And not just a built-in variable.

Bug: T237130
Bug: T237216
Change-Id: Ie1d86dc324993efcb863be23697732e6aa1dac10
2019-11-28 14:40:38 +00:00
Daimona Eaytoy 0672a8eb8a Ensure that a min/max arg count is available for all built-in functions
This is especially useful for old patches, created before the
introduction of FUNC_ARG_COUNT, where a rebase may break the parser.

Change-Id: Ib142438626a7305f102dc3e4cc9cb07ad33902b8
2019-11-22 17:59:00 +01:00
jenkins-bot a8c50150d6 Merge "Convert static arrays to constants" 2019-11-22 13:39:39 +00:00
jenkins-bot 2d2e524dca Merge "Tokenizer: don't strip backslashes from \x" 2019-11-22 13:36:49 +00:00
Daimona Eaytoy b3e58067ac Set the utf-8 flag for var dumps in the text table
This is not retroactive; that will be handled as part of T213006.

Bug: T34478
Change-Id: I2c532da71719a9ace1279bbf67d6e6e30e9a986c
2019-11-16 16:00:45 +00:00
Daimona Eaytoy c03f0a3b08 Convert static arrays to constants
Beloved PHP7!

Change-Id: Id5170662f7c5ceacfc0ac8d90787f2c92fd93464
2019-11-16 16:32:36 +01:00
Daimona Eaytoy c73381b6db Tokenizer: don't strip backslashes from \x
Bug: T238475
Change-Id: I8c2ea6ad369946df93440eece60d456dc1a3fd7a
2019-11-16 16:21:39 +01:00
Daimona Eaytoy 98bcad25c3 Also parse numbers with the new syntax and hard-deprecate the old one
This will allow people to switch their filters to the new syntax. The
deprecation warning is now more exhaustive, and the info() warning is
kept to ensure that everything proceeds smoothly.
The regex v2 has also been fixed to:
 - Consume all the digits/letters on the right (*)
 - Have named groups
 - Be created dynamically with other constants

(*) The previous version of v2 could complete the match and leave
digits/letters on the right when encountering numbers with the old
syntax, hence dropping support too early. We also cannot use a word
boundary (\b) because that would prevent matching numbers with trailing
dots (e.g. "5.").

Bug: T212730
Change-Id: Ibf6ac571f6b5c09149d69a19c38240ce6b024dff
2019-11-12 11:52:38 +00:00
Daimona Eaytoy a77a59b962 Hard-deprecate empty operands
This bumps the level to WARN, and makes it very clear that people should
fix the affected filters. It also removes the calling method, which was
mostly meant for debugging purposes, and changes the type to 'op_type'
to avoid conflicting with type:mediawiki in logstash.

Bug: T156096
Change-Id: Ie73f1604e8ed82bc2e1be9fc90fa065be37889a3
2019-11-12 11:39:25 +00:00
jenkins-bot 91bc961712 Merge "Check for 0-like floats passed to the modulo operator" 2019-11-10 11:51:28 +00:00
Daimona Eaytoy c0f8374624 Check for 0-like floats passed to the modulo operator
That throws an error in PHP.

Bug: T237459
Change-Id: Ia0b29d6a8b9f4aac6b5b72ce8f2f45afb03f4c99
2019-11-10 11:22:04 +00:00
jenkins-bot 7ff4b95aec Merge "Expand the list of types that can be cast to int" 2019-11-10 11:00:36 +00:00
Daimona Eaytoy 585d6cdb24 Make to sure to report division by zero when the LHS is undefined
Bug: T234339
Change-Id: I1575ec013c1e7e321a8f13f40804ebc5ab076268
2019-11-08 14:08:52 +00:00
Daimona Eaytoy 1abaff1aac Better handling of keywords and functions
Always run the keyword/function handler, even if there are DUNDEFINED
arguments, so that the handler can perform further validation on the
input and report any error to the user. However, replace DUNDEFINED with
DNULL before running the handler, to avoid special-casing DUNDEFINED in
every handler. If any argument was a DUNDEFINED, we will return
DUNDEFINED anyway.

Also centralize the keyword handling logic to a new method, like it
happens for functions.

Bug: T234339
Change-Id: I875cb77418a39790e91fe5867c49917bfe406ed4
2019-11-08 15:07:20 +01:00
Daimona Eaytoy b7c7ae168d Explicitly forbid negative indexes in arrays
This emits its own error because:
1- It's clearer to understand
2- It's easier to find where we're dealing with negative offsets, if
we'll ever want to allow that.

Note that trying to use a negative index already results in a hard PHP
error being thrown.

Bug: T237219
Change-Id: Ib11eaaca5e21f740269141c75e62bac48093e8d0
2019-11-08 05:55:56 +00:00
Daimona Eaytoy a7b28369ea Expand the list of types that can be cast to int
Bug: T237624
Change-Id: I2220cb8a8ec998a433a4469d7e0591ec0b4f2b12
2019-11-07 15:14:17 +01:00
Petr Pchelko 915b9a1538 Remove usages of deprecated User methods
Bug: T220191
Change-Id: I54e20870a32ff98b41a98495694ff563c4c4c5ca
2019-10-30 12:51:01 +00:00
Daimona Eaytoy 3a9eac9ad5 Unbreak filter edit form
In Ib7427e15f673a575738489476e604c387f449ddd, I thought that $parameters could've only been null if $action wasn't
enabled, but actually, they're null even if the action is just not set.
Which is true for all actions when creating a new filter, and all
non-set actions when editing an existing one.

Hence, revert the part that touched ViewEdit.

Also add a selenium test to ensure that warn parameters are visible.

Bug: T236286
Change-Id: I8150baa077208eb1fc54ebc1d8415a243d0f3bd3
2019-10-23 18:50:44 +02:00
James D. Forrester 4d988471be build: Upgrade mediawiki-codesniffer to v28.0.0
Change-Id: I7ef6ec1614718c016562281a166867ee3bd93df7
2019-10-09 18:34:07 +00:00
Max Semenik b3d11b48cb tests: setExpectedException() is deprecated
Bug: T192167
Change-Id: I899a8f03c6cc1f79f58bec09c2d8b2ba10b895d8
2019-10-08 16:31:15 -07:00
Daimona Eaytoy 0646b4fe7d Add config for Selenium and basic tests
This is some sort of Hello World for selenium. This patch adds the
config files and a couple of very basic tests.

Bug: T214478
Change-Id: I8193b4edb40332bea1d08e24ec020bf36004320d
2019-09-17 16:23:07 +00:00
jenkins-bot 48713c824b Merge "Throw AFPUserVisibleExceptions for empty operands in CachingParser" 2019-09-15 08:36:39 +00:00
Daimona Eaytoy a4e25c1ac9 Throw AFPUserVisibleExceptions for empty operands in CachingParser
Instead of TypeErrors. Basically, only empty parenthesis had to be
fixed.

Bug: T156096
Change-Id: I019615c7bfaa179c2184b5d3ea2c6b5da91366e3
2019-09-14 18:35:40 +00:00
jenkins-bot 45d7bd5971 Merge "CachingParser: ensure to catch errors inside short-circuited blocks" 2019-09-14 01:56:35 +00:00
jenkins-bot b8ad85cac7 Merge "Annotate the AST with var names before caching the AST" 2019-09-14 01:03:53 +00:00
Daimona Eaytoy 6e9a9a3bc2 CachingParser: ensure to catch errors inside short-circuited blocks
This is similar to the old parser: when discarding a node, actually
evaluate it if short-circuit is not allowed.
Add a whole lot of tests for all possible exceptions.
Move the logic to extract a message from an AFPUserVisibleException away
from the parser, to keep unit tests working.

Bug: T232498
Change-Id: I31ee4e255c6a87dd693b9bcd582539fdf57acd45
2019-09-13 21:13:15 +00:00
Daimona Eaytoy 004ccfdb5c Annotate the AST with var names before caching the AST
This implements T230982#5475400, and it should speed up the CachingParser by roughly 40%.

Bug: T230982
Change-Id: I803cc58637d50eb90e57decf243f5ca78075d63d
2019-09-13 19:43:50 +00:00
jenkins-bot cfad7d6f14 Merge "Actually return errors for action=edit API" 2019-09-10 19:59:03 +00:00
Bartosz Dziewoński 82b6f191d4 Actually return errors for action=edit API
Setting 'apiHookResult' results in a "successful" response; if we want
to report an error, we need to use ApiMessage. We already were doing
this for action=upload. Now our action=edit API responses will be
consistent with MediaWiki and other extensions, and will be able to
take advantage of errorformat=html.

Since this breaks compatibility anyway, also remove some redundant
backwards-compatibility values from the output.

To avoid user interface regressions in VisualEditor, the changes
I3b9c4fef (in VE) and I106dbd3c (in MediaWiki) should be merged first.

Before:
    {
        "edit": {
            "code": "abusefilter-disallowed",
            "message": {
                "key": "abusefilter-disallowed",
                "params": [ ... ]
            },
            "abusefilter": { ... },
            "info": "Hit AbuseFilter: Test filter disallow",
            "warning": "This action has been automatically identified ...",
            "result": "Failure"
        }
    }

After:
    {
        "errors": [
            {
                "code": "abusefilter-disallowed",
                "data": {
                    "abusefilter": { ... },
                },
                "module": "edit",
                "*": "This action has been automatically identified ..."
            }
        ],
        "*": "See http://localhost:3080/w/api.php for API usage. ..."
    }

For comparison, a 'readonly' error:
    {
        "errors": [
            {
                "code": "readonly",
                "data": {
                    "readonlyreason": "foo bar"
                },
                "module": "main",
                "*": "The wiki is currently in read-only mode."
            }
        ],
        "*": "See http://localhost:3080/w/api.php for API usage. ..."
    }

Bug: T229539
Depends-On: I106dbd3cbdbf7082b1d1f1c1106ece6b19c22a86
Depends-On: I3b9c4fefc0869ef7999c21cef754434febd852ec
Change-Id: I5424de387cbbcc9c85026b8cfeaf01635eee34a0
2019-09-09 20:15:19 +02:00
Daimona Eaytoy 7b06be0204 Allow dangling commas in variargs
This is because there are many filters using this feature. Moreover, it
could make it a little easier to add new arguments, just like dangling
commas in PHP arrays do.
Also re-align the CachingParser code of doLevelFunctions to the one in
the old Parser.

Bug: T153251
Change-Id: Ie4325159f47310788da57415a5e36e62aa4efad0
2019-09-07 11:19:14 +02:00
jenkins-bot 5be19f6f65 Merge "Add a 'strict' option to VariableHolder::getVar" 2019-09-05 19:23:23 +00:00
Daimona Eaytoy 489da0d229 Add a 'strict' option to VariableHolder::getVar
This will help mitigating problems like T230256 by enforcing that the
requested variables must exist. For now, it will only log bad usages,
thus providing a way to identify affected filters and fix them.

Bug: T230256
Change-Id: I7a61916576e444a56f0e07da7b6e5033346226bd
2019-09-04 18:19:23 +00:00
Daimona Eaytoy 13b1e880f2 Hotfix other DUNDEFINED casts to bool
These were spotted on testwiki with wmf.21.

Change-Id: Ic4d67a2b83aedfeb574fa1363a9fc618b2862f95
2019-09-04 18:06:22 +00:00
Daimona Eaytoy ce8539e2a5 Move parser tests back to /unit
Using `new LanguageEn()` involved a global, so use a MockObject instead.
Also fix LoggerFactory usage in Tokenizer to use DI instead.

Change-Id: I94d03f9459ab6444e239386eb96a0c2434bfe3dc
2019-09-03 13:23:11 +00:00
Kosta Harlan 984e06639d Move non-unit tests back into default (integration) directory
Follows-up Id8412e2b8a4e873fd4821ecc1a3c95710be9a870.

Change-Id: Ib92cfbb637e0143a5481212f11a6e511929d6801
2019-09-01 19:57:26 +00:00
Daimona Eaytoy f7812ea7a3 Remove redundant User::addToDatabase call in tests
We don't need to call it in the constructor, as long as the call in
setUp() is moved to before we start adding groups and checking blocks.

Follows-up Id8412e2b8a4e873fd4821ecc1a3c95710be9a.

Change-Id: I339363499f99295a83004074d6a44574cd622a58
2019-08-31 16:35:22 +00:00
Daimona Eaytoy d51ca862c6 Move parser tests to /unit
IMHO these can be considered unit tests; they were already fast, but now
they're executed in an instant.
This requires several changes: 1 - delay retrieving messages in
AFPUserVisibleException, to avoid having to deal with i18n whenever we
want to test exceptions; 2 - Use some DI for Parser and Tokenizer.
Equivset-dependend tests are also moved to a new class, thus helping to
fix the AF part of T189560.

Change-Id: If4585bf9bb696857005cf40a0d6985c36ac7e7a8
2019-08-28 16:36:37 +00:00
Daimona Eaytoy 8e166f10d6 Refactor and speed up non-parser tests
Some of these are transformed into real unit tests, while the
AbuseFilterSaveTest class is refactored to avoid using the DB and to use
a lot more of mocks and DI.

Depends-On: I22743557e162fd23b3b4e52951a649d8c21109c8
Change-Id: Id8412e2b8a4e873fd4821ecc1a3c95710be9a870
2019-08-27 16:24:27 +00:00
Daimona Eaytoy 71730f7d44 Warn if a function has been given too many parameters
While this is not as important as throwing for too few parameters, IMHO
it's still important to fail in this case. Mostly because if a function
receives too many parameters, chances are that who wrote the filter
didn't do that intendedly, and thus there may be a hidden bug.
Bonus: fix a few docblocks.

Bug: T230803
Change-Id: Iac2931f17b50ace8c8f4c2faa44b3f54ca134c54
2019-08-26 20:29:49 +02:00
Aryeh Gregor 4c8dac4dc6 Change config only before we've started testing
In general, it's not safe to change configuration in the middle of a
test, because services could wind up in an inconsistent state. In
particular, I'm trying to have setMwGlobals() reset services, which will
cause stuff to break if it happens in the middle of a test. So just
specify the settings you want up front, like in setUp().

Change-Id: I00e35ecea6a27468674b2a6e7d9d9eb6518e3bd5
2019-08-26 14:26:44 +03:00
jenkins-bot ff2f6ee26f Merge "Add a new class for the CachingParser's AST" 2019-08-25 18:00:24 +00:00