Commit graph

853 commits

Author SHA1 Message Date
Daimona Eaytoy c469fb4b76 Mostly remove $wgUser
There are lots of cases where we can inject a User object without
additional efforts. Now $wgUser is only used inside AFComputedVariable,
which is a little bit harder to handle because some instances of that
class are serialized in the DB, and thus we cannot easily change the
constructor until T213006 is resolved.

This partly copies what Ia474f02dfeee8c7d067ee7e555c08cbfef08f6a6 tried
to do, but adopting a different approach for various can*() methods:
they're now static methods in the AbuseFilter class, so future callers
don't need to instantiate an AbuseFilterView class. This also allows to
re-use those methods in an API module for editing filters (T213037).

Bug: T213037
Bug: T159299
Change-Id: I22743557e162fd23b3b4e52951a649d8c21109c8
2019-08-27 13:20:37 +02:00
Daimona Eaytoy 71730f7d44 Warn if a function has been given too many parameters
While this is not as important as throwing for too few parameters, IMHO
it's still important to fail in this case. Mostly because if a function
receives too many parameters, chances are that who wrote the filter
didn't do that intendedly, and thus there may be a hidden bug.
Bonus: fix a few docblocks.

Bug: T230803
Change-Id: Iac2931f17b50ace8c8f4c2faa44b3f54ca134c54
2019-08-26 20:29:49 +02:00
Daimona Eaytoy 4d86758a49 Add new number syntax as experimental
For now it will only report successful parse. Next step is formally
deprecating the old one (escalated to warning), then removing it in
favour of the new one (in another MW version).

Bug: T212730
Change-Id: I5dd11fd67d8e57d1d0c52ddfa026920ebfc5ee13
2019-08-26 08:15:55 +00:00
jenkins-bot 89524790d5 Merge "Add a hook to determine whether the current action should be filtered" 2019-08-25 18:45:07 +00:00
jenkins-bot ff2f6ee26f Merge "Add a new class for the CachingParser's AST" 2019-08-25 18:00:24 +00:00
Daimona Eaytoy d515af0ae6 Add a new class for the CachingParser's AST
This allows a little bit more of abstraction: we can store other data in the
tree, without having to store it in a specific node (e.g. the variables map,
which is still unused). It also adds a few typehints, and specializes
the return value of eval'ing the AST: previously, it was the one of
evalNode, which wasn't guaranteed to be an AFPData. Now we have this
guarantee. Last but not least, we can now measure runtime metrics for
evalTree, which doesn't recurse.
Bonus: fix a check in the old parser, which used the wrong variable when
reporting outofbounds errors.

Change-Id: Iff806793b1d968e9bb6220f1459f3d0ac587c7da
2019-08-25 17:29:16 +00:00
jenkins-bot 6196801178 Merge "Log more empty operands" 2019-08-24 20:53:01 +00:00
Daimona Eaytoy 2d031d0bee Log more empty operands
And fix a couple of minor bugs.

Bug: T156096
Depends-On: I3b85087677607573f4fa68681735dc35348dcd87
Change-Id: Ia4c713a1d45827f6a8bc5566a8d8835c49f8108a
2019-08-24 19:59:53 +00:00
Daimona Eaytoy 7f554734e6 Don't hardcode blockautopromote duration
As explained on phab, and add a script to fix broken entries.

Bug: T231131
Change-Id: I95d70acb936b5ca987af8f237d236fe47b663919
2019-08-24 11:40:11 +02:00
Huji Lee 1ddb65021b Add links to AbuseFilter logs on Special:Undelete
Depends-On: I671a0479e877e6c37606b688064cb9c893717709
Bug: T231055
Change-Id: Iebf832c513c6a4e954db0ba2633dd8ba6f27b412
2019-08-23 14:56:43 +00:00
Daimona Eaytoy bf61414f88 Don't show empty "Tools:" section in ViewEdit
After having removed the export link in
I72f46247f4323fb5bfe7fa74f332076dbd346187, we don't have any tool to
show for new filters. So avoid outputting an empty section.

Change-Id: Ia07bccdbadb7b874397135bc3f7468d6e0b9eb13
2019-08-21 17:32:43 +02:00
jenkins-bot 47838715fa Merge "Allow if without else" 2019-08-20 20:12:19 +00:00
jenkins-bot 5e605aaa62 Merge "Even better handling of DUNDEFINED" 2019-08-20 20:00:52 +00:00
jenkins-bot bf8ccccade Merge "Fix a bug in the return value of the CachingParser" 2019-08-20 19:58:38 +00:00
Daimona Eaytoy af7744781f Allow if without else
Bug: T230727
Depends-On: I8e7f7710b8cb37ada8531b631456a3ce7b27ee45
Change-Id: I3b85087677607573f4fa68681735dc35348dcd87
2019-08-20 19:36:14 +00:00
Daimona Eaytoy 963221ad6d Even better handling of DUNDEFINED
Ensure that the variable isn't set before marking it as DUNDEFINED:
that's only for when we cannot use a default, but if the variable is set
we already have one. Most notably, this fixes conditionals handling: right
now, if you have a conditional with an assignment in both
branches, the variable will be undefined. That's obviously wrong, so
it's fixed in this patch.
Plus: catch only AFPExceptions in a test to avoid unintentionally
catching the assert exception; simplify some assignments using wfSetVar.

Depends-On: I446a307e5395ea8cc8ec5ca5d5390b074bea2f24
Change-Id: I8e7f7710b8cb37ada8531b631456a3ce7b27ee45
2019-08-20 19:17:30 +00:00
Daimona Eaytoy fa76405ea7 Fix a bug in the return value of the CachingParser
This has always been wrong, and remained unnoticed. Also added a
typehint for added safety.

Change-Id: I8a3c31e7385283d95b4712d457784016239a0b3b
2019-08-20 20:54:19 +02:00
jenkins-bot a8e2071351 Merge "Better handling of function params in CachingParser" 2019-08-20 18:46:01 +00:00
jenkins-bot 8527a10774 Merge "Restyle edit box dimensions" 2019-08-20 16:33:16 +00:00
Daimona Eaytoy aa867bd370 Better handling of function params in CachingParser
This patch includes various fixes to how func arguments are handled in
CachingParser:
- Add a comment about a future improvement of checkSyntax, which we
  could limit to try building the AST.
- Having enough args for each function is now also checked when
  building the AST. This allows implementing the previous point without
  stopping to report notenoughargs at syntaxcheck-time (otherwise it'd be
  a runtime error). And it also ensure that we check for the params count
  inside skipped branches, e.g. inside if/else: these were already only
  discovered at runtime in CachingParser. The old parser is not affected
  by this change, because when checking syntax it will always execute
  all branches, and at runtime it will skip braces altogether.
- Fix arg count for CachingParser, which previously added a bogus param
  in case of a function called without parameters. This was fixed for
  the other parser in I484fe2994292970276150d2e417801453339e540, and I
  just ported the updated fix. Also note that the CachingParser was
  already failing for e.g. `count()`, but instead of complaining about
  missing arguments, it failed hard when trying to pass NULL to
  evalNode.
- Fixed some tests not to use setExpectedException, which caused the
  previous point to remain unnoticed: calling that method prevents the
  loop from continuing, and thus only the AbuseFilterParser part was
  being executed. The new implementation checks the exception ID and is
  thus more future-proof if the i18n message changes.
- Fixed some function names in error reporting for the old parser.
- The arg count is now checked outside of the function handlers, thus
  it's no more necessary to call checkEnoughArguments at the beginning
  of each handler. This also produces clearer error messages in case of
  aliases (e.g. set/set_var).
- Check the args count even if some of the args are DUNDEFINED. This is
  much easier now that the check is outside of the handler. This will
  make syntax check fail for e.g. `contains_any(added_lines)`.

Bug: T156095
Change-Id: I446a307e5395ea8cc8ec5ca5d5390b074bea2f24
2019-08-20 15:32:02 +00:00
jenkins-bot 7addec7b4a Merge "Make some other AFPData methods non-static" 2019-08-20 14:16:16 +00:00
jenkins-bot 1f45336157 Merge "Move keywords handlers to the Parser" 2019-08-20 14:16:10 +00:00
jenkins-bot f18d0814e2 Merge "Make several AFPData functions non-static" 2019-08-20 14:06:02 +00:00
jenkins-bot f1ab591d27 Merge "Avoid implicit casts from DUNDEFINED to something else" 2019-08-20 13:04:48 +00:00
jenkins-bot ea01809f5e Merge "Add the filter ID to empty operand logging" 2019-08-20 13:01:14 +00:00
jenkins-bot d32b03ca10 Merge "Increase cache hits for CachingParser" 2019-08-20 12:50:31 +00:00
jenkins-bot d0b30c2534 Merge "Make parser aware of the filter it is parsing" 2019-08-20 12:50:26 +00:00
jenkins-bot 1bfd182a2e Merge "Fix object to array cast, typehint array params" 2019-08-20 12:49:09 +00:00
jenkins-bot 4bebd22e3f Merge "Add test for multiple conditions inside conditionals" 2019-08-19 14:18:10 +00:00
Daimona Eaytoy e4b1df1521 Fix object to array cast, typehint array params
This was broken in I34c040dbeb3ab01158fb3db22496def6ccaf72d9. I thought
the members of that object were always arrays, but I was wrong.
Plus typehint a few array parameters and make a couple of methods
private since they're only used in this class.

Bug: T230639
Change-Id: I0c51359769c4b3054f95755a96e7e0a2d8e5bf15
2019-08-17 17:04:34 +00:00
Daimona Eaytoy b235e1040a Restyle edit box dimensions
Now it's always wider, and so is the "notes" field. Moreover, the
fallback textarea has the exact same size. Plus removed a parameter
which only made it hard to write a CSS rule for the textarea. Since the
textarea is generated by the same code, and we're always using it for
the same thing (filter syntax, regardless of the final goal), make it
always use the same name.

Bug: T230591
Change-Id: Ibb308e80d954c0e81aa09249c38c39572f157948
2019-08-17 18:53:13 +02:00
jenkins-bot 61bb3ff3e8 Merge "Various changes for blockautopromote" 2019-08-14 23:59:08 +00:00
Daimona Eaytoy 27578d7ba0 Various changes for blockautopromote
Problems fixed:
 - Don't hardcode duration in the message
 - Move duration to a constant
 - Fix wrong parameter order for AbuseFilter::blockAutopromote
 - Log a warning if we cannot block autopromotion
 - Remove the $performer parameter, as it should only and always be the
 filter user.

Bug: T230296
Change-Id: Ice9e4b21033c430cf1fd34182c63ca64ad2f5d3e
2019-08-14 18:50:43 +02:00
Daimona Eaytoy d715f6d2c0 Increase cache hits for CachingParser
If $parser->parse returns a falsey value (=null), that's because the
filter doesn't have any statement. But that's not a valid reason not to
cache the filter. Hence, return whatever parse() is returning inside the
callback, so that the result is always cached.

Change-Id: Ib6b0e72d882dc484456a3be6bbc74da36ef48bf7
2019-08-13 18:03:13 +02:00
Daimona Eaytoy d58b5930f8 Add the filter ID to empty operand logging
To make debugging a lot easier.

Bug: T156096
Bug: T153251
Change-Id: I1f905c6e1a524a745240b05709ef9d1dfc3c23a1
2019-08-13 15:22:55 +00:00
Daimona Eaytoy 1197eb6b41 Make parser aware of the filter it is parsing
This information will mostly be used for debugging purposes.

Change-Id: Ia1bcc2acc22aba97d855382b5b173ac3d5f2c54b
2019-08-13 15:22:38 +00:00
Santhosh Thottingal 1176e3a465
Fix the warning about permission name changes
Change-Id: I16463550328eb19d33270d8677404e012e5c80df
2019-08-13 14:40:17 +05:30
Daimona Eaytoy 7edf12dbc6 Add a hook to determine whether the current action should be filtered
Bug: T229252
Change-Id: I126013b6c516eebe7273fb85f2c5681844e757be
2019-08-13 10:31:29 +02:00
Daimona Eaytoy 430ba818d0 Add test for multiple conditions inside conditionals
The regression itself was fixed in
I980aec3481a52ecc35f1811a366014a5581a7cdb, so this patch only adds a
test for it.
Also remove a comment about CachingParser failures: we don't want to
encourage people to remove it from tests anymore.

Bug: T152281
Change-Id: I3ad49050ea49bf45d3226878e091da3c8dbefdb1
2019-08-12 18:18:05 +02:00
Daimona Eaytoy 4b0911ee01 Make some other AFPData methods non-static
Change-Id: I22ea337a36f911c57d3dadb9a3c45fc2c8b7c628
2019-08-12 14:40:51 +02:00
Daimona Eaytoy 3f171dc0a5 Move keywords handlers to the Parser
Just like we do for functions, it doesn't really make sense to have
keywords separately, in AFPData.

Change-Id: I208a9b1ce2bd12038e9fbcc515c48d604ec80eb8
2019-08-12 14:29:56 +02:00
Daimona Eaytoy 2fdf091eb9 Make several AFPData functions non-static
The keywords-related ones will be handled in a subsequent patch.

Change-Id: Ifcfad438023ef136dc6f2cd5529e867df9b23789
2019-08-12 14:12:16 +02:00
Daimona Eaytoy 1fe3647268 Avoid implicit casts from DUNDEFINED to something else
This patch keeps the current behaviour for everything (since DUNDEFINED
was always casted to boolean false), but handles the cast at a higher
level instead of relying on what AFPData::castTypes will do. This way
it's easier to spot places where we may get DUNDEFINED, and decide how
to handle them one by one.

Change-Id: I1070e15ea03c7dd4a4231b87afbc42240a558581
2019-08-12 11:18:15 +02:00
jenkins-bot 3748c41e79 Merge "Remove outdated comment, add a new one" 2019-08-12 08:26:30 +00:00
jenkins-bot eff4e208ca Merge "Don't show export link for new filters" 2019-08-11 17:02:56 +00:00
Daimona Eaytoy 200905f1cf Remove outdated comment, add a new one
As explained in T230093#5408119.

Bug: T230295
Change-Id: Ic0beaf9d126d14fbb7efbd2d0ec55e10c0fbcc99
2019-08-11 14:35:38 +00:00
Daimona Eaytoy c8b5b9321c Don't show export link for new filters
Bug: T230163
Change-Id: I72f46247f4323fb5bfe7fa74f332076dbd346187
2019-08-11 12:21:36 +02:00
Daimona Eaytoy 69ad23da98 Ban variable variables
As explained on phab, it's not worth the effort of keeping this feature.

Bug: T229947
Change-Id: Ic6067cab8e1ede98545e704888c99e2ed9a004e4
2019-08-11 01:47:35 +00:00
Daimona Eaytoy ff40204ef1 Gracefully handle blockautopromote failures
Instead of returning a successful message, return null and log a
warning. Also, make autopromoteBlockKey public + internal and use it
from Hooks instead of duplicating the logic.
Follow-up: I03feb05218789a3b73a31c9a94216daafcb7c145

Change-Id: I8ce96d1bd0239003f8ee6a45f412b9502d542a18
2019-08-10 15:18:30 +02:00
Daimona Eaytoy c34181e548 Add a new API module to retrieve private details from AbuseLog
Follow-up of Iaca492371f48fecf543268c179a651841ed12c3f. This patch adds
the new module, plus some technical changes to private details-related
methods and globals.

Bug: T210329
Depends-On: I613dbadb8f75c8c4116a362607563a436a73d321
Change-Id: I3c45b74c36c191083df184ed57416067a75f6591
2019-08-09 21:10:28 +00:00
Daimona Eaytoy c7ccb68058 Use "privatedetails" instead of "private" where needed
To keep a clear distinction between "private details" (i.e. user's ip)
and "private filters" (i.e. not publicly viewable). This patch renames
rights, i18n keys and methods names.
The patch for renaming globals and rights in WMF config is
I7e6b3d4453403edb6aa602587374b4ff5b6d625f.

Bug: T211004
Change-Id: I613dbadb8f75c8c4116a362607563a436a73d321
2019-08-09 21:10:22 +00:00
jenkins-bot 8ee442234f Merge "Move "block-autopromote" key from $wgMainStash to 'db-replicated'" 2019-08-07 23:07:45 +00:00
jenkins-bot 1fa5eef94c Merge "Overhaul Blockautopromote action" 2019-08-07 23:03:08 +00:00
Aaron Schulz 9e44f1a9e9 Move "block-autopromote" key from $wgMainStash to 'db-replicated'
Keep the key mutation methods in the AbuseFilter class

Bug: T227376
Change-Id: I03feb05218789a3b73a31c9a94216daafcb7c145
2019-08-07 01:09:13 +00:00
jenkins-bot 097b2da0ed Merge "Add a parent class for special pages" 2019-08-06 18:32:29 +00:00
Daimona Eaytoy 483dab1732 Add a parent class for special pages
This allows us to extract yet another static method from the AbuseFilter
class. This class should be expanded in the future, and an example use
case could be Ia5fd4f0b35fcabf045a7b49fa40fa85b72c92544.

Depends-On: I7c0170167b508132cd16e566c654a6c98dd683e9
Change-Id: I1bb45e47c3b42c01388b99778ce833e4e44419e1
2019-08-06 14:17:38 +00:00
Daimona Eaytoy 2ed6272bb2 Partly handle set and set_var in shortcircuit
This is more complicated than the := operator, because the var name
could be a complicated expression, and we have to handle a function
call. This patch only covers the case where the variable name is a
literal, which is enough for WMF production.

Bug: T214674
Change-Id: I6c0f8e95663919a0235b5ccf0c88ad0a539315a7
2019-08-06 16:14:34 +02:00
Daimona Eaytoy 2bdb44d58b Overhaul Blockautopromote action
As for all mostly unused consequences, blockautopromote has a couple of
major problems: first, it blocked the status for a random time between 3
and 7 days, which to me makes no sense at all (is it some sort of
casino?), and this patch fixes it to 5 days. Second, nothing was logged,
not the blocking nor the unblocking. Here I'm adding a LogHandler for
two new sub-actions of 'rights' to keep track of both action.

Bug: T49412
Change-Id: If48a48f5b8baaf9e77c0826466f5d03bb7f691d0
2019-08-05 22:27:49 -04:00
Daimona Eaytoy afeff7c222 Really avoid DEMPTY leak
Follow-up I7831f3ed9f7c0656e0e8f77ded049c20eca682ba, really avoid the leak. My addition was pointless because we need DUNDEFINED, not DEMPTY, and I spent way too much time trying to understand what was still wrong.
Still have to get used to these new names...

Change-Id: I332967f6fb00b67fd355547b19638c95ffa5bba7
2019-08-04 22:02:13 +00:00
jenkins-bot 44e427601f Merge "Avoid DEMPTY leak" 2019-08-04 21:16:22 +00:00
Daimona Eaytoy 6ef2cf523b Profiling: don't count time for operations shared with the edit
Parsing wikitext and retrieving its links are operations which we share
with the edit, so that if a filter does that, it won't be done later
upon saving.
Thus, add a static variable to subtract such time and avoid to erroneously log as slow any filter using those variables.

Bug: T219092
Depends-On: I24fbd41ac188a9cf6a7d3ca33dce349aedc9faa6
Change-Id: I7c0170167b508132cd16e566c654a6c98dd683e9
2019-08-04 20:12:10 +00:00
jenkins-bot 4347331692 Merge "Reset all filter profiling data at once" 2019-08-04 18:53:57 +00:00
jenkins-bot 19182606c1 Merge "Merge global profiling keys" 2019-08-04 18:40:14 +00:00
Daimona Eaytoy f977e858ab Avoid DEMPTY leak
As shown in the coverage reports [0], some empty operand logging lines are covered, but no test should have empty operands. I see one of the cause is skipOverBraces keeping $result as is, even if DEMPTY, so turn it into a DUNDEFINED.

[0] - https://doc.wikimedia.org/cover-extensions/AbuseFilter/includes/parser/AbuseFilterParser.php.html

Change-Id: I7831f3ed9f7c0656e0e8f77ded049c20eca682ba
2019-08-04 18:25:04 +00:00
Daimona Eaytoy 2742f24bca Reset all filter profiling data at once
Instead of scattering the process all over the code (and doing it
together with checking if the key already exists).
Wrap the logic in new methods for better readability.

Depends-On: Ib12e072a245fcad93c6c6bd452041f3441f68bb7
Change-Id: I24fbd41ac188a9cf6a7d3ca33dce349aedc9faa6
2019-08-04 18:03:14 +00:00
rarohde d022377578 Merge global profiling keys
The last step of the profiling overhaul. See T53294 for the original description by Dragons flight.

Note: Here I'm adding a FixMe for a problem which already exists in the code
and the child patch will fix it.

Bug: T53294
Depends-On: I2d8c8f8278073a9420e3eb373fb89a655925618a
Change-Id: Ib12e072a245fcad93c6c6bd452041f3441f68bb7
2019-08-04 17:59:58 +00:00
jenkins-bot 879ff59fe2 Merge "Split hook handlers related to filtering" 2019-08-04 17:55:51 +00:00
jenkins-bot be4b1c5bc1 Merge "Fix @deprecated since version" 2019-08-04 17:50:47 +00:00
jenkins-bot e733872e13 Merge "Allow accessing offsets of built-in variables" 2019-08-04 17:48:46 +00:00
jenkins-bot 790cd38fb0 Merge "Further deprecation for empty conditions" 2019-08-04 17:26:40 +00:00
Daimona Eaytoy 7d192cb1f2 Split hook handlers related to filtering
This adds a new get(Type)Vars method for every action type. The goal is
to 1-have shorter methods, which is always good; 2-try to make this code
a bit more testable.
I left as a todo moving all these methods to a separate class, the idea
being to make them non-static and thus easier to be tested.

Depends-On: I2eab2e50356eeb5224446ee2d0df9c787ae95b80
Change-Id: I6de2dd27a8f972b3f74c730a1516639f8c622166
2019-08-04 17:21:29 +00:00
Daimona Eaytoy 517919fca8 Allow accessing offsets of built-in variables
I5ec4ab44c4e88aaf18c0d7b73355d27050beeda7 almost fixed this bug, but we
also have to make it possible to access builtin variables as arrays.
This will only make sense for a few variables (e.g. added_lines and
removed_lines), but I don't think we should validate it when checking
syntax.

Bug: T198531
Change-Id: I417e1b8d4802bbfccd091ce5c7617659cfd1e4ea
2019-08-04 17:14:44 +00:00
jenkins-bot 534a282f7d Merge "Clarify "filter" field in SpecialAbuseLog and ApiQueryAbuseLog" 2019-08-04 17:13:47 +00:00
jenkins-bot 5fccb5a67b Merge "Change parameter order for newVariableHolderForEdit" 2019-08-04 17:07:27 +00:00
Daimona Eaytoy 71e3719e12 Clarify "filter" field in SpecialAbuseLog and ApiQueryAbuseLog
The "filter" fields can also accept a list of filters, and also global filters, so make it clear in the UI and in messages.

Change-Id: Ib258716d8e6792fd496938ebb4e8a2565d6370b7
2019-08-04 16:55:05 +00:00
Daimona Eaytoy c635f41697 Change parameter order for newVariableHolderForEdit
Make the old text non-optional, and typehint the old content.

Change-Id: I91be3c028e891d32fa8a61ed541336c85f8a9dfb
2019-08-04 16:48:21 +00:00
Daimona Eaytoy 4f8811bc3b Update cache key version for data in stashed edits
In I1dc3be6da1cc9e03bc47e8f8c867089ad0100f6f we added fields to the array.
Update the version to avoid PHP errors while upgrading the wikis, for edits
stashed before the upgrade, and saved afterwards.

Change-Id: I5489b556b1b0e9cb2af862dbfa0621909a5e355d
2019-08-04 16:36:23 +00:00
jenkins-bot c0b6267022 Merge "Use milliseconds for time profiling" 2019-08-04 16:12:59 +00:00
jenkins-bot fea7686724 Merge "Hide profiling for disabled filters" 2019-08-04 16:12:56 +00:00
jenkins-bot f7fd6a6daf Merge "Move per-filter matches profiling to per-filter data" 2019-08-04 16:07:58 +00:00
jenkins-bot d940ef63cd Merge "Specialize empty AFPData types" 2019-08-04 15:52:34 +00:00
Daimona Eaytoy e8866fee88 Fix @deprecated since version
The Runner was introduced in 1.34.

Change-Id: I9239705ef5628821b3ce6dbc27ac282cfc93e1e6
2019-08-04 15:39:54 +00:00
Daimona Eaytoy 5f4491f9aa Further deprecation for empty conditions
Start deprecating "empty" logic operators, and now that we have DEMPTY, simplify handling of empty function arguments introduced in Ica3e49f5b00595a95513d9683732e490aa7aae17.

Bug: T156096
Change-Id: Ied6b385e8690b6cc6e69afcf614389f737ab95bd
2019-08-04 15:33:49 +00:00
Daimona Eaytoy 9049be3609 Specialize empty AFPData types
As described in T156096#5389655.

Change-Id: Ifbf95a6b72a280cd77db6affbd8d642499bbfedc
2019-08-04 15:26:57 +00:00
Daimona Eaytoy c8ebb4956c Hide profiling for disabled filters
As data could be "old" and it may have no meaning.
Also remove a superfluous isset(), as $row->af_hidden is always set.

Depends-On: I2d8c8f8278073a9420e3eb373fb89a655925618a
Change-Id: I072363706c61f272c4c3691de4078e2a19148424
2019-08-03 23:28:42 +00:00
Daimona Eaytoy c3db63714e Use milliseconds for time profiling
Instead of seconds, and round the average condition at 1dp instead of 0.
Split from child patch by Dragons flight.

Depends-On: I2d8c8f8278073a9420e3eb373fb89a655925618a
Change-Id: I339aed5f8c1d49714e7927ce49286f9ce6c839f5
2019-08-03 23:24:46 +00:00
Daimona Eaytoy 0b7902fe6e Move per-filter matches profiling to per-filter data
They're currently stored separately, so move matches count together with
other per-filter data to keep it consistent. This also removes a
parameter from filterMatchesKey, as it's not needed anymore.
Split from child patch by Dragons flight.

Bug: T53294
Depends-On: I8f47beb73cfc1b63c4b3c809fc6d65a1e66ee334
Change-Id: I2d8c8f8278073a9420e3eb373fb89a655925618a
2019-08-03 23:22:20 +00:00
jenkins-bot 3b2b85b60d Merge "Store per-filter profiling in a single key" 2019-08-03 22:43:32 +00:00
Daimona Eaytoy 33bfe97d8c Move non-decimal numbers deprecation logging
Bug: T212730
Change-Id: Idb833c60541873bfe9c2b225009bd32e4a48cd60
2019-08-03 16:57:24 +00:00
Daimona Eaytoy d04a0d3afc Store per-filter profiling in a single key
Instead of having three keys, one for total actions, one for time and
one for conditions. This has several benefits: first, it avoids race
conditions which could happen having different keys. Second, it's much
more performant. Third, the code is also clearer to understand,
and more uniform with the one for global stats.
Split from child patch by Dragons flight.

Bug: T53294
Depends-On: I1dc3be6da1cc9e03bc47e8f8c867089ad0100f6f
Change-Id: I8f47beb73cfc1b63c4b3c809fc6d65a1e66ee334
2019-08-03 11:39:27 +00:00
Daimona Eaytoy a85e1ccc59 Make AbuseFilterParser::$funcCache non-static
Change-Id: I312efe3ce4d1f06e697aa4564aeec1bacbaf97d3
2019-08-03 09:19:49 +00:00
jenkins-bot 0e00654b7d Merge "Save profiling data and vars in cache when running filters" 2019-08-02 23:28:03 +00:00
Daimona Eaytoy 4acb266e90 Save profiling data and vars in cache when running filters
This is the proper solution to replace
Ia8e38ba25d1989fe71714d2b76891c4587921466, using a class member and an
additional method. Plus, change checkFilter not to accept a prefix, but a boolean indicating if the filter is global (as that's how it's used currently).

This change also fixes an issue which caused profiling data for local
filters to be mixed with profiling data for global filters with the same
ID.

Depends-On: Iafc142d2e5ba7aa0fb0d3265fa05cace27679738
Change-Id: I1dc3be6da1cc9e03bc47e8f8c867089ad0100f6f
2019-08-02 22:54:30 +00:00
Daimona Eaytoy 09d0254172 Better handling of DNONE
This patch includes:
 * Making it possible to access offsets of a DNONE (returning a DNONE)
 * Initializing user-defined variables as DNONE inside short-circuited branches
 * Make DNONE propagate with other operators
 * Make DNONE count as false for logic operators
 * Remove a now-outaded bit in doLevelAtom. In case of shortcircuit,
   $result is now DNONE instead of DNULL, and thus it's possible to
   access offsets of it. Performance++!
 * Don't allow modifying or adding an element of a DNONE as if it were an
    array (to avoid inconsistencies)

This re-applies Id85c673337fa90a3782fd22eb9690cd996967111 with several fixes.

NOTE: Haven't tested locally, although I'm pretty confident thanks to
the amount of tests added.

Bug: T214674
Bug: T228677
Change-Id: I5ec4ab44c4e88aaf18c0d7b73355d27050beeda7
2019-08-02 21:05:08 +00:00
jenkins-bot e3e157361d Merge "Revert "Initialize user-defined variables during shortcircuit"" 2019-07-29 23:30:50 +00:00
Daimona Eaytoy 13cdb86dd2 Revert "Initialize user-defined variables during shortcircuit"
Reason for revert: T214674#5374806

This reverts commit 56e6117afd.

Bug: T214674
Change-Id: Iccce248d2693cd9877a740b74e72a577e730435e
2019-07-29 23:06:23 +00:00
jenkins-bot dfa0109ba8 Merge "Rename old/new-(wiki)?text i18n keys" 2019-07-25 08:35:26 +00:00
Daimona Eaytoy eff4580a6f Add new method: AbuseFilterVariableHolder::newFromArray
Instead of duplicating code in several files.

Depends-On: I2eab2e50356eeb5224446ee2d0df9c787ae95b80
Change-Id: Iafc142d2e5ba7aa0fb0d3265fa05cace27679738
2019-07-24 18:41:32 +00:00
Daimona Eaytoy 4720c97530 Add a new class for methods related to running filters
Currently we strongly abuse (pardon the pun) the AbuseFilter class: its
purpose should be to hold static functions intended as generic utility
functions (e.g. to format messages, determine whether a filter is global
etc.), but we actually use it for all methods related to running filters.
This patch creates a new class, AbuseFilterRunner, containing all such
methods, which have been made non-static. This leads to several
improvements (also for related methods and the parser), and opens the
way to further improve the code.
Aside from making the code prettier, less global and easier to test,
this patch could also produce a performance improvement, although I
don't have tools to measure that.
Also note that many public methods have been removed, and almost any of
them has been made protected; a couple of them (the ones used from outside)
are left for back-compat, and will be removed in the future.

Change-Id: I2eab2e50356eeb5224446ee2d0df9c787ae95b80
2019-07-23 19:06:27 +00:00
Daimona Eaytoy 56e6117afd Initialize user-defined variables during shortcircuit
Bug: T214674
Depends-On: I5a14d4b2bc3ffd9caaaa095f16f36b9b6009db05
Change-Id: Id85c673337fa90a3782fd22eb9690cd996967111
2019-07-23 12:20:53 +00:00
Daimona Eaytoy 3258eeed69 Add normalizeThrottleParameters script to update.php
Bug: T203587
Bug: T203336
Bug: T203584
Bug: T203585
Depends-On: I7831dbb0bab55807392ac1f7915d6cb0cb713593
Change-Id: Ideaae7b58e0ffa606095aac4a9e5d21c6bdf11d2
2019-07-17 12:36:08 +00:00
Daimona Eaytoy 18d7d2ed62 Start using AFPData::DNONE
This should allow more flexibility when checking syntax, and a saner
behaviour overall.
Aside from not throwing exception in certain cases, the results should
be almost equal to the ones you would get without this patch. However,
there are still a few things to improve (which for convenience I wrote
inside the parser test) and many to test.

Bug: T204654
Depends-On: I69bfec45c76509fb1112641393f78e8d8834adcd
Change-Id: I5a14d4b2bc3ffd9caaaa095f16f36b9b6009db05
2019-07-14 08:48:47 +00:00
jenkins-bot d36dfaa951 Merge "build: Upgrade phan-taint-check-plugin from 1.5.x to 2.0.1" 2019-07-10 16:41:46 +00:00
James D. Forrester 70a03755e8 build: Upgrade phan-taint-check-plugin from 1.5.x to 2.0.1
Change-Id: Ica0439db5ec729c3b298db99fd89dd999f491457
2019-07-10 15:30:52 +00:00
Daimona Eaytoy 7bc566e635 Fix the regex for numbers, start deprecation of non-decimal numbers
Aside from the 14 thingy reported in the task, this syntax is awful! The
fix to the regex should only be intended as a temporary stopgap. A
proper fix would be to introduce a new syntax, like for instance the one
used in PHP.

Bug: T212726
Change-Id: Idc37a17ce539e6c63d67fc07d47d812569debe0e
2019-07-10 13:26:36 +00:00
Aaron Schulz fce600c3ee Fix bogus DB domain parameter in AbuseFilter::getCentralDB()
Follow-up to 2cf7b58434

Bug: T227613
Change-Id: I07b2d46389e6c8346d7c5848a00a1c2f8577acd8
2019-07-09 15:27:53 -07:00
jenkins-bot 6a5d5fc447 Merge "Really drop afl_log_id from update.php" 2019-07-09 17:03:24 +00:00
Daimona Eaytoy c6a9f3517a Really drop afl_log_id from update.php
Follow-up of 0b925da36e, somehow I forgot
to add the removal code for MySQL and SQLite to the Hooks.

Bug: T214592
Change-Id: If0d1d5430573273784ff6f6e338b0c2199f6d7bb
2019-07-09 16:51:28 +00:00
jenkins-bot c3dcd95733 Merge "Start making APFData members private" 2019-07-09 09:23:17 +00:00
jenkins-bot 9650f11d16 Merge "Fix error display on ViewEdit" 2019-07-09 09:02:12 +00:00
jenkins-bot 35ab35978b Merge "Add a new data type for non-initialized stuff" 2019-07-09 08:58:48 +00:00
Daimona Eaytoy 3aaeb20063 Start making APFData members private
$data and $type are meant to be read-only and should have getter
functions, but as usual they're just public. Add getter methods, a
comment with a @private annotation and remove usages in our codebase.

Change-Id: I5e51efc9f982a4e340b48d20cb1b38a75bb10021
2019-07-09 10:57:00 +02:00
jenkins-bot 6f0905541a Merge "Make AbuseFilterVariableHolder::mVars private" 2019-07-09 08:42:16 +00:00
jenkins-bot 69bebbb4ff Merge "Simplify action arrays" 2019-07-08 23:07:26 +00:00
Daimona Eaytoy 304b58d46a Make AbuseFilterVariableHolder::mVars private
This property is meant to be private, since it has all kinds of
getters/setters, aside from one which is introduced in this patch.

Change-Id: I217b1e22cabd3c0468c84b1d6a69a6ed3c6fa8e6
2019-07-08 16:25:10 +02:00
Daimona Eaytoy 5eff8f73b0 Fix error display on ViewEdit
This changes the buildFilterEditor function to be protected and to
behave consistently: so, instead of adding stuff to OutputPage inside it
and also returning other stuff to be added by the caller, the function
now adds everything itself.
Also, the message "you're editing an old version of the filter" is now
shown only if the user can see the filter.

Change-Id: I1f40af41c5de0f63aa6210a261928892da0b3f69
2019-07-08 16:11:33 +02:00
Daimona Eaytoy db193dad12 Rename old/new-(wiki)?text i18n keys
Now we have the key old-text for the old_wikitext variable, and the key
old-text-stripped for old_text. This can be confusing (see I61b2d252333ca634eae560d824f740f0f947b3d3), so use i18n keys more similar to the variable name.
NOTE: the keys will have to be changed on translatewiki if we want to avoid
confusing people.

Change-Id: Ie612350642ac4afc76f18639d988e72b4016b1e2
2019-07-08 15:55:02 +02:00
Daimona Eaytoy bc79962803 Add a new data type for non-initialized stuff
Split from I5a14d4b2bc3ffd9caaaa095f16f36b9b6009db05.
This adds a new data type to use for empty AFPDatas. Using NULL for that
makes it impossible to distinguish cases where we really got a null
value, and cases where there was nothing to parse.
For now, DNONE is the same as DNULL, but I've explicited DNULL where
necessary. A subsequent patch will make proper use of DNONE.

Bug: T156096
Change-Id: I69bfec45c76509fb1112641393f78e8d8834adcd
2019-07-08 15:35:02 +02:00
Daimona Eaytoy d8d4750e6a Simplify action arrays
The current form is awkward. They're all like
[ actionname => [ 'action' => actionname, 'parameters' => params ] ]
This is greatly confusing since adds a nesting level, and just
duplicates the actionname information (also, we actually never retrieve
it from the internal array). Instead, change all of them to be
[ actionname => params ]
which is a lot shorter and clearer (and easier to handle).
A similar case is handled in I8134ecc41fbecdbed99faf406e9e3ca91b6123b9
(see PS 8..10).

Change-Id: I34c040dbeb3ab01158fb3db22496def6ccaf72d9
2019-07-05 10:00:48 +02:00
Daimona Eaytoy b2af2f0bf5 Fix global caching
Use a more explicit TTL_WEEK, and add the version to avoid breaking the
world if we change the format.

Bug: T227299
Change-Id: I22705496ed8541c3dd9b643d78dff8886f4ff070
2019-07-05 09:17:57 +02:00
Aaron Schulz 2cf7b58434 Convert wfGetDB() calls to using getConnectionRef()
This handles the logic of calling reuseConnection() automatically

Change-Id: I9328e709fe5d81099338a31deef24d34db22d784
2019-07-04 15:09:32 -07:00
jenkins-bot 0d001e3d6f Merge "Disallow consecutive comparisons" 2019-07-04 20:43:57 +00:00
Daimona Eaytoy 85b46268f2 Rename the filterAction hook and add a parameter
The 'AbuseFilter-filterAction' hook is deprecated in favour of a new
'AbuseFilterAlterVariables' hook, which provides a User object and has a better name, since it reflects what it should be used for, and doesn't include the name of a function which will be removed. The hook will be hard deprecated in a subsequent patch, to avoid test failures.

Depends-On: Ic0ecc8746e2883c746bef815a0fee4131f1a0646
Change-Id: I212b1e09e9c05d487d96b2f4c28f2a613e6ff3cf
2019-07-04 18:10:47 +00:00
Daimona Eaytoy b7f1a7d459 Enforce saving of full abuse_filter row in cache
This is somehow a follow-up of
Ieb04f019453033c275e211cfc9fd68d5d7c392ef. A new method is introduced to
cache a filter, which checks that all fields are there.

Depends-On: I7c1ea17adf7f42cf9260d416906bfbf3b8a20688
Change-Id: Ic0ecc8746e2883c746bef815a0fee4131f1a0646
2019-07-04 18:10:33 +00:00
Daimona Eaytoy 7398730563 Disallow consecutive comparisons
As explained on phabricator, they don't work with shortcircuit, so they
already fail for all filters using them. Plus IMHO it's an unnecessary
deviation from PHP's behaviour, given that this syntax doesn't do what
users may expect.

Bug: T218906
Change-Id: If9e7545e14044c8dc3b4163bb6fca8ab0683b9fa
2019-07-04 19:15:07 +02:00
DannyS712 eb278479d5 Add help links to special pages
Bug: T226938
Change-Id: I50a76733b3b8d4ee72ccc6816b58a67a66b2f603
2019-07-03 16:06:16 +00:00
Daimona Eaytoy 6ea767f171 Tweak methods related to global filters
To make the switch to afl_filter_id and afl_global easier.

Bug: T227095
Depends-On: Ie550889495232b534c0f9aec31039cf21b2135b1
Change-Id: If557bad8f5c1a6d15e3556e4bfbd0330d7d49c59
2019-07-02 17:02:50 +02:00
Daimona Eaytoy 0b925da36e Drop afl_log_id
Field unused since its introduction.

Bug: T214592
Change-Id: I1f4f775e9678de5184672251631a490e4eb81764
2019-06-28 17:55:55 +00:00
Daimona Eaytoy 6100955242 Use more verbose names for filter IDs
Follow-up of Ie550889495232b534c0f9aec31039cf21b2135b1, suggested by
Krinkle.

Change-Id: Ia8f40644c7f4a6ed53186a5edab5df1bd313166a
2019-06-25 18:20:32 +00:00
Daimona Eaytoy 382751a707 Move conditions-related stuff inside AbuseFilterParser
Instead of relying on static methods and members in the AbuseFilter
class, move everything related to conditions inside the Parser, as the
amount of used conditions is something pertaining a single
AbuseFilter(Caching)Parser instance.
This change requires changing some signatures and adding parameters,
but will make introducing the new AbuseFilterRunner class easier (and
that will clean signatures, too).

Depends-On: I5b29ff556eca45fe59d15e2e3df4d06f1f6b3934
Change-Id: I7c1ea17adf7f42cf9260d416906bfbf3b8a20688
2019-06-19 15:14:17 +00:00
Daimona Eaytoy a8e8611509 Remove log_ids meta-variables
This is the second part of removing meta-variables. To achieve this, a
static property is added and another one removed.

Depends-On: I7f60df24dc8e706af289ebbbde7536c0baf8d5c3
Change-Id: I5b29ff556eca45fe59d15e2e3df4d06f1f6b3934
2019-06-19 14:44:56 +00:00
Daimona Eaytoy 246d8e78aa Improve getFirstFilterChange function
Fix the typehint, and use selectField instead of selectRow.
Follow-up of Ie550889495232b534c0f9aec31039cf21b2135b1 suggested by
Krinkle.

Change-Id: I7e74b7b484dfa487db96598ef7aef4895d7bf275
2019-06-17 13:01:56 +02:00
Daimona Eaytoy e7cd4b2a98 Rewrite AbuseFilter::decodeGlobalName
Now it returns an array with a bit more info, and has a different name
to reflect the fact that its input is now split in two parts. Plus, make
it throw whenever it gets an unexpected input, and add a bunch of test
cases for it.

Depends-On: Ib5fdeb75c1324f672b4ded39681f006fde34b4d1
Change-Id: Ie550889495232b534c0f9aec31039cf21b2135b1
2019-06-12 23:56:25 +00:00
Thalia 22ceae7e23 Use MediaWiki\Block\DatabaseBlock instead of Block
This follows the rename of the Block class in I6d96b63ca0.

Change-Id: I44cf9eb68c23a8299316effa4dee7f732486dd84
2019-05-31 16:08:19 +01:00
Daimona Eaytoy 53f03e5301 Tokenizer caching back to APC
Partial revert of I4dd81a723e2bdb828b90594ad66a3918d8ec5b6c.
Thinking again of it, I think it's not worth it to have this data over
the network. Plus, given that it's not-that-slow to be computed, I think
there can only be a performance gain in using APC (as opposed to e.g.
memcached/redis) for 99.9% of the filters.

Change-Id: I8c6a4a95ec12c18ede8e6419540f7a2ac943457c
2019-05-28 19:48:26 +02:00
Daimona Eaytoy c3ee5e7251 Simplify static properties in AbuseFilterHooks
Avoid all those data types (i.e. use null instead of false), use camelcase, make them private. Also, remove some logic to handle $lastEditPage being Article, as it can only be WikiPage.

Depends-On: I5a9db6e7c4356c9662a0b0a51e66252555b3d998
Depends-On: I359a618ffc4e45ce1fb70f2d1aa99a6668609e36
Change-Id: I7f60df24dc8e706af289ebbbde7536c0baf8d5c3
2019-05-25 16:27:21 +00:00
Daimona Eaytoy 864d2b7e03 Don't run filters with null title
As all title variables would be null, and the result pretty meaningless.

NOTE: Please vote V+2 and submit manually. I359a618ffc4e45ce1fb70f2d
should then be +2ed right after that. This way, there is no need to create
two more patches just for a handful of tests being broken for a minute.

Bug: T144265
Bug: T219030
Depends-On: If6b91711534c0d60e1aa27bd5748c3023e29f376
Change-Id: I5a9db6e7c4356c9662a0b0a51e66252555b3d998
2019-05-25 16:27:08 +00:00
jenkins-bot e5a15ab968 Merge "Add a parameter to generate(User|Title)Vars hooks to specify context" 2019-05-25 11:37:04 +00:00
jenkins-bot 2b1c62ecdd Merge "Restore unit tests for CachingParser and fix it" 2019-05-25 11:24:51 +00:00
Daimona Eaytoy 2b4ddd1096 Change a long if/elseif to switch
This is done for 3 reasons: 1-the code should hopefully be clearer;
2-FWIW, switch's are a little bit faster than elseifs (roughly 15%); 3-to
fix a bug with coverage driver which says those lines are not covered.
3 is a follow-up of I997576141943959d4602a9f839311108928ec766.

Change-Id: I2d69e421e384cb74a799c5c5f77d041a7e02d4c8
2019-05-25 10:59:37 +02:00
Daimona Eaytoy 39fc7c12af Restore unit tests for CachingParser and fix it
Added cachingParser back to *all* the parser tests, fixed a couple of
differences with the normal parser, and added a couple of tests so that
any cachingParser-related file has 100% coverage. Also move the remaining
get_matches tests inside parserTests, and specify the parser used in case of failure.
This also adds a new base class for parser-related tests with a couple
of util methods.

Bug: T201193
Change-Id: I980aec3481a52ecc35f1811a366014a5581a7cdb
2019-05-25 10:55:24 +02:00
jenkins-bot 1cb80be0ad Merge "Add tests for various data type casts" 2019-05-24 19:19:20 +00:00
jenkins-bot 058e215882 Merge "Refactor tokenizer caching" 2019-05-24 19:09:03 +00:00
Daimona Eaytoy f56562f583 Add tests for global filters
Another crucial part to have covered. Also clarify that
AbuseFilterCentralDB can be of the form "dbname-prefix".

Remove a filter used for profiling and replace it with a global one:
we're still fine, and the list is kept shorter.

Bug: T201193
Depends-On: I5ee7ba44a6cd82a5ddb24fb4127af04d96e647f4
Change-Id: If6b91711534c0d60e1aa27bd5748c3023e29f376
2019-05-24 16:58:23 +02:00
Daimona Eaytoy a766e39ade Add unit tests for profiling
Yet another important part to have covered. While for normal edits it
already works, for stashed ones it doesn't. That's why we need the patch
for checkAllFilters. Since for stashed edits profiling stats are all
zeros, this may explain T201334.
Changed the timestamp variable to use wfTimestamp instead of time() so
that we can fake it inside unit tests.
In a subsequent patch we should add average runtime conditions to tests
(really tricky).

Bug: T201193
Depends-On: Ib17821240b25c972a187e6b5eae42c5ada6c65e7
Change-Id: I5ee7ba44a6cd82a5ddb24fb4127af04d96e647f4
2019-05-23 08:47:40 +00:00
Daimona Eaytoy 33fca4e096 Ignore trailing commas in function calls
First step before removing this weird syntax. I'd love to add a unit
test for params count, but I couldn't find a way, since doLevelFunction
is protected, relies on class members, and the args count is local.

Bug: T153251
Change-Id: Ica3e49f5b00595a95513d9683732e490aa7aae17
2019-05-21 23:13:31 +00:00
jenkins-bot a9afdc1f80 Merge "Slightly improve degroup action, remove Phan suppressions" 2019-05-19 17:20:23 +00:00
jenkins-bot 48ac8c492b Merge "Temporarily catch another BadMethodCallException" 2019-05-19 17:13:10 +00:00
jenkins-bot e66d30d37c Merge "Don't send empty array to Database::makeList" 2019-05-18 12:27:50 +00:00
jenkins-bot 75e5c907fc Merge "Remove problematic array type hint from hook handler" 2019-05-18 09:01:48 +00:00