Commit graph

4960 commits

Author SHA1 Message Date
jenkins-bot 44e427601f Merge "Avoid DEMPTY leak" 2019-08-04 21:16:22 +00:00
Daimona Eaytoy 6ef2cf523b Profiling: don't count time for operations shared with the edit
Parsing wikitext and retrieving its links are operations which we share
with the edit, so that if a filter does that, it won't be done later
upon saving.
Thus, add a static variable to subtract such time and avoid to erroneously log as slow any filter using those variables.

Bug: T219092
Depends-On: I24fbd41ac188a9cf6a7d3ca33dce349aedc9faa6
Change-Id: I7c0170167b508132cd16e566c654a6c98dd683e9
2019-08-04 20:12:10 +00:00
jenkins-bot 4347331692 Merge "Reset all filter profiling data at once" 2019-08-04 18:53:57 +00:00
jenkins-bot 19182606c1 Merge "Merge global profiling keys" 2019-08-04 18:40:14 +00:00
Daimona Eaytoy f977e858ab Avoid DEMPTY leak
As shown in the coverage reports [0], some empty operand logging lines are covered, but no test should have empty operands. I see one of the cause is skipOverBraces keeping $result as is, even if DEMPTY, so turn it into a DUNDEFINED.

[0] - https://doc.wikimedia.org/cover-extensions/AbuseFilter/includes/parser/AbuseFilterParser.php.html

Change-Id: I7831f3ed9f7c0656e0e8f77ded049c20eca682ba
2019-08-04 18:25:04 +00:00
Daimona Eaytoy 2742f24bca Reset all filter profiling data at once
Instead of scattering the process all over the code (and doing it
together with checking if the key already exists).
Wrap the logic in new methods for better readability.

Depends-On: Ib12e072a245fcad93c6c6bd452041f3441f68bb7
Change-Id: I24fbd41ac188a9cf6a7d3ca33dce349aedc9faa6
2019-08-04 18:03:14 +00:00
rarohde d022377578 Merge global profiling keys
The last step of the profiling overhaul. See T53294 for the original description by Dragons flight.

Note: Here I'm adding a FixMe for a problem which already exists in the code
and the child patch will fix it.

Bug: T53294
Depends-On: I2d8c8f8278073a9420e3eb373fb89a655925618a
Change-Id: Ib12e072a245fcad93c6c6bd452041f3441f68bb7
2019-08-04 17:59:58 +00:00
jenkins-bot 879ff59fe2 Merge "Split hook handlers related to filtering" 2019-08-04 17:55:51 +00:00
jenkins-bot be4b1c5bc1 Merge "Fix @deprecated since version" 2019-08-04 17:50:47 +00:00
jenkins-bot e733872e13 Merge "Allow accessing offsets of built-in variables" 2019-08-04 17:48:46 +00:00
jenkins-bot 790cd38fb0 Merge "Further deprecation for empty conditions" 2019-08-04 17:26:40 +00:00
Daimona Eaytoy 7d192cb1f2 Split hook handlers related to filtering
This adds a new get(Type)Vars method for every action type. The goal is
to 1-have shorter methods, which is always good; 2-try to make this code
a bit more testable.
I left as a todo moving all these methods to a separate class, the idea
being to make them non-static and thus easier to be tested.

Depends-On: I2eab2e50356eeb5224446ee2d0df9c787ae95b80
Change-Id: I6de2dd27a8f972b3f74c730a1516639f8c622166
2019-08-04 17:21:29 +00:00
Daimona Eaytoy 517919fca8 Allow accessing offsets of built-in variables
I5ec4ab44c4e88aaf18c0d7b73355d27050beeda7 almost fixed this bug, but we
also have to make it possible to access builtin variables as arrays.
This will only make sense for a few variables (e.g. added_lines and
removed_lines), but I don't think we should validate it when checking
syntax.

Bug: T198531
Change-Id: I417e1b8d4802bbfccd091ce5c7617659cfd1e4ea
2019-08-04 17:14:44 +00:00
jenkins-bot 534a282f7d Merge "Clarify "filter" field in SpecialAbuseLog and ApiQueryAbuseLog" 2019-08-04 17:13:47 +00:00
jenkins-bot 5fccb5a67b Merge "Change parameter order for newVariableHolderForEdit" 2019-08-04 17:07:27 +00:00
Daimona Eaytoy 71e3719e12 Clarify "filter" field in SpecialAbuseLog and ApiQueryAbuseLog
The "filter" fields can also accept a list of filters, and also global filters, so make it clear in the UI and in messages.

Change-Id: Ib258716d8e6792fd496938ebb4e8a2565d6370b7
2019-08-04 16:55:05 +00:00
Daimona Eaytoy c635f41697 Change parameter order for newVariableHolderForEdit
Make the old text non-optional, and typehint the old content.

Change-Id: I91be3c028e891d32fa8a61ed541336c85f8a9dfb
2019-08-04 16:48:21 +00:00
Daimona Eaytoy 4f8811bc3b Update cache key version for data in stashed edits
In I1dc3be6da1cc9e03bc47e8f8c867089ad0100f6f we added fields to the array.
Update the version to avoid PHP errors while upgrading the wikis, for edits
stashed before the upgrade, and saved afterwards.

Change-Id: I5489b556b1b0e9cb2af862dbfa0621909a5e355d
2019-08-04 16:36:23 +00:00
jenkins-bot c0b6267022 Merge "Use milliseconds for time profiling" 2019-08-04 16:12:59 +00:00
jenkins-bot fea7686724 Merge "Hide profiling for disabled filters" 2019-08-04 16:12:56 +00:00
jenkins-bot f7fd6a6daf Merge "Move per-filter matches profiling to per-filter data" 2019-08-04 16:07:58 +00:00
jenkins-bot d940ef63cd Merge "Specialize empty AFPData types" 2019-08-04 15:52:34 +00:00
Daimona Eaytoy e8866fee88 Fix @deprecated since version
The Runner was introduced in 1.34.

Change-Id: I9239705ef5628821b3ce6dbc27ac282cfc93e1e6
2019-08-04 15:39:54 +00:00
Daimona Eaytoy 5f4491f9aa Further deprecation for empty conditions
Start deprecating "empty" logic operators, and now that we have DEMPTY, simplify handling of empty function arguments introduced in Ica3e49f5b00595a95513d9683732e490aa7aae17.

Bug: T156096
Change-Id: Ied6b385e8690b6cc6e69afcf614389f737ab95bd
2019-08-04 15:33:49 +00:00
Daimona Eaytoy 9049be3609 Specialize empty AFPData types
As described in T156096#5389655.

Change-Id: Ifbf95a6b72a280cd77db6affbd8d642499bbfedc
2019-08-04 15:26:57 +00:00
Daimona Eaytoy c8ebb4956c Hide profiling for disabled filters
As data could be "old" and it may have no meaning.
Also remove a superfluous isset(), as $row->af_hidden is always set.

Depends-On: I2d8c8f8278073a9420e3eb373fb89a655925618a
Change-Id: I072363706c61f272c4c3691de4078e2a19148424
2019-08-03 23:28:42 +00:00
Daimona Eaytoy c3db63714e Use milliseconds for time profiling
Instead of seconds, and round the average condition at 1dp instead of 0.
Split from child patch by Dragons flight.

Depends-On: I2d8c8f8278073a9420e3eb373fb89a655925618a
Change-Id: I339aed5f8c1d49714e7927ce49286f9ce6c839f5
2019-08-03 23:24:46 +00:00
Daimona Eaytoy 0b7902fe6e Move per-filter matches profiling to per-filter data
They're currently stored separately, so move matches count together with
other per-filter data to keep it consistent. This also removes a
parameter from filterMatchesKey, as it's not needed anymore.
Split from child patch by Dragons flight.

Bug: T53294
Depends-On: I8f47beb73cfc1b63c4b3c809fc6d65a1e66ee334
Change-Id: I2d8c8f8278073a9420e3eb373fb89a655925618a
2019-08-03 23:22:20 +00:00
jenkins-bot 3b2b85b60d Merge "Store per-filter profiling in a single key" 2019-08-03 22:43:32 +00:00
Daimona Eaytoy 33bfe97d8c Move non-decimal numbers deprecation logging
Bug: T212730
Change-Id: Idb833c60541873bfe9c2b225009bd32e4a48cd60
2019-08-03 16:57:24 +00:00
Daimona Eaytoy d04a0d3afc Store per-filter profiling in a single key
Instead of having three keys, one for total actions, one for time and
one for conditions. This has several benefits: first, it avoids race
conditions which could happen having different keys. Second, it's much
more performant. Third, the code is also clearer to understand,
and more uniform with the one for global stats.
Split from child patch by Dragons flight.

Bug: T53294
Depends-On: I1dc3be6da1cc9e03bc47e8f8c867089ad0100f6f
Change-Id: I8f47beb73cfc1b63c4b3c809fc6d65a1e66ee334
2019-08-03 11:39:27 +00:00
Daimona Eaytoy a85e1ccc59 Make AbuseFilterParser::$funcCache non-static
Change-Id: I312efe3ce4d1f06e697aa4564aeec1bacbaf97d3
2019-08-03 09:19:49 +00:00
jenkins-bot 0e00654b7d Merge "Save profiling data and vars in cache when running filters" 2019-08-02 23:28:03 +00:00
Daimona Eaytoy 4acb266e90 Save profiling data and vars in cache when running filters
This is the proper solution to replace
Ia8e38ba25d1989fe71714d2b76891c4587921466, using a class member and an
additional method. Plus, change checkFilter not to accept a prefix, but a boolean indicating if the filter is global (as that's how it's used currently).

This change also fixes an issue which caused profiling data for local
filters to be mixed with profiling data for global filters with the same
ID.

Depends-On: Iafc142d2e5ba7aa0fb0d3265fa05cace27679738
Change-Id: I1dc3be6da1cc9e03bc47e8f8c867089ad0100f6f
2019-08-02 22:54:30 +00:00
Daimona Eaytoy 09d0254172 Better handling of DNONE
This patch includes:
 * Making it possible to access offsets of a DNONE (returning a DNONE)
 * Initializing user-defined variables as DNONE inside short-circuited branches
 * Make DNONE propagate with other operators
 * Make DNONE count as false for logic operators
 * Remove a now-outaded bit in doLevelAtom. In case of shortcircuit,
   $result is now DNONE instead of DNULL, and thus it's possible to
   access offsets of it. Performance++!
 * Don't allow modifying or adding an element of a DNONE as if it were an
    array (to avoid inconsistencies)

This re-applies Id85c673337fa90a3782fd22eb9690cd996967111 with several fixes.

NOTE: Haven't tested locally, although I'm pretty confident thanks to
the amount of tests added.

Bug: T214674
Bug: T228677
Change-Id: I5ec4ab44c4e88aaf18c0d7b73355d27050beeda7
2019-08-02 21:05:08 +00:00
Translation updater bot 0f3f630f55 Localisation updates from https://translatewiki.net.
Change-Id: I39186630a2df8010e3703c037564720bff52c382
2019-08-01 09:14:50 +02:00
MarcoAurelio f07bf610c0 build: Update npm dependencies
- eslint-config-wikimedia  0.12.0  →  0.13.1
 - grunt-banana-checker      0.7.0  →   0.7.1
 - grunt-eslint             21.0.0  →  22.0.0
 - grunt-stylelint          0.10.1  →  0.11.0
 - stylelint                 9.9.0  →  10.1.0

Also npm audit (fix) them afterwards.

Addresses CVE-2019-10744.

Change-Id: I3153e269decab7f2637e2a41934e0ee07a5df760
2019-07-31 17:29:00 +00:00
Translation updater bot 180b8fc2d4 Localisation updates from https://translatewiki.net.
Change-Id: I6e31888d47c6c41eff51eb164859817304e2a5d6
2019-07-30 09:06:37 +02:00
jenkins-bot e3e157361d Merge "Revert "Initialize user-defined variables during shortcircuit"" 2019-07-29 23:30:50 +00:00
Daimona Eaytoy 13cdb86dd2 Revert "Initialize user-defined variables during shortcircuit"
Reason for revert: T214674#5374806

This reverts commit 56e6117afd.

Bug: T214674
Change-Id: Iccce248d2693cd9877a740b74e72a577e730435e
2019-07-29 23:06:23 +00:00
Translation updater bot e82e455ce3 Localisation updates from https://translatewiki.net.
Change-Id: I8d660e209d12599fc20f6d397aa7857e784de6c9
2019-07-29 09:13:06 +02:00
Translation updater bot aa26de8605 Localisation updates from https://translatewiki.net.
Change-Id: Ifd60e0abe5b26c516d799505c1bb99cdf5efa3d0
2019-07-27 09:29:18 +02:00
Translation updater bot f7a2b3d508 Localisation updates from https://translatewiki.net.
Change-Id: If4214fe88af12723b2faad6ef457eb14d3f8453b
2019-07-26 08:53:50 +02:00
jenkins-bot dfa0109ba8 Merge "Rename old/new-(wiki)?text i18n keys" 2019-07-25 08:35:26 +00:00
Translation updater bot f27a5887ac Localisation updates from https://translatewiki.net.
Change-Id: Id51fb12fd64ec236cbaedc58107370e337125186
2019-07-24 22:27:45 +02:00
jenkins-bot 1281cce4c9 Merge "Add new method: AbuseFilterVariableHolder::newFromArray" 2019-07-24 18:59:03 +00:00
jenkins-bot 7cb9b46112 Merge "Add a new class for methods related to running filters" 2019-07-24 18:55:15 +00:00
Daimona Eaytoy eff4580a6f Add new method: AbuseFilterVariableHolder::newFromArray
Instead of duplicating code in several files.

Depends-On: I2eab2e50356eeb5224446ee2d0df9c787ae95b80
Change-Id: Iafc142d2e5ba7aa0fb0d3265fa05cace27679738
2019-07-24 18:41:32 +00:00
Daimona Eaytoy 1860b4ca5a fixOldLogEntries: Don't print an extra line
Bug: T228655
Change-Id: Idce1f121e468ce0ac0720951f28613e6cc564113
2019-07-24 11:23:28 +00:00
jenkins-bot 37ef6574f4 Merge "Aesthetic changes for fixOldLogEntries script" 2019-07-23 19:33:07 +00:00