Commit graph

4934 commits

Author SHA1 Message Date
Daimona Eaytoy c3db63714e Use milliseconds for time profiling
Instead of seconds, and round the average condition at 1dp instead of 0.
Split from child patch by Dragons flight.

Depends-On: I2d8c8f8278073a9420e3eb373fb89a655925618a
Change-Id: I339aed5f8c1d49714e7927ce49286f9ce6c839f5
2019-08-03 23:24:46 +00:00
Daimona Eaytoy 0b7902fe6e Move per-filter matches profiling to per-filter data
They're currently stored separately, so move matches count together with
other per-filter data to keep it consistent. This also removes a
parameter from filterMatchesKey, as it's not needed anymore.
Split from child patch by Dragons flight.

Bug: T53294
Depends-On: I8f47beb73cfc1b63c4b3c809fc6d65a1e66ee334
Change-Id: I2d8c8f8278073a9420e3eb373fb89a655925618a
2019-08-03 23:22:20 +00:00
jenkins-bot 3b2b85b60d Merge "Store per-filter profiling in a single key" 2019-08-03 22:43:32 +00:00
Daimona Eaytoy 33bfe97d8c Move non-decimal numbers deprecation logging
Bug: T212730
Change-Id: Idb833c60541873bfe9c2b225009bd32e4a48cd60
2019-08-03 16:57:24 +00:00
Daimona Eaytoy d04a0d3afc Store per-filter profiling in a single key
Instead of having three keys, one for total actions, one for time and
one for conditions. This has several benefits: first, it avoids race
conditions which could happen having different keys. Second, it's much
more performant. Third, the code is also clearer to understand,
and more uniform with the one for global stats.
Split from child patch by Dragons flight.

Bug: T53294
Depends-On: I1dc3be6da1cc9e03bc47e8f8c867089ad0100f6f
Change-Id: I8f47beb73cfc1b63c4b3c809fc6d65a1e66ee334
2019-08-03 11:39:27 +00:00
Daimona Eaytoy a85e1ccc59 Make AbuseFilterParser::$funcCache non-static
Change-Id: I312efe3ce4d1f06e697aa4564aeec1bacbaf97d3
2019-08-03 09:19:49 +00:00
jenkins-bot 0e00654b7d Merge "Save profiling data and vars in cache when running filters" 2019-08-02 23:28:03 +00:00
Daimona Eaytoy 4acb266e90 Save profiling data and vars in cache when running filters
This is the proper solution to replace
Ia8e38ba25d1989fe71714d2b76891c4587921466, using a class member and an
additional method. Plus, change checkFilter not to accept a prefix, but a boolean indicating if the filter is global (as that's how it's used currently).

This change also fixes an issue which caused profiling data for local
filters to be mixed with profiling data for global filters with the same
ID.

Depends-On: Iafc142d2e5ba7aa0fb0d3265fa05cace27679738
Change-Id: I1dc3be6da1cc9e03bc47e8f8c867089ad0100f6f
2019-08-02 22:54:30 +00:00
Daimona Eaytoy 09d0254172 Better handling of DNONE
This patch includes:
 * Making it possible to access offsets of a DNONE (returning a DNONE)
 * Initializing user-defined variables as DNONE inside short-circuited branches
 * Make DNONE propagate with other operators
 * Make DNONE count as false for logic operators
 * Remove a now-outaded bit in doLevelAtom. In case of shortcircuit,
   $result is now DNONE instead of DNULL, and thus it's possible to
   access offsets of it. Performance++!
 * Don't allow modifying or adding an element of a DNONE as if it were an
    array (to avoid inconsistencies)

This re-applies Id85c673337fa90a3782fd22eb9690cd996967111 with several fixes.

NOTE: Haven't tested locally, although I'm pretty confident thanks to
the amount of tests added.

Bug: T214674
Bug: T228677
Change-Id: I5ec4ab44c4e88aaf18c0d7b73355d27050beeda7
2019-08-02 21:05:08 +00:00
Translation updater bot 0f3f630f55 Localisation updates from https://translatewiki.net.
Change-Id: I39186630a2df8010e3703c037564720bff52c382
2019-08-01 09:14:50 +02:00
MarcoAurelio f07bf610c0 build: Update npm dependencies
- eslint-config-wikimedia  0.12.0  →  0.13.1
 - grunt-banana-checker      0.7.0  →   0.7.1
 - grunt-eslint             21.0.0  →  22.0.0
 - grunt-stylelint          0.10.1  →  0.11.0
 - stylelint                 9.9.0  →  10.1.0

Also npm audit (fix) them afterwards.

Addresses CVE-2019-10744.

Change-Id: I3153e269decab7f2637e2a41934e0ee07a5df760
2019-07-31 17:29:00 +00:00
Translation updater bot 180b8fc2d4 Localisation updates from https://translatewiki.net.
Change-Id: I6e31888d47c6c41eff51eb164859817304e2a5d6
2019-07-30 09:06:37 +02:00
jenkins-bot e3e157361d Merge "Revert "Initialize user-defined variables during shortcircuit"" 2019-07-29 23:30:50 +00:00
Daimona Eaytoy 13cdb86dd2 Revert "Initialize user-defined variables during shortcircuit"
Reason for revert: T214674#5374806

This reverts commit 56e6117afd.

Bug: T214674
Change-Id: Iccce248d2693cd9877a740b74e72a577e730435e
2019-07-29 23:06:23 +00:00
Translation updater bot e82e455ce3 Localisation updates from https://translatewiki.net.
Change-Id: I8d660e209d12599fc20f6d397aa7857e784de6c9
2019-07-29 09:13:06 +02:00
Translation updater bot aa26de8605 Localisation updates from https://translatewiki.net.
Change-Id: Ifd60e0abe5b26c516d799505c1bb99cdf5efa3d0
2019-07-27 09:29:18 +02:00
Translation updater bot f7a2b3d508 Localisation updates from https://translatewiki.net.
Change-Id: If4214fe88af12723b2faad6ef457eb14d3f8453b
2019-07-26 08:53:50 +02:00
jenkins-bot dfa0109ba8 Merge "Rename old/new-(wiki)?text i18n keys" 2019-07-25 08:35:26 +00:00
Translation updater bot f27a5887ac Localisation updates from https://translatewiki.net.
Change-Id: Id51fb12fd64ec236cbaedc58107370e337125186
2019-07-24 22:27:45 +02:00
jenkins-bot 1281cce4c9 Merge "Add new method: AbuseFilterVariableHolder::newFromArray" 2019-07-24 18:59:03 +00:00
jenkins-bot 7cb9b46112 Merge "Add a new class for methods related to running filters" 2019-07-24 18:55:15 +00:00
Daimona Eaytoy eff4580a6f Add new method: AbuseFilterVariableHolder::newFromArray
Instead of duplicating code in several files.

Depends-On: I2eab2e50356eeb5224446ee2d0df9c787ae95b80
Change-Id: Iafc142d2e5ba7aa0fb0d3265fa05cace27679738
2019-07-24 18:41:32 +00:00
Daimona Eaytoy 1860b4ca5a fixOldLogEntries: Don't print an extra line
Bug: T228655
Change-Id: Idce1f121e468ce0ac0720951f28613e6cc564113
2019-07-24 11:23:28 +00:00
jenkins-bot 37ef6574f4 Merge "Aesthetic changes for fixOldLogEntries script" 2019-07-23 19:33:07 +00:00
Daimona Eaytoy 4720c97530 Add a new class for methods related to running filters
Currently we strongly abuse (pardon the pun) the AbuseFilter class: its
purpose should be to hold static functions intended as generic utility
functions (e.g. to format messages, determine whether a filter is global
etc.), but we actually use it for all methods related to running filters.
This patch creates a new class, AbuseFilterRunner, containing all such
methods, which have been made non-static. This leads to several
improvements (also for related methods and the parser), and opens the
way to further improve the code.
Aside from making the code prettier, less global and easier to test,
this patch could also produce a performance improvement, although I
don't have tools to measure that.
Also note that many public methods have been removed, and almost any of
them has been made protected; a couple of them (the ones used from outside)
are left for back-compat, and will be removed in the future.

Change-Id: I2eab2e50356eeb5224446ee2d0df9c787ae95b80
2019-07-23 19:06:27 +00:00
Daimona Eaytoy 40ebf9ad6d Aesthetic changes for fixOldLogEntries script
Bug: T228655
Change-Id: Ic94a32ec906eb49db85c96212718bb3cf03f86af
2019-07-23 13:21:00 +00:00
Daimona Eaytoy 56e6117afd Initialize user-defined variables during shortcircuit
Bug: T214674
Depends-On: I5a14d4b2bc3ffd9caaaa095f16f36b9b6009db05
Change-Id: Id85c673337fa90a3782fd22eb9690cd996967111
2019-07-23 12:20:53 +00:00
Translation updater bot 1665e76109 Localisation updates from https://translatewiki.net.
Change-Id: Ib35b213829c043262141041d151500629039ad31
2019-07-23 08:53:22 +02:00
Translation updater bot 9c81738e6c Localisation updates from https://translatewiki.net.
Change-Id: I6969f7ad3f05ac761f7fb29430d61be49bb4c006
2019-07-22 22:38:57 +02:00
Daimona Eaytoy 8f55230a0b Merge "Add a maintenance script to fix broken log entries" 2019-07-22 12:32:40 +00:00
Translation updater bot 15d5d13277 Localisation updates from https://translatewiki.net.
Change-Id: I7f62474782c80aae25bba6a834edba269f94e725
2019-07-18 08:51:37 +02:00
Daimona Eaytoy 3258eeed69 Add normalizeThrottleParameters script to update.php
Bug: T203587
Bug: T203336
Bug: T203584
Bug: T203585
Depends-On: I7831dbb0bab55807392ac1f7915d6cb0cb713593
Change-Id: Ideaae7b58e0ffa606095aac4a9e5d21c6bdf11d2
2019-07-17 12:36:08 +00:00
Daimona Eaytoy 07e5168eb9 Add a maintenance script to fix broken log entries
A very simple fix for some old entries imploded with '\n' instead of
"\n". As for other scripts, this extends LoggedUpdateMaintenance but it
includes a dry run option to be tested, and a subsequent patch (to be
merged after testing) will add it to update.php.

Bug: T208931
Change-Id: I16d629c05c3b9c9055c68dafd261b0a5c0db9c25
2019-07-15 11:59:45 +00:00
Daimona Eaytoy 9937f8b050 Remove extra file from parser tests
Added in I5a14d4b2bc3ffd9caaaa095f16f36b9b6009db05, but .r files aren't
used anymore since I6c06e596587750c4ebaabafbd277bc75eeb436a5, and I
forgot to remove the file upon rebasing.

Change-Id: Id688d215b1136bd0a04b8c0d8d8d16de5da1295e
2019-07-15 12:22:09 +02:00
Translation updater bot 3c2b399f57 Localisation updates from https://translatewiki.net.
Change-Id: Iec96646405c046b959d05adffa10da2b2e6c313f
2019-07-15 08:32:54 +02:00
jenkins-bot 7a2d5ba94c Merge "Start using AFPData::DNONE" 2019-07-14 17:33:10 +00:00
Daimona Eaytoy 18d7d2ed62 Start using AFPData::DNONE
This should allow more flexibility when checking syntax, and a saner
behaviour overall.
Aside from not throwing exception in certain cases, the results should
be almost equal to the ones you would get without this patch. However,
there are still a few things to improve (which for convenience I wrote
inside the parser test) and many to test.

Bug: T204654
Depends-On: I69bfec45c76509fb1112641393f78e8d8834adcd
Change-Id: I5a14d4b2bc3ffd9caaaa095f16f36b9b6009db05
2019-07-14 08:48:47 +00:00
libraryupgrader a1203293bd build: Updating npm dependencies
* set-value: 2.0.0 → 2.0.1
  * https://npmjs.com/advisories/1012
  * CVE-2019-10747
* union-value: 1.0.0 → 1.0.1
  * https://npmjs.com/advisories/1012
  * CVE-2019-10747
* mixin-deep: 1.3.1 → 1.3.2
  * https://npmjs.com/advisories/1013
  * CVE-2019-10746

Change-Id: I36781c59229e416074783f0dc80b7e96bc730e1e
2019-07-14 07:35:03 +00:00
Translation updater bot 502a7541fb Localisation updates from https://translatewiki.net.
Change-Id: Iff5ccd0346b228f1a785e7ec04ba8c7636faa638
2019-07-12 08:41:58 +02:00
jenkins-bot d36dfaa951 Merge "build: Upgrade phan-taint-check-plugin from 1.5.x to 2.0.1" 2019-07-10 16:41:46 +00:00
James D. Forrester 70a03755e8 build: Upgrade phan-taint-check-plugin from 1.5.x to 2.0.1
Change-Id: Ica0439db5ec729c3b298db99fd89dd999f491457
2019-07-10 15:30:52 +00:00
Daimona Eaytoy 7bc566e635 Fix the regex for numbers, start deprecation of non-decimal numbers
Aside from the 14 thingy reported in the task, this syntax is awful! The
fix to the regex should only be intended as a temporary stopgap. A
proper fix would be to introduce a new syntax, like for instance the one
used in PHP.

Bug: T212726
Change-Id: Idc37a17ce539e6c63d67fc07d47d812569debe0e
2019-07-10 13:26:36 +00:00
Aaron Schulz fce600c3ee Fix bogus DB domain parameter in AbuseFilter::getCentralDB()
Follow-up to 2cf7b58434

Bug: T227613
Change-Id: I07b2d46389e6c8346d7c5848a00a1c2f8577acd8
2019-07-09 15:27:53 -07:00
jenkins-bot 6a5d5fc447 Merge "Really drop afl_log_id from update.php" 2019-07-09 17:03:24 +00:00
Daimona Eaytoy c6a9f3517a Really drop afl_log_id from update.php
Follow-up of 0b925da36e, somehow I forgot
to add the removal code for MySQL and SQLite to the Hooks.

Bug: T214592
Change-Id: If0d1d5430573273784ff6f6e338b0c2199f6d7bb
2019-07-09 16:51:28 +00:00
jenkins-bot c3dcd95733 Merge "Start making APFData members private" 2019-07-09 09:23:17 +00:00
jenkins-bot 9650f11d16 Merge "Fix error display on ViewEdit" 2019-07-09 09:02:12 +00:00
jenkins-bot 35ab35978b Merge "Add a new data type for non-initialized stuff" 2019-07-09 08:58:48 +00:00
Daimona Eaytoy 3aaeb20063 Start making APFData members private
$data and $type are meant to be read-only and should have getter
functions, but as usual they're just public. Add getter methods, a
comment with a @private annotation and remove usages in our codebase.

Change-Id: I5e51efc9f982a4e340b48d20cb1b38a75bb10021
2019-07-09 10:57:00 +02:00
jenkins-bot 6f0905541a Merge "Make AbuseFilterVariableHolder::mVars private" 2019-07-09 08:42:16 +00:00