Commit graph

36 commits

Author SHA1 Message Date
thiemowmde 10dcd1f847 Replace preg_replace_callback with strtr in CommentParser
It does the same as before.

I think performance is not a concern here, and wasn't my motivation
either. But I hope this makes the code easier to read and to reason
with.

I added a pure unit test case (without involving an actual Language
object) to cover the previously uncovered digits feature.

Change-Id: I6a0fc86035817eabb42b55e58183ae094c052aa6
2023-10-31 08:55:40 +01:00
Bartosz Dziewoński 1d6ba985be Fix parsing of non-English titles in JS tests
Same issue as b68832ace0 in PHP.

Change-Id: Ide6ebc47929b597e6dc23df696d87bb3130ce276
2022-10-17 17:10:53 +02:00
Bartosz Dziewoński 4b38d72c2e Re-enable fixed JS parser tests
The changes from I39f9b994ce5636d70fea2e935a7c87c7d56dcb26 also make
most of the broken tests pass again.

Change-Id: I165f36235b47dee4bcef115e518bc1e81d2f83a5
2022-10-15 20:24:09 +00:00
Bartosz Dziewoński 361283a332 Ship HTML test files for JS using 'packageFiles' instead of 'templates'
We originally used 'templates' because it seemed like an obvious
choice for HTML files, and because 'packageFiles' requires extra code
to include anything that isn't a .js or .json file.

However, the templates are expected to be HTML fragments rather than
whole documents, and they are parsed in a particular way that takes a
lot of code to clean up (which we needed to do, because we use the
same test files for testing PHP code).

I tried doing it in the 'packageFiles' way, and the extra code doesn't
seem that bad in comparison after all. Moreover, the 'templates'
mechanism (when used the intended way) feels vaguely deprecated in
favor of Vue.js, and I'd rather move away from it.

This makes the tests faster too (probably mostly thanks to the removal
of the clean up code) – on my machine they go from 1800ms to 1500ms.

(Simplify linearWalk tests, as we no longer need to do weird things
with document fragments to get consistent outputs in PHP and JS.)

Change-Id: I39f9b994ce5636d70fea2e935a7c87c7d56dcb26
2022-10-12 22:45:41 +00:00
Ed Sanders 6d03608b7c parser.test.js: New test for each case, as in modifier.test.js
Explicitly skip the old parser tests as these are not yet working.

Change-Id: I1a8fbb9b177acd97db9c42250dd6f5226f879ac0
2022-10-11 17:23:40 +01:00
Bartosz Dziewoński 77614a2d02 tests: Fix root node / container handling
Since times immemorial, and for reasons lost to history, our test code
was adding an extra <div> wrapper before parsing the HTML used for
tests. This wasn't a problem, until now, because I want to add some
tests for T303396 that need to check that the *real* wrappers present
in some test cases are handled correctly.

Changes to test cases mostly remove a leading "0/" from serialized
ranges, corresponding to removing the extra wrapper.

Change-Id: Ia50e3590538c8cd274b02d2a937ba1a3fbb4ac89
2022-03-10 18:43:58 +01:00
Bartosz Dziewoński 4c29304484 CommentParser: Avoid using a dynamic undeclared property
Change-Id: Iefa8dea83bc0d31b9c6b3509189eeaa652dd9ea0
2022-03-08 23:30:11 +00:00
Bartosz Dziewoński 8e44b43df0 Split off ThreadItemSet from CommentParser
Goal:
-----
Finishing the work from Iadb7757debe000025e52770ca51ebcf24ca8ee66
by changing CommentParser::parse() to return a data object, instead of
the whole parser.

Changes:
--------
ThreadItemSet.php:
ThreadItemSet.js:
* New data class to access the results of parsing a discussion. Most
  methods and properties are moved from CommentParser with no changes.

CommentParser.php:
Parser.js:
* parse() returns a new ThreadItemSet.
* Remove methods moved to ThreadItemSet.
* Placeholder headings are generated slightly differently, as we process
  things in a different order.
* Grouping threads and computing IDs/names is no longer lazy. We always
  needed IDs/names anyway.
* computeId() explicitly uses a ThreadItemSet to check the existing IDs
  when de-duplicating.

controller.js:
* Move the code for turning some nodes annotated by CommentFormatter
  into a ThreadItemSet (previously a Parser) from controller#init to
  ThreadItemSet.static.newFromAnnotatedNodes, and rewrite it to handle
  assigning parents/replies and recalculating legacy IDs more nicely.
* mw.dt.pageThreads is now a ThreadItemSet.

Change-Id: I49bfe019aa460651447fd383f73eafa9d7180a92
2022-02-21 16:22:32 +00:00
Bartosz Dziewoński 4613ae78e7 Change CommentParser into a service
Goal:
-----
To have a method like CommentParser::parse(), which just takes a node
to parse and a title and returns plain data, so that we don't need to
keep track of the config to construct a CommentParser object (the
required config like content language is provided by services) and
we don't need to keep that object around after parsing.

Changes:
--------
CommentParser.php:
* …is now a service. Constructor only takes services as arguments.
  The node and title are passed to a new parse() method.
* parse() should return plain data, but I split this part to a separate
  patch for ease of review: I49bfe019aa460651447fd383f73eafa9d7180a92.
* CommentParser still cheats and accesses global state in a few places,
  e.g. calling Title::makeTitleSafe or CommentUtils::getTitleFromUrl,
  so we can't turn its tests into true unit tests. This work is left
  for future commits.

LanguageData.php:
* …is now a service, instead of a static class.

Parser.js:
* …is not a real service, but it's changed to behave in a similar way.
  Constructor takes only the required config as argument,
  and node and title are instead passed to a new parse() method.

CommentParserTest.php:
parser.test.js:
* Can be simplified, now that we don't need a useless node and title
  to test internal methods that don't use them.

testUtils.js:
* Can be simplified, now that we don't need to override internal
  ResourceLoader stuff just to change the parser config.

Change-Id: Iadb7757debe000025e52770ca51ebcf24ca8ee66
2022-02-19 19:51:57 +01:00
Ed Sanders 34011b7a07 Parser: Pass in title of page being parsed
Will be used to parse selflinks in the future.

Change-Id: I2bc29d1c5c69cb6309f582f162f9af7d96ce8913
2022-01-12 21:17:59 +00:00
Ed Sanders 3f5756f495 List methods not covered by unit tests in TODO sections
Change-Id: Ia06c3c726b7bc9758454aa8b2f7ea34cbada4c09
2021-11-13 15:27:38 +00:00
Ed Sanders 7c3e583bec build: Update eslint-config-wikimedia to 0.21.0
Change-Id: I72de463d5a878e555eeed0e7ce2772e1d3a46f06
2021-11-08 19:03:40 +00:00
Ed Sanders 1893405635 Code style: Move var declarations inline
Change-Id: I1686603388b050ba4ec22eff23e4806cdf262b87
2021-04-22 17:43:46 +00:00
Bartosz Dziewoński 329df8c953 Parsing discussions converted to language variants
* Export parser data (date format, digits, timezone names, and
  messages for weekday/month names) converted to language variants
* Update the parsers to try matching using every variant, in case
  the page is displayed in non-default variant (and to avoid
  problems with incomplete variant conversion)

Bug: T259818
Change-Id: I04d73992cd31ce06fa79f87df0c0a53d7efc3c58
2020-09-16 22:07:07 +00:00
Ed Sanders a2431fe006 Refactor CommentParser
* Pass rootNode to the constructor
* Rename getters to match CommentItem/HeadingItem/ThreadItem
  value classes.
* Always build the thread tree so CommentItem's always have
  and ID and replies/parent.

Change-Id: I508be9534de59016ff806e3d84edcbb1c76cb0c6
2020-07-20 23:38:10 +01:00
Ed Sanders a4636d39fc Move #getTranscludedFrom from parser to ThreadItem
Also requires moving getTitleFromUrl to CommentUtils

Change-Id: I9cb83a3fdd456eba66899433b866ce7a7f00eeb5
2020-07-20 15:56:48 +01:00
Ed Sanders 7ae5bbf384 Move #getAuthors from parser to ThreadItem
Change-Id: I16e513000e5366b3044b17a99da07d8d0f47a61f
2020-07-20 15:13:59 +01:00
Bartosz Dziewoński c64bb6b5b7 Add the test for getAuthors() in JS too
Change-Id: Id7dabc535b6bb688602c0d55fc3696f662cb10c7
2020-05-19 21:13:52 +02:00
Ed Sanders c5d1029b25 Move /cases and /data up to /tests
Theses are no longer QUnit specific.

Change-Id: I5f3cca1ff686922e0cdaaedb80858f37df04799a
2020-05-18 21:47:17 +01:00
Ed Sanders b1427163af Parser.php: Add tests for getTranscludedFrom
Requires an implementation of unwrapParsoidSections

Change-Id: I96c929b1117ba652dbd5af6a1ee37a5f9e87ed1e
2020-05-18 19:53:01 +00:00
Bartosz Dziewoński c0002be7cd tests: Fix computing ranges in Parsoid documents
In JS tests, we load the documents via mw.template, which apparently
causes the <html>, <head> and <body> tags to disappear, resulting
in the ranges not matching in PHP tests (and the real document).

Put in a big hack that makes them match, and update the JSON files.

Change-Id: I8194752cd5f82c3716c99e76a37226af5d4a0ec1
2020-05-14 01:11:44 +02:00
Ed Sanders 41d5aa1d37 Separate out more data from parser.tests.js
Change-Id: Ib1c3ec30796dd6bff36835e4555b6974e2a5b376
2020-05-08 12:12:11 +01:00
Bartosz Dziewoński 0269626adc Handle &nbsp; and other entities in the timestamp/timezone
Bug: T251838
Change-Id: Iba8d7c71e332c63229eec4bc7c80b10627135784
2020-05-05 22:38:16 +02:00
Bartosz Dziewoński 55b43a70d4 Fix skipping to the end of the paragraph when there is no wrapper
Bug: T249217
Change-Id: I77847901d98f56dbcbcf379c6c17f41e06a3a939
2020-04-03 18:11:59 +00:00
Bartosz Dziewoński a14293c8e7 Remove Parsoid section wrappers before finding transclusions
The section wrappers can be marked as template-affected when the
previous or next section is transcluded, causing comments to be
unnecessarily uneditable. The new test case demonstrates this.

Depends-On: I03bc455d5484a6c51f3fa2397c64936b829fe7e3
Change-Id: I895a04990d79a3475d778b4fef054ea0bb076f0b
2020-03-24 23:31:21 +01:00
Bartosz Dziewoński 216516f719 Rename tests/qunit/utils.js to .../testUtils.js
We now also have modules/utils.js and I found the identical name confusing.

Change-Id: I6895748528241e0368a79eab9dcf3123afaf5f87
2020-03-24 22:29:48 +00:00
Bartosz Dziewoński 04365c0188 Merge RL modules which are only loaded by 'ext.discussionTools.init'
Bug: T240474
Change-Id: I1b83aa18666be8f1ea6a3602b299f92574d42cb7
2020-03-14 14:33:23 +01:00
Bartosz Dziewoński e3e4ef9de4 parser: Detect comments transcluded from another page
When trying to reply to a comment that is inside a transclusion,
detect if it's transcluded from a subpage or simply wrapped in a
template, and show appropriate error messages.

References:
* VisualEditor ve.dm.Converter#getAboutGroup()
* VisualEditor ve.dm.ModelRegistry#matchElement()
* Parsoid Linter#findEnclosingTemplateName()

Bug: T245694
Change-Id: If3dd1ebbf1d02ee4379c200019bfc3a8ec02325b
2020-03-09 20:28:56 +01:00
Bartosz Dziewoński ea26009896 Work around mw.Uri crash on fallback encoding in links
Bug: T245889
Change-Id: I182f9ffa84a3b3cf4afafd536360572eda9d2714
2020-02-29 19:08:01 +01:00
Bartosz Dziewoński e9c401e3aa Ignore LRM and RLM before timezone indicator
They are not generated by MediaWiki, but they often appear when users
sign others' unsigned comments by copy-pasting the timestamp from the
history page.

Add test config data for nlwiki, exported by running this in the
browser console:

  copy(
    JSON.stringify( { wgArticlePath, wgNamespaceIds, wgFormattedNamespaces }, null, 2 ) + '\n' +
    JSON.stringify( mw.loader.moduleRegistry['ext.discussionTools.parser'].packageExports['data.json'], null, 2 )
  );

Bug: T245784
Change-Id: Icbcdc5a028e9ce2cb09173f87769e525ec3082fc
2020-02-25 00:20:00 +00:00
Bartosz Dziewoński 2a93ed11b4 Rearrange test cases
I think directories like this make more sense for adding more test cases.

Depends-On: I9153851fe162c012967fda00d3e1f81964a8dce9
Change-Id: Ibc72b747a75c72643c1fc04eae49bd15656e8104
2020-02-25 00:19:30 +00:00
Bartosz Dziewoński e29b8173bf Handle comments before first section heading
The loop in parser.js assumed that there was always a heading before
any comments (not counting the page title, only section headings).

Bug: T243869
Change-Id: I3a0bb06716e75d4a17e25c40748673a071ee5f30
2020-01-30 00:14:46 -08:00
Bartosz Dziewoński 8b86d1abbc Document how to update the test JSON files
(And rebuild them using this method, the properties are in different
order, it's actually nice for readability to have 'replies' last)

Change-Id: Ib586e1081fa36cb9125db1b0b1d41f092350641c
2019-12-03 21:07:16 +01:00
Bartosz Dziewoński 4021ca1642 Add unit tests for parser#getTimestampParser
Change-Id: I03cba04489194539d6ff3a32acdb9a8fe3d499e5
2019-10-30 00:13:56 +01:00
Bartosz Dziewoński 112ea056cd Integration tests for parsing
Tests that handle a specific case and describe what they are testing
would be nice… but tests that just document the current status to
avoid regressions are also okay and easier to add.

Change-Id: I0b3530ae0e77de70932aaf623f5290d1876699a0
2019-10-30 00:13:56 +01:00
Bartosz Dziewoński 9efe8b1dd4 Add unit tests for parser#getTimestampRegexp
Depends-On: I6c3d186de1877f73d4a4e3fec7d6d632a5d5fa83
Change-Id: Icdb44f793a8f5e56666ec635bb8b0125041b5aab
2019-10-24 23:21:29 +02:00