Commit graph

206 commits

Author SHA1 Message Date
Ed Sanders b1427163af Parser.php: Add tests for getTranscludedFrom
Requires an implementation of unwrapParsoidSections

Change-Id: I96c929b1117ba652dbd5af6a1ee37a5f9e87ed1e
2020-05-18 19:53:01 +00:00
Ed Sanders d1e58841af Rename removeListItem to removeAddedListItem and remove in PHP
This method shouldn't be required on the server. Leave comments
relating to it in addListItem so JS & PHP can be kept in sync.

Change-Id: I849fac660faf6e750272c20776f96b9250f96b1b
2020-05-18 19:25:08 +00:00
Reedy 234988155e Add leading \ to covers
Change-Id: I1ed4cd28bf630c6aae238e548410d1293a8b71f1
2020-05-15 22:08:25 +01:00
Ed Sanders e6e0b1ead9 PHP: Add missing typehints
Change-Id: I5639f8cbdae9aaa9cfa06136e19cc94f9fad10ea
2020-05-15 22:04:47 +02:00
Ed Sanders b78fb3f4c1 Move all PHP to the MediaWiki\Extension\DiscussionTools namespace
Change-Id: I654ebb3e646a6d8d62f7bd14d48805e39f836d7e
2020-05-15 21:57:13 +02:00
Ed Sanders 340572bc05 Create a Utils class in PHP
Also move htmlTrim to utils in JS.

Change-Id: Ia5356d713c1c5d521c396cc28bcd4ecc7ee5bbbb
2020-05-15 00:25:32 +01:00
Ed Sanders a3889fd400 Port modifier.js to PHP
Change-Id: I03b9e4377cb3ce6a5ca9d06e49dca9b2516f4979
2020-05-15 00:20:41 +01:00
Bartosz Dziewoński 6f32369b6a tests: Fix comparing PHP and JS ranges
In JS, strings are internally encoded as UTF-16, and properties like
.length return values in UTF-16 code units.

In PHP, strings are internally encoded as UTF-8, and we have the
option of using methods that return bytes like strlen() or UTF-8 code
units like mb_strlen().

However, the values produced by preg_match( …, PREG_OFFSET_CAPTURE )
are in bytes, and there's nothing we can do about that. So let's use
bytes throughout, mixing the two types results in meaningless numbers.

Then in the test code, we have to calculate UTF-16 code units offsets
based on the UTF-8 byte offsets.

We also have to copy the entire workaround for mw:Entity nodes… Maybe
the parser should be fixed to return the real nodes for ranges' ends
in this case.

Change-Id: I05804489d7de0d60be6e9f84e6a49a885e9fb870
2020-05-14 22:37:34 +00:00
Bartosz Dziewoński 33d69e26c9 tests: Fix different whitespace trimming in PHP and JS
Notably, JS trims the no-break space, while PHP doesn't. There are
some other differences that don't come up in our tests. What we really
want is to trim the ASCII whitespace as defined in the HTML spec.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/trim
https://www.php.net/manual/en/function.trim.php
https://infra.spec.whatwg.org/#ascii-whitespace

Change-Id: I95b8fb38878716a2fa7ec84c9f2e8065ebe77c0d
2020-05-14 21:37:26 +00:00
Bartosz Dziewoński c0002be7cd tests: Fix computing ranges in Parsoid documents
In JS tests, we load the documents via mw.template, which apparently
causes the <html>, <head> and <body> tags to disappear, resulting
in the ranges not matching in PHP tests (and the real document).

Put in a big hack that makes them match, and update the JSON files.

Change-Id: I8194752cd5f82c3716c99e76a37226af5d4a0ec1
2020-05-14 01:11:44 +02:00
Bartosz Dziewoński 95a87911eb tests: Check ranges in PHP parser tests
Comment out only the cases that still fail, so it's easier to fix them.

Change-Id: I85d205731d572c93ababa7dd66e674321969edb7
2020-05-13 23:58:37 +02:00
Bartosz Dziewoński b8d7a75c34 Fix performance of DiscussionToolsCommentParser::childIndexOf()
Profiling reveals that >87% of the run time of our test suite is spent
in this tiny method. Apparently, DOMNodeList::item() is extremely slow
(possibly it's linear time instead of constant time?).

Profiled using XDebug and KCacheGrind:
https://phabricator.wikimedia.org/F31815264

We can calculate the child's index in its parent by counting its
precending siblings instead, which turns out to be much faster.

Before:
 1. 275444ms to run DiscussionToolsCommentParserTest:testGetComments with data set #2
 2. 12668ms to run DiscussionToolsCommentParserTest:testGetComments with data set #3
 ...
After:
 1. 9545ms to run DiscussionToolsCommentParserTest:testGetComments with data set #2
 2. 5549ms to run DiscussionToolsCommentParserTest:testGetComments with data set #3
 ...

That's still kind of slow but now it's bearable to run the test suite.

Change-Id: I49155f7aa2e231a9a20bf282cf6aaa28fc902e0b
2020-05-13 02:56:39 +02:00
Ed Sanders 745101c02b PHP tests: Move some test utils to a parent class
Change-Id: I6ae5aa85f8aaa02e1b9323820a841c06c5d62b64
2020-05-12 12:33:04 +01:00
Ed Sanders a000b8c7b1 Add comment tests to PHP
TODO: Make the assertions less slow (currently ~50s)

Change-Id: I1d774e353c070484b5bae18e2ec3e3e41da68202
2020-05-12 12:33:04 +01:00
Roan Kattouw 7b7a2cd69c The Great Parser JS to PHP port of 2020!*
* Not to be confused with the Parsing Team's
"Great Parser JS to PHP port of 2019"

Gasp as OR hacks are changed to null coalescing operators.
Applaud as variable declarations are dropped.
Cheer as parameters and return values are type-hinted.
Shudder as DomNodeLists have no indexOf method.

Moving discussion parsing to the server should allow
us to implement much cleaner APIs for commenting.

Bug: T252252
Co-authored-by: Ed Sanders <esanders@wikimedia.org>
Change-Id: Ic1438d516e223db462cb227f6668e856672f538c
2020-05-12 12:33:04 +01:00
Ed Sanders 4397bda8cb Modifier: Separate out test data
Change-Id: I63d868a47daebee180c229b424311d10b80d2e89
2020-05-11 16:58:00 +01:00
Ed Sanders 41d5aa1d37 Separate out more data from parser.tests.js
Change-Id: Ib1c3ec30796dd6bff36835e4555b6974e2a5b376
2020-05-08 12:12:11 +01:00
jenkins-bot f0f457562d Merge "Handle &nbsp; and other entities in the timestamp/timezone" 2020-05-07 19:44:12 +00:00
Ed Sanders 56d8721fd7 Switching between VE & wikitext (plain)
Bug: T234403
Change-Id: I84a639ab8e31ec9f683a63b0ecf184f426df09dd
2020-05-06 18:03:36 +01:00
Bartosz Dziewoński 0269626adc Handle &nbsp; and other entities in the timestamp/timezone
Bug: T251838
Change-Id: Iba8d7c71e332c63229eec4bc7c80b10627135784
2020-05-05 22:38:16 +02:00
Bartosz Dziewoński 047e8eb82c Fix inserting replies inside a <pre>
In the future, we should think about a better solution that also
handles other elements (T250126), but this is an easy fix for now.

Bug: T250512
Change-Id: I1321f0da523ddb4a999b8c453b9094a267b38ae2
2020-04-27 23:23:31 +02:00
Ed Sanders d3c46a2967 eslint: Disable max-len
Change-Id: I7e49dc170dba62b88f1879ae670f806f6b54d4e9
2020-04-21 15:50:19 +01:00
Ed Sanders d40e427b5b Remove extra linebreaks from wikitext
Extra lines currently render empty list items which are invisible.
Remove these as the user probably didn't intend from them to appear
in the wikiext.

Bug: T249867
Change-Id: I75c1192a94b918783d362d38cd91a508bb169d56
2020-04-14 20:29:38 +00:00
Ed Sanders f04abd02d4 Improve autosign handling of 3/5 tilde sigs and add tests
3 or 5 tilde signatures will be assumed to be erroneous and fixed
to 4 tilde signatures. This will be visible in the preview so shouldn't
come as a suprise to users.

Bug: T245628
Change-Id: I741f0761a6fb10c99cf3239ac5c6c7e1a2b872c7
2020-04-14 20:29:31 +00:00
Bartosz Dziewoński 55b43a70d4 Fix skipping to the end of the paragraph when there is no wrapper
Bug: T249217
Change-Id: I77847901d98f56dbcbcf379c6c17f41e06a3a939
2020-04-03 18:11:59 +00:00
Bartosz Dziewoński a14293c8e7 Remove Parsoid section wrappers before finding transclusions
The section wrappers can be marked as template-affected when the
previous or next section is transcluded, causing comments to be
unnecessarily uneditable. The new test case demonstrates this.

Depends-On: I03bc455d5484a6c51f3fa2397c64936b829fe7e3
Change-Id: I895a04990d79a3475d778b4fef054ea0bb076f0b
2020-03-24 23:31:21 +01:00
Bartosz Dziewoński 216516f719 Rename tests/qunit/utils.js to .../testUtils.js
We now also have modules/utils.js and I found the identical name confusing.

Change-Id: I6895748528241e0368a79eab9dcf3123afaf5f87
2020-03-24 22:29:48 +00:00
Bartosz Dziewoński dab37fd7b4 parser: Make #getTranscludedFrom return page title in text form
It's more convenient for display or comparing it with other things.

Depends-On: I03bc455d5484a6c51f3fa2397c64936b829fe7e3
Change-Id: I88d7aa68977210b16860075ed52983a5e99ee0f7
2020-03-24 22:29:35 +00:00
Bartosz Dziewoński e9b583d1c3 parser: Improve merging multiple comments on one line
Now also works if the "follow-up" comment is wrapped in e.g. `<small>`.

Change-Id: Ic37cb6afdb42021f109a1818f5c4299d907ed094
2020-03-14 13:34:42 +00:00
Bartosz Dziewoński 04365c0188 Merge RL modules which are only loaded by 'ext.discussionTools.init'
Bug: T240474
Change-Id: I1b83aa18666be8f1ea6a3602b299f92574d42cb7
2020-03-14 14:33:23 +01:00
jenkins-bot 2e0c299a1f Merge "Fix signatureRanges overlapping for some comments" 2020-03-09 20:31:54 +00:00
Bartosz Dziewoński e3e4ef9de4 parser: Detect comments transcluded from another page
When trying to reply to a comment that is inside a transclusion,
detect if it's transcluded from a subpage or simply wrapped in a
template, and show appropriate error messages.

References:
* VisualEditor ve.dm.Converter#getAboutGroup()
* VisualEditor ve.dm.ModelRegistry#matchElement()
* Parsoid Linter#findEnclosingTemplateName()

Bug: T245694
Change-Id: If3dd1ebbf1d02ee4379c200019bfc3a8ec02325b
2020-03-09 20:28:56 +01:00
Bartosz Dziewoński b4029c3c58 Fix signatureRanges overlapping for some comments
If two signatures for a single comment were near each other,
we would sometimes treat them as one huge signature.

Change-Id: Ied4b3aa535a9ca6bebef8a004ae48b7d5a8f2f9b
2020-03-09 13:28:22 +00:00
Bartosz Dziewoński 0ca851aa92 parser: Return signature and timestamp ranges
Currently not used for anything. May be used later for editing
comments (T245225) or reformatting timestamps (T240360).

Note that a comment may have multiple signatures+timestamps,
and we return them all so that you have to deal with that.

Fix some unrelated incorrect documentation comments.

Bug: T245220
Change-Id: I51b8bf4a3bb7968f35e32c7e44c95c2ab079d9ac
2020-03-05 14:28:17 +01:00
Bartosz Dziewoński 606d6b34ec Add reply links at the end of a line, even if the signature is in the middle
Previously they were added at the end of the text node containing the
timestamp, which was usually the end of the line, but not always.

And also fix the same problem for inserting the actual replies (or
reply widgets). This replaces an undocumented hack that prevented our
own reply links from triggering this bug (without it, the reply widget
would be inserted before the reply link rather than after).

While we're here, remove unintentional spacing that appeared before
some reply links, caused by trailing whitespace in text nodes.

Add tests for all of the above.

Bug: T245695
Change-Id: I354b63e2446bb996176a2e3d76abf944127f307e
2020-03-02 21:39:37 +01:00
Bartosz Dziewoński ea26009896 Work around mw.Uri crash on fallback encoding in links
Bug: T245889
Change-Id: I182f9ffa84a3b3cf4afafd536360572eda9d2714
2020-02-29 19:08:01 +01:00
jenkins-bot 0e22b4c79f Merge "Clean up the interface after discarding a reply" 2020-02-27 23:05:00 +00:00
Bartosz Dziewoński d068d2ef2c Clean up the interface after discarding a reply
Bug: T245574
Change-Id: I016a7a5c44e0d15a153143177976cceb8d6d3d1b
2020-02-27 18:09:34 +01:00
Bartosz Dziewoński e9c401e3aa Ignore LRM and RLM before timezone indicator
They are not generated by MediaWiki, but they often appear when users
sign others' unsigned comments by copy-pasting the timestamp from the
history page.

Add test config data for nlwiki, exported by running this in the
browser console:

  copy(
    JSON.stringify( { wgArticlePath, wgNamespaceIds, wgFormattedNamespaces }, null, 2 ) + '\n' +
    JSON.stringify( mw.loader.moduleRegistry['ext.discussionTools.parser'].packageExports['data.json'], null, 2 )
  );

Bug: T245784
Change-Id: Icbcdc5a028e9ce2cb09173f87769e525ec3082fc
2020-02-25 00:20:00 +00:00
Bartosz Dziewoński 2a93ed11b4 Rearrange test cases
I think directories like this make more sense for adding more test cases.

Depends-On: I9153851fe162c012967fda00d3e1f81964a8dce9
Change-Id: Ibc72b747a75c72643c1fc04eae49bd15656e8104
2020-02-25 00:19:30 +00:00
Bartosz Dziewoński ff0386239f Only detect comments with real signatures
Consequences of this are visible in the test cases:

* (en) Tech News posts are not detected.
  Examples: "21:22, 1 July 2019 (UTC)", "21:42, 29 July 2019 (UTC)"

* (en) Comments by users who customize the timestamp are not detected.
  Examples: "10:49, 28 June 2019 (UTC)", "21:34, 14 July 2019 (UTC)"

* (en) Comments with signatures missing a username are not detected.
  This sometimes happens if a comment is accidentally signed with
  '~~~~~' (five tildes), which only inserts the timestamp.
  Examples: "17:17, 27 July 2019 (UTC)", "10:25, 29 July 2019 (UTC)"

* (pl) A lone timestamp at the beginning of a thread is not detected.
  It's not part of a post, it was added to aid automatic archiving.
  Example: "21:03, 18 paź 2018 (CET)"

Bug: T245692
Change-Id: I0767bb239a1800f2e538917b5995fc4f0fa4d043
2020-02-21 01:30:54 +01:00
Bartosz Dziewoński 7761f62b42 Fix edit summary for comments in 0th section (no heading)
Bug: T245765
Change-Id: I9eb4726ef096b8d7459cc1409814514ec1dc89ae
2020-02-21 00:44:42 +01:00
Bartosz Dziewoński e29b8173bf Handle comments before first section heading
The loop in parser.js assumed that there was always a heading before
any comments (not counting the page title, only section headings).

Bug: T243869
Change-Id: I3a0bb06716e75d4a17e25c40748673a071ee5f30
2020-01-30 00:14:46 -08:00
Bartosz Dziewoński 890588f36a Pick reply insertion point based on parser tree, not DOM tree
I don't like that I had to special-case `<p>` tags (top-level
comments) in this code. I feel like it should be possible to handle
top-level comments and replies in a generic way, but I couldn't find
a way to do it that actually worked.

Notes about changes to the behavior, based on the test cases:

* Given a top-level comment A, if there was a "list gap" in the
  replies to it: previously new replies would be incorrectly added at
  the location of the gap; now they are added after the last reply.
  (T242822)

  Example: "pl", comment at "08:23, 29 wrz 2018 (CEST)"

* Given a top-level comment A and a reply to it B that skips an
  indentation level: previously new replies to A would be added with
  the same indentation level as B; now they are added with the
  indentation level of A plus one. (The old behavior wasn't a bug, and
  this is an accidental effect of other changes, but it seems okay.)

  Example: "pl", comment at "03:22, 30 wrz 2018 (CEST)"
    and reply at "09:43, 30 wrz 2018 (CEST)"

* Given a top-level comment A, a reply to it B, and a following
  top-level comment C that starts at the same indentation level as B:
  previously new replies to A would be incorrectly added in the middle
  of the comment C, due to the DOM list structure; now they are added
  before C. (T241391)

  (It seems that comment C was supposed to be a multi-line reply that
  was wrongly indented. Unfortunately we have no way to distinguish
  this case from a top-level multi-line comment that just happens to
  start with a bullet list.)

  Example: "pl", comments at "03:36, 24 paź 2018 (CEST)",
    "08:35, 24 paź 2018 (CEST)", "17:14, 24 paź 2018 (CEST)"

* In the "en" example, there are some other changes where funnily
  nested tags result in slightly different results with the new code.
  They don't look important.

* In rare cases, we must split an existing list to add a reply in the
  right place. (Basically add `</ul>` before the reply and `<ul>`
  after, but it's a bit awkward in DOM terms.)

  Example: split-list.html, comment "aaa"; also split-list2.html
    (which is the result of saving the previous reply), comment "aaa"

* The modifier can no longer generate DOM that is invalid HTML, fixing
  a FIXME in modifier.test.js (or at least, it doesn't happen in these
  test cases any more).

Bug: T241391
Bug: T242822
Change-Id: I2a70db01e9a8916c5636bc59ea8490166966d5ec
2020-01-23 21:13:12 +01:00
Bartosz Dziewoński 30fcfec1fd parser: Merge multiple comments on one line
Even when you have multiple signatures by multiple users in one
paragraph (or list item), it's still basically a single comment.
We don't want to offer multiple buttons to reply to it.

The changed parser test cases are illustrative:
* All affected comments in the "pl" example are comments with a
  "post-scriptum", which is now more intuitively treated as part of
  the main comment.
* The first comment in the "en" example would probably have been
  better if it wasn't merged, but a weird use of the outdent template
  causes us to not be able to distinguish that the two parts of the
  comment display on separate lines.
* The last comment in the "en" example (isn't that neat?) was previously
  incorrectly treated as two comments, because there's a timestamp in
  the middle of it (the user is referring to another comment).
* Remaining affected comments in the "en" example are also comments
  with a "post-scriptum" and their treatment is clearly better now.

It also accidentally fixes some problems with modifier tests (but not
all), where previously <dl> nodes would be inserted in the middle of
<p> nodes, to reply to the comments which are now merged.

Bug: T240640
Change-Id: I0f2d9238aff75d78286250affd323cd145661a11
2020-01-22 02:21:43 +01:00
Bartosz Dziewoński da732843f3 Integration tests for the modifier
Document the current behavior of the modifier (which inserts the
replies into the DOM tree), so that we can more easily see the effect
of changes in I2a70db01e9a8916c5636bc59ea8490166966d5ec.

Basically, add a reply to every comment, and dump the resulting HTML,
comparing it to previously generated expected HTML (which can be
checked visually). Have a look at the new HTML files.

Notably, the very first section in the "pl" example demonstrates a
case of wrong reply location due to list gap :) (T242822).

Change-Id: I4aed0f0b112f53d98e3fe1da4d40db8687c7e537
2020-01-22 00:58:06 +01:00
Bartosz Dziewoński da668b72d5 Identify comments by username+timestamp+seq
Possible use cases:
* Matching comments between PHP and Parsoid HTML [implemented here]
* Finding the same comment in a different revision of a page
  (e.g. while resolving an edit conflict, or to allow resuming
  composition of autosaved comments) [implemented for highlighting
  user's own posted comment only]
* Permanent links to comments [future]

The reasoning for this form of ID is:
* _Timestamp_ by itself is a nearly unique identifier, so it's a good
  thing to start with
* Users may post multiple comments in one edit (or in many edits in
  one minute), so we need the _sequential number_ to distinguish them
* _Username_ is probably not required, but it may reduce the need
  for sequential numbers, and will help with human-readability if we
  add permanent links

The ID remains stable when a new comment is added anywhere by anyone
(excepts comments within the same minute by the same user), or when a
section is renamed.

It's not always stable when a comment is moved or when an entire
section is moved or deleted (archived), but you can't have everything.

Change-Id: Idaae6427d659d12b82e37f1791bd03833632c7c0
2019-12-09 13:45:31 +00:00
Bartosz Dziewoński 8b86d1abbc Document how to update the test JSON files
(And rebuild them using this method, the properties are in different
order, it's actually nice for readability to have 'replies' last)

Change-Id: Ib586e1081fa36cb9125db1b0b1d41f092350641c
2019-12-03 21:07:16 +01:00
Bartosz Dziewoński 4021ca1642 Add unit tests for parser#getTimestampParser
Change-Id: I03cba04489194539d6ff3a32acdb9a8fe3d499e5
2019-10-30 00:13:56 +01:00
Bartosz Dziewoński fc34556b04 Fix parsing links to subpages in user signatures
Change-Id: I381087c252eeb7530e63c4d99cecc1b2ee041b0a
2019-10-30 00:13:56 +01:00
Bartosz Dziewoński c83201b10c Fix parsing non-standard case in links to user contribs
Change-Id: I2da72e2731019ad5be0ba33aa229ad914a7aaf10
2019-10-30 00:13:56 +01:00
Bartosz Dziewoński e8012b7094 Fix incorrectly detecting a section heading inside the table of contents
Change-Id: I7209b523c3322b3b379e6aa82a4b2014cc39c404
2019-10-30 00:13:56 +01:00
Bartosz Dziewoński 112ea056cd Integration tests for parsing
Tests that handle a specific case and describe what they are testing
would be nice… but tests that just document the current status to
avoid regressions are also okay and easier to add.

Change-Id: I0b3530ae0e77de70932aaf623f5290d1876699a0
2019-10-30 00:13:56 +01:00
Ed Sanders a9340f0c84 build: Update linters
Change-Id: Iee2e91b24218c6120f4a20d6ea38ea03c6292ef0
2019-10-28 12:29:56 +00:00
Bartosz Dziewoński 9efe8b1dd4 Add unit tests for parser#getTimestampRegexp
Depends-On: I6c3d186de1877f73d4a4e3fec7d6d632a5d5fa83
Change-Id: Icdb44f793a8f5e56666ec635bb8b0125041b5aab
2019-10-24 23:21:29 +02:00
Ed Sanders fe0a05ad49 Add required files for an extension
Change-Id: I626e607e6ae7efc2521f9dd1da83c6fd90b0767d
2019-10-10 19:42:56 +01:00