Use the same logic for marking ranges in the document, and ensure
that the heading range does not include section edit links or
section numberings.
Change-Id: I782caafc34fee2a822b0a17b24dd6b9528202eca
We avoided fixing these because it causes changes in just about all of
the test data, which is annoying when reviewing or blaming changes.
But the previous several commits also caused changes in just about all
of the test data, so we might as well do this too.
Change-Id: I83b64d83b6f12c04dc06c0cadff7cdd89417e137
To avoid old threads re-appearing on popular pages when someone
uses a vague title (e.g. dozens of threads titled "question" on
[[Wikipedia:Help desk]]: https://w.wiki/fbN), include the oldest
timestamp in the thread (i.e. date the thread was started) in the
heading ID.
Bug: T264478
Change-Id: If918bfd5e025248923d1939bc86916697ead95a0
Sequential numbers aren't great because they change when an earlier
comment is archived. Parent comment/heading IDs should change less
often.
This also makes much more sense for disambiguating subsections,
e.g. a dozen identical ===Votes=== sections for a dozen proposals.
Bug: T264478
Change-Id: I466454984fd919ebef35f2b37ddb5d86dc842996
Our threads now also contain all replies to their sub-threads.
This is similar to how sections work in MediaWiki, where the parent
section also contains the content of all the lower-level sections.
We're going to need this for notifications about replies in a thread.
Bug: T264478
Change-Id: I241fc58e2088a7555942824b0f184ed21e3a8b6f
Previously, only comments could have IDs, because we only needed IDs
for replying. But we might also use them for notifications soon.
Bug: T264478
Change-Id: I1bcad02bf17ab54bc5028a959543c10f0430836b
I haven't really reviewed the outputs, but at least a) they don't
crash b) they will fail if the output suddenly changes (which could
cause problems due to caching).
Bug: T252555
Change-Id: I1bbcbc5dd17ce1e24b3622062f5e8df4baf5f389
Also, add tests covering this and the previous bug fixes in this code
(T259818, T261706).
Note that the test data added in tests/cases/ doesn't exactly match
the entire configuration of the wiki, only the parts we want to cover.
This is unlike the data in tests/data/, which was literally copied
from the relevant wikis, and which is used as input for other tests.
Bug: T265500
Change-Id: I29a59a5952f6dc9fb5910434bb6bcc9dcdaa01a9
The wikitext parser outputs `<p><br></p>` for empty paragraphs, so we
need to ignore `<br>` tags when searching for an "interesting" node
that marks the beginning of a comment. Otherwise the empty paragraphs
mess up the detection of indentation levels.
Bug: T264116
Change-Id: I84a97ab577baa7336b78935ccdc48041ecfc231a
The tests pass either way because this trivial difference is ignored,
but it's annoying when trying to update changed tests.
Change-Id: I7ad1acad43e30a63843cfda4d0d08ff663ef3252
* Export parser data (date format, digits, timezone names, and
messages for weekday/month names) converted to language variants
* Update the parsers to try matching using every variant, in case
the page is displayed in non-default variant (and to avoid
problems with incomplete variant conversion)
Bug: T259818
Change-Id: I04d73992cd31ce06fa79f87df0c0a53d7efc3c58
This is primarily to cover the handling of localised digits,
which previously wasn't being tested, leading to T261706.
Bug: T261706
Change-Id: I9de7f01f77e767e9048c85604b559af4bca0de91
* Remove 'wgMetaNamespace' and 'wgMetaNamespaceTalk', the same data
exists in 'wgFormattedNamespaces'.
* Rename 'wgContentLang' to 'wgContentLanguage', to match its real
name in JS config. MediaWiki doesn't use 'wgContentLang' anywhere,
although the related PHP global is called $wgContLang.
* Document how I made these files, previously only mentioned in the
commit message of e9c401e3aa.
Change-Id: I67f962812c155aedf41154e0d837e7feb5af972d
The PHP code incorrectly assumed that the digits are single-byte in
UTF-8, which is never the case (except for 0-9).
The JS code worked correctly because it uses UTF-16 strings, so the
bug would only affect non-BMP digits there. This was noted in a TODO
comment, but we overlooked it when reimplementing in PHP.
Instead of a string of 10 characters, use an array of 10
single-character strings.
Bug: T261706
Change-Id: Ic5421382474c88f003424799c53ff473d99cce92
When a comment ended before the end of a paragraph, the next
comment would begin right there in the middle of the paragraph.
This could result in the detected indentation level of that
comment being incorrect, and replies being inserted in wrong
places, as seen in the 'signatures-funny' test case.
The code moved to the parser was previously repeated twice in
addListItem() and addReplyLink(), which should have been a hint
that something isn't quite right.
Also, fix the code guarding against overlapping signatures,
now that signatures may not be at the end of a comment.
Bug: T260855
Change-Id: Ic26a87642f8a15d5de2f7073d4d8176b299c7f94
Expand the 'signatures-funny' test case with more examples, which
don't behave correctly.
Follow-up commits I04a8ea09401e06f2d4bb1f226f17eb528b29ed95 and
Ic26a87642f8a15d5de2f7073d4d8176b299c7f94 fix them.
Bug: T255738
Change-Id: I0fdd8bdf11b497ffeed37c37953c5730f6e4f3b7
Previously, parser would output offsets that don't exist in their
containers, because we were pretending that entities are parts of
their neighboring text nodes.
Turns out it's much easier to do it right when going backwards.
Change-Id: I9bccca2d403f1a976ae517449989170cdd99721e
Follow-up to ccd9e411d2.
* Fix variable name in CommentTestCase::overwriteHtmlFile()
* Overwrite before assertions, because they abort execution if they fail
Change-Id: I5bba016ba93f9dd1994325ae82c3105ba11cf033
The latter results in lots of extra HTML entity encoding.
The former is built by the Parsing team and appears to result
in no unexpected changes elsewhere in the document.
As Parsoid's selser relies on HTML fragments being byte-for-byte
equal, these changes were resulting in wikitext normalisations
in untouched parts of the document ("dirty diffs").
Bug: T259855
Change-Id: Ib3cb605911e690ec3e8c2f9df25fd1a2e2849d7e
When adding a reply, we take a node at the end of the previous comment,
compare that comment's indentation level to the expected indentation level
of the reply, and add (or remove) that number of wrapper lists.
The existing code did not consider that comments may have lists within
them, and so the indentation of that node may not match the indentation
of the comment.
Bug: T252702
Change-Id: Icc5ff19783d2b213bff99f283cb0599a8b5c1ab4
Previously we preferred that, but used '*' (<ul><li>) when the parent
comment or the previous reply also used it.
Bug: T252708
Change-Id: I3abf606da6693905764f1be745fad999fdf57fbe
This is similar to the code we already have in JS tests, but instead
of printing to the console where you have to copy-paste from, it just
overwrites the files.
Also, update all of the expected results by this method.
Changes in the expected outputs:
* In JSON files, the "warnings" are now always in the same place
regardless of the type of the warning.
* In all HTML files, self-closing tags now include the trailing slash,
some characters are no longer encoded as entities when not necessary,
and attributes may be single-quoted when that makes them shorter.
* In Parsoid HTML files, the header is no longer terribly mangled.
Other notes:
* CommentParserTest.php: Change the output of serializeComments()
to be in similar order as in JS, to reduce the diffs in this commit
and because it's a better order for humans.
* modifier.test.js: Remove some hacks that were working around small
inconsistencies between the previous expected outputs and the actual
outputs.
Change-Id: I9f764640dae823321c0ac35898fa4db03f1ca364
* Remove the existing approach for detecting signatures that only
worked in source mode; remove autoSignWikitext()
* Use the same approach for auto-signing in source mode as we have
already used in visual
* In both modes, detect whether the user has already typed a signature
at the end of their comment in the modifier, and if so, don't add a
signature
* Add test cases for the detection
Bug: T255738
Change-Id: I791d3035cb1ffc33ce3966d4617a25d08700c35b
* Pass rootNode to the constructor
* Rename getters to match CommentItem/HeadingItem/ThreadItem
value classes.
* Always build the thread tree so CommentItem's always have
and ID and replies/parent.
Change-Id: I508be9534de59016ff806e3d84edcbb1c76cb0c6
Rather than the <body> node, we were passing <body>'s first child.
Current implementation of CommentParser::getComments() doesn't fail
the tests in spite of this because the XPath query incorrectly returns
results relative to the document's real root node, but these tests
would start failing after I2441f33e6e7bad753ac830d277e6a2e81ee8c93d.
Follow-up to 3e6ab2c4d2.
Change-Id: Ic26e0a1ee4443987e215c5f26ef1f084ccd0b40b
The expected HTML was wrong, a '<br />' tag inside 'data-mw' was
somehow turned into '<br ></span>'. No idea how that happened.
Something must be wrong with the HTML parsing in JS tests, which
were used to generate this file.
Change-Id: I69caa68fe70e706df81e8adf29889254704f601e
I was wondering if the different approach to childIndexOf()
implemented in PHP in b8d7a75c34 would
be faster in JS as well, and yes, it is.
Our test suite now takes (on my machine):
* Chrome: 8337 ms → 7355 ms (average over 5 tries)
* Firefox: 5321 ms → 5044 ms (average over 5 tries)
Change-Id: I71963eeb92dcea9bfd59cbf01a7aa0b7de5d9cf1
When there is a wrapper element whose range matches the range of
a comment, any replies will now be added outside of that wrapper,
instead of directly after the comment (inside the wrapper).
Bug: T250126
Change-Id: I6b42c4db019ae998e91eebd324f9cbd2aa791b4f
It was useful when I was debugging those parts of the code, but now
it's usually annoying.
The warnings can still sometimes be useful for understanding how the
tool parses some discussion, though. To keep that functionality, add
displaying warnings for each comment in the debug mode.
Change-Id: I2d218a8a394f179bcc0990ff988a0567c275ccf2
This method shouldn't be required on the server. Leave comments
relating to it in addListItem so JS & PHP can be kept in sync.
Change-Id: I849fac660faf6e750272c20776f96b9250f96b1b
In JS, strings are internally encoded as UTF-16, and properties like
.length return values in UTF-16 code units.
In PHP, strings are internally encoded as UTF-8, and we have the
option of using methods that return bytes like strlen() or UTF-8 code
units like mb_strlen().
However, the values produced by preg_match( …, PREG_OFFSET_CAPTURE )
are in bytes, and there's nothing we can do about that. So let's use
bytes throughout, mixing the two types results in meaningless numbers.
Then in the test code, we have to calculate UTF-16 code units offsets
based on the UTF-8 byte offsets.
We also have to copy the entire workaround for mw:Entity nodes… Maybe
the parser should be fixed to return the real nodes for ranges' ends
in this case.
Change-Id: I05804489d7de0d60be6e9f84e6a49a885e9fb870
In JS tests, we load the documents via mw.template, which apparently
causes the <html>, <head> and <body> tags to disappear, resulting
in the ranges not matching in PHP tests (and the real document).
Put in a big hack that makes them match, and update the JSON files.
Change-Id: I8194752cd5f82c3716c99e76a37226af5d4a0ec1
Profiling reveals that >87% of the run time of our test suite is spent
in this tiny method. Apparently, DOMNodeList::item() is extremely slow
(possibly it's linear time instead of constant time?).
Profiled using XDebug and KCacheGrind:
https://phabricator.wikimedia.org/F31815264
We can calculate the child's index in its parent by counting its
precending siblings instead, which turns out to be much faster.
Before:
1. 275444ms to run DiscussionToolsCommentParserTest:testGetComments with data set #2
2. 12668ms to run DiscussionToolsCommentParserTest:testGetComments with data set #3
...
After:
1. 9545ms to run DiscussionToolsCommentParserTest:testGetComments with data set #2
2. 5549ms to run DiscussionToolsCommentParserTest:testGetComments with data set #3
...
That's still kind of slow but now it's bearable to run the test suite.
Change-Id: I49155f7aa2e231a9a20bf282cf6aaa28fc902e0b
* Not to be confused with the Parsing Team's
"Great Parser JS to PHP port of 2019"
Gasp as OR hacks are changed to null coalescing operators.
Applaud as variable declarations are dropped.
Cheer as parameters and return values are type-hinted.
Shudder as DomNodeLists have no indexOf method.
Moving discussion parsing to the server should allow
us to implement much cleaner APIs for commenting.
Bug: T252252
Co-authored-by: Ed Sanders <esanders@wikimedia.org>
Change-Id: Ic1438d516e223db462cb227f6668e856672f538c
In the future, we should think about a better solution that also
handles other elements (T250126), but this is an easy fix for now.
Bug: T250512
Change-Id: I1321f0da523ddb4a999b8c453b9094a267b38ae2
Extra lines currently render empty list items which are invisible.
Remove these as the user probably didn't intend from them to appear
in the wikiext.
Bug: T249867
Change-Id: I75c1192a94b918783d362d38cd91a508bb169d56
3 or 5 tilde signatures will be assumed to be erroneous and fixed
to 4 tilde signatures. This will be visible in the preview so shouldn't
come as a suprise to users.
Bug: T245628
Change-Id: I741f0761a6fb10c99cf3239ac5c6c7e1a2b872c7
The section wrappers can be marked as template-affected when the
previous or next section is transcluded, causing comments to be
unnecessarily uneditable. The new test case demonstrates this.
Depends-On: I03bc455d5484a6c51f3fa2397c64936b829fe7e3
Change-Id: I895a04990d79a3475d778b4fef054ea0bb076f0b
It's more convenient for display or comparing it with other things.
Depends-On: I03bc455d5484a6c51f3fa2397c64936b829fe7e3
Change-Id: I88d7aa68977210b16860075ed52983a5e99ee0f7
When trying to reply to a comment that is inside a transclusion,
detect if it's transcluded from a subpage or simply wrapped in a
template, and show appropriate error messages.
References:
* VisualEditor ve.dm.Converter#getAboutGroup()
* VisualEditor ve.dm.ModelRegistry#matchElement()
* Parsoid Linter#findEnclosingTemplateName()
Bug: T245694
Change-Id: If3dd1ebbf1d02ee4379c200019bfc3a8ec02325b
If two signatures for a single comment were near each other,
we would sometimes treat them as one huge signature.
Change-Id: Ied4b3aa535a9ca6bebef8a004ae48b7d5a8f2f9b
Currently not used for anything. May be used later for editing
comments (T245225) or reformatting timestamps (T240360).
Note that a comment may have multiple signatures+timestamps,
and we return them all so that you have to deal with that.
Fix some unrelated incorrect documentation comments.
Bug: T245220
Change-Id: I51b8bf4a3bb7968f35e32c7e44c95c2ab079d9ac
Previously they were added at the end of the text node containing the
timestamp, which was usually the end of the line, but not always.
And also fix the same problem for inserting the actual replies (or
reply widgets). This replaces an undocumented hack that prevented our
own reply links from triggering this bug (without it, the reply widget
would be inserted before the reply link rather than after).
While we're here, remove unintentional spacing that appeared before
some reply links, caused by trailing whitespace in text nodes.
Add tests for all of the above.
Bug: T245695
Change-Id: I354b63e2446bb996176a2e3d76abf944127f307e
They are not generated by MediaWiki, but they often appear when users
sign others' unsigned comments by copy-pasting the timestamp from the
history page.
Add test config data for nlwiki, exported by running this in the
browser console:
copy(
JSON.stringify( { wgArticlePath, wgNamespaceIds, wgFormattedNamespaces }, null, 2 ) + '\n' +
JSON.stringify( mw.loader.moduleRegistry['ext.discussionTools.parser'].packageExports['data.json'], null, 2 )
);
Bug: T245784
Change-Id: Icbcdc5a028e9ce2cb09173f87769e525ec3082fc
I think directories like this make more sense for adding more test cases.
Depends-On: I9153851fe162c012967fda00d3e1f81964a8dce9
Change-Id: Ibc72b747a75c72643c1fc04eae49bd15656e8104
Consequences of this are visible in the test cases:
* (en) Tech News posts are not detected.
Examples: "21:22, 1 July 2019 (UTC)", "21:42, 29 July 2019 (UTC)"
* (en) Comments by users who customize the timestamp are not detected.
Examples: "10:49, 28 June 2019 (UTC)", "21:34, 14 July 2019 (UTC)"
* (en) Comments with signatures missing a username are not detected.
This sometimes happens if a comment is accidentally signed with
'~~~~~' (five tildes), which only inserts the timestamp.
Examples: "17:17, 27 July 2019 (UTC)", "10:25, 29 July 2019 (UTC)"
* (pl) A lone timestamp at the beginning of a thread is not detected.
It's not part of a post, it was added to aid automatic archiving.
Example: "21:03, 18 paź 2018 (CET)"
Bug: T245692
Change-Id: I0767bb239a1800f2e538917b5995fc4f0fa4d043
The loop in parser.js assumed that there was always a heading before
any comments (not counting the page title, only section headings).
Bug: T243869
Change-Id: I3a0bb06716e75d4a17e25c40748673a071ee5f30
I don't like that I had to special-case `<p>` tags (top-level
comments) in this code. I feel like it should be possible to handle
top-level comments and replies in a generic way, but I couldn't find
a way to do it that actually worked.
Notes about changes to the behavior, based on the test cases:
* Given a top-level comment A, if there was a "list gap" in the
replies to it: previously new replies would be incorrectly added at
the location of the gap; now they are added after the last reply.
(T242822)
Example: "pl", comment at "08:23, 29 wrz 2018 (CEST)"
* Given a top-level comment A and a reply to it B that skips an
indentation level: previously new replies to A would be added with
the same indentation level as B; now they are added with the
indentation level of A plus one. (The old behavior wasn't a bug, and
this is an accidental effect of other changes, but it seems okay.)
Example: "pl", comment at "03:22, 30 wrz 2018 (CEST)"
and reply at "09:43, 30 wrz 2018 (CEST)"
* Given a top-level comment A, a reply to it B, and a following
top-level comment C that starts at the same indentation level as B:
previously new replies to A would be incorrectly added in the middle
of the comment C, due to the DOM list structure; now they are added
before C. (T241391)
(It seems that comment C was supposed to be a multi-line reply that
was wrongly indented. Unfortunately we have no way to distinguish
this case from a top-level multi-line comment that just happens to
start with a bullet list.)
Example: "pl", comments at "03:36, 24 paź 2018 (CEST)",
"08:35, 24 paź 2018 (CEST)", "17:14, 24 paź 2018 (CEST)"
* In the "en" example, there are some other changes where funnily
nested tags result in slightly different results with the new code.
They don't look important.
* In rare cases, we must split an existing list to add a reply in the
right place. (Basically add `</ul>` before the reply and `<ul>`
after, but it's a bit awkward in DOM terms.)
Example: split-list.html, comment "aaa"; also split-list2.html
(which is the result of saving the previous reply), comment "aaa"
* The modifier can no longer generate DOM that is invalid HTML, fixing
a FIXME in modifier.test.js (or at least, it doesn't happen in these
test cases any more).
Bug: T241391
Bug: T242822
Change-Id: I2a70db01e9a8916c5636bc59ea8490166966d5ec
Even when you have multiple signatures by multiple users in one
paragraph (or list item), it's still basically a single comment.
We don't want to offer multiple buttons to reply to it.
The changed parser test cases are illustrative:
* All affected comments in the "pl" example are comments with a
"post-scriptum", which is now more intuitively treated as part of
the main comment.
* The first comment in the "en" example would probably have been
better if it wasn't merged, but a weird use of the outdent template
causes us to not be able to distinguish that the two parts of the
comment display on separate lines.
* The last comment in the "en" example (isn't that neat?) was previously
incorrectly treated as two comments, because there's a timestamp in
the middle of it (the user is referring to another comment).
* Remaining affected comments in the "en" example are also comments
with a "post-scriptum" and their treatment is clearly better now.
It also accidentally fixes some problems with modifier tests (but not
all), where previously <dl> nodes would be inserted in the middle of
<p> nodes, to reply to the comments which are now merged.
Bug: T240640
Change-Id: I0f2d9238aff75d78286250affd323cd145661a11
Document the current behavior of the modifier (which inserts the
replies into the DOM tree), so that we can more easily see the effect
of changes in I2a70db01e9a8916c5636bc59ea8490166966d5ec.
Basically, add a reply to every comment, and dump the resulting HTML,
comparing it to previously generated expected HTML (which can be
checked visually). Have a look at the new HTML files.
Notably, the very first section in the "pl" example demonstrates a
case of wrong reply location due to list gap :) (T242822).
Change-Id: I4aed0f0b112f53d98e3fe1da4d40db8687c7e537
Possible use cases:
* Matching comments between PHP and Parsoid HTML [implemented here]
* Finding the same comment in a different revision of a page
(e.g. while resolving an edit conflict, or to allow resuming
composition of autosaved comments) [implemented for highlighting
user's own posted comment only]
* Permanent links to comments [future]
The reasoning for this form of ID is:
* _Timestamp_ by itself is a nearly unique identifier, so it's a good
thing to start with
* Users may post multiple comments in one edit (or in many edits in
one minute), so we need the _sequential number_ to distinguish them
* _Username_ is probably not required, but it may reduce the need
for sequential numbers, and will help with human-readability if we
add permanent links
The ID remains stable when a new comment is added anywhere by anyone
(excepts comments within the same minute by the same user), or when a
section is renamed.
It's not always stable when a comment is moved or when an entire
section is moved or deleted (archived), but you can't have everything.
Change-Id: Idaae6427d659d12b82e37f1791bd03833632c7c0
(And rebuild them using this method, the properties are in different
order, it's actually nice for readability to have 'replies' last)
Change-Id: Ib586e1081fa36cb9125db1b0b1d41f092350641c
Tests that handle a specific case and describe what they are testing
would be nice… but tests that just document the current status to
avoid regressions are also okay and easier to add.
Change-Id: I0b3530ae0e77de70932aaf623f5290d1876699a0
