Change the order of checks to ensure that we have at least one comment
before we try comparing ranges, to avoid issues with empty headings
having collapsed ranges. It should be a tiny bit faster this way, too.
Bug: T304377
Change-Id: I59ad30cfc075dcec882e048d2d199744efec2114
We were calling Title::newFromText() before setupEnv(), which meant
that the title for each test case was parsed using the default rules
for English, rather than the rules for the specified wiki.
This only makes a practical difference for tests with self-links.
Changed the only such test to demonstrate the fix.
Change-Id: I45561f1c9f0d149e2b743f0000b742bf6fc014af
Also fix a bug where headings would be ignored while checking for
comment frames. See task for detailed explanation.
Bug: T303396
Change-Id: I6495826b4b050ea80680e0798ac6ab4497a7c09e
Since times immemorial, and for reasons lost to history, our test code
was adding an extra <div> wrapper before parsing the HTML used for
tests. This wasn't a problem, until now, because I want to add some
tests for T303396 that need to check that the *real* wrappers present
in some test cases are handled correctly.
Changes to test cases mostly remove a leading "0/" from serialized
ranges, corresponding to removing the extra wrapper.
Change-Id: Ia50e3590538c8cd274b02d2a937ba1a3fbb4ac89
It is friendlier for static analysis tools like Phan, which can't
infer anything from the `->nodeType === …` checks, and we were already
using it in most places.
Fix newly revealed Phan failures (and one unneeded suppression).
Change-Id: Id789f05e16a210f7ba22ca7514587c392fac0741
Also special-case thumbnail wrappers generated by
MediaTransformOutput::linkWrap, for compatibility with
TimedMediaHandler.
Bug: T301427
Bug: T302296
Change-Id: I7f48d8b2261507c5a33526c54109f5187d062ed3
Previously, we required a signature at the end of the comment.
This was a pretty rough heuristic that did not correctly handle
many comments that we would consider entirely properly signed
in CommentParser (e.g. comments wrapped in formatting like
<small>…</small>, comments with a post-scriptum or in parentheses,
or comments generated by various templates).
Now we process the user input using the same code that adds reply
links, and only add a signature when we detect that there really
isn't a signature (including template-generated), or if the signature
is in the wrong place and would result in the reply link showing up
in the wrong place as well (not at the end of the comment).
Bug: T278442
Bug: T268558
Bug: T278355
Bug: T291421
Bug: T282983
Change-Id: I46b6110af328ebdf93b7dfc2bd941e04391a1599
We were rendering the preview in a completely different way from how
we would add the real reply, and the results would be different
sometimes, particularly for multi-line comments with messed-up markup.
Render it server-side instead, in a very similar way to real replies
(generating a DOM list node and transforming it through Parsoid),
although without the whole context of the page to improve performance.
We can remove a lot of client-side code that was used solely for this.
This will allow the preview to accurately display the signatures when
we change how they are added (T278442), without us having to implement
those changes again from scratch for the preview.
Change-Id: I53341f4d4075c25b67ec3b3032bff9b8a880dcd3
Goal:
-----
Finishing the work from Iadb7757debe000025e52770ca51ebcf24ca8ee66
by changing CommentParser::parse() to return a data object, instead of
the whole parser.
Changes:
--------
ThreadItemSet.php:
ThreadItemSet.js:
* New data class to access the results of parsing a discussion. Most
methods and properties are moved from CommentParser with no changes.
CommentParser.php:
Parser.js:
* parse() returns a new ThreadItemSet.
* Remove methods moved to ThreadItemSet.
* Placeholder headings are generated slightly differently, as we process
things in a different order.
* Grouping threads and computing IDs/names is no longer lazy. We always
needed IDs/names anyway.
* computeId() explicitly uses a ThreadItemSet to check the existing IDs
when de-duplicating.
controller.js:
* Move the code for turning some nodes annotated by CommentFormatter
into a ThreadItemSet (previously a Parser) from controller#init to
ThreadItemSet.static.newFromAnnotatedNodes, and rewrite it to handle
assigning parents/replies and recalculating legacy IDs more nicely.
* mw.dt.pageThreads is now a ThreadItemSet.
Change-Id: I49bfe019aa460651447fd383f73eafa9d7180a92
Goal:
-----
To have a method like CommentParser::parse(), which just takes a node
to parse and a title and returns plain data, so that we don't need to
keep track of the config to construct a CommentParser object (the
required config like content language is provided by services) and
we don't need to keep that object around after parsing.
Changes:
--------
CommentParser.php:
* …is now a service. Constructor only takes services as arguments.
The node and title are passed to a new parse() method.
* parse() should return plain data, but I split this part to a separate
patch for ease of review: I49bfe019aa460651447fd383f73eafa9d7180a92.
* CommentParser still cheats and accesses global state in a few places,
e.g. calling Title::makeTitleSafe or CommentUtils::getTitleFromUrl,
so we can't turn its tests into true unit tests. This work is left
for future commits.
LanguageData.php:
* …is now a service, instead of a static class.
Parser.js:
* …is not a real service, but it's changed to behave in a similar way.
Constructor takes only the required config as argument,
and node and title are instead passed to a new parse() method.
CommentParserTest.php:
parser.test.js:
* Can be simplified, now that we don't need a useless node and title
to test internal methods that don't use them.
testUtils.js:
* Can be simplified, now that we don't need to override internal
ResourceLoader stuff just to change the parser config.
Change-Id: Iadb7757debe000025e52770ca51ebcf24ca8ee66
The Data class contained utilities for two unrelated purposes.
Split each half to a separate class.
Notably, this improves the signature of the getLocalData() function.
Change-Id: Icde615fb9d483fee1f352c34909b37f8ffde8081
(suggested by PhpStorm)
composer.json:
* Document required PHP extensions
Parser.js:
* Remove incorrect param documentation
* Fix some typos in comments (missing parentheses)
CommentParser.php:
* Fix some typos in comments (missing parentheses)
ImmutableRange.php:
* Remove unused property
* Add a `throw` to indicate that code path is unreachable
SubscribedNewCommentPresentationModel.php:
* Add missing `return false`
CommentParserTest.php:
* Remove unnecessary pass-by-reference
CommentModifierTest.php:
* Remove unused variable
CommentParserTest.php:
* Don't construct Element objects directly. PHP's DOMElement allows
it, but Parsoid/Dodo's doesn't, and we use the latter for static
analysis. This generates all kinds of confusing warnings.
Change-Id: Ia9598ebea0e99830dd485296e94a9d96acc4b258
It adds white-space between block tags and strips invisible tags.
It may be slightly slower (it takes HTML as input rather than DOM, so
we need to serialize the HTML first and then call it, rather than only
find and concatenate text nodes), but the difference is negligible,
and it seems better to use this method than to try to re-implement it.
Test runtime went from ~9.0s to ~9.5s locally, when testing using:
php tests/phpunit/phpunit.php \
extensions/DiscussionTools/tests/phpunit/ThreadItemTest.php \
--filter getText
Bug: T219138
Change-Id: I0cb89ebd2160e1ef499b78573c6688f493a4c42f
Goal: To be able to re-use or test the transformations we previously
performed in addWikitextReply() / addHtmlReply(), without requiring
a Comment object or adding the result as a reply.
Change-Id: I040c4be9b6b9bddba661f30fd0566f8850673074
* We don't need it anywhere in JS, since we're not sending that HTML
to Parsoid.
* We only need it on the nodes directly containing our reply lines in
PHP, not all over the place.
Change-Id: I0a04388225f32654dda2f599442cd27a303b5d0a
When we encounter a node that doesn't represent comment contents, e.g.:
* a [reply] link we inserted (T297034#7641334)
* an {{outdent}} template (see changed test case)
…we should ignore it together with its descendants (like in
Parser#nextInterestingLeafNode), instead of processing descendants
and possibly detecting comment contents in them.
Follow-up to 8de940b587,
72b9c2c6f5.
Bug: T297034
Change-Id: Ib2fa40c5fa389572b0e88ef558728fa06e3621b0
Prior to 8de940b5, the comments on this page would not be marked as
partially transcluded.
Bug: T298408
Change-Id: Ib7eb8b4113151048c0e778b3530600d98dd8f705
These are not used for anything yet, but soon the parser will
want to know the title of the page it is parsing.
Change-Id: I02fa5d63fae78f3e92032d93bc27ac5c744faecb
The following values for configuration variables are supported:
$wgDiscussionToolsReplyIndentation = 'invisible'; (default)
$wgDiscussionToolsReplyIndentation = 'bullet';
Bug: T259864
Change-Id: Icefad79630adc6ed35687498614e6a03ede1451b
Reimplement getFullyCoveredSiblings() using compareRanges(), which
checks basically the same thing, but works better and I like it more.
Bug: T297034
Change-Id: I33dc1d088bdee984064315290e378bfbfa830b10
Previously: 569db3603c (2020-06).
Unfortunately we've found cases where the previous implementation
doesn't work correctly, resulting in comments being added to the wrong
pages or page corruption.
Bug: T289873
Bug: T298051
Change-Id: Id867b3005ebc46906d6df852a525fcaec9e6b19b
Comment boundaries are stored as a DOM parent node plus a child index.
Because of that, inserting anything into the DOM before a comment –
such as another comment's start/end markers – would cause us to insert
subsequent comments' markers into the wrong places.
This issue didn't affect many pages, because usually any parent node
would have just one comment in it. Only pages with comment boundaries
outside of any wrappers (directly inside the root node) were broken.
Just process the list in reverse to fix this.
Bug: T298096
Change-Id: Iccffc36b71e9fcf3d72c4db2b9459d39042f7a2d
Servers as another test case for partially transcluded comments,
and a test case for comment start markers placed outside of
paragraphs.
Bug: T298051
Bug: T298096
Change-Id: Id07d2f57708c037578cb653c83848490c9a15fc6
We're probably not going to use it again, and I don't want to make the
effort of rewriting it in Id867b3005ebc46906d6df852a525fcaec9e6b19b.
Change-Id: I0b02533f7c9b8c1b0df271e03a74063f123d0dff
Data used for the tests assumes there are no variants for English,
and some tests fail when there are. Correct behavior with language
variants is tested using other languages.
Change-Id: I348a0ba0389c2a18644ce5e05c7f37d8f26a8c55
Usually this isn't a problem, because the comments are marked as
template-generated and we don't allow replying to them. But we had a
special case where we were trying to skip over some invisible
elements, which was causing us to skip into the middle of the
about-group in some cases. When Parsoid sees that, it serializes the
contents twice.
Bug: T290940
Change-Id: I9fe0b8d43ab874ccef371990799f77bfc46bc954
We do something similar in CommentItem.js with a moment object.
The object can be converted to a string when required.
Change-Id: Id7221e9201db0d89c3b771574634c878c9515ca0
These types can be passed a parameters to any file without
creating a dependency, so it makes more sense to allow
the globally.
Change-Id: I5504465fd997b46547642e7046993b370b85586e
The PHP DOM extension measures lengths and offsets in Unicode codepoints.
Our PHP code used UTF-8 bytes, causing some offsets to be slightly off.
Now it mostly uses Unicode codepoints as well (we're forced to use bytes
in a few places, because preg_match returns offsets in bytes).
In practice, this had no visible effect to the user. It caused the
markers `<span data-mw-comment-end="..."></span>` to be placed at
the end of their container instead of the correct position when the
timestamp contained multibyte characters (e.g. "ź" in Polish); but
the correct position is usually at the end of the container anyway.
In the test cases, the only difference is placing these markers before
a trailing line break inside `<p>...</p>` tags rather than before it.
The patch also accidentally fixes another bug, where element nodes
with no children (mostly <img>) were incorrectly excluded when calling
cloneContents(), because they were treated as if they were text nodes.
Change-Id: Iccdccf1078598f4b62cab96225e9c85a4c0e93ee
If the user talk edit or mention coincides with exactly one new comment:
* Change the primary link to be a direct link to the comment
* Add a text snippet to notifications that don't already include one
(user talk edits that are not new sections).
This is done for all such notifications, regardless of whether anyone
has topic subscriptions enabled.
Bug: T281590
Bug: T253082
Change-Id: I98fbca8e57845cd7c82ad533c393db953e4e5643
* ThreadItem::getText
* CommentItem::getBodyText (used when generating notifications)
* ThreadItem::getHTML (may soon be used in API)
* CommentItem::getBodyHTML (may soon be used in API)
* ImmutableRange::cloneContents (the common implementation for all
of the above)
The outputs are only lightly reviewed. This is mostly meant to
document the current behavior rather than the expected behavior,
to avoid making unintentional changes while refactoring.
Change-Id: I14471ee4969aa3d0b5577d9de2a6d4462fab4d09
The code (prior to d25825a754) assumed
that level 3+ headings would always follow a level 2 heading or the
placeholder heading, but we don't generate a placeholder heading if
there are no comments in section zero.
Add more tests to confirm that comments under level 3+ headings (that
are not sub-headings of level 2), and level 1 headings, are ignored
when generating notifications, and do not mess with normal headings.
Bug: T288775
Change-Id: Ic57b56752a4797cb01234f66e0ed7b849752bd70
This DOMDocument property has no effect, because we do not use
DOMDocument methods for parsing HTML, but rather DOMUtils::parseHTML()
provided by Parsoid.
Change-Id: I1d9e73e53f2d44f41cf9dcda4f06ac8647671096
This reverts commit f075e37303.
No longer needed after Iaf786cd0f1d870cbcf0b968b7adce616c82df3d8
in MediaWiki core, and now causes exceptions because
UNSAFE_restoreLegacyHtmlPrefilter is undefined.
Bug: T280944
Change-Id: I0dbd6fcb5dce939de334815e9fe371425cf5641f
Use `DOMCompat::getBody( ... )` as a nicer getter than
`->getElementsByTagName( 'body' )->item( 0 )`.
Remove overly defensive checks and redundant annotations on its
return value. Since we're dealing with HTML documents throughout,
the document body is guaranteed to exist.
We previously needed some of them to convince Phan when it thought
the body may be null, but this seems to no longer be needed.
Change-Id: If7aee7b6adbfa78269c7ba28b26a6eaa21fe935b
Adding test cases in a separate commit to make it easier to review how
the test results change after I98fbca8e.
* For mentions, the 'mentioned-users' extra parameter is copied to our
event (which is then used to avoid duplicate notifications).
* For user talk page edit, nothing special happens right now (we use
the target page title to avoid duplicate notifications, but this is
not apparent from the test case, since page titles are not present).
Bug: T281590
Bug: T253082
Change-Id: I153e7735f63f1e2643ed881281d807313cd699c3
In case 4 and case 6, no notifications are expected. In all other
cases we now get the expected notifications.
Bug: T285528
Change-Id: I9e813bb3a053bc1232783f9eae1ad75672b4fa7e
Adding test cases in a separate commit to make it easier to review how
the test results change.
As expected, in every case, no notifications are generated right now.
Bug: T285528
Change-Id: I25308754112c521d2db8c54ef0c82373456d9e31
These changes ensure that DiscussionTools is independent of DOM
library choice, and will not break if/when Parsoid switches to an
alternate (more standards-compliant) DOM library.
We run `phan` against the Dodo standards-compliant DOM library,
so this ends up flagging uses of non-standard PHP extensions to
the DOM. These will be suppressed for now with a "Nonstandard DOM"
comment that can be grepped for, since they will eventually
will need to be rewritten or worked around.
Most frequent issues:
* Node::nodeValue and Node::textContent and Element::getAttribute()
can return null in a spec-compliant implementation. Add `?? ''` to
make spec-compliant results consistent w/ what PHP returns.
* DOMXPath doesn't accept anything except DOMDocument. These uses
should be replaced with DOMCompat::querySelectorAll() or similar
(which end up using DOMXPath under the covers for DOMDocument any way,
but are implemented more efficiently in a spec-compliant
implementation).
* A couple of times we have code like:
`while ($node->firstChild!==null) { $node = $node->firstChild; }`
and phan's analysis isn't strong enough to determine that $node is still
non-null after the while. This same issue should appear with DOMDocument
but phan doesn't complain for some reason.
One apparently legit issue:
* Node::insertBefore() is once called in a funny way which leans on
the fact that the second option is optional in PHP. This seems to be
a workaround for an ancient PHP bug, and can probably be safely
removed.
Bug: T287611
Bug: T217867
Change-Id: I3c4f41c3819770f85d68157c9f690d650b7266a3
For compatibility with Parsoid's document abstraction (Parsoid may
switch to an alternate DOM library in the future), don't explicitly
create a new document object using `new DOMDocument`; instead use
the Parsoid wrapper `DOMCompat::newDocument()`. This ensures that
the Document object created will be compatible with Parsoid.
There are a number of other subtle dependencies on the PHP `dom`
extension in DiscussionTools, like explicit `instanceof` tests; those
will be tweaked in a follow-up patch
(I3c4f41c3819770f85d68157c9f690d650b7266a3) since they do not affect
correctness so long as Parsoid is aliasing Document to a subclass of
the built-in DOMDocument. Similarly, the Phan warnings we suppress
do not cause runtime errors (because of the fixes included in
c5265341afd9efde6b54ba56dc009aab88eff83c) but phan will be happier
once the follow-up patch lands and aligns all the DOM types.
Bug: T287611
Depends-On: If0671255779571a91d3472a9d90d0f2d69dd1f7d
Change-Id: Ib98bd5b76de7a0d32a29840d1ce04379c72ef486
The user interface only allows you to subscribe to level 2 headings.
But we would generate events for whatever heading was the closest,
If it was e.g. level 3, no one would receive that notification.
Now we generate events for the closest level 2 heading, or we don't
generate the event at all if there isn't one (if the only headings are
of level 3 and below, or level 1, or if the comment is added before
the first heading on the page).
Bug: T286736
Change-Id: Iae99853070e353ab81c9cc29ef1d53c877adfc66
Disable the legacy htmlPrefilter from jquery.migrate.js, which is
causing noisy warnings when running tests because we use HTML
templates with wacky content in the test module. They look like this:
"JQMIGRATE: HTML tags must be properly nested and closed: <200 KB of HTML>"
Change-Id: Ic9bbd56e24b5769988a52f28d26d8b6d5922b1b4
The issue occurred when replying to a comment consisting of multiple
list items, starting with a <dt> (instead of the expected <dd>), so
that the comment is considered to be unindented.
Modifier tried to add the reply directly inside the list (<dl>) rather
than inside the last list item (<dt>), which caused it to be confused
about indentation levels and try to un-indent more times than there
were indentations.
The simplest solution, given the existing code, is to add the reply
outside the list instead, in a new list. This results in a "list gap"
(<dl><dt>...</dt><dd>...</dd></dl><dl><dd>...</dd></dl>), but I think
it's acceptable for this rare case.
There are separate tests cases for old Parser and for Parsoid HTML,
because they parse the original wikitext differently (with the old
Parser producing HTML with a list gap too).
Bug: T279445
Change-Id: Ie0ee960e7090cf051ee547b480c980e9530eda51
We added it because the initial designs for the subscribe action were
much easier to implement like this, and topic "containers" (T269950)
would have required it.
However, the latest design of the subscribe action will not need it
(T279149), and topic containers are still very far away, so let's
remove it for now.
Bug: T280433
Change-Id: I21a23e9bea43f24d265750926fbd62b99038d3f1
The existing comment IDs can't be used to find the same comment on
a different revision or page (when it's transcluded), because they
depend on the comment's parent and its position on the page.
Comment names depend only on the author and timestamp. The trade-off
is that they can't distinguish comments posted within the same minute,
or in the same edit, so we will still need the IDs sometimes.
Prefer using comment names when replying, if they're not ambiguous.
This fixes T273413 and T275821.
Heading names depend on the author and timestamp of the oldest comment.
This way we don't have to detect changes to the heading text, but we
can't distinguish headings without any comments.
Bug: T274685
Bug: T273413
Bug: T275821
Change-Id: Id85c50ba38d1e532cec106708c077b908a3fcd49
Longer, but follows the style guide and less likely to conflict.
We need to account for init classes in the cache being around for
a while.
Change-Id: I738bc93393850db320fdbda2b003ca8ac40556da
The code we're testing already produces a string of serialized HTML,
no need to parse and re-serialize it.
Also, we recently learned that the precise format matters here
(T274709), and now this test *actually* covers the fix for that bug.
Follow-up to 5b26e9664b.
As a downside, this test might now spuriously fail if the format of
the output of Parsoid's XMLSerializer changes. Hopefully that won't
happen too often.
Change-Id: I69b514f545e47dcb437fb39a83edb8e2f19ed99b
Now it detect signatures generated by en.wp's {{Undated}} template,
and signatures of people who do weird stuff to the timestamps.
Bug: T275938
Change-Id: I27b07f6786ca5433a3c02a5fe68e4716d41401bb
The horrendous 11-line if() condition did not correctly handle
signatures wrapped in inline formatting markup, like <small>.
Instead, implement this logic in the code for skipping to the end
of a paragraph, which didn't exist yet when that condition was
added, but seems like a much better place to check this now.
Bug: T275934
Change-Id: I5cccff889b5e15b5f8fde0538bf4bccb22e762cf
This code expected $container->firstChild to be a
<div class="mw-parser-output">, but that element is not present
when we're running on HTML to be saved in parser cache.
We ended up inserting the marker inside whatever node was the
first on the page, and if it was a <style> element, both our
marker and the styles would be lost when serializing, like in
6c7a0ca9a2.
When we're running on final HTML, the marker will now be outside
of <div class="mw-parser-output">, but that seems to be fine. Only
early versions of I4e60fdbc098c1a74757d6e60fec6bcf8e5db37c1 had
problems with that (see comments on patchset 41), but it works now.
The added test case also covers the fix for T274709.
Bug: T275440
Change-Id: I38d45dd8686919be51e1d307ded12b0afe185eb5
Top-level comments that start or end with a list (inconsistent
indentation) would not have triggered the logic for detecting
wrappers.
Bug: T273692
Change-Id: Idcb4eed73e391f5f86eca2eb05cb3cea0d86f30a
For a moment I doubted if we handle this case correctly, but in fact
I didn't botch that code *this* badly.
Change-Id: I5a9d142e4bd97ac40aa388bb43b65ab1286e3f18
Documentation:
https://www.mediawiki.org/wiki/Manual:PHP_unit_testing/Writing_unit_tests_for_extensions#Two_types_of_tests
We can do this because the tested methods do not depend on any globals
or on MediaWiki being installed.
In addition to being the new hotness, MediaWikiUnitTestCase allows the
test classes that use it instead of MediaWikiTestCase to start up much
faster. In my testing, running this test case individually now takes
0.35s, compared to 1.1s before.
Try:
* With new code:
time php tests/phpunit/phpunit.php extensions/DiscussionTools/tests/phpunit/unit/CommentUtilsTest.php
* With old code:
time php tests/phpunit/phpunit.php extensions/DiscussionTools/tests/phpunit/CommentUtilsTest.php
Change-Id: I771b1f3d101a394ee869e42547d9ae7839397752
* Ignore rendering-transparent nodes between discussion comments.
* Improve isRenderingTransparentNode() so that <link> nodes
representing TemplateStyles are not considered transparent,
otherwise this would undo ae920b831f.
Using a regexp from Parsoid.
Bug: T272746
Change-Id: I0b3c3251156ba6c4826abf5ba44ea93f80ebc01d
Add yet another tree walking utility: CommentUtils::linearWalk().
Unlike TreeWalker, it allows handling the beginnings and ends of nodes
separately – kind of like parsing a XML token stream, or kind of like
VisualEditor's linear model.
(Add unit tests for this utility. The simple.html test case is copied
from [VisualEditor/VisualEditor]/demos/ve/pages/simple.html.)
Use this utility to stop skipping when we reach either a closing or
opening block node tag. Previously we'd skip over such tags inside
nested "transparent" nodes (like <a>, <del>, or apparently <font>).
Bug: T271385
Change-Id: I201a942eb3a56335e84d94e150ec2c33f8b4f4e0
As a result of 0fc71f60cd, "empty" text
nodes (containing only whitespace) at the end of the comment may be
inside the comment's range, and trying to ignore them caused the
ranges not to match and the frame not to be detected.
Now the code works whether they're inside the comment's range or not.
Add a test case for wrapped discussion comments with HTML comments and
with whitespace.
Bug: T250126
Bug: T268407
Change-Id: I2217ff5a635fd1c9c9e803f46795b1bfb3d17535
While working on T270009, I noticed that <style> and <link> nodes
are treated differently, which seemed weird. Rewrite this again,
hopefully this is the last time.
The changed test cases also involve <area> and <input> nodes,
and the new results make more sense to me.
Bug: T264116
Change-Id: I3af90c84768a4b3dc53446927f4dba6f72175a2f
We've recently decided that we want to "extend" comments until
the end of the paragraph (e36dc8e78a,
d0ae6c4e44).
However, we still had this special case that did the opposite: it
ensured that if a comment ended in the middle of a text node, the
comment would not be extended to the end of the node. Remove it.
Note the change in the test file signatures-funny-formattedreply.html,
which actually covered this case specifically.
Change-Id: Id1384bb0c6e1a5f0c70f55efcb4caa240f230f07
The end marker is skipped forward until an open or close
block tag is reached. In tree traversal terms this means
moving either to the next sibling, or the parent (to skip
over close tags).
Bug: T256033
Change-Id: Iaa2c588698790d576ac4f9ecc126f58a082ef6b3
The general rule is that comments start after their preceding
thread item, but when that is a heading we should skip past
the entire <h[1-6]> node to avoid making section edit links
part of the first comment.
Bug: T267988
Change-Id: Ia7f1b27e0a69a9aab7c7da743bf8549479304096
As CommentFormatter no longer needs HTMLFormatter, remove
the inheritance and make addReplyLinks a static method.
Testing locally this is marginally slower, going from 2.55s
to 2.9s for the CommentFormatterTest case.
Bug: T266317
Bug: T267973
Change-Id: If69749cae678a1647a138d782a32032189f55cec
After recent changes allowing ThreadItems to have IDs, they can now
also have warnings about duplicate IDs.
Bug: T267035
Change-Id: If3edfe34e6e29741e29fac8946a3c88badc4ab7f
Use the same logic for marking ranges in the document, and ensure
that the heading range does not include section edit links or
section numberings.
Change-Id: I782caafc34fee2a822b0a17b24dd6b9528202eca
We avoided fixing these because it causes changes in just about all of
the test data, which is annoying when reviewing or blaming changes.
But the previous several commits also caused changes in just about all
of the test data, so we might as well do this too.
Change-Id: I83b64d83b6f12c04dc06c0cadff7cdd89417e137
To avoid old threads re-appearing on popular pages when someone
uses a vague title (e.g. dozens of threads titled "question" on
[[Wikipedia:Help desk]]: https://w.wiki/fbN), include the oldest
timestamp in the thread (i.e. date the thread was started) in the
heading ID.
Bug: T264478
Change-Id: If918bfd5e025248923d1939bc86916697ead95a0
Sequential numbers aren't great because they change when an earlier
comment is archived. Parent comment/heading IDs should change less
often.
This also makes much more sense for disambiguating subsections,
e.g. a dozen identical ===Votes=== sections for a dozen proposals.
Bug: T264478
Change-Id: I466454984fd919ebef35f2b37ddb5d86dc842996
Our threads now also contain all replies to their sub-threads.
This is similar to how sections work in MediaWiki, where the parent
section also contains the content of all the lower-level sections.
We're going to need this for notifications about replies in a thread.
Bug: T264478
Change-Id: I241fc58e2088a7555942824b0f184ed21e3a8b6f
Previously, only comments could have IDs, because we only needed IDs
for replying. But we might also use them for notifications soon.
Bug: T264478
Change-Id: I1bcad02bf17ab54bc5028a959543c10f0430836b
I haven't really reviewed the outputs, but at least a) they don't
crash b) they will fail if the output suddenly changes (which could
cause problems due to caching).
Bug: T252555
Change-Id: I1bbcbc5dd17ce1e24b3622062f5e8df4baf5f389
Also, add tests covering this and the previous bug fixes in this code
(T259818, T261706).
Note that the test data added in tests/cases/ doesn't exactly match
the entire configuration of the wiki, only the parts we want to cover.
This is unlike the data in tests/data/, which was literally copied
from the relevant wikis, and which is used as input for other tests.
Bug: T265500
Change-Id: I29a59a5952f6dc9fb5910434bb6bcc9dcdaa01a9
The wikitext parser outputs `<p><br></p>` for empty paragraphs, so we
need to ignore `<br>` tags when searching for an "interesting" node
that marks the beginning of a comment. Otherwise the empty paragraphs
mess up the detection of indentation levels.
Bug: T264116
Change-Id: I84a97ab577baa7336b78935ccdc48041ecfc231a
The tests pass either way because this trivial difference is ignored,
but it's annoying when trying to update changed tests.
Change-Id: I7ad1acad43e30a63843cfda4d0d08ff663ef3252
* Export parser data (date format, digits, timezone names, and
messages for weekday/month names) converted to language variants
* Update the parsers to try matching using every variant, in case
the page is displayed in non-default variant (and to avoid
problems with incomplete variant conversion)
Bug: T259818
Change-Id: I04d73992cd31ce06fa79f87df0c0a53d7efc3c58
This is primarily to cover the handling of localised digits,
which previously wasn't being tested, leading to T261706.
Bug: T261706
Change-Id: I9de7f01f77e767e9048c85604b559af4bca0de91
* Remove 'wgMetaNamespace' and 'wgMetaNamespaceTalk', the same data
exists in 'wgFormattedNamespaces'.
* Rename 'wgContentLang' to 'wgContentLanguage', to match its real
name in JS config. MediaWiki doesn't use 'wgContentLang' anywhere,
although the related PHP global is called $wgContLang.
* Document how I made these files, previously only mentioned in the
commit message of e9c401e3aa.
Change-Id: I67f962812c155aedf41154e0d837e7feb5af972d
The PHP code incorrectly assumed that the digits are single-byte in
UTF-8, which is never the case (except for 0-9).
The JS code worked correctly because it uses UTF-16 strings, so the
bug would only affect non-BMP digits there. This was noted in a TODO
comment, but we overlooked it when reimplementing in PHP.
Instead of a string of 10 characters, use an array of 10
single-character strings.
Bug: T261706
Change-Id: Ic5421382474c88f003424799c53ff473d99cce92
When a comment ended before the end of a paragraph, the next
comment would begin right there in the middle of the paragraph.
This could result in the detected indentation level of that
comment being incorrect, and replies being inserted in wrong
places, as seen in the 'signatures-funny' test case.
The code moved to the parser was previously repeated twice in
addListItem() and addReplyLink(), which should have been a hint
that something isn't quite right.
Also, fix the code guarding against overlapping signatures,
now that signatures may not be at the end of a comment.
Bug: T260855
Change-Id: Ic26a87642f8a15d5de2f7073d4d8176b299c7f94
Expand the 'signatures-funny' test case with more examples, which
don't behave correctly.
Follow-up commits I04a8ea09401e06f2d4bb1f226f17eb528b29ed95 and
Ic26a87642f8a15d5de2f7073d4d8176b299c7f94 fix them.
Bug: T255738
Change-Id: I0fdd8bdf11b497ffeed37c37953c5730f6e4f3b7
Previously, parser would output offsets that don't exist in their
containers, because we were pretending that entities are parts of
their neighboring text nodes.
Turns out it's much easier to do it right when going backwards.
Change-Id: I9bccca2d403f1a976ae517449989170cdd99721e
Follow-up to ccd9e411d2.
* Fix variable name in CommentTestCase::overwriteHtmlFile()
* Overwrite before assertions, because they abort execution if they fail
Change-Id: I5bba016ba93f9dd1994325ae82c3105ba11cf033
The latter results in lots of extra HTML entity encoding.
The former is built by the Parsing team and appears to result
in no unexpected changes elsewhere in the document.
As Parsoid's selser relies on HTML fragments being byte-for-byte
equal, these changes were resulting in wikitext normalisations
in untouched parts of the document ("dirty diffs").
Bug: T259855
Change-Id: Ib3cb605911e690ec3e8c2f9df25fd1a2e2849d7e
When adding a reply, we take a node at the end of the previous comment,
compare that comment's indentation level to the expected indentation level
of the reply, and add (or remove) that number of wrapper lists.
The existing code did not consider that comments may have lists within
them, and so the indentation of that node may not match the indentation
of the comment.
Bug: T252702
Change-Id: Icc5ff19783d2b213bff99f283cb0599a8b5c1ab4
Previously we preferred that, but used '*' (<ul><li>) when the parent
comment or the previous reply also used it.
Bug: T252708
Change-Id: I3abf606da6693905764f1be745fad999fdf57fbe
This is similar to the code we already have in JS tests, but instead
of printing to the console where you have to copy-paste from, it just
overwrites the files.
Also, update all of the expected results by this method.
Changes in the expected outputs:
* In JSON files, the "warnings" are now always in the same place
regardless of the type of the warning.
* In all HTML files, self-closing tags now include the trailing slash,
some characters are no longer encoded as entities when not necessary,
and attributes may be single-quoted when that makes them shorter.
* In Parsoid HTML files, the header is no longer terribly mangled.
Other notes:
* CommentParserTest.php: Change the output of serializeComments()
to be in similar order as in JS, to reduce the diffs in this commit
and because it's a better order for humans.
* modifier.test.js: Remove some hacks that were working around small
inconsistencies between the previous expected outputs and the actual
outputs.
Change-Id: I9f764640dae823321c0ac35898fa4db03f1ca364
* Remove the existing approach for detecting signatures that only
worked in source mode; remove autoSignWikitext()
* Use the same approach for auto-signing in source mode as we have
already used in visual
* In both modes, detect whether the user has already typed a signature
at the end of their comment in the modifier, and if so, don't add a
signature
* Add test cases for the detection
Bug: T255738
Change-Id: I791d3035cb1ffc33ce3966d4617a25d08700c35b
* Pass rootNode to the constructor
* Rename getters to match CommentItem/HeadingItem/ThreadItem
value classes.
* Always build the thread tree so CommentItem's always have
and ID and replies/parent.
Change-Id: I508be9534de59016ff806e3d84edcbb1c76cb0c6
Rather than the <body> node, we were passing <body>'s first child.
Current implementation of CommentParser::getComments() doesn't fail
the tests in spite of this because the XPath query incorrectly returns
results relative to the document's real root node, but these tests
would start failing after I2441f33e6e7bad753ac830d277e6a2e81ee8c93d.
Follow-up to 3e6ab2c4d2.
Change-Id: Ic26e0a1ee4443987e215c5f26ef1f084ccd0b40b
The expected HTML was wrong, a '<br />' tag inside 'data-mw' was
somehow turned into '<br ></span>'. No idea how that happened.
Something must be wrong with the HTML parsing in JS tests, which
were used to generate this file.
Change-Id: I69caa68fe70e706df81e8adf29889254704f601e
I was wondering if the different approach to childIndexOf()
implemented in PHP in b8d7a75c34 would
be faster in JS as well, and yes, it is.
Our test suite now takes (on my machine):
* Chrome: 8337 ms → 7355 ms (average over 5 tries)
* Firefox: 5321 ms → 5044 ms (average over 5 tries)
Change-Id: I71963eeb92dcea9bfd59cbf01a7aa0b7de5d9cf1
When there is a wrapper element whose range matches the range of
a comment, any replies will now be added outside of that wrapper,
instead of directly after the comment (inside the wrapper).
Bug: T250126
Change-Id: I6b42c4db019ae998e91eebd324f9cbd2aa791b4f
It was useful when I was debugging those parts of the code, but now
it's usually annoying.
The warnings can still sometimes be useful for understanding how the
tool parses some discussion, though. To keep that functionality, add
displaying warnings for each comment in the debug mode.
Change-Id: I2d218a8a394f179bcc0990ff988a0567c275ccf2
This method shouldn't be required on the server. Leave comments
relating to it in addListItem so JS & PHP can be kept in sync.
Change-Id: I849fac660faf6e750272c20776f96b9250f96b1b
In JS, strings are internally encoded as UTF-16, and properties like
.length return values in UTF-16 code units.
In PHP, strings are internally encoded as UTF-8, and we have the
option of using methods that return bytes like strlen() or UTF-8 code
units like mb_strlen().
However, the values produced by preg_match( …, PREG_OFFSET_CAPTURE )
are in bytes, and there's nothing we can do about that. So let's use
bytes throughout, mixing the two types results in meaningless numbers.
Then in the test code, we have to calculate UTF-16 code units offsets
based on the UTF-8 byte offsets.
We also have to copy the entire workaround for mw:Entity nodes… Maybe
the parser should be fixed to return the real nodes for ranges' ends
in this case.
Change-Id: I05804489d7de0d60be6e9f84e6a49a885e9fb870
In JS tests, we load the documents via mw.template, which apparently
causes the <html>, <head> and <body> tags to disappear, resulting
in the ranges not matching in PHP tests (and the real document).
Put in a big hack that makes them match, and update the JSON files.
Change-Id: I8194752cd5f82c3716c99e76a37226af5d4a0ec1
Profiling reveals that >87% of the run time of our test suite is spent
in this tiny method. Apparently, DOMNodeList::item() is extremely slow
(possibly it's linear time instead of constant time?).
Profiled using XDebug and KCacheGrind:
https://phabricator.wikimedia.org/F31815264
We can calculate the child's index in its parent by counting its
precending siblings instead, which turns out to be much faster.
Before:
1. 275444ms to run DiscussionToolsCommentParserTest:testGetComments with data set #2
2. 12668ms to run DiscussionToolsCommentParserTest:testGetComments with data set #3
...
After:
1. 9545ms to run DiscussionToolsCommentParserTest:testGetComments with data set #2
2. 5549ms to run DiscussionToolsCommentParserTest:testGetComments with data set #3
...
That's still kind of slow but now it's bearable to run the test suite.
Change-Id: I49155f7aa2e231a9a20bf282cf6aaa28fc902e0b
* Not to be confused with the Parsing Team's
"Great Parser JS to PHP port of 2019"
Gasp as OR hacks are changed to null coalescing operators.
Applaud as variable declarations are dropped.
Cheer as parameters and return values are type-hinted.
Shudder as DomNodeLists have no indexOf method.
Moving discussion parsing to the server should allow
us to implement much cleaner APIs for commenting.
Bug: T252252
Co-authored-by: Ed Sanders <esanders@wikimedia.org>
Change-Id: Ic1438d516e223db462cb227f6668e856672f538c
