Commit graph

77 commits

Author SHA1 Message Date
jenkins-bot 0eb37a87df Merge "Don't detect comments within quotes" 2021-02-28 22:56:20 +00:00
Bartosz Dziewoński 024a978ffd Don't detect comments within quotes
Bug: T275881
Change-Id: I8f7a4279837bd95ebf5b604ff350c0a3f29c2c05
2021-02-28 22:49:48 +00:00
Bartosz Dziewoński efe95494a8 Improve signature detection to handle formatting on the timestamp
Now it detect signatures generated by en.wp's {{Undated}} template,
and signatures of people who do weird stuff to the timestamps.

Bug: T275938
Change-Id: I27b07f6786ca5433a3c02a5fe68e4716d41401bb
2021-02-27 02:33:30 +01:00
Bartosz Dziewoński af082908a5 Improve merging multiple comments on one paragraph
The horrendous 11-line if() condition did not correctly handle
signatures wrapped in inline formatting markup, like <small>.

Instead, implement this logic in the code for skipping to the end
of a paragraph, which didn't exist yet when that condition was
added, but seems like a much better place to check this now.

Bug: T275934
Change-Id: I5cccff889b5e15b5f8fde0538bf4bccb22e762cf
2021-02-27 02:21:36 +01:00
Bartosz Dziewoński 9fafe72fc7 Broken test cases for comments with double signatures
Bug: T275934
Change-Id: I00edc63b723053e7ac7d26ce5fc1dde4b824380d
2021-02-27 00:12:42 +01:00
Bartosz Dziewoński 5b26e9664b CommentFormatter: Fix inserting placeholder heading marker
This code expected $container->firstChild to be a
<div class="mw-parser-output">, but that element is not present
when we're running on HTML to be saved in parser cache.

We ended up inserting the marker inside whatever node was the
first on the page, and if it was a <style> element, both our
marker and the styles would be lost when serializing, like in
6c7a0ca9a2.

When we're running on final HTML, the marker will now be outside
of <div class="mw-parser-output">, but that seems to be fine. Only
early versions of I4e60fdbc098c1a74757d6e60fec6bcf8e5db37c1 had
problems with that (see comments on patchset 41), but it works now.

The added test case also covers the fix for T274709.

Bug: T275440
Change-Id: I38d45dd8686919be51e1d307ded12b0afe185eb5
2021-02-24 20:32:48 +01:00
jenkins-bot d5b2389ffa Merge "Fix replying outside wrappers for partially indented comments" 2021-02-11 00:09:48 +00:00
jenkins-bot 89dd44bb35 Merge "More test cases for comment wrappers (multiple siblings)" 2021-02-09 18:07:14 +00:00
Bartosz Dziewoński 9d2b35828d Fix replying outside wrappers for partially indented comments
Top-level comments that start or end with a list (inconsistent
indentation) would not have triggered the logic for detecting
wrappers.

Bug: T273692
Change-Id: Idcb4eed73e391f5f86eca2eb05cb3cea0d86f30a
2021-02-08 22:18:37 +01:00
Ed Sanders d05109b24d Truncate user generated part of IDs to 80 characters
This ensures that IDs fit in a 255 character database field.

Bug: T273658
Change-Id: I3cfe4fce6a865b4343f0f01121cd696aa5f98b22
2021-02-03 15:04:58 +00:00
Bartosz Dziewoński 0f4db60e72 More test cases for comment wrappers (multiple siblings)
For a moment I doubted if we handle this case correctly, but in fact
I didn't botch that code *this* badly.

Change-Id: I5a9d142e4bd97ac40aa388bb43b65ab1286e3f18
2021-02-03 00:26:52 +01:00
Ed Sanders 47aea0b160 Use tabs for indentation in JSON test files
Change-Id: I1d8f8b33b19bcff249ad08dfe687f87f5e5bf9bf
2021-01-27 00:25:15 +00:00
Bartosz Dziewoński c781b127c9 Handle category links at ends of comments affecting indentation
* Ignore rendering-transparent nodes between discussion comments.
* Improve isRenderingTransparentNode() so that <link> nodes
  representing TemplateStyles are not considered transparent,
  otherwise this would undo ae920b831f.
  Using a regexp from Parsoid.

Bug: T272746
Change-Id: I0b3c3251156ba6c4826abf5ba44ea93f80ebc01d
2021-01-26 04:55:03 +01:00
Bartosz Dziewoński 8f42c74985 Fix skipping to the end of paragraph, now it considers nested tags
Add yet another tree walking utility: CommentUtils::linearWalk().
Unlike TreeWalker, it allows handling the beginnings and ends of nodes
separately – kind of like parsing a XML token stream, or kind of like
VisualEditor's linear model.

(Add unit tests for this utility. The simple.html test case is copied
from [VisualEditor/VisualEditor]/demos/ve/pages/simple.html.)

Use this utility to stop skipping when we reach either a closing or
opening block node tag. Previously we'd skip over such tags inside
nested "transparent" nodes (like <a>, <del>, or apparently <font>).

Bug: T271385
Change-Id: I201a942eb3a56335e84d94e150ec2c33f8b4f4e0
2021-01-18 18:20:20 +00:00
Bartosz Dziewoński 0da00d44be Broken test cases for a big mess of <font> tags
Bug: T271385
Change-Id: I26b08923583593f40a372ef6614524f69781a87a
2021-01-18 18:20:13 +00:00
Ed Sanders 8b71a2b5dc Load site config data in CommentFormatter tests
This fixes missing reply links in arwiki test output.

Change-Id: I24d3b8371a8343c4445c716fadf0692be0924eed
2021-01-08 23:03:33 +00:00
jenkins-bot 8a6bb8efd0 Merge "Ignore outdent templates at the beginning of comments" 2021-01-04 21:48:27 +00:00
jenkins-bot fa9d729728 Merge "Change which nodes are ignored at the beginning of comments again" 2021-01-04 21:47:40 +00:00
Bartosz Dziewoński 6e37a172ae Fix detecting decorative comment frames with whitespace
As a result of 0fc71f60cd, "empty" text
nodes (containing only whitespace) at the end of the comment may be
inside the comment's range, and trying to ignore them caused the
ranges not to match and the frame not to be detected.

Now the code works whether they're inside the comment's range or not.

Add a test case for wrapped discussion comments with HTML comments and
with whitespace.

Bug: T250126
Bug: T268407
Change-Id: I2217ff5a635fd1c9c9e803f46795b1bfb3d17535
2021-01-04 20:31:33 +01:00
Bartosz Dziewoński efccc28b5d Add test case for trailing void tags (<br>)
Bug: T266288
Change-Id: I9385a5b6804fa199327f7af2cfd8275f30727f66
2021-01-04 19:22:18 +00:00
Bartosz Dziewoński 50ad5bb2b4 Ignore outdent templates at the beginning of comments
Bug: T264116
Change-Id: Iae9dbb30a1aead897cc274f655d3ecff4b297dbd
2020-12-14 21:35:56 +01:00
Bartosz Dziewoński ae920b831f Change which nodes are ignored at the beginning of comments again
While working on T270009, I noticed that <style> and <link> nodes
are treated differently, which seemed weird. Rewrite this again,
hopefully this is the last time.

The changed test cases also involve <area> and <input> nodes,
and the new results make more sense to me.

Bug: T264116
Change-Id: I3af90c84768a4b3dc53446927f4dba6f72175a2f
2020-12-14 21:33:50 +01:00
Bartosz Dziewoński 6c7a0ca9a2 Fix trying to insert start/end markers in impossible locations
Bug: T270009
Bug: T266288
Change-Id: I962128e7d9290e7b5eb49bfdb5847fd17714bae1
2020-12-14 21:09:56 +01:00
Ed Sanders fb0cc01ff8 Skip over empty inline templates (e.g. tracking templates)
Bug: T269036
Change-Id: I15e56041c1f1ecb85e9e368a9fbb07882438bf8d
2020-12-09 18:51:41 +00:00
Bartosz Dziewoński 8c9230fa10 Handle category links like comments (rendering-transparent nodes)
Bug: T269036
Change-Id: Id4321ad09907b5030881456c93da90a39bdfdd75
2020-12-08 21:39:16 +00:00
Bartosz Dziewoński 366dc2387e Add tests covering it.wp unsigned comment template
Bug: T268178
Bug: T268589
Change-Id: Icd353275b3b8191eddbff4d72984a8c2c8e6cdd6
2020-11-25 02:08:03 +01:00
Bartosz Dziewoński 0fc71f60cd Skip to the end of the paragraph if it's just text, too
We've recently decided that we want to "extend" comments until
the end of the paragraph (e36dc8e78a,
d0ae6c4e44).

However, we still had this special case that did the opposite: it
ensured that if a comment ended in the middle of a text node, the
comment would not be extended to the end of the node. Remove it.

Note the change in the test file signatures-funny-formattedreply.html,
which actually covered this case specifically.

Change-Id: Id1384bb0c6e1a5f0c70f55efcb4caa240f230f07
2020-11-25 00:48:53 +01:00
Ed Sanders d0ae6c4e44 Skip end marker "forward" until a block tag is reached
The end marker is skipped forward until an open or close
block tag is reached. In tree traversal terms this means
moving either to the next sibling, or the parent (to skip
over close tags).

Bug: T256033
Change-Id: Iaa2c588698790d576ac4f9ecc126f58a082ef6b3
2020-11-23 15:08:29 +00:00
Ed Sanders 44a1bbcc59 Fix start node for comments following headings
The general rule is that comments start after their preceding
thread item, but when that is a heading we should skip past
the entire <h[1-6]> node to avoid making section edit links
part of the first comment.

Bug: T267988
Change-Id: Ia7f1b27e0a69a9aab7c7da743bf8549479304096
2020-11-19 23:48:30 +00:00
Ed Sanders 32cd64ec6a Use Parsoid DOMCompat/DOMUtils in CommentFormatter
As CommentFormatter no longer needs HTMLFormatter, remove
the inheritance and make addReplyLinks a static method.

Testing locally this is marginally slower, going from 2.55s
to 2.9s for the CommentFormatterTest case.

Bug: T266317
Bug: T267973
Change-Id: If69749cae678a1647a138d782a32032189f55cec
2020-11-16 22:28:07 +00:00
Bartosz Dziewoński 1626242863 Don't detect comments within headings
Bug: T267068
Change-Id: Id134f15e086fd070801c4b1d836dbfbf9bf444ad
2020-11-04 21:57:33 +01:00
Bartosz Dziewoński 31f6d44bf6 Move warnings stuff from CommentItem to ThreadItem
After recent changes allowing ThreadItems to have IDs, they can now
also have warnings about duplicate IDs.

Bug: T267035
Change-Id: If3edfe34e6e29741e29fac8946a3c88badc4ab7f
2020-11-02 20:07:23 +00:00
Ed Sanders 3aca622894 Treat headings like comments now they have IDs
Use the same logic for marking ranges in the document, and ensure
that the heading range does not include section edit links or
section numberings.

Change-Id: I782caafc34fee2a822b0a17b24dd6b9528202eca
2020-10-28 12:38:18 +00:00
Bartosz Dziewoński 044bc50fb6 Fix some TODOs about test data
We avoided fixing these because it causes changes in just about all of
the test data, which is annoying when reviewing or blaming changes.

But the previous several commits also caused changes in just about all
of the test data, so we might as well do this too.

Change-Id: I83b64d83b6f12c04dc06c0cadff7cdd89417e137
2020-10-22 00:21:04 +00:00
Bartosz Dziewoński 0ddc171c8a Add oldest timestamp in the thread to heading IDs
To avoid old threads re-appearing on popular pages when someone
uses a vague title (e.g. dozens of threads titled "question" on
[[Wikipedia:Help desk]]: https://w.wiki/fbN), include the oldest
timestamp in the thread (i.e. date the thread was started) in the
heading ID.

Bug: T264478
Change-Id: If918bfd5e025248923d1939bc86916697ead95a0
2020-10-22 02:19:21 +02:00
Bartosz Dziewoński b09bbfe668 Disambiguate comments by parent ID, rather than sequential numbers
Sequential numbers aren't great because they change when an earlier
comment is archived. Parent comment/heading IDs should change less
often.

This also makes much more sense for disambiguating subsections,
e.g. a dozen identical ===Votes=== sections for a dozen proposals.

Bug: T264478
Change-Id: I466454984fd919ebef35f2b37ddb5d86dc842996
2020-10-22 02:19:21 +02:00
Bartosz Dziewoński 3137d76f40 Connect sub-threads to their parent threads
Our threads now also contain all replies to their sub-threads.
This is similar to how sections work in MediaWiki, where the parent
section also contains the content of all the lower-level sections.
We're going to need this for notifications about replies in a thread.

Bug: T264478
Change-Id: I241fc58e2088a7555942824b0f184ed21e3a8b6f
2020-10-22 02:05:02 +02:00
Bartosz Dziewoński 9ee0fd69f5 Allow headings to have IDs
Previously, only comments could have IDs, because we only needed IDs
for replying. But we might also use them for notifications soon.

Bug: T264478
Change-Id: I1bcad02bf17ab54bc5028a959543c10f0430836b
2020-10-22 02:04:28 +02:00
Bartosz Dziewoński 284115a184 Add tests for CommentFormatter
I haven't really reviewed the outputs, but at least a) they don't
crash b) they will fail if the output suddenly changes (which could
cause problems due to caching).

Bug: T252555
Change-Id: I1bbcbc5dd17ce1e24b3622062f5e8df4baf5f389
2020-10-20 04:13:25 +02:00
Bartosz Dziewoński a29c49ae70 Better way to update expected test outputs
Use an environment variable "DISCUSSIONTOOLS_OVERWRITE_TESTS".

Change-Id: I017112b7d6b1df9497f01f3f97f34e0935ca16f8
2020-10-19 23:53:30 +02:00
Bartosz Dziewoński a1dc3a4896 Correctly generate timezone abbreviations for parsing
Also, add tests covering this and the previous bug fixes in this code
(T259818, T261706).

Note that the test data added in tests/cases/ doesn't exactly match
the entire configuration of the wiki, only the parts we want to cover.
This is unlike the data in tests/data/, which was literally copied
from the relevant wikis, and which is used as input for other tests.

Bug: T265500
Change-Id: I29a59a5952f6dc9fb5910434bb6bcc9dcdaa01a9
2020-10-15 12:11:25 +00:00
jenkins-bot 485f9a4f8c Merge "Add 'id' attributes in the "wrappers" test case" 2020-10-06 22:39:42 +00:00
Bartosz Dziewoński 9775a60ace Add 'id' attributes in the "wrappers" test case
Change-Id: I40ca69586a915590e36dea45a90ab3f332931656
2020-10-02 23:47:47 +02:00
Bartosz Dziewoński ed17f640b6 Ignore other empty-ish things at the beginning of comments
Follow-up to 432a959436.

Bug: T264116
Change-Id: I0685cafab70c7e9d22f504f1a1309c9a28d6f2e1
2020-09-30 23:42:47 +02:00
jenkins-bot f7c7ba3c44 Merge "Ignore empty paragraphs at the beginning of comments" 2020-09-30 18:16:36 +00:00
jenkins-bot 5c61b23be1 Merge "Update test case to match actual output" 2020-09-30 18:16:35 +00:00
Bartosz Dziewoński 432a959436 Ignore empty paragraphs at the beginning of comments
The wikitext parser outputs `<p><br></p>` for empty paragraphs, so we
need to ignore `<br>` tags when searching for an "interesting" node
that marks the beginning of a comment. Otherwise the empty paragraphs
mess up the detection of indentation levels.

Bug: T264116
Change-Id: I84a97ab577baa7336b78935ccdc48041ecfc231a
2020-09-29 22:22:35 +02:00
Bartosz Dziewoński 385a5c1622 Update test case to match actual output
The tests pass either way because this trivial difference is ignored,
but it's annoying when trying to update changed tests.

Change-Id: I7ad1acad43e30a63843cfda4d0d08ff663ef3252
2020-09-29 22:22:35 +02:00
Ed Sanders 6b8312e610 Ignore HTML comments which are more than two lines from a reply
Bug: T264026
Change-Id: I989132d7599a7fa156dba46d87a9ed4b76322c0c
2020-09-29 11:30:03 +01:00
Bartosz Dziewoński f934e9aefd Add integration tests using pages from sr.wp
For testing our handling of language variants.

Bug: T259818
Change-Id: Id25b537fecd789640c209ff7f30e777455a3aece
2020-09-16 22:07:16 +00:00