One particular case is that Cite.php considers equal a name and
its encoding, i.e. "a & b" === "a & b". Added a new test for
this case, but blacklisted it on html2wt, wt2wt and html2html due
to a different problem with how Parsoid encodes entities. This
will be investigated separately, as a simple fix could break
unrelated cases.
Also updated tests and blacklist to the new ids.
Change-Id: I87637a1dc812a3a8f29327b9e6c0040b22a651c4
Also encode cite ids properly as now they can contain arbitrary
text. Change in blacklist due to this.
TODO: Investigate if it would be better to do this directly in
the tokenizer.
Change-Id: Ic112124e90d256d73a351d0d57fe3c7546fa065f
* Although this resolves the crashes, I'm unsatisfied with it as a
proper fix to the underlying issue. There are many places throughout
the codebase where we serialize and then parse document fragments
that should be instrumented to store and unpack data-* attributes.
Bug: T76518
Change-Id: Idca1b0a37ec924a71cb51160d000c7de9717d422
* Currently, this mimics Cite.php behavior where "a b" and "a_b"
are considered identical ids.
* Added new parser test.
* Fixed output of another test.
* Fixed section name of a commented out test.
Change-Id: I0c51404c3e659bbddfe9a8909aa6a109d368b762
* This is both faster and consistent with how we're accessing other
parsoid attributes. It's also a step towards not having this data in
the html output.
* Changes to parserTests and the blacklist are for attribute order.
* Requires upgrading domino to 1.0.18
https://github.com/fgnass/domino/pull/48
Change-Id: I1edbc260887d480adf04763b15043c374e27cceb
General changes
---------------
* Replaced the hacky 'inBlockNode' parser pipeline option with
a cleaner 'noPWrapping' option that suppresses paragraph wrapping
in sub-pipelines (ex: recursive link content, ref tags, attribute
content, etc.).
Changes to wt2html pipeline
---------------------------
* Fixed paragraph-wrapping code to ensure that there are no bare
text nodes left behind, but without removing the line-based block-tag
influences on p-wrapping. Some simplifications as well.
TODO: There are still some discrepancies around <blockquote>
p-wrapping behavior. These will be investigated and addressed
in a future patch.
* Fixed foster parenting code to ensure that fostered content is
added in p-tags where necessary rather than span-tags.
Changes to html2wt/selser pipeline
----------------------------------
* Fixed DOMDiff to tag mw:DiffMarker nodes with a is-block-node
attribute when the deleted node is a block node. This is used
during selective serialization to discard original separators
between adjacent p-nodes if either of their neighbors is a
deleted block node.
* Fixed serialization to account for changes to p-wrapping.
- Updated tag handlers for the <p> tag.
- Updated separator handling to deal with deleted block tags
and their influence on separators around adjacent p-tags.
- Updated selser output code to test whether a deleted block
tag forces nowiki escaping on unedited content from adjacent
p-tags.
Changes to parser tests / test setup
------------------------------------
* Tweaked selser test generation to ensure that text nodes are always
inserted in p-wrappers where necessary.
* Updated parser test output for several tests to introduce p-tags
instead of span-tags or missing p-tags, add html/parsoid section,
or in one case, add missing HTML output.
Parser Test Result changes
--------------------------
Newly passing
- 12 wt2html
- 1 wt2wt
- 3 html2html
- 3 html2wt
Newly failing
- 1 html2wt
"3. Leading whitespace in indent-pre suppressing contexts should not be escaped"
This is just normalization of output where multiple HTML forms
serialize to the same wikitext with a newline difference. It is not
worth the complexity to fix this.
- 1 wt2wt
""Trailing newlines in a deep dom-subtree that ends a wikitext line"
This is again normalization during serialization where an extra
unnecessary newline is introduced.
- A bunch of selser test changes.
182 +add, 188 -add => 6 fewer selser failures
- That is a lot of changes to sift through, and I didn't look at every
one of those, but a number of changes seem to be harmless, and just
a change to previously "failing" tests.
- "Media link with nasty text" test seems to have a lot of selser
changes, but the HTML generated by Parsoid seems to be "buggy" with
interesting DSR values as well. That needs investigation separately.
- "HTML nested bullet list, closed tags (bug 5497) [[3,3,4,[0,1,4],3]]"
has seen a degradation where a dirty diff got introduced.
Haven't investigated carefully why that is so.
Change-Id: Ia9c9950717120fbcd03abfe4e09168e787669ac4
* Nested <ref> tag support was broken in 69b6ec4d since that patch
effectively processed <refs> only on the final DOM rather than
extracted <ref> information in sub-pipelines. In doing so, it
broke support for <ref> in <ref> tags which are supported by {{efn}}
templates and used as follows.
{{efn|A clarification.{{sfn|Smith|2009|p=2}}}}
The {{sfn}} tpl generates a <ref> tag inside another <ref> tag
generated by the {{efn}} tpl.
* This patch fixes the breakage by processing <ref> content after it
is extracted (in case it is known to have nested <ref> tags).
* This fixes the rendering on enwiki/Otto_I%2C_Holy_Roman_Emperor
and is an improvement over when it was broken. The nested <ref>
gets id 18 (just like the Cite.php handling) whereas it used to
get id 1 before the breakage.
* TODO:
- Figure out a way to add a test for this.
Change-Id: Ib82b75f66249b2133a123adbe6fd7acbfd8ec8fb
* Process <ref> and <references> tag on the top-level DOM only
and ignore the generateRefs pass when processing other content.
* This required a few fixes:
- ensure that DOMPostProcessor knows about the top-level.
- ensure that DOMVisitor knows about the top-level.
- cleanup pass leaves behind the ref-marker metas from DOMs from
non top-level content.
- process nested references content.
* One of the references tests had incorrect parsed output. That test
has been updated to reflect the correct output from this patch.
* Barack Obama seems to now have the correct numbering on references.
Change-Id: I5465721d2fc715f2168f267e773a446bc37d198b
* Keep track of table nesting in token stream patcher and use it to
convert <td>, <tr>, and <th> tags to plain strings.
* This fix is only enabled on the top-level token stream.
To support this, fixed the resetState function in the parser
construction code to pass in a toplevel flag which lets the
token stream patcher know the context it is in.
* Fixes 29 (wt2html,wt2wt,html2html,selser) tests and improves
results of 1 previously blacklisted tests. The failing selser
test is actually a false failure because selser is more accurate
than non-selser wts.
* Consolidated a few separate tests into a single test that covers
all this functionality.
- This new test fails wt2wt and html2wt modes because serializer
uses tokenizer information which continues to return table tokens
and results in <nowiki> wrappers.
Bug: 66489
Bug: 66498
Change-Id: I9f42354ea9efb0f8adfc96c23760012220d00dd4
The Cite extension does not currently handle resetState calls in
sub-pipelines, and relies on sharing a single Cite instance between all
pipelines. Fixing this is a longer project, so this patch works around the
issue for now by passing a flag indicating resetState calls in sub-pipelines
and ignoring the call in Cite in that case.
Change-Id: If3d426a5311a55d1c1530860d2b665d3681f1aa9
* Entities in ref name weren't expected
* Fixes the crash from arwiki:تأثير_الدمعة_السوداء
* Makes use of the fix from de3642b8dd4a804ac654f2943a900496f2c8b3f3
Bug: 63790
Change-Id: Icb8781b4d9decc5a8b115d0b11def4d18f5d5025
* Thus far, <references> tag content was being parsed to
stage 2 and merged into main pipeline. This patch takes
this all the way to DOM. This required some tweaks to
handling of <ref>s nested inside <references>.
* Fixed up a buggy parser test in the bargain -- the old parsoid
result was buggy as well. I verified output in the enwp
sandbox.
Change-Id: Iff6c528066b71ce1b00dd769910a04ee66623340
* The current fix is a hack to fix dsr issues right away.
Meanwhile, will invesitigate a fix that will involve processing
<references> in its own subpipeline and persisting state into
the top-level page.
* Fixes a known selser failure from bug 62025.
Change-Id: I0f80d68e927f500939a44af401cc73c07e24721f
* Renamed buildDOMFragmentForTokenStream --> buildDOMFragmentTokens
and made env. the first arg.
* Added documentation to buildDOMFragmentTokens and encapsulateHTML
Change-Id: I7eccfd3f4dc5b4b91d20d1d24d98ec514df6dfbc
* Removed manager and passed in env and parent-frame to all
utilities that process content in new pipelines.
* Added more documentation to mediawiki.Util.js.
* Renamed processAttributeToDOM to a more appropriate name.
* Added pipelineFactory property to env and used that to
construct parsing pipelines everywhere.
Change-Id: Ic612e5630d19d4e3f5d6388bc5cd117d337fd799
* This patch adds a flag to DOM fragment unpacking to update
fragment DSR in cases where the fragment needs it (references
block + reused template/extension/images content). In other
cases of dom-fragment use, the DSR should not be updated.
Ex: All cases where fragments are used to implement parsing
scopes (all A-tag content currently) since DSR computation
is set up with offsets in the top-level source.
This is not just an optimization, but a correctness issue
since the fragment unwrapper always sets the fragment DSR
on the first node which would be incorrect in scenarios where
the fragment DOM has multiple top-level nodes.
* Parser test runs now have better DSR values in certain cases.
Change-Id: If1f5bf98dab246a3c8a1869b38335e90268cb5c5
This reads better than manually testing the constructor, and often
leads to terser code since we don't have to check whether the argument
is an non-null object before querying the constructor field.
Change-Id: I53ec87d6e80d658aa3d26dc2b613dc6c58e2d026
In particular, use `Array.isArray` instead of `$.isArray`, and
`Object.assign` instead of `$.extend`. `Object.assign` operates only on own
properties, so use `Object.create` on the prototype where necessary to
get inherited properties. `Object.assign` does a simple assignment and
is appropriate in most places, but be careful if we ever install
getters/setters on a prototype.
Implement `Util.clone()` from scratch to have a jquery-compatible deep
clone operation. In particular, this needs to ignore objects which
aren't "plain objects", so we don't try to clone DOM nodes. Our
definition of a "plain object" is compatible with jquery/zepto.js, and
is thus something of a hack. We should eventually replace this with a
`console.assert()` and remove/rewrite the places where we try to clone
objects which contain DOM trees and other cruft.
Change-Id: I88c8fe41a9be84c167d5a0ea1187fd258f077968
* For now it stores in .dataobject, freeing up .data to
handle <object> elements.
* Adds a test that crashes master.
Bug: 57394
Change-Id: I4207d76ad9dab660e890008b2ee5014554ce52c8
* Adds an index of all the references on a page in order to avoid
repeating attrs when multiple <references /> tags are present.
* Update tests to reflect the new behaviour.
Bug: 59782
Change-Id: Ia44bf59a9304788aca170041d3b85f53557151fc
