* Although this resolves the crashes, I'm unsatisfied with it as a
proper fix to the underlying issue. There are many places throughout
the codebase where we serialize and then parse document fragments
that should be instrumented to store and unpack data-* attributes.
Bug: T76518
Change-Id: Idca1b0a37ec924a71cb51160d000c7de9717d422
* Currently, this mimics Cite.php behavior where "a b" and "a_b"
are considered identical ids.
* Added new parser test.
* Fixed output of another test.
* Fixed section name of a commented out test.
Change-Id: I0c51404c3e659bbddfe9a8909aa6a109d368b762
* This is both faster and consistent with how we're accessing other
parsoid attributes. It's also a step towards not having this data in
the html output.
* Changes to parserTests and the blacklist are for attribute order.
* Requires upgrading domino to 1.0.18
https://github.com/fgnass/domino/pull/48
Change-Id: I1edbc260887d480adf04763b15043c374e27cceb
General changes
---------------
* Replaced the hacky 'inBlockNode' parser pipeline option with
a cleaner 'noPWrapping' option that suppresses paragraph wrapping
in sub-pipelines (ex: recursive link content, ref tags, attribute
content, etc.).
Changes to wt2html pipeline
---------------------------
* Fixed paragraph-wrapping code to ensure that there are no bare
text nodes left behind, but without removing the line-based block-tag
influences on p-wrapping. Some simplifications as well.
TODO: There are still some discrepancies around <blockquote>
p-wrapping behavior. These will be investigated and addressed
in a future patch.
* Fixed foster parenting code to ensure that fostered content is
added in p-tags where necessary rather than span-tags.
Changes to html2wt/selser pipeline
----------------------------------
* Fixed DOMDiff to tag mw:DiffMarker nodes with a is-block-node
attribute when the deleted node is a block node. This is used
during selective serialization to discard original separators
between adjacent p-nodes if either of their neighbors is a
deleted block node.
* Fixed serialization to account for changes to p-wrapping.
- Updated tag handlers for the <p> tag.
- Updated separator handling to deal with deleted block tags
and their influence on separators around adjacent p-tags.
- Updated selser output code to test whether a deleted block
tag forces nowiki escaping on unedited content from adjacent
p-tags.
Changes to parser tests / test setup
------------------------------------
* Tweaked selser test generation to ensure that text nodes are always
inserted in p-wrappers where necessary.
* Updated parser test output for several tests to introduce p-tags
instead of span-tags or missing p-tags, add html/parsoid section,
or in one case, add missing HTML output.
Parser Test Result changes
--------------------------
Newly passing
- 12 wt2html
- 1 wt2wt
- 3 html2html
- 3 html2wt
Newly failing
- 1 html2wt
"3. Leading whitespace in indent-pre suppressing contexts should not be escaped"
This is just normalization of output where multiple HTML forms
serialize to the same wikitext with a newline difference. It is not
worth the complexity to fix this.
- 1 wt2wt
""Trailing newlines in a deep dom-subtree that ends a wikitext line"
This is again normalization during serialization where an extra
unnecessary newline is introduced.
- A bunch of selser test changes.
182 +add, 188 -add => 6 fewer selser failures
- That is a lot of changes to sift through, and I didn't look at every
one of those, but a number of changes seem to be harmless, and just
a change to previously "failing" tests.
- "Media link with nasty text" test seems to have a lot of selser
changes, but the HTML generated by Parsoid seems to be "buggy" with
interesting DSR values as well. That needs investigation separately.
- "HTML nested bullet list, closed tags (bug 5497) [[3,3,4,[0,1,4],3]]"
has seen a degradation where a dirty diff got introduced.
Haven't investigated carefully why that is so.
Change-Id: Ia9c9950717120fbcd03abfe4e09168e787669ac4
* Nested <ref> tag support was broken in 69b6ec4d since that patch
effectively processed <refs> only on the final DOM rather than
extracted <ref> information in sub-pipelines. In doing so, it
broke support for <ref> in <ref> tags which are supported by {{efn}}
templates and used as follows.
{{efn|A clarification.{{sfn|Smith|2009|p=2}}}}
The {{sfn}} tpl generates a <ref> tag inside another <ref> tag
generated by the {{efn}} tpl.
* This patch fixes the breakage by processing <ref> content after it
is extracted (in case it is known to have nested <ref> tags).
* This fixes the rendering on enwiki/Otto_I%2C_Holy_Roman_Emperor
and is an improvement over when it was broken. The nested <ref>
gets id 18 (just like the Cite.php handling) whereas it used to
get id 1 before the breakage.
* TODO:
- Figure out a way to add a test for this.
Change-Id: Ib82b75f66249b2133a123adbe6fd7acbfd8ec8fb
* Process <ref> and <references> tag on the top-level DOM only
and ignore the generateRefs pass when processing other content.
* This required a few fixes:
- ensure that DOMPostProcessor knows about the top-level.
- ensure that DOMVisitor knows about the top-level.
- cleanup pass leaves behind the ref-marker metas from DOMs from
non top-level content.
- process nested references content.
* One of the references tests had incorrect parsed output. That test
has been updated to reflect the correct output from this patch.
* Barack Obama seems to now have the correct numbering on references.
Change-Id: I5465721d2fc715f2168f267e773a446bc37d198b
* Keep track of table nesting in token stream patcher and use it to
convert <td>, <tr>, and <th> tags to plain strings.
* This fix is only enabled on the top-level token stream.
To support this, fixed the resetState function in the parser
construction code to pass in a toplevel flag which lets the
token stream patcher know the context it is in.
* Fixes 29 (wt2html,wt2wt,html2html,selser) tests and improves
results of 1 previously blacklisted tests. The failing selser
test is actually a false failure because selser is more accurate
than non-selser wts.
* Consolidated a few separate tests into a single test that covers
all this functionality.
- This new test fails wt2wt and html2wt modes because serializer
uses tokenizer information which continues to return table tokens
and results in <nowiki> wrappers.
Bug: 66489
Bug: 66498
Change-Id: I9f42354ea9efb0f8adfc96c23760012220d00dd4
The Cite extension does not currently handle resetState calls in
sub-pipelines, and relies on sharing a single Cite instance between all
pipelines. Fixing this is a longer project, so this patch works around the
issue for now by passing a flag indicating resetState calls in sub-pipelines
and ignoring the call in Cite in that case.
Change-Id: If3d426a5311a55d1c1530860d2b665d3681f1aa9
* Entities in ref name weren't expected
* Fixes the crash from arwiki:تأثير_الدمعة_السوداء
* Makes use of the fix from de3642b8dd4a804ac654f2943a900496f2c8b3f3
Bug: 63790
Change-Id: Icb8781b4d9decc5a8b115d0b11def4d18f5d5025
* Thus far, <references> tag content was being parsed to
stage 2 and merged into main pipeline. This patch takes
this all the way to DOM. This required some tweaks to
handling of <ref>s nested inside <references>.
* Fixed up a buggy parser test in the bargain -- the old parsoid
result was buggy as well. I verified output in the enwp
sandbox.
Change-Id: Iff6c528066b71ce1b00dd769910a04ee66623340
* The current fix is a hack to fix dsr issues right away.
Meanwhile, will invesitigate a fix that will involve processing
<references> in its own subpipeline and persisting state into
the top-level page.
* Fixes a known selser failure from bug 62025.
Change-Id: I0f80d68e927f500939a44af401cc73c07e24721f
* Renamed buildDOMFragmentForTokenStream --> buildDOMFragmentTokens
and made env. the first arg.
* Added documentation to buildDOMFragmentTokens and encapsulateHTML
Change-Id: I7eccfd3f4dc5b4b91d20d1d24d98ec514df6dfbc
* Removed manager and passed in env and parent-frame to all
utilities that process content in new pipelines.
* Added more documentation to mediawiki.Util.js.
* Renamed processAttributeToDOM to a more appropriate name.
* Added pipelineFactory property to env and used that to
construct parsing pipelines everywhere.
Change-Id: Ic612e5630d19d4e3f5d6388bc5cd117d337fd799
* This patch adds a flag to DOM fragment unpacking to update
fragment DSR in cases where the fragment needs it (references
block + reused template/extension/images content). In other
cases of dom-fragment use, the DSR should not be updated.
Ex: All cases where fragments are used to implement parsing
scopes (all A-tag content currently) since DSR computation
is set up with offsets in the top-level source.
This is not just an optimization, but a correctness issue
since the fragment unwrapper always sets the fragment DSR
on the first node which would be incorrect in scenarios where
the fragment DOM has multiple top-level nodes.
* Parser test runs now have better DSR values in certain cases.
Change-Id: If1f5bf98dab246a3c8a1869b38335e90268cb5c5
This reads better than manually testing the constructor, and often
leads to terser code since we don't have to check whether the argument
is an non-null object before querying the constructor field.
Change-Id: I53ec87d6e80d658aa3d26dc2b613dc6c58e2d026
In particular, use `Array.isArray` instead of `$.isArray`, and
`Object.assign` instead of `$.extend`. `Object.assign` operates only on own
properties, so use `Object.create` on the prototype where necessary to
get inherited properties. `Object.assign` does a simple assignment and
is appropriate in most places, but be careful if we ever install
getters/setters on a prototype.
Implement `Util.clone()` from scratch to have a jquery-compatible deep
clone operation. In particular, this needs to ignore objects which
aren't "plain objects", so we don't try to clone DOM nodes. Our
definition of a "plain object" is compatible with jquery/zepto.js, and
is thus something of a hack. We should eventually replace this with a
`console.assert()` and remove/rewrite the places where we try to clone
objects which contain DOM trees and other cruft.
Change-Id: I88c8fe41a9be84c167d5a0ea1187fd258f077968
* For now it stores in .dataobject, freeing up .data to
handle <object> elements.
* Adds a test that crashes master.
Bug: 57394
Change-Id: I4207d76ad9dab660e890008b2ee5014554ce52c8
* Adds an index of all the references on a page in order to avoid
repeating attrs when multiple <references /> tags are present.
* Update tests to reflect the new behaviour.
Bug: 59782
Change-Id: Ia44bf59a9304788aca170041d3b85f53557151fc
- Make sure void element attributes are sanitized
- Drop attributes on end tags
- Fix Cite to use data-parsoid instead of invalid attributes to pass
information from the token stream to the DOM
Change-Id: If236d4c9197b12ff86a607763f25ed7677623bc4
* isForeignContent now flags content that is not originally
present in the top-level page => transclusion and extension
content. During DOM fragment unpacking, top-level children
of the fragment are always re-assigned the fragment wrapper's
about-id.
* For content that is not flagged isForeignContent (figures that
are reused from cache, and all other users of dom-fragment
including scoped parsing), the fragment DOM is walked and all
about-ids are reassigned fresh about ids.
* From an earlier patchset (now unrelated to the bug fixes here)
* Renamed older badly named isEncapsulatedElt helper
to isFirstEncapsulationWrapperNode.
* Left behind helper to recognize when a node is an
encapsulation wrapper (in case it comes handy later on).
* No change in parser tests.
TODO: We need a parser test with extensions in them.
{{echo|<div>foo</div> <math>1+2</math> <div>bar</div>}} would
not RT properly before this patch because the inner <math> node
did not have the about-id as the surrounding <div>s. This patch
fixes this.
This should also fix a lot of the <timeline> regressions seen
in latest RT testing.
Change-Id: I595e9f46d8ce789a05e4b7dd8b58e443e4c02f98
* Smaller data-mw with adaptive quoting.
* Updated parserTests with new output
* Also add a serializeChildren method that works similar to innerHTML.
* Move HTML normalization to DOMUtils to avoid cyclic dependencies
* Disable non-IE WS to space normalization pass, needs to be improved. See bug
55588. This leads to some new passes and some new fails.
Change-Id: I392881bd8a0cfd9f116e70e9a52d1ef14bbd568b
* Got rid of unnecessary/buggy concat of EOFTk in ext.Cite.js
to wikitext string (concat is only required when processing
a token array from stage 2 onwards).
* Moved temporary node.data.parsoid.wrapper flag to a tmp object
which is then stripped in the cleanup pass. Other temporary
flags from other passes could also be moved here.
* Added a helper to test if a typeOf has a parsoid-generated
value.
Change-Id: I2024c575b5293e5155fc8e2915a34b5fb8394671
* Added mechanism to parse a set of tokens in its own parsing
scope all the way to DOM (which in turn restricts token
transformations to just that scope). This is equivalent to
enforcing well-balanced requirements in restricted contexts
(Ex: link content, image captions for starters). This also
provides an option of enforcing balanced templates in certain
contexts.
This patch applies it to link content and image captions.
Deleted the hacky closeUnclosedBlockTags code for dealing
with bad HTML in captions.
* Refactored common/duplicate code out of Cite handler and
Template handler.
* Updated DSR handling for dom-fragments which eliminates the
warnings about cs/s dsr mismatches seen on image reuse.
it:Dalmine used to get a bunch of DSR inconsistency warnings
when dom-fragments were reused (reported in bug 53071) and
are now fixed with this patch.
* There is still one big hole in DOM fragment unpacking that
has to be fixed. This pertains to improper tag nesting that
will be broken up by the tree builder. The DOM fragment unpacker
has to recognize these scenarios and fix up the DOM (either
by fixing up the final DOM or by stripping mis-nested tags
in the DOM fragment being unpacked).
This patch has an incomplete hack for this that addresses
the common-use case of nested a-tags generated by wikitext
of this form: "[http://foo.bar a [[Wikilink] here]".
Bug: 54454
Bug: 49942
Bug: 44476
Bug: 47326
Change-Id: I33ee38bc43743125b705ac821b339586593dbef7
* This eliminates repetitive code from a few places and might be
useful for future patches as well.
Change-Id: I383ed253a2fa20c1b7429689d17cde176751e29a
* The initialization of nestedRefCollectionMap[references-id]
in handleReferences was the wrong place to initialize it since
on pages with multiple <references /> tags, the map can get
reset after a references tag is processed. So, for later
references tags, the entry for those will get cleared which
led to crashes.
Ex: es:Abderramán_I and couple others found in RT testing.
* This patch fixes the crasher.
* Added a new parser test with multiple references tags that
captures this scenario. Regenerated selser changes file.
Change-Id: I64402e77996486b099acda114f2e250258b0048e
* Track <ref>s birthed in <references> tags and capture their
HTML and set <references>.data-mw.body.html to the concatenated
HTML for all those child <ref> tags.
* No change in parser-tests because our test normalization code
strips data-mw before comparing results. Since this patch
only changes data-mw output for "References: 5. ..." test,
there is no change. In any case, this patch updates the
data-mw output for the test in question so that if we do
modify our normalization routines, the test will still pass.
Change-Id: Iaea752dec03e5e7f0d36baf3b7a9973f1b85e8a5
* This patch now supports any template that generates nested refs
where the top-level token is just a single-ref.
Ex: {{efn|New timetable{{sfn|Vallance|1991|p=31}}}}
* The solution in this patch removes the hack from the previous
version (53bbcbb3). As documented in the bug report, this
solution essentially "trusts" template authors and lets
nested refs through when they come from templates.
But, top-level nested refs are still not allowed.
<ref>foo<ref>bar</ref>baz</ref> in the toplevel page
will still generate a single reference with content
"foo<ref>bar</ref>baz"
* Updated the nested ref test output
- Regenerated selser changes because of the changed output.
- Regenerated blacklist because of the changed selser changes.
* This now handles:
- https://en.wikipedia.org/wiki/User:Edgepedia/VE/GNoSR
- https://en.wikipedia.org/wiki/Phellinus ellipsoideus
Change-Id: I627955f0be1c5e2bafc49647c94c2be68ce711a8
* There is another unrelated issue on that page ==> It has
a couple instances of <ref>text<ref> which slurps the entire
rest of the page into a 10K string (on which the regexp failed).
Change-Id: Ia4c498b28042de5bfe9b0945c0efaa8c6cb81ea0
* source and group attributes were being added as HTML attrs.
on the ol-node which is invalid. Plus, after refererences
generation, source attr was being deleted which causes crashers
in production if references output is reused from cache.
* Moved both attributes to data.parsoid
* Removed group="" from parserTests that I erroneously added
in that previous commit.
* Minor cleanup in DOMPostProcessor.
Change-Id: I85f4ad0a311d653d7ba3e010c7a5db028b643e9f
* So far, references wikitext was being replaced with a meta marker
token that participated as an inline token in p-wrapping. As a
result, in some cases, a p-tag was wrapping the references ol-node
which is buggy HTML. When VE loaded this DOM, the p-node was split
around the ol-node and duplicated DSR which led to duplicated
output when selser ran on this.
* One way of fixing this would have been to add a special case in
the paragraph wrapper to treat the mw:Extensions/References/Marker
meta tag as a block token.
* A better solution with an eye towards the longer-term is to emit
mw:DOMFragment wrapper for the references tag with the content
being an ol-node. The dom-fragment unpacking code then takes care
of p-wrapping issues for the ol-node. This fixes the bug in this
case.
* Also fixed parser tests output where group attribute was missing
on some tests (which indicated that the older code was not emitting
the group attribute on references tag).
* No change in parser tests.
Change-Id: I073b2e68667d577c75cad07d19cea2b19d0e89fe
* After trying various hacks, came up with a relatively simple
fix/hack to support nested refs.
The fix looks for {{#tag:ref..}} and short-circuits full
pipeline expansion and converts that to an extension tag
in place.
* Tested with the following wikitext which parses and RTs correctly
A <ref name='foo' />
B {{#tag:ref|nested ref <ref>bar</ref> |name=foo}}
<references />
* Also tested on en:Fomitiporia_ellipsoidea from the bug report
and verified correct parse and round tripping.
* Verified that the nested ref in <ref> foo <ref>bar</ref> </ref>
continues to be parsed as plain text.
* No change in parser test results -- have to make another
round of updates to parser tests.
Change-Id: I43bb8b710bd10a9ddbea27818ff8aaf97ddb3fdc
* See example below that clarifies the problem before this patch
-------------------
$ echo "{{#tag:ref|foo}} {{echo|<references />}}" | node parse --extensions ref,references --wt2wt --editMode true
<ref>foo</ref> <references />
-------------------
* The problem is that references are generated after dom fragments
are unpacked in the DOM which lost the mw:Transclusion typeof and
data-mw that had been set on the ref and references tags.
* This is a quick fix to prevent some dirty diffs with #tag:ref being
reported on en-wp. Longer term, we do have a plan to use DOMFragment
encapsulation for refs and references as well and splitting up
references processing differently than is being done currently.
Change-Id: I4186cae93b9882d367c7d4efecc092607fe17c61
* While debugging reports of dirty diffs on some pages (en:Bleak House
specifically), it took me a while to notice that some mw:Transclusion
had about="mwt5" style ids (introduced during template expansion reuse)
whereas all other about ids had about="#mwt5" (note the # char) style ids.
While this by itself shouldn't cause dirty diffs since DOM-diff ignores
about ids, this could potentially introduce introduce other bugs elsewhere
if we start using/comparing about ids.
Fixed all uses of "#" + env.newObjectId() with env.newAboutId and let
env prefix the "#" key.
Change-Id: I74d50ae155f5d24af95c07da15b14eb990cf2891
* Nested ref tags are not supported anymore.
* Turned off pre and p-wrap handlers on ref content since the native
cite extension seems to not do any of this on ref content.
* No change in parser test results (because there are no tests yet).
Other cleanup:
* Removed the 'inBlockToken' hack from Cite since this is not
necessary anymore.
TODO: The use of this flag in TemplateHandler may not be needed
either. Verify and get rid of it.
* Leading whitespace in ref-content is still removed but this may
not be strictly necessary.
Change-Id: I3406236032abe36099a1e420f443277a95fe597b
* The page in question "es:Estadio_Deportivo_Cali" had a ref
with name "constructor", and this name was used as an object
property key and this clashed with the predefined property
constructor.
* The reliable solution here is to prefix ref-name and ref-group
with a string and use it to prevent clashes.
Change-Id: Ib5cf7cce6fa4acd88e3d49ca9d4390a61bfddd7e
* Temporarily hacked sanitizer to pass through typeof attribute
so that mw:DOMFragment wrapper for extension tags can get to
the DOM post processor and get unwrapped.
* Implemented getArgDict for the extension handler since data-mw
for extensions has a different form than that for templates.
* Extracted common functionality into Util.js and used it in Cite.js
and ExtensionHandler.js
* Tested with timeline extension (test snippet below) and verified
that it parses and RTs both with editMode true and false.
TODO: Long overdue. Extension testing.
--------
<timeline>
ImageSize = width:250 height:200
PlotArea = left:40 right:10 top:10 bottom:20
TimeAxis = orientation:horizontal
AlignBars = justify
Colors =
id:gray1 value:gray(0.9)
DateFormat = yyyy
Period = from:1960 till:2010
ScaleMajor = unit:year increment:10 start:1960
PlotData =
bar:3000 color:gray1 width:1
from:start till:end
bar:2000 color:gray1
from:start till:end
bar:1000 color:gray1
from:start till:end
bar:0 color:gray1
LineData =
layer:front
points:(48,96)(84,111) color:blue width:2 #1962 tot 1968. Inwonertal 1962: 1348 1968: 1610
points:(84,111)(100,112) color:blue width:2 #1975: 1627
points:(100,112)(128,116) color:blue width:2 #1982: 1699
points:(128,116)(160,135) color:blue width:2 #1990: 2036
points:(160,135)(196,146) color:blue width:2 #1999: 2217
points:(196,146)(228,158) color:blue width:2 #2004/5
</timeline>
--------
Change-Id: Ia8d2f82e893047e2447cf809e04cc7f508f5899b
* data-mw wasn't being emitted for references -- there was a FIXME
for it.
* Tested fixes on example from 3c88b310.
* Removed meta-placeholder that was being emitted for <references />
tags without any refs to emit since VE might add references and
this wont be valid anymore. Serializer can also handle references
output without any content. So, no need for that hack anymore.
Verified by testing with "<references />" input
Change-Id: I3d2852f2c6a88bf22145add9b2173fd99d152775
