* So far, references wikitext was being replaced with a meta marker
token that participated as an inline token in p-wrapping. As a
result, in some cases, a p-tag was wrapping the references ol-node
which is buggy HTML. When VE loaded this DOM, the p-node was split
around the ol-node and duplicated DSR which led to duplicated
output when selser ran on this.
* One way of fixing this would have been to add a special case in
the paragraph wrapper to treat the mw:Extensions/References/Marker
meta tag as a block token.
* A better solution with an eye towards the longer-term is to emit
mw:DOMFragment wrapper for the references tag with the content
being an ol-node. The dom-fragment unpacking code then takes care
of p-wrapping issues for the ol-node. This fixes the bug in this
case.
* Also fixed parser tests output where group attribute was missing
on some tests (which indicated that the older code was not emitting
the group attribute on references tag).
* No change in parser tests.
Change-Id: I073b2e68667d577c75cad07d19cea2b19d0e89fe
* After trying various hacks, came up with a relatively simple
fix/hack to support nested refs.
The fix looks for {{#tag:ref..}} and short-circuits full
pipeline expansion and converts that to an extension tag
in place.
* Tested with the following wikitext which parses and RTs correctly
A <ref name='foo' />
B {{#tag:ref|nested ref <ref>bar</ref> |name=foo}}
<references />
* Also tested on en:Fomitiporia_ellipsoidea from the bug report
and verified correct parse and round tripping.
* Verified that the nested ref in <ref> foo <ref>bar</ref> </ref>
continues to be parsed as plain text.
* No change in parser test results -- have to make another
round of updates to parser tests.
Change-Id: I43bb8b710bd10a9ddbea27818ff8aaf97ddb3fdc
* See example below that clarifies the problem before this patch
-------------------
$ echo "{{#tag:ref|foo}} {{echo|<references />}}" | node parse --extensions ref,references --wt2wt --editMode true
<ref>foo</ref> <references />
-------------------
* The problem is that references are generated after dom fragments
are unpacked in the DOM which lost the mw:Transclusion typeof and
data-mw that had been set on the ref and references tags.
* This is a quick fix to prevent some dirty diffs with #tag:ref being
reported on en-wp. Longer term, we do have a plan to use DOMFragment
encapsulation for refs and references as well and splitting up
references processing differently than is being done currently.
Change-Id: I4186cae93b9882d367c7d4efecc092607fe17c61
* While debugging reports of dirty diffs on some pages (en:Bleak House
specifically), it took me a while to notice that some mw:Transclusion
had about="mwt5" style ids (introduced during template expansion reuse)
whereas all other about ids had about="#mwt5" (note the # char) style ids.
While this by itself shouldn't cause dirty diffs since DOM-diff ignores
about ids, this could potentially introduce introduce other bugs elsewhere
if we start using/comparing about ids.
Fixed all uses of "#" + env.newObjectId() with env.newAboutId and let
env prefix the "#" key.
Change-Id: I74d50ae155f5d24af95c07da15b14eb990cf2891
* Nested ref tags are not supported anymore.
* Turned off pre and p-wrap handlers on ref content since the native
cite extension seems to not do any of this on ref content.
* No change in parser test results (because there are no tests yet).
Other cleanup:
* Removed the 'inBlockToken' hack from Cite since this is not
necessary anymore.
TODO: The use of this flag in TemplateHandler may not be needed
either. Verify and get rid of it.
* Leading whitespace in ref-content is still removed but this may
not be strictly necessary.
Change-Id: I3406236032abe36099a1e420f443277a95fe597b
* The page in question "es:Estadio_Deportivo_Cali" had a ref
with name "constructor", and this name was used as an object
property key and this clashed with the predefined property
constructor.
* The reliable solution here is to prefix ref-name and ref-group
with a string and use it to prevent clashes.
Change-Id: Ib5cf7cce6fa4acd88e3d49ca9d4390a61bfddd7e
* Temporarily hacked sanitizer to pass through typeof attribute
so that mw:DOMFragment wrapper for extension tags can get to
the DOM post processor and get unwrapped.
* Implemented getArgDict for the extension handler since data-mw
for extensions has a different form than that for templates.
* Extracted common functionality into Util.js and used it in Cite.js
and ExtensionHandler.js
* Tested with timeline extension (test snippet below) and verified
that it parses and RTs both with editMode true and false.
TODO: Long overdue. Extension testing.
--------
<timeline>
ImageSize = width:250 height:200
PlotArea = left:40 right:10 top:10 bottom:20
TimeAxis = orientation:horizontal
AlignBars = justify
Colors =
id:gray1 value:gray(0.9)
DateFormat = yyyy
Period = from:1960 till:2010
ScaleMajor = unit:year increment:10 start:1960
PlotData =
bar:3000 color:gray1 width:1
from:start till:end
bar:2000 color:gray1
from:start till:end
bar:1000 color:gray1
from:start till:end
bar:0 color:gray1
LineData =
layer:front
points:(48,96)(84,111) color:blue width:2 #1962 tot 1968. Inwonertal 1962: 1348 1968: 1610
points:(84,111)(100,112) color:blue width:2 #1975: 1627
points:(100,112)(128,116) color:blue width:2 #1982: 1699
points:(128,116)(160,135) color:blue width:2 #1990: 2036
points:(160,135)(196,146) color:blue width:2 #1999: 2217
points:(196,146)(228,158) color:blue width:2 #2004/5
</timeline>
--------
Change-Id: Ia8d2f82e893047e2447cf809e04cc7f508f5899b
* data-mw wasn't being emitted for references -- there was a FIXME
for it.
* Tested fixes on example from 3c88b310.
* Removed meta-placeholder that was being emitted for <references />
tags without any refs to emit since VE might add references and
this wont be valid anymore. Serializer can also handle references
output without any content. So, no need for that hack anymore.
Verified by testing with "<references />" input
Change-Id: I3d2852f2c6a88bf22145add9b2173fd99d152775
* Serializer was complaining about missing source for these tags.
Tweaked code to serialize these back to self-closed tag form.
* Tested with which rts this with the regular serializer in edit mode.
echo "one <ref name='foo'>bar</ref> two<ref name='foo' />\n<references/>" | node parse --extensions ref,references --wt2wt
Change-Id: Iaac63b26660d8b03af937d00d04540c3b8c0c86c
This patch adds the capability to expand individual transclusions all the way
to DOM, which enforces properly nested templates. This is also needed for
DOM-based template re-expansion, which the VE folks would like to use for
template editing.
In normal parsing, DOM-based expansion is currently disabled by default for
transclusions, and enabled by default for extensions. The decision whether to
balance a particular transclusion can be based on a statistics-based
classification of templates into balanced and unbalanced ones. The advantage
of this approach is consistency in behavior for old revisions. Another
alternative is to wrap unbalanced transclusions into <domparse> tags which
disable DOM-based parsing for all transclusions in its content. This has the
advantage that all special cases can be handled after tag insertion, and that
balancing could also be enforced in the PHP parser using an extension.
Other major changes:
* Renamed transclusion content typeof from mw:Object/Template to
mw:Transclusion
* Renamed extension typeof from mw:Object/Extensions/foo to mw:Extension/foo
and switched Cite to use lower-case ref/references too
Other minor changes:
* Only apply template encapsulation algorithm on DOM to mw:Transclusion and
mw:Param objects, and no longer to all mw:Object/* elements. We should
probably switch to a more explicit encapsulation type instead. Maybe
something like mw:EncapStart/Transclusion and mw:EncapEnd/Transclusion
instead of the current mw:Transclusion and mw:Transclusion/End, so that
stripping those metas out selectively is easy?
* Changed the DOMTraverser logic to let handlers explicitly return the next
element to handle. Useful when several siblings are handled, as is the case
for the fragment unwrapper for example, and avoids some cases where deleted
nodes were still being processed.
* Changed Cite to use mw:Extension/Ref{,erences} as all other extensions.
* Make sure we round-trip gallery when the PHP preprocessor is not used
Five parsoid-specific wt2html tests are failing due to the typeof change.
They will be fine once those tests are adjusted. For now, adding them
to parser tests blacklist.
TODO:
* Switch the Cite extension to the generic DOMFragment mechanism instead of
marker tokens.
Change-Id: I64b560a12695256915d2196d0646347381400e80
* PHP parser normalizes whitespace in attribute values of all
extension tags. This patch mimics that behavior.
* Removed special case whitespace stripping for references.group
attribute since that is now handled by the general case.
* As a result, <ref name='foo' /> and <ref name=' foo ' /> are
and treated as being identical.
Change-Id: I34009ab6662d05453fe46379c58d6e989f296958
* Extensions were being serialized back to original wikitext src.
* This patch now uses data-mw content to serialize to extension
wikitext.
* Fixed Cite to add attributes for ref to data-mw so that attributes
roundtrip back (WS and quotes are normalized).
* TODO:
1. add data-mw for <references> in Cite.js
2. add data-mw for all other extensions in the extension handler
Change-Id: If3c35398a409f4775204407b3dcf2442ec2fb4e5
* Updated references output DOM to match spec better.
* Also refactored some repetitive DOM code into helper functions.
* Turned on tpl encapsulation inside extensions.
* Verified output on the example in the spec and the example in
commit 3c88b310.
Change-Id: I1b894759d2cd89b6453156e6b0246f6cc8f0d60b
'jshint --show-non-errors' shows unused variables. Some of these are unused
function arguments, which are rarely bugs. But try to fix the ones which
are forgotten/dead code and unnecessary require()s.
Change-Id: I685ccd8e388fd3acf75053a07e2f729398fa2855
* When I last fixed Cite in commit 0164b846, I didn't take it
all the way through. I was assigning ref indexes in the
async pipeline which is incorrect since ref-tokens and
references-tokens should be processed in the same order
as they show up in the input.
* I now moved ref-index assignment to the DOM post processor
phase where they always belonged.
* Fixed references and cite reset functions to only reset a
specific group, if necessary.
* Pipelines processing template transclusions should propagate
the extension tag if the transclusion showed up in the context of
extension source so that any extension-specific constraints are
respected. (Ex: ref-tags in reference-extension context are handled
differently -- and this should continue to be true even when the
tags show up via transcluded content)
* These fixes were tested on the following snippet from 0164b846.
Where this snippet was generating buggy output earlier, it is now
being handled correctly.
---------------------------
X1<ref name="x" /> X2<ref name="x" />
<references>
<ref name="x">x</ref>
</references>
Y<ref name="y">{{echo|y}}</ref>
Z<ref name="z" />
<references>
{{echo|<ref name="z">z</ref>}}
</references>
A<ref name="a">a</ref>
B<ref name="b" />
<references>
{{echo|<ref name="b">b</ref>}}
</references>
C<ref name="c">c</ref>
D<ref name="d" />
<references>
<ref name="d">{{echo|d}}</ref>
</references>
---------------------------
X1<ref name="x" /> X2<ref name="x" />
Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
This code is being run on:
<meta property="mw:PageProp/categorydefaultsort" content="0 126">
which doesn't have a 'typeof' attribute, and so the match is crashing
with a null pointer dereference.
Turn the expression around. RegExp.test is more efficient, and it handles
a null argument with no problems. Add start/end anchors to the regexp
while we're at it.
Change-Id: I11bc9fdc902a21ce0bc1c05f08d3b615d5483e2e
Most of these warnings are globals defined in mediawiki.parser.defines.js,
which we turn into proper local variables. But there are a grab page of
other use-before-def, missing semicolon, bad line break, and other minor
warnings.
No tricky stuff attempted in this patch, this should be safe obvious
fixes only... although there is a nasty circular dependency between
Util and parser.defines which had to be broken.
Change-Id: I8810da3a5ec287d7ae9078463ba01472a87a492e
* Updated native implementations of the <ref> and <references>
tag implementations of the cite extension.
* <references> tag was not being processed properly by Parsoid.
This led to lost references on the BO page. This patch fixes
it which fills out references and more closely matches output
en:WP.
* Extracted extension content processing code into a helper and
reused it for both <ref> and <references> handler.
- Leading ws-only lines are unconditionally stripped. Is this
accurate or is this extension-specific? Given that this code
is right now tied to <ref> and <references> tag, this is not
yet a problem, but if made more generic, this issue has to
be addressed.
* PreHandler should not run when processing the refs-tag. Right
now, this is a hard check in the pre-handler. Probably worth
making this more generic by letting every stage in the pipeline
get a chance at turning themselves on/off based on the extension
being processed by the pipeline (can use the _applyToStage fn.
on ParserPipeline). TO BE DONE.
* <ref> extension needs to be reset after each <references> tag
is processed to duplicate behavior of existing cite extension.
TO BE DONE.
* Updated refs group index to start at 1.
* No change in parser tests. References section output on the
en:Barack Obama page now more closely matches the refs output
on enwp.
* In addition to the en:BO page, the following wikitext was used to
fix bugs and test the implementation.
CMD: "node parse --extensions ref,references < /tmp/test"
----------------------------------------------
X1<ref name="x" /> X2<ref name="x" />
<references>
<ref name="x">x</ref>
</references>
Y<ref name="y">{{echo|y}}</ref>
Z<ref name="z" />
<references>
{{echo|<ref name="z">z</ref>}}
</references>
A<ref name="a">a</ref>
B<ref name="b" />
<references>
{{echo|<ref name="b">b</ref>}}
</references>
C<ref name="c">c</ref>
D<ref name="d" />
<references>
<ref name="d">{{echo|d}}</ref>
</references>
----------------------------------------------
Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
Tokenization
------------
* Fixed tokenizer to correctly parse extension tags in different
contexts: with start and end tags, in self-closing tag mode,
and to correctly handle scenarios when the exension end-tag is
followed by a '#' (the special char used to skip over extension
content).
* Removed the distinction in the tokenizer between installed
extensions and natively supported extension tags (<ref> and
<references> for ex.). They all tokenize and get processed
identically and get handled by different paths in the extension
handler.
* Template and TemplateArg tokens now carry tpl. transclusion
source alongwith them since tsr information will not be
accurate when they show up in extension contexts that in turn
showed up in template context that were expanded by the php
preprocessor.
Ex: {{echo|<ref>{{echo|foo}}</ref>}}
The tsr for the ref-tag will correspond to the
template-source of the echo-template, NOT the original top-level
page. So, env.page.src.substring(..) will return incorrect
source for the innermost {{echo|foo}}. This fix of carrying
along tpl transclusion source in the token itself eliminates
this problem.
Knowledge of native extensions
------------------------------
* Natively implemented extension tags (<ref> and <references>)
are hardcoded in env.conf.parsoid. At some point, it would
be good to have a registration mechanism for parsoid-native
extensions.
Extension handling
------------------
* Extracted extension handling out of the template handler into
its own handler class. Right now, this class inherits from the
template handler in order to be able to reuse a lot of the
expansion and encapsulation functionality currently in the
Template Handler.
* This handler now handles extensions that are:
(a) natively implemented and registered with Parsoid.
(b) implemented as a PHP extension and expanded by relying on
the PHP preprocessor.
For (a), it uses information from env.conf.parsoid to find
ext-handlers for natively implemented ext-tags. However, this
can be cleaned up at some point by making available a registration
mechanism.
Cite/Ref/References
-------------------
* Reworked the cite handler to split up ref-token processing
and references token processing.
* The handler now processes ref-tokens, parses content all
the way to html output and encapsulates the html in an
attribute of a meta-token that serves as a placeholder for
where the ref-token occured.
* References are handled as a DOM post-pass where these meta
placeholder tokens are collected, content extracted from
the attribute and spit out at the site of a references tag.
The DOM walking is in DOMPostProcessor.js, but the actual
processing is part of the Cite.js to keep all cite extension
handling code in one place.
Parser pipeline
---------------
* Restructured parser pipeline recipes based on changes to Cite,
TemplateHandler, and ExtensionHandler.
* Added a couple functions to the parser pipeline:
1. resetState to reset state before starting a new top-level parse
when pipelines are reused across top-level parses (ex: parser
tests)
2. setSourceOffsets to set start/end offsets of the source being
handled by the pipeline. This is required to correctly set tsr
values when extension content (which is a substring of original
top-level text) is parsed in its own pipeline.
Other fixes
-----------
* Removed env parameter from the Params object since it was not
being used and seemed like unnecessary state propagation.
* Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code
in the tokenizer that converts "\n" in text to NlTks.
* Cleanup of Util.shiftTokenTSR.
* ext.util.TokenCollection is now no longer used by anything.
Added a FIXME and left around in case we are able to improve
tokenizing and handling of *include* tags that can eliminate the
need for the messy TokenAndAttrCollector.
Test results
------------
* No change in parser tests results.
* Tested with a few different files.
- en:BO page seems to be parsed about 10% faster than before
(needs verification).
- Referencs on the en:BO page seem to be more accurate than
before.
Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
* A references tag for a group can appear multiple times on a page.
Ex: en:List_of_number-one_albums_from_the_1960s_%28UK%29 uses
{{Reflist|group=lower-alpha}} more than once, and each reflist
tag emits references accumulated since the previous references
output. This requires clearing the refs for the group after it
is emitted.
* Parsoid wasn't clearing the group's refs. This resulted in
duplicate ref output which in turn crashed template encapsulation
code that relies on template ids being unique.
* This patch fixes it which fixes the crasher on
en:List_of_number-one_albums_from_the_1960s_%28UK%29
* No change in parser test results.
Change-Id: I48ba9c3fe2249c6ef1c1bf068c22c7985fa16e8f
* When named-ref-tags are used multiple times in a document, the
Cite.onReferences function was outputting the ref-content once
for each use in the references section. While this is "harmless"
for regular ref-content, when the ref-content has templates
(ex: {[cite .. }}), that template content is duplicated several
times -- this also means that template expansions seen in the
output no longer has unique ids. Occasionally, this trips up
DOM passes (template encapsulation) which make the reasonable
assumption that template ids are unique. It introduced an
infinite loop in the code that detects template range nestings
and overlaps. This is not triggered always, but only in certain
scenarios. en:Euphoria (Enrique Iglesias album) was a page that
sent the parser into an infinite loop.
* Now fixed by outputting ref-content once.
* In addition, added an error-check in the DOM post-pass that might
get stuck in an infinite loop to output an error message.
* No change in parser test results.
* en:Euphoria (Enrique Iglesias album) now parses and rts without
getting stuck in an infinite loop. Possibly other pages as well.
Change-Id: I81a055999eefe3ca36dfb8a9cda3fee538ae6c65
The 'fakejquery' module just defines the bits we use: $.each, $.map,
$.extend, $.isArray, $.isEmptyObject (and the internal functions required
to implement those).
Change-Id: I626b2f3a3d3a8d395d0c9679e01d3399403636dd
* Added a tokens/x-mediawiki/post-expansion sub-pipeline that
is used to process ref-content. This same pipeline type
could also be used to post-process tokenized content from
other extensions.
* Cleaned up transform handler ranks a bit.
* Fixed paragraph-handler to accept initial state and also
fixed it to do a complete reset on encountering EOFTk.
Cite ref-content handling uses the sub-pipeline while
passing the inBlockToken flag to prevent top-level p-wrapping
of ref-content.
* Fixes references output on en:Barack Obama and introduces
lists in individual ref items.
* No change in parser tests -- how do we add tests for testing
extensions like cite/ref?
Change-Id: I928a3a0f5f989ec28bbe73383794fe6f32599fd1
* Updated #tag parser function code to set tag attributes.
This lets {{Reflist|group=n}} to pass the group=n attribute
to the <references> tag. In turn, this fixes incorrect
reference listing in the above template call. This then
fixes crashers in en:Prophet Muhammed, en:The PS3 and such
pages.
- TODO: parser output for Prophet Muhammed is still broken
where content from body ends up in a table that has
display:none css set.
* Minor code cleanup in AttributeTransformManager.process
and added comment about lazy processing.
Change-Id: If528a3a5c6e22d3ff691dfc6b5b2f00201a26384
The PHP preprocessor returns a references tag with an empty group attribute,
which we did not map to the null reference group. This caused the references
listing to vanish when using the PHP preprocessor.
This patch makes sure the attribute is converted to a string, stripped of
leading/trailing whitespace and finally set to null if no string remains.
No changes in parser tests, since Cite is in a separate extension.
Change-Id: Ib3de375225947a78c362370be8d78504fa24def2
* Replaced empty (mw:Placeholder) span tags with meta-tag.
* Minor code refactoring in DSR computation.
* Removed hardcoding of "mwt" prefix for object ids.
* Added rt-info for html tags that are not lower-case in the
source wikitext.
- Additional 2 wt2wt tests green.
Change-Id: I9bb6a075bc60b6d78186d23000128eef6a04c787
This should hopefully fix some spurious semantic diffs that are purely based
on ref numbering, which changes if the references are not reset.
Change-Id: I37b1f4cfe69538e1c5ccc81fdad3084c01151700
* Run cite handler first thing in phase 3 so that ref-content
is pulled out of the main token stream. This effectively
prevents unclosed/unbalanced tags within ref-tags from
affecting the main stream.
* No change in parser tests.
* Fixes rt diffs on the following pages:
1. en:Bandaid surgeries (zero rt diffs)
2. en:Gondi bank (zero rt diffs)
3. en:Ashland, oh (1 syntactic diff)
The reverted commit (78989e8) had fixed rt-diffs in 1. above,
hadn't changed diffs in 2. and introduced regressions in 3.
Change-Id: I6ad498d9d3711b71e7f427eea214acd10da8647b
* Setting tsr to (endTsr || [null, text.length]) was causing
serious duplication of rt-src on the following page:
en:Almirante Latorre-class battleship
This is the case for ref-tokens that come from templates
which have their TSR stripped. So, we shouldn't attempt
to set it to [null, text.length] again.
* No change in parserTests.
* No change in errors for en:L'Opéra national de Paris.
Change-Id: If6a79638c349c9a3fab42094aad51e2e476f9632
Also remove the src, as we re-set it based on DSR anyway.
No change in parserTests, but three less diffs in L'Opéra national de Paris.
Change-Id: Ia6784efe7d76c41f24a183465393bda78641a8eb
This exposed a few bugs (primarily missing var declarations), which are fixed.
It would for example have caught the shallow clone bug we just fixed, and
should catch similar bugs early in the future.
TODO / issues this exposed:
Avoid attempted tsr modification in ext.core.TemplateHandler.js:306:29. This
is fixed for now by cloning the tokens, but should really be avoided by
stripping the tsr before caching the tokens (anything in phase 1 will work for
example).
Change-Id: I6a1a07de7ac333f31da9cf0ae9ed20e5507bacba
* There was a fixme for this in ext.Cite.js which was triggering
an error in the serializer and introducing rt-diffs.
* No change in parser tests.
* Eliminates semantic error in en:Birkenhauer Island and other
pages.
Change-Id: I9fe6ea26fdf1859d751b02a288393af610be5474
- Fixed several bugs in the DSR computation.
- Deleted comment tokens from template output.
- Eliminated TSR info from template output.
- Rearranged order of DOM post passes.
- Updated template encapsulation to use DSR values.
- There are still a few missing pieces and scenarios that are
not handled properly.
Change-Id: I644914902ec1d27418138b107abe765f81178398
* Only one instance of Cite handler is created and it deals with the
final expanded token stream. So, effectively there is no information
about whether we are processing in a top-level or nested template
context. So, we instead use tsr.
Change-Id: I002f421fc78576e869cc8fbcdc386ef0abb09ca9
* Added template-wrapping information in cite-expansion with
Object/Ext/Cite RDFa type.
* Added 'ref' tag to whitelisted tags (temporary workaround to
let <ref name='blah' /> to RT correctly till we handle this
properly).
* Added support for self-closing, but non-void tags.
* Deleted Object/Template/Content type attribute since it was
not used.
* Renamed Object/Template/Attributes to ExpandedAttrs/Template
to distinguish template attribute code paths from content
code paths -- could possibly be renamed to something more
suitable later on.
Results:
- With this experimental support, RT diffs on the en:Barack_Obama
page are vastly reduced.
Change-Id: I018f03df102764d3e39442ae359b751056a38c2f
