'jshint --show-non-errors' shows unused variables. Some of these are unused
function arguments, which are rarely bugs. But try to fix the ones which
are forgotten/dead code and unnecessary require()s.
Change-Id: I685ccd8e388fd3acf75053a07e2f729398fa2855
* When I last fixed Cite in commit 0164b846, I didn't take it
all the way through. I was assigning ref indexes in the
async pipeline which is incorrect since ref-tokens and
references-tokens should be processed in the same order
as they show up in the input.
* I now moved ref-index assignment to the DOM post processor
phase where they always belonged.
* Fixed references and cite reset functions to only reset a
specific group, if necessary.
* Pipelines processing template transclusions should propagate
the extension tag if the transclusion showed up in the context of
extension source so that any extension-specific constraints are
respected. (Ex: ref-tags in reference-extension context are handled
differently -- and this should continue to be true even when the
tags show up via transcluded content)
* These fixes were tested on the following snippet from 0164b846.
Where this snippet was generating buggy output earlier, it is now
being handled correctly.
---------------------------
X1<ref name="x" /> X2<ref name="x" />
<references>
<ref name="x">x</ref>
</references>
Y<ref name="y">{{echo|y}}</ref>
Z<ref name="z" />
<references>
{{echo|<ref name="z">z</ref>}}
</references>
A<ref name="a">a</ref>
B<ref name="b" />
<references>
{{echo|<ref name="b">b</ref>}}
</references>
C<ref name="c">c</ref>
D<ref name="d" />
<references>
<ref name="d">{{echo|d}}</ref>
</references>
---------------------------
X1<ref name="x" /> X2<ref name="x" />
Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
This code is being run on:
<meta property="mw:PageProp/categorydefaultsort" content="0 126">
which doesn't have a 'typeof' attribute, and so the match is crashing
with a null pointer dereference.
Turn the expression around. RegExp.test is more efficient, and it handles
a null argument with no problems. Add start/end anchors to the regexp
while we're at it.
Change-Id: I11bc9fdc902a21ce0bc1c05f08d3b615d5483e2e
Most of these warnings are globals defined in mediawiki.parser.defines.js,
which we turn into proper local variables. But there are a grab page of
other use-before-def, missing semicolon, bad line break, and other minor
warnings.
No tricky stuff attempted in this patch, this should be safe obvious
fixes only... although there is a nasty circular dependency between
Util and parser.defines which had to be broken.
Change-Id: I8810da3a5ec287d7ae9078463ba01472a87a492e
The cloned Cite object in the ParserCloned hook must be cleared, even if
the clone occurs inside a <ref> or <references> tag. The simple way to
do this is to set mInCite and mInReferences to false before calling
clearState().
While Cite could probably use a major cleanup, let's fix this bug now
instead of waiting for someone to get around to it.
Bug: 47291
Change-Id: Id3e91c41dc33a703b5326961fd57e1fe8ac61f5b
* Updated native implementations of the <ref> and <references>
tag implementations of the cite extension.
* <references> tag was not being processed properly by Parsoid.
This led to lost references on the BO page. This patch fixes
it which fills out references and more closely matches output
en:WP.
* Extracted extension content processing code into a helper and
reused it for both <ref> and <references> handler.
- Leading ws-only lines are unconditionally stripped. Is this
accurate or is this extension-specific? Given that this code
is right now tied to <ref> and <references> tag, this is not
yet a problem, but if made more generic, this issue has to
be addressed.
* PreHandler should not run when processing the refs-tag. Right
now, this is a hard check in the pre-handler. Probably worth
making this more generic by letting every stage in the pipeline
get a chance at turning themselves on/off based on the extension
being processed by the pipeline (can use the _applyToStage fn.
on ParserPipeline). TO BE DONE.
* <ref> extension needs to be reset after each <references> tag
is processed to duplicate behavior of existing cite extension.
TO BE DONE.
* Updated refs group index to start at 1.
* No change in parser tests. References section output on the
en:Barack Obama page now more closely matches the refs output
on enwp.
* In addition to the en:BO page, the following wikitext was used to
fix bugs and test the implementation.
CMD: "node parse --extensions ref,references < /tmp/test"
----------------------------------------------
X1<ref name="x" /> X2<ref name="x" />
<references>
<ref name="x">x</ref>
</references>
Y<ref name="y">{{echo|y}}</ref>
Z<ref name="z" />
<references>
{{echo|<ref name="z">z</ref>}}
</references>
A<ref name="a">a</ref>
B<ref name="b" />
<references>
{{echo|<ref name="b">b</ref>}}
</references>
C<ref name="c">c</ref>
D<ref name="d" />
<references>
<ref name="d">{{echo|d}}</ref>
</references>
----------------------------------------------
Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
Tokenization
------------
* Fixed tokenizer to correctly parse extension tags in different
contexts: with start and end tags, in self-closing tag mode,
and to correctly handle scenarios when the exension end-tag is
followed by a '#' (the special char used to skip over extension
content).
* Removed the distinction in the tokenizer between installed
extensions and natively supported extension tags (<ref> and
<references> for ex.). They all tokenize and get processed
identically and get handled by different paths in the extension
handler.
* Template and TemplateArg tokens now carry tpl. transclusion
source alongwith them since tsr information will not be
accurate when they show up in extension contexts that in turn
showed up in template context that were expanded by the php
preprocessor.
Ex: {{echo|<ref>{{echo|foo}}</ref>}}
The tsr for the ref-tag will correspond to the
template-source of the echo-template, NOT the original top-level
page. So, env.page.src.substring(..) will return incorrect
source for the innermost {{echo|foo}}. This fix of carrying
along tpl transclusion source in the token itself eliminates
this problem.
Knowledge of native extensions
------------------------------
* Natively implemented extension tags (<ref> and <references>)
are hardcoded in env.conf.parsoid. At some point, it would
be good to have a registration mechanism for parsoid-native
extensions.
Extension handling
------------------
* Extracted extension handling out of the template handler into
its own handler class. Right now, this class inherits from the
template handler in order to be able to reuse a lot of the
expansion and encapsulation functionality currently in the
Template Handler.
* This handler now handles extensions that are:
(a) natively implemented and registered with Parsoid.
(b) implemented as a PHP extension and expanded by relying on
the PHP preprocessor.
For (a), it uses information from env.conf.parsoid to find
ext-handlers for natively implemented ext-tags. However, this
can be cleaned up at some point by making available a registration
mechanism.
Cite/Ref/References
-------------------
* Reworked the cite handler to split up ref-token processing
and references token processing.
* The handler now processes ref-tokens, parses content all
the way to html output and encapsulates the html in an
attribute of a meta-token that serves as a placeholder for
where the ref-token occured.
* References are handled as a DOM post-pass where these meta
placeholder tokens are collected, content extracted from
the attribute and spit out at the site of a references tag.
The DOM walking is in DOMPostProcessor.js, but the actual
processing is part of the Cite.js to keep all cite extension
handling code in one place.
Parser pipeline
---------------
* Restructured parser pipeline recipes based on changes to Cite,
TemplateHandler, and ExtensionHandler.
* Added a couple functions to the parser pipeline:
1. resetState to reset state before starting a new top-level parse
when pipelines are reused across top-level parses (ex: parser
tests)
2. setSourceOffsets to set start/end offsets of the source being
handled by the pipeline. This is required to correctly set tsr
values when extension content (which is a substring of original
top-level text) is parsed in its own pipeline.
Other fixes
-----------
* Removed env parameter from the Params object since it was not
being used and seemed like unnecessary state propagation.
* Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code
in the tokenizer that converts "\n" in text to NlTks.
* Cleanup of Util.shiftTokenTSR.
* ext.util.TokenCollection is now no longer used by anything.
Added a FIXME and left around in case we are able to improve
tokenizing and handling of *include* tags that can eliminate the
need for the messy TokenAndAttrCollector.
Test results
------------
* No change in parser tests results.
* Tested with a few different files.
- en:BO page seems to be parsed about 10% faster than before
(needs verification).
- Referencs on the en:BO page seem to be more accurate than
before.
Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
* A references tag for a group can appear multiple times on a page.
Ex: en:List_of_number-one_albums_from_the_1960s_%28UK%29 uses
{{Reflist|group=lower-alpha}} more than once, and each reflist
tag emits references accumulated since the previous references
output. This requires clearing the refs for the group after it
is emitted.
* Parsoid wasn't clearing the group's refs. This resulted in
duplicate ref output which in turn crashed template encapsulation
code that relies on template ids being unique.
* This patch fixes it which fixes the crasher on
en:List_of_number-one_albums_from_the_1960s_%28UK%29
* No change in parser test results.
Change-Id: I48ba9c3fe2249c6ef1c1bf068c22c7985fa16e8f
If the parser is cloned, we need to clone our data object and
reassociate the hooks. Otherwise references may be lost when the clone
has ParserClearState called.
Change-Id: Ic1d75850e8d610ea890e34c38d062a358fd55178
Depends-On: Ieec65c908d71e89b9a66f83b9a626f842aadacbb
* When named-ref-tags are used multiple times in a document, the
Cite.onReferences function was outputting the ref-content once
for each use in the references section. While this is "harmless"
for regular ref-content, when the ref-content has templates
(ex: {[cite .. }}), that template content is duplicated several
times -- this also means that template expansions seen in the
output no longer has unique ids. Occasionally, this trips up
DOM passes (template encapsulation) which make the reasonable
assumption that template ids are unique. It introduced an
infinite loop in the code that detects template range nestings
and overlaps. This is not triggered always, but only in certain
scenarios. en:Euphoria (Enrique Iglesias album) was a page that
sent the parser into an infinite loop.
* Now fixed by outputting ref-content once.
* In addition, added an error-check in the DOM post-pass that might
get stuck in an infinite loop to output an error message.
* No change in parser test results.
* en:Euphoria (Enrique Iglesias album) now parses and rts without
getting stuck in an infinite loop. Possibly other pages as well.
Change-Id: I81a055999eefe3ca36dfb8a9cda3fee538ae6c65
The 'fakejquery' module just defines the bits we use: $.each, $.map,
$.extend, $.isArray, $.isEmptyObject (and the internal functions required
to implement those).
Change-Id: I626b2f3a3d3a8d395d0c9679e01d3399403636dd
