wikimedia/mediawiki-extensions-Cite
1
0
Fork 0
mirror of https://gerrit.wikimedia.org/r/mediawiki/extensions/Cite synced 2024-11-24 23:05:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
mediawiki-extensions-Cite/lib/ext.Cite.js

551 lines
16 KiB
JavaScript
Raw Normal View History

Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
/* ----------------------------------------------------------------------
* This file implements <ref> and <references> extension tag handling
* natively in Parsoid.
* ---------------------------------------------------------------------- */
Deep-freeze cache returns and enable strict mode This exposed a few bugs (primarily missing var declarations), which are fixed. It would for example have caught the shallow clone bug we just fixed, and should catch similar bugs early in the future. TODO / issues this exposed: Avoid attempted tsr modification in ext.core.TemplateHandler.js:306:29. This is fixed for now by cloning the tokens, but should really be avoided by stripping the tsr before caching the tokens (anything in phase 1 will work for example). Change-Id: I6a1a07de7ac333f31da9cf0ae9ed20e5507bacba
2012-10-30 00:51:07 +00:00
"use strict";
Use es6 methods instead of fakejquery. In particular, use `Array.isArray` instead of `$.isArray`, and `Object.assign` instead of `$.extend`. `Object.assign` operates only on own properties, so use `Object.create` on the prototype where necessary to get inherited properties. `Object.assign` does a simple assignment and is appropriate in most places, but be careful if we ever install getters/setters on a prototype. Implement `Util.clone()` from scratch to have a jquery-compatible deep clone operation. In particular, this needs to ignore objects which aren't "plain objects", so we don't try to clone DOM nodes. Our definition of a "plain object" is compatible with jquery/zepto.js, and is thus something of a hack. We should eventually replace this with a `console.assert()` and remove/rewrite the places where we try to clone objects which contain DOM trees and other cruft. Change-Id: I88c8fe41a9be84c167d5a0ea1187fd258f077968
2014-01-22 20:24:23 +00:00
require('./core-upgrade.js');
Deep-freeze cache returns and enable strict mode This exposed a few bugs (primarily missing var declarations), which are fixed. It would for example have caught the shallow clone bug we just fixed, and should catch similar bugs early in the future. TODO / issues this exposed: Avoid attempted tsr modification in ext.core.TemplateHandler.js:306:29. This is fixed for now by cloning the tokens, but should really be avoided by stripping the tsr before caching the tokens (anything in phase 1 will work for example). Change-Id: I6a1a07de7ac333f31da9cf0ae9ed20e5507bacba
2012-10-30 00:51:07 +00:00
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
var Util = require( './mediawiki.Util.js' ).Util,
Updates to cite to reflect spec for references and ref * Updated references output DOM to match spec better. * Also refactored some repetitive DOM code into helper functions. * Turned on tpl encapsulation inside extensions. * Verified output on the example in the spec and the example in commit 3c88b310. Change-Id: I1b894759d2cd89b6453156e6b0246f6cc8f0d60b
2013-05-02 16:17:08 +00:00
DU = require( './mediawiki.DOMUtils.js').DOMUtils,
(Bug 50218) Generated DOMFragment wrapping <ol> for references tag * So far, references wikitext was being replaced with a meta marker token that participated as an inline token in p-wrapping. As a result, in some cases, a p-tag was wrapping the references ol-node which is buggy HTML. When VE loaded this DOM, the p-node was split around the ol-node and duplicated DSR which led to duplicated output when selser ran on this. * One way of fixing this would have been to add a special case in the paragraph wrapper to treat the mw:Extensions/References/Marker meta tag as a block token. * A better solution with an eye towards the longer-term is to emit mw:DOMFragment wrapper for the references tag with the content being an ol-node. The dom-fragment unpacking code then takes care of p-wrapping issues for the ol-node. This fixes the bug in this case. * Also fixed parser tests output where group attribute was missing on some tests (which indicated that the older code was not emitting the group attribute on references tag). * No change in parser tests. Change-Id: I073b2e68667d577c75cad07d19cea2b19d0e89fe
2013-06-26 17:35:08 +00:00
coreutil = require('util'),
[Cite] Generate the same ids for <ref>s and notes as Cite.php One particular case is that Cite.php considers equal a name and its encoding, i.e. "a & b" === "a &amp; b". Added a new test for this case, but blacklisted it on html2wt, wt2wt and html2html due to a different problem with how Parsoid encodes entities. This will be investigated separately, as a simple fix could break unrelated cases. Also updated tests and blacklist to the new ids. Change-Id: I87637a1dc812a3a8f29327b9e6c0040b22a651c4
2014-12-20 12:49:12 +00:00
defines = require('./mediawiki.parser.defines.js'),
entities = require('entities');
Updates to cite to reflect spec for references and ref * Updated references output DOM to match spec better. * Also refactored some repetitive DOM code into helper functions. * Turned on tpl encapsulation inside extensions. * Verified output on the example in the spec and the example in commit 3c88b310. Change-Id: I1b894759d2cd89b6453156e6b0246f6cc8f0d60b
2013-05-02 16:17:08 +00:00
Fix a bunch of jshint warnings. Most of these warnings are globals defined in mediawiki.parser.defines.js, which we turn into proper local variables. But there are a grab page of other use-before-def, missing semicolon, bad line break, and other minor warnings. No tricky stuff attempted in this patch, this should be safe obvious fixes only... although there is a nasty circular dependency between Util and parser.defines which had to be broken. Change-Id: I8810da3a5ec287d7ae9078463ba01472a87a492e
2013-04-22 19:51:09 +00:00
// define some constructor shortcuts
var KV = defines.KV,
Added utility method to process content in pipeline. * This eliminates repetitive code from a few places and might be useful for future patches as well. Change-Id: I383ed253a2fa20c1b7429689d17cde176751e29a
2013-09-30 17:37:07 +00:00
EOFTk = defines.EOFTk,
Remove unused variables. 'jshint --show-non-errors' shows unused variables. Some of these are unused function arguments, which are rarely bugs. But try to fix the ones which are forgotten/dead code and unnecessary require()s. Change-Id: I685ccd8e388fd3acf75053a07e2f729398fa2855
2013-04-25 13:43:49 +00:00
SelfclosingTagTk = defines.SelfclosingTagTk;
Forward-port Cite extension * Adapted Cite extension to use current interfaces and token formats * Improved TokenCollector Change-Id: I20419b19edd9bbad2c2abf17a2ff1411b99c0c04
2012-05-03 11:05:28 +00:00
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
// FIXME: Move out to some common helper file?
// Helper function to process extension source
function processExtSource(manager, extToken, opts) {
var extSrc = extToken.getAttribute('source'),
tagWidths = extToken.dataAttribs.tagWidths,
content = extSrc.substring(tagWidths[0], extSrc.length - tagWidths[1]);
// FIXME: Should this be specific to the extension
Bunch of unrelated cleanup * Got rid of unnecessary/buggy concat of EOFTk in ext.Cite.js to wikitext string (concat is only required when processing a token array from stage 2 onwards). * Moved temporary node.data.parsoid.wrapper flag to a tmp object which is then stripped in the cleanup pass. Other temporary flags from other passes could also be moved here. * Added a helper to test if a typeOf has a parsoid-generated value. Change-Id: I2024c575b5293e5155fc8e2915a34b5fb8394671
2013-10-07 19:55:59 +00:00
// Or is it okay to do this unconditionally for all?
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
// Right now, this code is run only for ref and references,
// so not a real problem, but if this is used on other extensions,
// requires addressing.
//
Fix failures from v2 endpoint * Although this resolves the crashes, I'm unsatisfied with it as a proper fix to the underlying issue. There are many places throughout the codebase where we serialize and then parse document fragments that should be instrumented to store and unpack data-* attributes. Bug: T76518 Change-Id: Idca1b0a37ec924a71cb51160d000c7de9717d422
2014-12-03 00:47:32 +00:00
// FIXME: SSS: This stripping maybe be unnecessary after all.
Bunch of unrelated cleanup * Got rid of unnecessary/buggy concat of EOFTk in ext.Cite.js to wikitext string (concat is only required when processing a token array from stage 2 onwards). * Moved temporary node.data.parsoid.wrapper flag to a tmp object which is then stripped in the cleanup pass. Other temporary flags from other passes could also be moved here. * Added a helper to test if a typeOf has a parsoid-generated value. Change-Id: I2024c575b5293e5155fc8e2915a34b5fb8394671
2013-10-07 19:55:59 +00:00
//
Start aligning Parsoid cite with PHP cite behavior some more * Nested ref tags are not supported anymore. * Turned off pre and p-wrap handlers on ref content since the native cite extension seems to not do any of this on ref content. * No change in parser test results (because there are no tests yet). Other cleanup: * Removed the 'inBlockToken' hack from Cite since this is not necessary anymore. TODO: The use of this flag in TemplateHandler may not be needed either. Verify and get rid of it. * Leading whitespace in ref-content is still removed but this may not be strictly necessary. Change-Id: I3406236032abe36099a1e420f443277a95fe597b
2013-06-13 16:27:27 +00:00
// Strip all leading white-space
(Bug 50394) Fixed bad regexp * There is another unrelated issue on that page ==> It has a couple instances of <ref>text<ref> which slurps the entire rest of the page into a 10K string (on which the regexp failed). Change-Id: Ia4c498b28042de5bfe9b0945c0efaa8c6cb81ea0
2013-06-28 20:54:15 +00:00
var wsMatch = content.match(/^(\s*)([^]*)$/),
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
leadingWS = wsMatch[1];
// Update content to normalized form
content = wsMatch[2];
if (!content || content.length === 0) {
opts.emptyContentCB(opts.res);
} else {
// Pass an async signal since the ext-content is not processed completely.
opts.parentCB({tokens: opts.res, async: true});
Added utility method to process content in pipeline. * This eliminates repetitive code from a few places and might be useful for future patches as well. Change-Id: I383ed253a2fa20c1b7429689d17cde176751e29a
2013-09-30 17:37:07 +00:00
// Wrap templates always
opts.pipelineOpts = Util.extendProps({}, opts.pipelineOpts, { wrapTemplates: true });
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
var tsr = extToken.dataAttribs.tsr;
[jscs] Enforce requireSpaceBeforeBinaryOperators * This was auto-corrected with jscs -x . Change-Id: I13df3896e8130743eb31581e80d4c7909e48768d
2015-04-06 21:07:04 +00:00
opts.srcOffsets = [ tsr[0] +tagWidths[0] +leadingWS.length, tsr[1] -tagWidths[1] ];
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
Added utility method to process content in pipeline. * This eliminates repetitive code from a few places and might be useful for future patches as well. Change-Id: I383ed253a2fa20c1b7429689d17cde176751e29a
2013-09-30 17:37:07 +00:00
// Process ref content
Code cleanup: Pipeline utilities now take env and parent frame. * Removed manager and passed in env and parent-frame to all utilities that process content in new pipelines. * Added more documentation to mediawiki.Util.js. * Renamed processAttributeToDOM to a more appropriate name. * Added pipelineFactory property to env and used that to construct parsing pipelines everywhere. Change-Id: Ic612e5630d19d4e3f5d6388bc5cd117d337fd799
2014-02-11 22:38:28 +00:00
Util.processContentInPipeline(manager.env, manager.frame, content, opts);
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
}
}
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
/**
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
* Simple token transform version of the Ref extension tag
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
*
Clean up and comment the Cite extension a bit.
2011-12-13 18:45:09 +00:00
* @class
* @constructor
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
*/
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
function Ref(cite) {
this.cite = cite;
this.reset();
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
}
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
/**
* Reset state before each top-level parse -- this lets us share a pipeline
* to parse unrelated pages.
*/
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
Ref.prototype.reset = function() { };
Reset the Cite extension state with the end token This should hopefully fix some spurious semantic diffs that are purely based on ref numbering, which changes if the references are not reset. Change-Id: I37b1f4cfe69538e1c5ccc81fdad3084c01151700
2012-11-16 18:03:01 +00:00
(Bug 69827): Supported nested <ref> tags once more * Nested <ref> tag support was broken in 69b6ec4d since that patch effectively processed <refs> only on the final DOM rather than extracted <ref> information in sub-pipelines. In doing so, it broke support for <ref> in <ref> tags which are supported by {{efn}} templates and used as follows. {{efn|A clarification.{{sfn|Smith|2009|p=2}}}} The {{sfn}} tpl generates a <ref> tag inside another <ref> tag generated by the {{efn}} tpl. * This patch fixes the breakage by processing <ref> content after it is extracted (in case it is known to have nested <ref> tags). * This fixes the rendering on enwiki/Otto_I%2C_Holy_Roman_Emperor and is an improvement over when it was broken. The nested <ref> gets id 18 (just like the Cite.php handling) whereas it used to get id 1 before the breakage. * TODO: - Figure out a way to add a test for this. Change-Id: Ib82b75f66249b2133a123adbe6fd7acbfd8ec8fb
2014-08-21 23:16:32 +00:00
function hasRef(node) {
var c = node.firstChild;
while (c) {
if (DU.isElt(c)) {
var typeOf = c.getAttribute('typeof');
if ((/(?:^|\s)mw:Extension\/ref\/Marker(?=$|\s)/).test(typeOf)) {
return true;
}
if (hasRef(c)) {
return true;
}
}
c = c.nextSibling;
}
return false;
}
Clean up and comment the Cite extension a bit.
2011-12-13 18:45:09 +00:00
/**
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
* Handle ref tokens
Clean up and comment the Cite extension a bit.
2011-12-13 18:45:09 +00:00
*/
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
Ref.prototype.handleRef = function ( manager, pipelineOpts, refTok, cb ) {
Take #2: (Bug 49555): Support all nested ref scenarios * This patch now supports any template that generates nested refs where the top-level token is just a single-ref. Ex: {{efn|New timetable{{sfn|Vallance|1991|p=31}}}} * The solution in this patch removes the hack from the previous version (53bbcbb3). As documented in the bug report, this solution essentially "trusts" template authors and lets nested refs through when they come from templates. But, top-level nested refs are still not allowed. <ref>foo<ref>bar</ref>baz</ref> in the toplevel page will still generate a single reference with content "foo<ref>bar</ref>baz" * Updated the nested ref test output - Regenerated selser changes because of the changed output. - Regenerated blacklist because of the changed selser changes. * This now handles: - https://en.wikipedia.org/wiki/User:Edgepedia/VE/GNoSR - https://en.wikipedia.org/wiki/Phellinus ellipsoideus Change-Id: I627955f0be1c5e2bafc49647c94c2be68ce711a8
2013-07-15 22:31:20 +00:00
// Nested <ref> tags at the top level are considered errors
// But, inside templates, they are supported
if (!pipelineOpts.inTemplate && pipelineOpts.extTag === "ref") {
Start aligning Parsoid cite with PHP cite behavior some more * Nested ref tags are not supported anymore. * Turned off pre and p-wrap handlers on ref content since the native cite extension seems to not do any of this on ref content. * No change in parser test results (because there are no tests yet). Other cleanup: * Removed the 'inBlockToken' hack from Cite since this is not necessary anymore. TODO: The use of this flag in TemplateHandler may not be needed either. Verify and get rid of it. * Leading whitespace in ref-content is still removed but this may not be strictly necessary. Change-Id: I3406236032abe36099a1e420f443277a95fe597b
2013-06-13 16:27:27 +00:00
cb({ tokens: [refTok.getAttribute("source")] });
return;
}
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
(Bug 67237): Fix citation numbering issue * Process <ref> and <references> tag on the top-level DOM only and ignore the generateRefs pass when processing other content. * This required a few fixes: - ensure that DOMPostProcessor knows about the top-level. - ensure that DOMVisitor knows about the top-level. - cleanup pass leaves behind the ref-marker metas from DOMs from non top-level content. - process nested references content. * One of the references tests had incorrect parsed output. That test has been updated to reflect the correct output from this patch. * Barack Obama seems to now have the correct numbering on references. Change-Id: I5465721d2fc715f2168f267e773a446bc37d198b
2014-07-01 18:04:31 +00:00
var refOpts = Object.assign({
Temp fix for handling non-string ref attribute vals * Entities in ref name weren't expected * Fixes the crash from arwiki:تأثير_الدمعة_السوداء * Makes use of the fix from de3642b8dd4a804ac654f2943a900496f2c8b3f3 Bug: 63790 Change-Id: Icb8781b4d9decc5a8b115d0b11def4d18f5d5025
2014-04-14 23:04:31 +00:00
name: null, group: null
}, Util.KVtoHash(refTok.getAttribute("options"), true)),
Make <ref>s nested in <references> editable. * Track <ref>s birthed in <references> tags and capture their HTML and set <references>.data-mw.body.html to the concatenated HTML for all those child <ref> tags. * No change in parser-tests because our test normalization code strips data-mw before comparing results. Since this patch only changes data-mw output for "References: 5. ..." test, there is no change. In any case, this patch updates the data-mw output for the test in question so that if we do modify our normalization routines, the test will still pass. Change-Id: Iaea752dec03e5e7f0d36baf3b7a9973f1b85e8a5
2013-07-24 00:15:45 +00:00
about = manager.env.newAboutId(),
(Bug 69827): Supported nested <ref> tags once more * Nested <ref> tag support was broken in 69b6ec4d since that patch effectively processed <refs> only on the final DOM rather than extracted <ref> information in sub-pipelines. In doing so, it broke support for <ref> in <ref> tags which are supported by {{efn}} templates and used as follows. {{efn|A clarification.{{sfn|Smith|2009|p=2}}}} The {{sfn}} tpl generates a <ref> tag inside another <ref> tag generated by the {{efn}} tpl. * This patch fixes the breakage by processing <ref> content after it is extracted (in case it is known to have nested <ref> tags). * This fixes the rendering on enwiki/Otto_I%2C_Holy_Roman_Emperor and is an improvement over when it was broken. The nested <ref> gets id 18 (just like the Cite.php handling) whereas it used to get id 1 before the breakage. * TODO: - Figure out a way to add a test for this. Change-Id: Ib82b75f66249b2133a123adbe6fd7acbfd8ec8fb
2014-08-21 23:16:32 +00:00
finalCB = function(toks, contentBody) {
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
// Marker meta with ref content
var da = Util.clone(refTok.dataAttribs);
// Clear stx='html' so that sanitizer doesn't barf
da.stx = undefined;
(Bug 67237): Fix citation numbering issue * Process <ref> and <references> tag on the top-level DOM only and ignore the generateRefs pass when processing other content. * This required a few fixes: - ensure that DOMPostProcessor knows about the top-level. - ensure that DOMVisitor knows about the top-level. - cleanup pass leaves behind the ref-marker metas from DOMs from non top-level content. - process nested references content. * One of the references tests had incorrect parsed output. That test has been updated to reflect the correct output from this patch. * Barack Obama seems to now have the correct numbering on references. Change-Id: I5465721d2fc715f2168f267e773a446bc37d198b
2014-07-01 18:04:31 +00:00
da.group = refOpts.group || '';
da.name = refOpts.name || '';
(Bug 69827): Supported nested <ref> tags once more * Nested <ref> tag support was broken in 69b6ec4d since that patch effectively processed <refs> only on the final DOM rather than extracted <ref> information in sub-pipelines. In doing so, it broke support for <ref> in <ref> tags which are supported by {{efn}} templates and used as follows. {{efn|A clarification.{{sfn|Smith|2009|p=2}}}} The {{sfn}} tpl generates a <ref> tag inside another <ref> tag generated by the {{efn}} tpl. * This patch fixes the breakage by processing <ref> content after it is extracted (in case it is known to have nested <ref> tags). * This fixes the rendering on enwiki/Otto_I%2C_Holy_Roman_Emperor and is an improvement over when it was broken. The nested <ref> gets id 18 (just like the Cite.php handling) whereas it used to get id 1 before the breakage. * TODO: - Figure out a way to add a test for this. Change-Id: Ib82b75f66249b2133a123adbe6fd7acbfd8ec8fb
2014-08-21 23:16:32 +00:00
da.content = contentBody ? DU.serializeChildren(contentBody) : '';
da.hasRefInRef = contentBody ? hasRef(contentBody) : false;
Improved template roundtripping based on WT ranges on DOM nodes. - Fixed several bugs in the DSR computation. - Deleted comment tokens from template output. - Eliminated TSR info from template output. - Rearranged order of DOM post passes. - Updated template encapsulation to use DSR values. - There are still a few missing pieces and scenarios that are not handled properly. Change-Id: I644914902ec1d27418138b107abe765f81178398
2012-09-17 19:46:44 +00:00
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
toks.push(new SelfclosingTagTk( 'meta', [
Add capability for DOM-based template expansion, and use DOM fragments for extensions This patch adds the capability to expand individual transclusions all the way to DOM, which enforces properly nested templates. This is also needed for DOM-based template re-expansion, which the VE folks would like to use for template editing. In normal parsing, DOM-based expansion is currently disabled by default for transclusions, and enabled by default for extensions. The decision whether to balance a particular transclusion can be based on a statistics-based classification of templates into balanced and unbalanced ones. The advantage of this approach is consistency in behavior for old revisions. Another alternative is to wrap unbalanced transclusions into <domparse> tags which disable DOM-based parsing for all transclusions in its content. This has the advantage that all special cases can be handled after tag insertion, and that balancing could also be enforced in the PHP parser using an extension. Other major changes: * Renamed transclusion content typeof from mw:Object/Template to mw:Transclusion * Renamed extension typeof from mw:Object/Extensions/foo to mw:Extension/foo and switched Cite to use lower-case ref/references too Other minor changes: * Only apply template encapsulation algorithm on DOM to mw:Transclusion and mw:Param objects, and no longer to all mw:Object/* elements. We should probably switch to a more explicit encapsulation type instead. Maybe something like mw:EncapStart/Transclusion and mw:EncapEnd/Transclusion instead of the current mw:Transclusion and mw:Transclusion/End, so that stripping those metas out selectively is easy? * Changed the DOMTraverser logic to let handlers explicitly return the next element to handle. Useful when several siblings are handled, as is the case for the fragment unwrapper for example, and avoids some cases where deleted nodes were still being processed. * Changed Cite to use mw:Extension/Ref{,erences} as all other extensions. * Make sure we round-trip gallery when the PHP preprocessor is not used Five parsoid-specific wt2html tests are failing due to the typeof change. They will be fine once those tests are adjusted. For now, adding them to parser tests blacklist. TODO: * Switch the Cite extension to the generic DOMFragment mechanism instead of marker tokens. Change-Id: I64b560a12695256915d2196d0646347381400e80
2013-05-16 22:23:05 +00:00
new KV('typeof', 'mw:Extension/ref/Marker'),
Sanitize void tags and drop attributes for end tags - Make sure void element attributes are sanitized - Drop attributes on end tags - Fix Cite to use data-parsoid instead of invalid attributes to pass information from the token stream to the DOM Change-Id: If236d4c9197b12ff86a607763f25ed7677623bc4
2013-10-17 22:30:04 +00:00
new KV('about', about)
Add capability for DOM-based template expansion, and use DOM fragments for extensions This patch adds the capability to expand individual transclusions all the way to DOM, which enforces properly nested templates. This is also needed for DOM-based template re-expansion, which the VE folks would like to use for template editing. In normal parsing, DOM-based expansion is currently disabled by default for transclusions, and enabled by default for extensions. The decision whether to balance a particular transclusion can be based on a statistics-based classification of templates into balanced and unbalanced ones. The advantage of this approach is consistency in behavior for old revisions. Another alternative is to wrap unbalanced transclusions into <domparse> tags which disable DOM-based parsing for all transclusions in its content. This has the advantage that all special cases can be handled after tag insertion, and that balancing could also be enforced in the PHP parser using an extension. Other major changes: * Renamed transclusion content typeof from mw:Object/Template to mw:Transclusion * Renamed extension typeof from mw:Object/Extensions/foo to mw:Extension/foo and switched Cite to use lower-case ref/references too Other minor changes: * Only apply template encapsulation algorithm on DOM to mw:Transclusion and mw:Param objects, and no longer to all mw:Object/* elements. We should probably switch to a more explicit encapsulation type instead. Maybe something like mw:EncapStart/Transclusion and mw:EncapEnd/Transclusion instead of the current mw:Transclusion and mw:Transclusion/End, so that stripping those metas out selectively is easy? * Changed the DOMTraverser logic to let handlers explicitly return the next element to handle. Useful when several siblings are handled, as is the case for the fragment unwrapper for example, and avoids some cases where deleted nodes were still being processed. * Changed Cite to use mw:Extension/Ref{,erences} as all other extensions. * Make sure we round-trip gallery when the PHP preprocessor is not used Five parsoid-specific wt2html tests are failing due to the typeof change. They will be fine once those tests are adjusted. For now, adding them to parser tests blacklist. TODO: * Switch the Cite extension to the generic DOMFragment mechanism instead of marker tokens. Change-Id: I64b560a12695256915d2196d0646347381400e80
2013-05-16 22:23:05 +00:00
], da));
RT <references> as well as </references> tag. * Fixes RT diff in en:Standing Sushi Bar. Change-Id: Ia05f9d01503e3003260a10663d1dd845a261beab
2012-11-16 23:36:07 +00:00
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
// All done!
cb({tokens: toks, async: false});
};
RT <references> as well as </references> tag. * Fixes RT diff in en:Standing Sushi Bar. Change-Id: Ia05f9d01503e3003260a10663d1dd845a261beab
2012-11-16 23:36:07 +00:00
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
processExtSource(manager, refTok, {
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
// Full pipeline for processing ref-content
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
pipelineType: 'text/x-mediawiki/full',
pipelineOpts: {
(Bug 64901) Fix paragraph-wrapping to match PHP parser + Tidy combo General changes --------------- * Replaced the hacky 'inBlockNode' parser pipeline option with a cleaner 'noPWrapping' option that suppresses paragraph wrapping in sub-pipelines (ex: recursive link content, ref tags, attribute content, etc.). Changes to wt2html pipeline --------------------------- * Fixed paragraph-wrapping code to ensure that there are no bare text nodes left behind, but without removing the line-based block-tag influences on p-wrapping. Some simplifications as well. TODO: There are still some discrepancies around <blockquote> p-wrapping behavior. These will be investigated and addressed in a future patch. * Fixed foster parenting code to ensure that fostered content is added in p-tags where necessary rather than span-tags. Changes to html2wt/selser pipeline ---------------------------------- * Fixed DOMDiff to tag mw:DiffMarker nodes with a is-block-node attribute when the deleted node is a block node. This is used during selective serialization to discard original separators between adjacent p-nodes if either of their neighbors is a deleted block node. * Fixed serialization to account for changes to p-wrapping. - Updated tag handlers for the <p> tag. - Updated separator handling to deal with deleted block tags and their influence on separators around adjacent p-tags. - Updated selser output code to test whether a deleted block tag forces nowiki escaping on unedited content from adjacent p-tags. Changes to parser tests / test setup ------------------------------------ * Tweaked selser test generation to ensure that text nodes are always inserted in p-wrappers where necessary. * Updated parser test output for several tests to introduce p-tags instead of span-tags or missing p-tags, add html/parsoid section, or in one case, add missing HTML output. Parser Test Result changes -------------------------- Newly passing - 12 wt2html - 1 wt2wt - 3 html2html - 3 html2wt Newly failing - 1 html2wt "3. Leading whitespace in indent-pre suppressing contexts should not be escaped" This is just normalization of output where multiple HTML forms serialize to the same wikitext with a newline difference. It is not worth the complexity to fix this. - 1 wt2wt ""Trailing newlines in a deep dom-subtree that ends a wikitext line" This is again normalization during serialization where an extra unnecessary newline is introduced. - A bunch of selser test changes. 182 +add, 188 -add => 6 fewer selser failures - That is a lot of changes to sift through, and I didn't look at every one of those, but a number of changes seem to be harmless, and just a change to previously "failing" tests. - "Media link with nasty text" test seems to have a lot of selser changes, but the HTML generated by Parsoid seems to be "buggy" with interesting DSR values as well. That needs investigation separately. - "HTML nested bullet list, closed tags (bug 5497) [[3,3,4,[0,1,4],3]]" has seen a degradation where a dirty diff got introduced. Haven't investigated carefully why that is so. Change-Id: Ia9c9950717120fbcd03abfe4e09168e787669ac4
2014-08-29 05:59:20 +00:00
extTag: "ref",
Take #2: (Bug 49555): Support all nested ref scenarios * This patch now supports any template that generates nested refs where the top-level token is just a single-ref. Ex: {{efn|New timetable{{sfn|Vallance|1991|p=31}}}} * The solution in this patch removes the hack from the previous version (53bbcbb3). As documented in the bug report, this solution essentially "trusts" template authors and lets nested refs through when they come from templates. But, top-level nested refs are still not allowed. <ref>foo<ref>bar</ref>baz</ref> in the toplevel page will still generate a single reference with content "foo<ref>bar</ref>baz" * Updated the nested ref test output - Regenerated selser changes because of the changed output. - Regenerated blacklist because of the changed selser changes. * This now handles: - https://en.wikipedia.org/wiki/User:Edgepedia/VE/GNoSR - https://en.wikipedia.org/wiki/Phellinus ellipsoideus Change-Id: I627955f0be1c5e2bafc49647c94c2be68ce711a8
2013-07-15 22:31:20 +00:00
inTemplate: pipelineOpts.inTemplate,
Introduce Parsing/DOM scopes for content fragments * Added mechanism to parse a set of tokens in its own parsing scope all the way to DOM (which in turn restricts token transformations to just that scope). This is equivalent to enforcing well-balanced requirements in restricted contexts (Ex: link content, image captions for starters). This also provides an option of enforcing balanced templates in certain contexts. This patch applies it to link content and image captions. Deleted the hacky closeUnclosedBlockTags code for dealing with bad HTML in captions. * Refactored common/duplicate code out of Cite handler and Template handler. * Updated DSR handling for dom-fragments which eliminates the warnings about cs/s dsr mismatches seen on image reuse. it:Dalmine used to get a bunch of DSR inconsistency warnings when dom-fragments were reused (reported in bug 53071) and are now fixed with this patch. * There is still one big hole in DOM fragment unpacking that has to be fixed. This pertains to improper tag nesting that will be broken up by the tree builder. The DOM fragment unpacker has to recognize these scenarios and fix up the DOM (either by fixing up the final DOM or by stripping mis-nested tags in the DOM fragment being unpacked). This patch has an incomplete hack for this that addresses the common-use case of nested a-tags generated by wikitext of this form: "[http://foo.bar a [[Wikilink] here]". Bug: 54454 Bug: 49942 Bug: 44476 Bug: 47326 Change-Id: I33ee38bc43743125b705ac821b339586593dbef7
2013-09-24 01:07:29 +00:00
noPre: true,
(Bug 64901) Fix paragraph-wrapping to match PHP parser + Tidy combo General changes --------------- * Replaced the hacky 'inBlockNode' parser pipeline option with a cleaner 'noPWrapping' option that suppresses paragraph wrapping in sub-pipelines (ex: recursive link content, ref tags, attribute content, etc.). Changes to wt2html pipeline --------------------------- * Fixed paragraph-wrapping code to ensure that there are no bare text nodes left behind, but without removing the line-based block-tag influences on p-wrapping. Some simplifications as well. TODO: There are still some discrepancies around <blockquote> p-wrapping behavior. These will be investigated and addressed in a future patch. * Fixed foster parenting code to ensure that fostered content is added in p-tags where necessary rather than span-tags. Changes to html2wt/selser pipeline ---------------------------------- * Fixed DOMDiff to tag mw:DiffMarker nodes with a is-block-node attribute when the deleted node is a block node. This is used during selective serialization to discard original separators between adjacent p-nodes if either of their neighbors is a deleted block node. * Fixed serialization to account for changes to p-wrapping. - Updated tag handlers for the <p> tag. - Updated separator handling to deal with deleted block tags and their influence on separators around adjacent p-tags. - Updated selser output code to test whether a deleted block tag forces nowiki escaping on unedited content from adjacent p-tags. Changes to parser tests / test setup ------------------------------------ * Tweaked selser test generation to ensure that text nodes are always inserted in p-wrappers where necessary. * Updated parser test output for several tests to introduce p-tags instead of span-tags or missing p-tags, add html/parsoid section, or in one case, add missing HTML output. Parser Test Result changes -------------------------- Newly passing - 12 wt2html - 1 wt2wt - 3 html2html - 3 html2wt Newly failing - 1 html2wt "3. Leading whitespace in indent-pre suppressing contexts should not be escaped" This is just normalization of output where multiple HTML forms serialize to the same wikitext with a newline difference. It is not worth the complexity to fix this. - 1 wt2wt ""Trailing newlines in a deep dom-subtree that ends a wikitext line" This is again normalization during serialization where an extra unnecessary newline is introduced. - A bunch of selser test changes. 182 +add, 188 -add => 6 fewer selser failures - That is a lot of changes to sift through, and I didn't look at every one of those, but a number of changes seem to be harmless, and just a change to previously "failing" tests. - "Media link with nasty text" test seems to have a lot of selser changes, but the HTML generated by Parsoid seems to be "buggy" with interesting DSR values as well. That needs investigation separately. - "HTML nested bullet list, closed tags (bug 5497) [[3,3,4,[0,1,4],3]]" has seen a degradation where a dirty diff got introduced. Haven't investigated carefully why that is so. Change-Id: Ia9c9950717120fbcd03abfe4e09168e787669ac4
2014-08-29 05:59:20 +00:00
noPWrapping: true
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
},
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
res: [],
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
parentCB: cb,
emptyContentCB: finalCB,
documentCB: function(refContentDoc) {
(Bug 69827): Supported nested <ref> tags once more * Nested <ref> tag support was broken in 69b6ec4d since that patch effectively processed <refs> only on the final DOM rather than extracted <ref> information in sub-pipelines. In doing so, it broke support for <ref> in <ref> tags which are supported by {{efn}} templates and used as follows. {{efn|A clarification.{{sfn|Smith|2009|p=2}}}} The {{sfn}} tpl generates a <ref> tag inside another <ref> tag generated by the {{efn}} tpl. * This patch fixes the breakage by processing <ref> content after it is extracted (in case it is known to have nested <ref> tags). * This fixes the rendering on enwiki/Otto_I%2C_Holy_Roman_Emperor and is an improvement over when it was broken. The nested <ref> gets id 18 (just like the Cite.php handling) whereas it used to get id 1 before the breakage. * TODO: - Figure out a way to add a test for this. Change-Id: Ib82b75f66249b2133a123adbe6fd7acbfd8ec8fb
2014-08-21 23:16:32 +00:00
finalCB([], refContentDoc.body);
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
}
});
};
Added a new parsing pipeline for processing ref tokens. * Added a tokens/x-mediawiki/post-expansion sub-pipeline that is used to process ref-content. This same pipeline type could also be used to post-process tokenized content from other extensions. * Cleaned up transform handler ranks a bit. * Fixed paragraph-handler to accept initial state and also fixed it to do a complete reset on encountering EOFTk. Cite ref-content handling uses the sub-pipeline while passing the inBlockToken flag to prevent top-level p-wrapping of ref-content. * Fixes references output on en:Barack Obama and introduces lists in individual ref items. * No change in parser tests -- how do we add tests for testing extensions like cite/ref? Change-Id: I928a3a0f5f989ec28bbe73383794fe6f32599fd1
2012-11-29 20:09:06 +00:00
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
/**
* Helper class used by <references> implementation
*/
function RefGroup(group) {
this.name = group || '';
this.refs = [];
Use maps where wanting Comments seemed to indicate so. Change-Id: I63d0524d134d8be166403cfa9d6400cecc1749ed
2014-01-10 03:33:57 +00:00
this.indexByName = new Map();
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
}
(Bug 57252) Generate HTML5-compliant cite id/about attr values * Currently, this mimics Cite.php behavior where "a b" and "a_b" are considered identical ids. * Added new parser test. * Fixed output of another test. * Fixed section name of a commented out test. Change-Id: I0c51404c3e659bbddfe9a8909aa6a109d368b762
2014-01-03 23:21:46 +00:00
function makeValidIdAttr(val) {
// Looks like Cite.php doesn't try to fix ids that already have
// a "_" in them. Ex: name="a b" and name="a_b" are considered
// identical. Not sure if this is a feature or a bug.
[Cite] Generate the same ids for <ref>s and notes as Cite.php One particular case is that Cite.php considers equal a name and its encoding, i.e. "a & b" === "a &amp; b". Added a new test for this case, but blacklisted it on html2wt, wt2wt and html2html due to a different problem with how Parsoid encodes entities. This will be investigated separately, as a simple fix could break unrelated cases. Also updated tests and blacklist to the new ids. Change-Id: I87637a1dc812a3a8f29327b9e6c0040b22a651c4
2014-12-20 12:49:12 +00:00
// It also considers entities equal to their encoding (i.e. '&' === '&amp;')
// and then substitutes % with .
var v = entities.decodeHTML(val).replace(/\s/g, '_');
return encodeURIComponent(v).replace(/%/g,".");
(Bug 57252) Generate HTML5-compliant cite id/about attr values * Currently, this mimics Cite.php behavior where "a b" and "a_b" are considered identical ids. * Added new parser test. * Fixed output of another test. * Fixed section name of a commented out test. Change-Id: I0c51404c3e659bbddfe9a8909aa6a109d368b762
2014-01-03 23:21:46 +00:00
}
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
RefGroup.prototype.renderLine = function(refsList, ref) {
var ownerDoc = refsList.ownerDocument,
arrow = ownerDoc.createTextNode('↑'),
T88290: Only output <ref> contents in <references> Instead of outputting a <ref>'s HTML in both data-mw and in <references>, output it only in the later and point to it from data-mw.body.id. Also preserve data-parsoid for <ref>s text in <references>, as now that's the only representation of it. To correctly do html2wt when there are <ref>s inside <references> we need access to the main document DOM when serializing, so also ensure that env.page.dom is correctly set (it was only set in v2 before). Updated tests results and blacklist (some tests now pass). Change-Id: I0fa7ad692585af19136909bfec39db9868b137c5
2015-02-19 18:06:29 +00:00
li, a, textSpan;
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
Updates to cite to reflect spec for references and ref * Updated references output DOM to match spec better. * Also refactored some repetitive DOM code into helper functions. * Turned on tpl encapsulation inside extensions. * Verified output on the example in the spec and the example in commit 3c88b310. Change-Id: I1b894759d2cd89b6453156e6b0246f6cc8f0d60b
2013-05-02 16:17:08 +00:00
// Generate the li and set ref content first, so the HTML gets parsed.
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
// We then append the rest of the ref nodes before the first node
Updates to cite to reflect spec for references and ref * Updated references output DOM to match spec better. * Also refactored some repetitive DOM code into helper functions. * Turned on tpl encapsulation inside extensions. * Verified output on the example in the spec and the example in commit 3c88b310. Change-Id: I1b894759d2cd89b6453156e6b0246f6cc8f0d60b
2013-05-02 16:17:08 +00:00
li = ownerDoc.createElement('li');
DU.addAttributes(li, {
'about': "#" + ref.target,
'id': ref.target
});
T88290: Only output <ref> contents in <references> Instead of outputting a <ref>'s HTML in both data-mw and in <references>, output it only in the later and point to it from data-mw.body.id. Also preserve data-parsoid for <ref>s text in <references>, as now that's the only representation of it. To correctly do html2wt when there are <ref>s inside <references> we need access to the main document DOM when serializing, so also ensure that env.page.dom is correctly set (it was only set in v2 before). Updated tests results and blacklist (some tests now pass). Change-Id: I0fa7ad692585af19136909bfec39db9868b137c5
2015-02-19 18:06:29 +00:00
textSpan = ownerDoc.createElement('span');
DU.addAttributes(textSpan, {
'id': "mw-reference-text-" + ref.target,
'class': "mw-reference-text"
});
textSpan.innerHTML = ref.content;
li.appendChild(textSpan);
Updates to cite to reflect spec for references and ref * Updated references output DOM to match spec better. * Also refactored some repetitive DOM code into helper functions. * Turned on tpl encapsulation inside extensions. * Verified output on the example in the spec and the example in commit 3c88b310. Change-Id: I1b894759d2cd89b6453156e6b0246f6cc8f0d60b
2013-05-02 16:17:08 +00:00
// 'mw:referencedBy' span wrapper
var span = ownerDoc.createElement('span');
span.setAttribute('rel', 'mw:referencedBy');
T88290: Only output <ref> contents in <references> Instead of outputting a <ref>'s HTML in both data-mw and in <references>, output it only in the later and point to it from data-mw.body.id. Also preserve data-parsoid for <ref>s text in <references>, as now that's the only representation of it. To correctly do html2wt when there are <ref>s inside <references> we need access to the main document DOM when serializing, so also ensure that env.page.dom is correctly set (it was only set in v2 before). Updated tests results and blacklist (some tests now pass). Change-Id: I0fa7ad692585af19136909bfec39db9868b137c5
2015-02-19 18:06:29 +00:00
li.insertBefore(span, textSpan);
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
// Generate leading linkbacks
if (ref.linkbacks.length === 1) {
a = ownerDoc.createElement('a');
Updates to cite to reflect spec for references and ref * Updated references output DOM to match spec better. * Also refactored some repetitive DOM code into helper functions. * Turned on tpl encapsulation inside extensions. * Verified output on the example in the spec and the example in commit 3c88b310. Change-Id: I1b894759d2cd89b6453156e6b0246f6cc8f0d60b
2013-05-02 16:17:08 +00:00
DU.addAttributes(a, {
[Cite] Generate the same ids for <ref>s and notes as Cite.php One particular case is that Cite.php considers equal a name and its encoding, i.e. "a & b" === "a &amp; b". Added a new test for this case, but blacklisted it on html2wt, wt2wt and html2html due to a different problem with how Parsoid encodes entities. This will be investigated separately, as a simple fix could break unrelated cases. Also updated tests and blacklist to the new ids. Change-Id: I87637a1dc812a3a8f29327b9e6c0040b22a651c4
2014-12-20 12:49:12 +00:00
'href': '#' + ref.id
Updates to cite to reflect spec for references and ref * Updated references output DOM to match spec better. * Also refactored some repetitive DOM code into helper functions. * Turned on tpl encapsulation inside extensions. * Verified output on the example in the spec and the example in commit 3c88b310. Change-Id: I1b894759d2cd89b6453156e6b0246f6cc8f0d60b
2013-05-02 16:17:08 +00:00
});
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
a.appendChild(arrow);
Updates to cite to reflect spec for references and ref * Updated references output DOM to match spec better. * Also refactored some repetitive DOM code into helper functions. * Turned on tpl encapsulation inside extensions. * Verified output on the example in the spec and the example in commit 3c88b310. Change-Id: I1b894759d2cd89b6453156e6b0246f6cc8f0d60b
2013-05-02 16:17:08 +00:00
span.appendChild(a);
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
} else {
Updates to cite to reflect spec for references and ref * Updated references output DOM to match spec better. * Also refactored some repetitive DOM code into helper functions. * Turned on tpl encapsulation inside extensions. * Verified output on the example in the spec and the example in commit 3c88b310. Change-Id: I1b894759d2cd89b6453156e6b0246f6cc8f0d60b
2013-05-02 16:17:08 +00:00
span.appendChild(arrow);
Use es6 methods instead of fakejquery. In particular, use `Array.isArray` instead of `$.isArray`, and `Object.assign` instead of `$.extend`. `Object.assign` operates only on own properties, so use `Object.create` on the prototype where necessary to get inherited properties. `Object.assign` does a simple assignment and is appropriate in most places, but be careful if we ever install getters/setters on a prototype. Implement `Util.clone()` from scratch to have a jquery-compatible deep clone operation. In particular, this needs to ignore objects which aren't "plain objects", so we don't try to clone DOM nodes. Our definition of a "plain object" is compatible with jquery/zepto.js, and is thus something of a hack. We should eventually replace this with a `console.assert()` and remove/rewrite the places where we try to clone objects which contain DOM trees and other cruft. Change-Id: I88c8fe41a9be84c167d5a0ea1187fd258f077968
2014-01-22 20:24:23 +00:00
ref.linkbacks.forEach(function(linkback, i) {
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
a = ownerDoc.createElement('a');
Updates to cite to reflect spec for references and ref * Updated references output DOM to match spec better. * Also refactored some repetitive DOM code into helper functions. * Turned on tpl encapsulation inside extensions. * Verified output on the example in the spec and the example in commit 3c88b310. Change-Id: I1b894759d2cd89b6453156e6b0246f6cc8f0d60b
2013-05-02 16:17:08 +00:00
DU.addAttributes(a, {
'href': '#' + ref.linkbacks[i]
});
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
a.appendChild(ownerDoc.createTextNode(ref.groupIndex + '.' + i));
// Separate linkbacks with a space
Updates to cite to reflect spec for references and ref * Updated references output DOM to match spec better. * Also refactored some repetitive DOM code into helper functions. * Turned on tpl encapsulation inside extensions. * Verified output on the example in the spec and the example in commit 3c88b310. Change-Id: I1b894759d2cd89b6453156e6b0246f6cc8f0d60b
2013-05-02 16:17:08 +00:00
span.appendChild(ownerDoc.createTextNode(' '));
span.appendChild(a);
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
});
}
(Bug 49314) Add space before ref-content Change-Id: Ie67ba1c332451b60107f6fb946efeafc15485b50
2013-06-12 21:07:01 +00:00
// Space before content node
T88290: Only output <ref> contents in <references> Instead of outputting a <ref>'s HTML in both data-mw and in <references>, output it only in the later and point to it from data-mw.body.id. Also preserve data-parsoid for <ref>s text in <references>, as now that's the only representation of it. To correctly do html2wt when there are <ref>s inside <references> we need access to the main document DOM when serializing, so also ensure that env.page.dom is correctly set (it was only set in v2 before). Updated tests results and blacklist (some tests now pass). Change-Id: I0fa7ad692585af19136909bfec39db9868b137c5
2015-02-19 18:06:29 +00:00
li.insertBefore(ownerDoc.createTextNode(' '), textSpan);
(Bug 49314) Add space before ref-content Change-Id: Ie67ba1c332451b60107f6fb946efeafc15485b50
2013-06-12 21:07:01 +00:00
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
// Add it to the ref list
refsList.appendChild(li);
};
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
function ReferencesData() {
this.index = 0;
this.refGroups = new Map();
Fix shoddy work I did in fdf5fa4b when I broke references * I will work on adding testing for refs and references next. Change-Id: Ib9be770d70587cd36ecad4df8067e646eac6d750
2013-06-12 21:01:27 +00:00
}
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
ReferencesData.prototype.getRefGroup = function (groupName, allocIfMissing) {
groupName = groupName || '';
if ( !this.refGroups.has( groupName ) && allocIfMissing ) {
this.refGroups.set( groupName, new RefGroup( groupName ) );
}
return this.refGroups.get( groupName );
};
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
ReferencesData.prototype.removeRefGroup = function (groupName) {
if (groupName !== null && groupName !== undefined) {
Cite: Parse <references> content all the way to DOM. * Thus far, <references> tag content was being parsed to stage 2 and merged into main pipeline. This patch takes this all the way to DOM. This required some tweaks to handling of <ref>s nested inside <references>. * Fixed up a buggy parser test in the bargain -- the old parsoid result was buggy as well. I verified output in the enwp sandbox. Change-Id: Iff6c528066b71ce1b00dd769910a04ee66623340
2014-03-04 23:13:43 +00:00
// '' is a valid group (the default group)
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
this.refGroups.delete(groupName);
}
};
ReferencesData.prototype.add = function(groupName, refName, about, skipLinkback) {
var group = this.getRefGroup(groupName, true),
ref;
refName = makeValidIdAttr(refName);
if ( refName && group.indexByName.has( refName ) ) {
ref = group.indexByName.get( refName );
if (ref.content) {
ref.hasMultiples = true;
}
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
} else {
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
// The ids produced Cite.php have some particulars:
// Simple refs get 'cite_ref-' + index
// Refs with names get 'cite_ref-' + name + '_' + index + (backlink num || 0)
// Notes (references) whose ref doesn't have a name are 'cite_note-' + index
// Notes whose ref has a name are 'cite_note-' + name + '-' + index
var n = this.index,
[jscs] Enforce requireSpaceBeforeBinaryOperators * This was auto-corrected with jscs -x . Change-Id: I13df3896e8130743eb31581e80d4c7909e48768d
2015-04-06 21:07:04 +00:00
refKey = (1 +n) + '',
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
refIdBase = 'cite_ref-' + (refName ? refName + '_' + refKey : refKey),
noteId = 'cite_note-' + (refName ? refName + '-' + refKey : refKey);
// bump index
this.index += 1;
ref = {
about: about,
content: null,
group: group.name,
groupIndex: group.refs.length + 1,
index: n,
key: refIdBase,
id: (refName ? refIdBase + '-0' : refIdBase),
linkbacks: [],
name: refName,
target: noteId
};
group.refs.push( ref );
if (refName) {
group.indexByName.set( refName, ref );
}
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
}
Properly bump reference anchors * Adds an index of all the references on a page in order to avoid repeating attrs when multiple <references /> tags are present. * Update tests to reflect the new behaviour. Bug: 59782 Change-Id: Ia44bf59a9304788aca170041d3b85f53557151fc
2014-01-10 21:13:20 +00:00
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
if (!skipLinkback) {
ref.linkbacks.push(ref.key + '-' + ref.linkbacks.length);
Properly bump reference anchors * Adds an index of all the references on a page in order to avoid repeating attrs when multiple <references /> tags are present. * Update tests to reflect the new behaviour. Bug: 59782 Change-Id: Ia44bf59a9304788aca170041d3b85f53557151fc
2014-01-10 21:13:20 +00:00
}
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
return ref;
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
};
Improved template roundtripping based on WT ranges on DOM nodes. - Fixed several bugs in the DSR computation. - Deleted comment tokens from template output. - Eliminated TSR info from template output. - Rearranged order of DOM post passes. - Updated template encapsulation to use DSR values. - There are still a few missing pieces and scenarios that are not handled properly. Change-Id: I644914902ec1d27418138b107abe765f81178398
2012-09-17 19:46:44 +00:00
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
function References(cite) {
this.cite = cite;
this.reset();
}
References.prototype.reset = function() {};
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
/**
* Sanitize the references tag and convert it into a meta-token
*/
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
References.prototype.handleReferences = function ( manager, pipelineOpts, refsTok, cb ) {
Only pass document.body to the post-processors * With some other cleanup pulled out of the external ids patch. Change-Id: I8cee994079579d922da97a4c1c793540bfd8d1b1
2014-03-14 19:39:00 +00:00
var env = manager.env;
Handle multiply-referenced ref-tags without duplicate ref-content. * When named-ref-tags are used multiple times in a document, the Cite.onReferences function was outputting the ref-content once for each use in the references section. While this is "harmless" for regular ref-content, when the ref-content has templates (ex: {[cite .. }}), that template content is duplicated several times -- this also means that template expansions seen in the output no longer has unique ids. Occasionally, this trips up DOM passes (template encapsulation) which make the reasonable assumption that template ids are unique. It introduced an infinite loop in the code that detects template range nestings and overlaps. This is not triggered always, but only in certain scenarios. en:Euphoria (Enrique Iglesias album) was a page that sent the parser into an infinite loop. * Now fixed by outputting ref-content once. * In addition, added an error-check in the DOM post-pass that might get stuck in an infinite loop to output an error message. * No change in parser test results. * en:Euphoria (Enrique Iglesias album) now parses and rts without getting stuck in an infinite loop. Possibly other pages as well. Change-Id: I81a055999eefe3ca36dfb8a9cda3fee538ae6c65
2013-02-20 23:50:52 +00:00
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
// group is the only recognized option?
Temp fix for handling non-string ref attribute vals * Entities in ref name weren't expected * Fixes the crash from arwiki:تأثير_الدمعة_السوداء * Makes use of the fix from de3642b8dd4a804ac654f2943a900496f2c8b3f3 Bug: 63790 Change-Id: Icb8781b4d9decc5a8b115d0b11def4d18f5d5025
2014-04-14 23:04:31 +00:00
var refsOpts = Object.assign({
group: null
}, Util.KVtoHash(refsTok.getAttribute("options"), true));
Make Cite <references> group attribute handling more robust The PHP preprocessor returns a references tag with an empty group attribute, which we did not map to the null reference group. This caused the references listing to vanish when using the PHP preprocessor. This patch makes sure the attribute is converted to a string, stripped of leading/trailing whitespace and finally set to null if no string remains. No changes in parser tests, since Cite is in a separate extension. Change-Id: Ib3de375225947a78c362370be8d78504fa24def2
2012-11-27 23:13:32 +00:00
Make <ref>s nested in <references> editable. * Track <ref>s birthed in <references> tags and capture their HTML and set <references>.data-mw.body.html to the concatenated HTML for all those child <ref> tags. * No change in parser-tests because our test normalization code strips data-mw before comparing results. Since this patch only changes data-mw output for "References: 5. ..." test, there is no change. In any case, this patch updates the data-mw output for the test in question so that if we do modify our normalization routines, the test will still pass. Change-Id: Iaea752dec03e5e7f0d36baf3b7a9973f1b85e8a5
2013-07-24 00:15:45 +00:00
// Assign an about id and intialize the nested refs html
Only pass document.body to the post-processors * With some other cleanup pulled out of the external ids patch. Change-Id: I8cee994079579d922da97a4c1c793540bfd8d1b1
2014-03-14 19:39:00 +00:00
var referencesId = env.newAboutId();
Make <ref>s nested in <references> editable. * Track <ref>s birthed in <references> tags and capture their HTML and set <references>.data-mw.body.html to the concatenated HTML for all those child <ref> tags. * No change in parser-tests because our test normalization code strips data-mw before comparing results. Since this patch only changes data-mw output for "References: 5. ..." test, there is no change. In any case, this patch updates the data-mw output for the test in question so that if we do modify our normalization routines, the test will still pass. Change-Id: Iaea752dec03e5e7f0d36baf3b7a9973f1b85e8a5
2013-07-24 00:15:45 +00:00
(Bug 50218) Generated DOMFragment wrapping <ol> for references tag * So far, references wikitext was being replaced with a meta marker token that participated as an inline token in p-wrapping. As a result, in some cases, a p-tag was wrapping the references ol-node which is buggy HTML. When VE loaded this DOM, the p-node was split around the ol-node and duplicated DSR which led to duplicated output when selser ran on this. * One way of fixing this would have been to add a special case in the paragraph wrapper to treat the mw:Extensions/References/Marker meta tag as a block token. * A better solution with an eye towards the longer-term is to emit mw:DOMFragment wrapper for the references tag with the content being an ol-node. The dom-fragment unpacking code then takes care of p-wrapping issues for the ol-node. This fixes the bug in this case. * Also fixed parser tests output where group attribute was missing on some tests (which indicated that the older code was not emitting the group attribute on references tag). * No change in parser tests. Change-Id: I073b2e68667d577c75cad07d19cea2b19d0e89fe
2013-06-26 17:35:08 +00:00
// Emit a marker mw:DOMFragment for the references
// token so that the dom post processor can generate
// and emit references at this point in the DOM.
Cite: Parse <references> content all the way to DOM. * Thus far, <references> tag content was being parsed to stage 2 and merged into main pipeline. This patch takes this all the way to DOM. This required some tweaks to handling of <ref>s nested inside <references>. * Fixed up a buggy parser test in the bargain -- the old parsoid result was buggy as well. I verified output in the enwp sandbox. Change-Id: Iff6c528066b71ce1b00dd769910a04ee66623340
2014-03-04 23:13:43 +00:00
var emitReferencesFragment = function(toks, refsBody) {
Added a flag to DOM-fragment unpacking to update fragment DSR * This patch adds a flag to DOM fragment unpacking to update fragment DSR in cases where the fragment needs it (references block + reused template/extension/images content). In other cases of dom-fragment use, the DSR should not be updated. Ex: All cases where fragments are used to implement parsing scopes (all A-tag content currently) since DSR computation is set up with offsets in the top-level source. This is not just an optimization, but a correctness issue since the fragment unwrapper always sets the fragment DSR on the first node which would be incorrect in scenarios where the fragment DOM has multiple top-level nodes. * Parser test runs now have better DSR values in certain cases. Change-Id: If1f5bf98dab246a3c8a1869b38335e90268cb5c5
2014-01-21 22:03:26 +00:00
var type = refsTok.getAttribute('typeof');
var olHTML = "<ol class='references'" +
" typeof='mw:Extension/references'" +
Cite: Parse <references> content all the way to DOM. * Thus far, <references> tag content was being parsed to stage 2 and merged into main pipeline. This patch takes this all the way to DOM. This required some tweaks to handling of <ref>s nested inside <references>. * Fixed up a buggy parser test in the bargain -- the old parsoid result was buggy as well. I verified output in the enwp sandbox. Change-Id: Iff6c528066b71ce1b00dd769910a04ee66623340
2014-03-04 23:13:43 +00:00
" about='" + referencesId + "'" + ">" + (refsBody || "") + "</ol>";
Added a flag to DOM-fragment unpacking to update fragment DSR * This patch adds a flag to DOM fragment unpacking to update fragment DSR in cases where the fragment needs it (references block + reused template/extension/images content). In other cases of dom-fragment use, the DSR should not be updated. Ex: All cases where fragments are used to implement parsing scopes (all A-tag content currently) since DSR computation is set up with offsets in the top-level source. This is not just an optimization, but a correctness issue since the fragment unwrapper always sets the fragment DSR on the first node which would be incorrect in scenarios where the fragment DOM has multiple top-level nodes. * Parser test runs now have better DSR values in certain cases. Change-Id: If1f5bf98dab246a3c8a1869b38335e90268cb5c5
2014-01-21 22:03:26 +00:00
var olProcessor = function(ol) {
var dp = DU.getDataParsoid( ol );
dp.src = refsTok.getAttribute('source');
Temp fix for handling non-string ref attribute vals * Entities in ref name weren't expected * Fixes the crash from arwiki:تأثير_الدمعة_السوداء * Makes use of the fix from de3642b8dd4a804ac654f2943a900496f2c8b3f3 Bug: 63790 Change-Id: Icb8781b4d9decc5a8b115d0b11def4d18f5d5025
2014-04-14 23:04:31 +00:00
if ( refsOpts.group ) {
dp.group = refsOpts.group;
Added a flag to DOM-fragment unpacking to update fragment DSR * This patch adds a flag to DOM fragment unpacking to update fragment DSR in cases where the fragment needs it (references block + reused template/extension/images content). In other cases of dom-fragment use, the DSR should not be updated. Ex: All cases where fragments are used to implement parsing scopes (all A-tag content currently) since DSR computation is set up with offsets in the top-level source. This is not just an optimization, but a correctness issue since the fragment unwrapper always sets the fragment DSR on the first node which would be incorrect in scenarios where the fragment DOM has multiple top-level nodes. * Parser test runs now have better DSR values in certain cases. Change-Id: If1f5bf98dab246a3c8a1869b38335e90268cb5c5
2014-01-21 22:03:26 +00:00
}
(Bug 53109) Move data-mw away from manual json attr loading * This is both faster and consistent with how we're accessing other parsoid attributes. It's also a step towards not having this data in the html output. * Changes to parserTests and the blacklist are for attribute order. * Requires upgrading domino to 1.0.18 https://github.com/fgnass/domino/pull/48 Change-Id: I1edbc260887d480adf04763b15043c374e27cceb
2014-09-08 20:10:25 +00:00
DU.storeDataParsoid( ol, dp );
(Bug 50218) Generated DOMFragment wrapping <ol> for references tag * So far, references wikitext was being replaced with a meta marker token that participated as an inline token in p-wrapping. As a result, in some cases, a p-tag was wrapping the references ol-node which is buggy HTML. When VE loaded this DOM, the p-node was split around the ol-node and duplicated DSR which led to duplicated output when selser ran on this. * One way of fixing this would have been to add a special case in the paragraph wrapper to treat the mw:Extensions/References/Marker meta tag as a block token. * A better solution with an eye towards the longer-term is to emit mw:DOMFragment wrapper for the references tag with the content being an ol-node. The dom-fragment unpacking code then takes care of p-wrapping issues for the ol-node. This fixes the bug in this case. * Also fixed parser tests output where group attribute was missing on some tests (which indicated that the older code was not emitting the group attribute on references tag). * No change in parser tests. Change-Id: I073b2e68667d577c75cad07d19cea2b19d0e89fe
2013-06-26 17:35:08 +00:00
};
Added a flag to DOM-fragment unpacking to update fragment DSR * This patch adds a flag to DOM fragment unpacking to update fragment DSR in cases where the fragment needs it (references block + reused template/extension/images content). In other cases of dom-fragment use, the DSR should not be updated. Ex: All cases where fragments are used to implement parsing scopes (all A-tag content currently) since DSR computation is set up with offsets in the top-level source. This is not just an optimization, but a correctness issue since the fragment unwrapper always sets the fragment DSR on the first node which would be incorrect in scenarios where the fragment DOM has multiple top-level nodes. * Parser test runs now have better DSR values in certain cases. Change-Id: If1f5bf98dab246a3c8a1869b38335e90268cb5c5
2014-01-21 22:03:26 +00:00
cb({
async: false,
Renamed function name & reorder args for clarity + added docs. * Renamed buildDOMFragmentForTokenStream --> buildDOMFragmentTokens and made env. the first arg. * Added documentation to buildDOMFragmentTokens and encapsulateHTML Change-Id: I7eccfd3f4dc5b4b91d20d1d24d98ec514df6dfbc
2014-02-20 23:03:09 +00:00
tokens: DU.buildDOMFragmentTokens(
manager.env,
Added a flag to DOM-fragment unpacking to update fragment DSR * This patch adds a flag to DOM fragment unpacking to update fragment DSR in cases where the fragment needs it (references block + reused template/extension/images content). In other cases of dom-fragment use, the DSR should not be updated. Ex: All cases where fragments are used to implement parsing scopes (all A-tag content currently) since DSR computation is set up with offsets in the top-level source. This is not just an optimization, but a correctness issue since the fragment unwrapper always sets the fragment DSR on the first node which would be incorrect in scenarios where the fragment DOM has multiple top-level nodes. * Parser test runs now have better DSR values in certain cases. Change-Id: If1f5bf98dab246a3c8a1869b38335e90268cb5c5
2014-01-21 22:03:26 +00:00
refsTok,
olHTML,
olProcessor,
Cite: Parse <references> content all the way to DOM. * Thus far, <references> tag content was being parsed to stage 2 and merged into main pipeline. This patch takes this all the way to DOM. This required some tweaks to handling of <ref>s nested inside <references>. * Fixed up a buggy parser test in the bargain -- the old parsoid result was buggy as well. I verified output in the enwp sandbox. Change-Id: Iff6c528066b71ce1b00dd769910a04ee66623340
2014-03-04 23:13:43 +00:00
// The <ol> HTML above is wrapper HTML added on and doesn't
// have any DSR on it. We want DSR added to it.
Added a flag to DOM-fragment unpacking to update fragment DSR * This patch adds a flag to DOM fragment unpacking to update fragment DSR in cases where the fragment needs it (references block + reused template/extension/images content). In other cases of dom-fragment use, the DSR should not be updated. Ex: All cases where fragments are used to implement parsing scopes (all A-tag content currently) since DSR computation is set up with offsets in the top-level source. This is not just an optimization, but a correctness issue since the fragment unwrapper always sets the fragment DSR on the first node which would be incorrect in scenarios where the fragment DOM has multiple top-level nodes. * Parser test runs now have better DSR values in certain cases. Change-Id: If1f5bf98dab246a3c8a1869b38335e90268cb5c5
2014-01-21 22:03:26 +00:00
{ aboutId: referencesId, setDSR: true, isForeignContent: true }
)
});
};
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
processExtSource(manager, refsTok, {
// Partial pipeline for processing ref-content
// Expand till stage 2 so that all embedded
// ref tags get processed
Cite: Parse <references> content all the way to DOM. * Thus far, <references> tag content was being parsed to stage 2 and merged into main pipeline. This patch takes this all the way to DOM. This required some tweaks to handling of <ref>s nested inside <references>. * Fixed up a buggy parser test in the bargain -- the old parsoid result was buggy as well. I verified output in the enwp sandbox. Change-Id: Iff6c528066b71ce1b00dd769910a04ee66623340
2014-03-04 23:13:43 +00:00
pipelineType: 'text/x-mediawiki/full',
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
pipelineOpts: {
Cite: Parse <references> content all the way to DOM. * Thus far, <references> tag content was being parsed to stage 2 and merged into main pipeline. This patch takes this all the way to DOM. This required some tweaks to handling of <ref>s nested inside <references>. * Fixed up a buggy parser test in the bargain -- the old parsoid result was buggy as well. I verified output in the enwp sandbox. Change-Id: Iff6c528066b71ce1b00dd769910a04ee66623340
2014-03-04 23:13:43 +00:00
// In order to associated ref-tags nested here with this references
// object, we have to pass along the references id.
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
extTag: "references",
Cite: Parse <references> content all the way to DOM. * Thus far, <references> tag content was being parsed to stage 2 and merged into main pipeline. This patch takes this all the way to DOM. This required some tweaks to handling of <ref>s nested inside <references>. * Fixed up a buggy parser test in the bargain -- the old parsoid result was buggy as well. I verified output in the enwp sandbox. Change-Id: Iff6c528066b71ce1b00dd769910a04ee66623340
2014-03-04 23:13:43 +00:00
extTagId: referencesId,
<references> handler should respect inTemplate pipeline option Change-Id: Ibfc1ea97aef8681bfd3cf04dba0aa32824c076f0
2014-03-07 17:35:58 +00:00
wrapTemplates: pipelineOpts.wrapTemplates,
inTemplate: pipelineOpts.inTemplate
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
},
res: [],
parentCB: cb,
(Bug 50218) Generated DOMFragment wrapping <ol> for references tag * So far, references wikitext was being replaced with a meta marker token that participated as an inline token in p-wrapping. As a result, in some cases, a p-tag was wrapping the references ol-node which is buggy HTML. When VE loaded this DOM, the p-node was split around the ol-node and duplicated DSR which led to duplicated output when selser ran on this. * One way of fixing this would have been to add a special case in the paragraph wrapper to treat the mw:Extensions/References/Marker meta tag as a block token. * A better solution with an eye towards the longer-term is to emit mw:DOMFragment wrapper for the references tag with the content being an ol-node. The dom-fragment unpacking code then takes care of p-wrapping issues for the ol-node. This fixes the bug in this case. * Also fixed parser tests output where group attribute was missing on some tests (which indicated that the older code was not emitting the group attribute on references tag). * No change in parser tests. Change-Id: I073b2e68667d577c75cad07d19cea2b19d0e89fe
2013-06-26 17:35:08 +00:00
emptyContentCB: emitReferencesFragment,
Cite: Parse <references> content all the way to DOM. * Thus far, <references> tag content was being parsed to stage 2 and merged into main pipeline. This patch takes this all the way to DOM. This required some tweaks to handling of <ref>s nested inside <references>. * Fixed up a buggy parser test in the bargain -- the old parsoid result was buggy as well. I verified output in the enwp sandbox. Change-Id: Iff6c528066b71ce1b00dd769910a04ee66623340
2014-03-04 23:13:43 +00:00
endCB: emitReferencesFragment,
documentCB: function(refsDoc) {
(Bug 53109) Move data-mw away from manual json attr loading * This is both faster and consistent with how we're accessing other parsoid attributes. It's also a step towards not having this data in the html output. * Changes to parserTests and the blacklist are for attribute order. * Requires upgrading domino to 1.0.18 https://github.com/fgnass/domino/pull/48 Change-Id: I1edbc260887d480adf04763b15043c374e27cceb
2014-09-08 20:10:25 +00:00
emitReferencesFragment([], DU.serializeChildren(refsDoc.body));
Cite: Parse <references> content all the way to DOM. * Thus far, <references> tag content was being parsed to stage 2 and merged into main pipeline. This patch takes this all the way to DOM. This required some tweaks to handling of <ref>s nested inside <references>. * Fixed up a buggy parser test in the bargain -- the old parsoid result was buggy as well. I verified output in the enwp sandbox. Change-Id: Iff6c528066b71ce1b00dd769910a04ee66623340
2014-03-04 23:13:43 +00:00
}
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
});
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
};
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
References.prototype.extractRefFromNode = function(node, refsData,
refInRefProcessor, referencesAboutId, referencesGroup, refsInReferencesHTML) {
(Bug 69827): Supported nested <ref> tags once more * Nested <ref> tag support was broken in 69b6ec4d since that patch effectively processed <refs> only on the final DOM rather than extracted <ref> information in sub-pipelines. In doing so, it broke support for <ref> in <ref> tags which are supported by {{efn}} templates and used as follows. {{efn|A clarification.{{sfn|Smith|2009|p=2}}}} The {{sfn}} tpl generates a <ref> tag inside another <ref> tag generated by the {{efn}} tpl. * This patch fixes the breakage by processing <ref> content after it is extracted (in case it is known to have nested <ref> tags). * This fixes the rendering on enwiki/Otto_I%2C_Holy_Roman_Emperor and is an improvement over when it was broken. The nested <ref> gets id 18 (just like the Cite.php handling) whereas it used to get id 1 before the breakage. * TODO: - Figure out a way to add a test for this. Change-Id: Ib82b75f66249b2133a123adbe6fd7acbfd8ec8fb
2014-08-21 23:16:32 +00:00
var nestedInReferences = referencesAboutId !== undefined,
(Bug 67237): Fix citation numbering issue * Process <ref> and <references> tag on the top-level DOM only and ignore the generateRefs pass when processing other content. * This required a few fixes: - ensure that DOMPostProcessor knows about the top-level. - ensure that DOMVisitor knows about the top-level. - cleanup pass leaves behind the ref-marker metas from DOMs from non top-level content. - process nested references content. * One of the references tests had incorrect parsed output. That test has been updated to reflect the correct output from this patch. * Barack Obama seems to now have the correct numbering on references. Change-Id: I5465721d2fc715f2168f267e773a446bc37d198b
2014-07-01 18:04:31 +00:00
dp = DU.getDataParsoid( node ),
// SSS FIXME: Need to clarify semantics here.
// If both the containing <references> elt as well as the nested <ref> elt has
// a group attribute, what takes precedence?
group = dp.group || referencesGroup || '',
refName = dp.name,
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
about = node.getAttribute("about"),
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
ref = refsData.add(group, refName, about, nestedInReferences),
Roundtrip {{#tag:ref|..}} properly in editMode * See example below that clarifies the problem before this patch ------------------- $ echo "{{#tag:ref|foo}} {{echo|<references />}}" | node parse --extensions ref,references --wt2wt --editMode true <ref>foo</ref> <references /> ------------------- * The problem is that references are generated after dom fragments are unpacked in the DOM which lost the mw:Transclusion typeof and data-mw that had been set on the ref and references tags. * This is a quick fix to prevent some dirty diffs with #tag:ref being reported on en-wp. Longer term, we do have a plan to use DOMFragment encapsulation for refs and references as well and splitting up references processing differently than is being done currently. Change-Id: I4186cae93b9882d367c7d4efecc092607fe17c61
2013-06-16 22:14:25 +00:00
nodeType = (node.getAttribute("typeof") || '').replace(/mw:Extension\/ref\/Marker/, '');
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
// Add ref-index linkback
Make <ref>s nested in <references> editable. * Track <ref>s birthed in <references> tags and capture their HTML and set <references>.data-mw.body.html to the concatenated HTML for all those child <ref> tags. * No change in parser-tests because our test normalization code strips data-mw before comparing results. Since this patch only changes data-mw output for "References: 5. ..." test, there is no change. In any case, this patch updates the data-mw output for the test in question so that if we do modify our normalization routines, the test will still pass. Change-Id: Iaea752dec03e5e7f0d36baf3b7a9973f1b85e8a5
2013-07-24 00:15:45 +00:00
var doc = node.ownerDocument,
span = doc.createElement('span'),
(Bug 67237): Fix citation numbering issue * Process <ref> and <references> tag on the top-level DOM only and ignore the generateRefs pass when processing other content. * This required a few fixes: - ensure that DOMPostProcessor knows about the top-level. - ensure that DOMVisitor knows about the top-level. - cleanup pass leaves behind the ref-marker metas from DOMs from non top-level content. - process nested references content. * One of the references tests had incorrect parsed output. That test has been updated to reflect the correct output from this patch. * Barack Obama seems to now have the correct numbering on references. Change-Id: I5465721d2fc715f2168f267e773a446bc37d198b
2014-07-01 18:04:31 +00:00
content = dp.content,
T88290: Keep separate content for multiple <ref>s with same name Change-Id: Iae4dcdb405ce8d233af3105e19bab2b32938ed96
2015-02-27 19:11:37 +00:00
dataMW = Util.clone(DU.getDataMw(node)),
body;
Roundtrip {{#tag:ref|..}} properly in editMode * See example below that clarifies the problem before this patch ------------------- $ echo "{{#tag:ref|foo}} {{echo|<references />}}" | node parse --extensions ref,references --wt2wt --editMode true <ref>foo</ref> <references /> ------------------- * The problem is that references are generated after dom fragments are unpacked in the DOM which lost the mw:Transclusion typeof and data-mw that had been set on the ref and references tags. * This is a quick fix to prevent some dirty diffs with #tag:ref being reported on en-wp. Longer term, we do have a plan to use DOMFragment encapsulation for refs and references as well and splitting up references processing differently than is being done currently. Change-Id: I4186cae93b9882d367c7d4efecc092607fe17c61
2013-06-16 22:14:25 +00:00
(Bug 69827): Supported nested <ref> tags once more * Nested <ref> tag support was broken in 69b6ec4d since that patch effectively processed <refs> only on the final DOM rather than extracted <ref> information in sub-pipelines. In doing so, it broke support for <ref> in <ref> tags which are supported by {{efn}} templates and used as follows. {{efn|A clarification.{{sfn|Smith|2009|p=2}}}} The {{sfn}} tpl generates a <ref> tag inside another <ref> tag generated by the {{efn}} tpl. * This patch fixes the breakage by processing <ref> content after it is extracted (in case it is known to have nested <ref> tags). * This fixes the rendering on enwiki/Otto_I%2C_Holy_Roman_Emperor and is an improvement over when it was broken. The nested <ref> gets id 18 (just like the Cite.php handling) whereas it used to get id 1 before the breakage. * TODO: - Figure out a way to add a test for this. Change-Id: Ib82b75f66249b2133a123adbe6fd7acbfd8ec8fb
2014-08-21 23:16:32 +00:00
if (dp.hasRefInRef) {
var html = DU.parseHTML(content).body;
Bug fix in processRefs DOM pass refactoring Change-Id: Icd4ce9919de4ba50e5186b8ec3b1923ae9eeb18e
2015-03-26 16:47:59 +00:00
refInRefProcessor(html);
(Bug 69827): Supported nested <ref> tags once more * Nested <ref> tag support was broken in 69b6ec4d since that patch effectively processed <refs> only on the final DOM rather than extracted <ref> information in sub-pipelines. In doing so, it broke support for <ref> in <ref> tags which are supported by {{efn}} templates and used as follows. {{efn|A clarification.{{sfn|Smith|2009|p=2}}}} The {{sfn}} tpl generates a <ref> tag inside another <ref> tag generated by the {{efn}} tpl. * This patch fixes the breakage by processing <ref> content after it is extracted (in case it is known to have nested <ref> tags). * This fixes the rendering on enwiki/Otto_I%2C_Holy_Roman_Emperor and is an improvement over when it was broken. The nested <ref> gets id 18 (just like the Cite.php handling) whereas it used to get id 1 before the breakage. * TODO: - Figure out a way to add a test for this. Change-Id: Ib82b75f66249b2133a123adbe6fd7acbfd8ec8fb
2014-08-21 23:16:32 +00:00
content = DU.serializeChildren(html);
}
T88290: Keep separate content for multiple <ref>s with same name Change-Id: Iae4dcdb405ce8d233af3105e19bab2b32938ed96
2015-02-27 19:11:37 +00:00
if (content) {
// If there are multiple <ref>s with the same name, but different
// content, we need to record this one's content instead of
// linking to <references>.
if (ref.hasMultiples && content !== ref.content) {
body = {'html': content};
} else {
body = {'id': "mw-reference-text-" + ref.target};
}
}
(Bug 53109) Move data-mw away from manual json attr loading * This is both faster and consistent with how we're accessing other parsoid attributes. It's also a step towards not having this data in the html output. * Changes to parserTests and the blacklist are for attribute order. * Requires upgrading domino to 1.0.18 https://github.com/fgnass/domino/pull/48 Change-Id: I1edbc260887d480adf04763b15043c374e27cceb
2014-09-08 20:10:25 +00:00
if (!Object.keys(dataMW).length) {
dataMW = {
Make <ref>s nested in <references> editable. * Track <ref>s birthed in <references> tags and capture their HTML and set <references>.data-mw.body.html to the concatenated HTML for all those child <ref> tags. * No change in parser-tests because our test normalization code strips data-mw before comparing results. Since this patch only changes data-mw output for "References: 5. ..." test, there is no change. In any case, this patch updates the data-mw output for the test in question so that if we do modify our normalization routines, the test will still pass. Change-Id: Iaea752dec03e5e7f0d36baf3b7a9973f1b85e8a5
2013-07-24 00:15:45 +00:00
'name': 'ref',
// Dont set body if this is a reused reference
// like <ref name='..' /> with empty content.
T88290: Keep separate content for multiple <ref>s with same name Change-Id: Iae4dcdb405ce8d233af3105e19bab2b32938ed96
2015-02-27 19:11:37 +00:00
'body': body,
Make <ref>s nested in <references> editable. * Track <ref>s birthed in <references> tags and capture their HTML and set <references>.data-mw.body.html to the concatenated HTML for all those child <ref> tags. * No change in parser-tests because our test normalization code strips data-mw before comparing results. Since this patch only changes data-mw output for "References: 5. ..." test, there is no change. In any case, this patch updates the data-mw output for the test in question so that if we do modify our normalization routines, the test will still pass. Change-Id: Iaea752dec03e5e7f0d36baf3b7a9973f1b85e8a5
2013-07-24 00:15:45 +00:00
'attrs': {
(Bug 67237): Fix citation numbering issue * Process <ref> and <references> tag on the top-level DOM only and ignore the generateRefs pass when processing other content. * This required a few fixes: - ensure that DOMPostProcessor knows about the top-level. - ensure that DOMVisitor knows about the top-level. - cleanup pass leaves behind the ref-marker metas from DOMs from non top-level content. - process nested references content. * One of the references tests had incorrect parsed output. That test has been updated to reflect the correct output from this patch. * Barack Obama seems to now have the correct numbering on references. Change-Id: I5465721d2fc715f2168f267e773a446bc37d198b
2014-07-01 18:04:31 +00:00
// 1. Use 'dp.group' (which is the group attribute that the ref node had)
// rather than use 'group' (which could be the group from an enclosing
// <references> tag).
// 2. Dont emit empty keys
'group': dp.group || undefined,
Make <ref>s nested in <references> editable. * Track <ref>s birthed in <references> tags and capture their HTML and set <references>.data-mw.body.html to the concatenated HTML for all those child <ref> tags. * No change in parser-tests because our test normalization code strips data-mw before comparing results. Since this patch only changes data-mw output for "References: 5. ..." test, there is no change. In any case, this patch updates the data-mw output for the test in question so that if we do modify our normalization routines, the test will still pass. Change-Id: Iaea752dec03e5e7f0d36baf3b7a9973f1b85e8a5
2013-07-24 00:15:45 +00:00
'name': refName || undefined
Add capability for DOM-based template expansion, and use DOM fragments for extensions This patch adds the capability to expand individual transclusions all the way to DOM, which enforces properly nested templates. This is also needed for DOM-based template re-expansion, which the VE folks would like to use for template editing. In normal parsing, DOM-based expansion is currently disabled by default for transclusions, and enabled by default for extensions. The decision whether to balance a particular transclusion can be based on a statistics-based classification of templates into balanced and unbalanced ones. The advantage of this approach is consistency in behavior for old revisions. Another alternative is to wrap unbalanced transclusions into <domparse> tags which disable DOM-based parsing for all transclusions in its content. This has the advantage that all special cases can be handled after tag insertion, and that balancing could also be enforced in the PHP parser using an extension. Other major changes: * Renamed transclusion content typeof from mw:Object/Template to mw:Transclusion * Renamed extension typeof from mw:Object/Extensions/foo to mw:Extension/foo and switched Cite to use lower-case ref/references too Other minor changes: * Only apply template encapsulation algorithm on DOM to mw:Transclusion and mw:Param objects, and no longer to all mw:Object/* elements. We should probably switch to a more explicit encapsulation type instead. Maybe something like mw:EncapStart/Transclusion and mw:EncapEnd/Transclusion instead of the current mw:Transclusion and mw:Transclusion/End, so that stripping those metas out selectively is easy? * Changed the DOMTraverser logic to let handlers explicitly return the next element to handle. Useful when several siblings are handled, as is the case for the fragment unwrapper for example, and avoids some cases where deleted nodes were still being processed. * Changed Cite to use mw:Extension/Ref{,erences} as all other extensions. * Make sure we round-trip gallery when the PHP preprocessor is not used Five parsoid-specific wt2html tests are failing due to the typeof change. They will be fine once those tests are adjusted. For now, adding them to parser tests blacklist. TODO: * Switch the Cite extension to the generic DOMFragment mechanism instead of marker tokens. Change-Id: I64b560a12695256915d2196d0646347381400e80
2013-05-16 22:23:05 +00:00
}
(Bug 53109) Move data-mw away from manual json attr loading * This is both faster and consistent with how we're accessing other parsoid attributes. It's also a step towards not having this data in the html output. * Changes to parserTests and the blacklist are for attribute order. * Requires upgrading domino to 1.0.18 https://github.com/fgnass/domino/pull/48 Change-Id: I1edbc260887d480adf04763b15043c374e27cceb
2014-09-08 20:10:25 +00:00
};
Make <ref>s nested in <references> editable. * Track <ref>s birthed in <references> tags and capture their HTML and set <references>.data-mw.body.html to the concatenated HTML for all those child <ref> tags. * No change in parser-tests because our test normalization code strips data-mw before comparing results. Since this patch only changes data-mw output for "References: 5. ..." test, there is no change. In any case, this patch updates the data-mw output for the test in question so that if we do modify our normalization routines, the test will still pass. Change-Id: Iaea752dec03e5e7f0d36baf3b7a9973f1b85e8a5
2013-07-24 00:15:45 +00:00
}
DU.addAttributes(span, {
'about': about,
'class': 'reference',
[Cite] Generate the same ids for <ref>s and notes as Cite.php One particular case is that Cite.php considers equal a name and its encoding, i.e. "a & b" === "a &amp; b". Added a new test for this case, but blacklisted it on html2wt, wt2wt and html2html due to a different problem with how Parsoid encodes entities. This will be investigated separately, as a simple fix could break unrelated cases. Also updated tests and blacklist to the new ids. Change-Id: I87637a1dc812a3a8f29327b9e6c0040b22a651c4
2014-12-20 12:49:12 +00:00
'id': nestedInReferences ? undefined :
(ref.name ? ref.linkbacks[ref.linkbacks.length - 1] : ref.id),
Make <ref>s nested in <references> editable. * Track <ref>s birthed in <references> tags and capture their HTML and set <references>.data-mw.body.html to the concatenated HTML for all those child <ref> tags. * No change in parser-tests because our test normalization code strips data-mw before comparing results. Since this patch only changes data-mw output for "References: 5. ..." test, there is no change. In any case, this patch updates the data-mw output for the test in question so that if we do modify our normalization routines, the test will still pass. Change-Id: Iaea752dec03e5e7f0d36baf3b7a9973f1b85e8a5
2013-07-24 00:15:45 +00:00
'rel': 'dc:references',
'typeof': nodeType
});
DU.addTypeOf(span, "mw:Extension/ref");
(Bug 53109) Move data-mw away from manual json attr loading * This is both faster and consistent with how we're accessing other parsoid attributes. It's also a step towards not having this data in the html output. * Changes to parserTests and the blacklist are for attribute order. * Requires upgrading domino to 1.0.18 https://github.com/fgnass/domino/pull/48 Change-Id: I1edbc260887d480adf04763b15043c374e27cceb
2014-09-08 20:10:25 +00:00
var dataParsoid = {
src: dp.src,
dsr: dp.dsr
};
DU.setDataParsoid( span, dataParsoid );
DU.setDataMw( span, dataMW );
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
Make <ref>s nested in <references> editable. * Track <ref>s birthed in <references> tags and capture their HTML and set <references>.data-mw.body.html to the concatenated HTML for all those child <ref> tags. * No change in parser-tests because our test normalization code strips data-mw before comparing results. Since this patch only changes data-mw output for "References: 5. ..." test, there is no change. In any case, this patch updates the data-mw output for the test in question so that if we do modify our normalization routines, the test will still pass. Change-Id: Iaea752dec03e5e7f0d36baf3b7a9973f1b85e8a5
2013-07-24 00:15:45 +00:00
// refIndex-a
var refIndex = doc.createElement('a');
refIndex.setAttribute('href', '#' + ref.target);
refIndex.appendChild(doc.createTextNode(
'[' + ((group === '') ? '' : group + ' ') + ref.groupIndex + ']'
));
span.appendChild(refIndex);
(Bug 69827): Supported nested <ref> tags once more * Nested <ref> tag support was broken in 69b6ec4d since that patch effectively processed <refs> only on the final DOM rather than extracted <ref> information in sub-pipelines. In doing so, it broke support for <ref> in <ref> tags which are supported by {{efn}} templates and used as follows. {{efn|A clarification.{{sfn|Smith|2009|p=2}}}} The {{sfn}} tpl generates a <ref> tag inside another <ref> tag generated by the {{efn}} tpl. * This patch fixes the breakage by processing <ref> content after it is extracted (in case it is known to have nested <ref> tags). * This fixes the rendering on enwiki/Otto_I%2C_Holy_Roman_Emperor and is an improvement over when it was broken. The nested <ref> gets id 18 (just like the Cite.php handling) whereas it used to get id 1 before the breakage. * TODO: - Figure out a way to add a test for this. Change-Id: Ib82b75f66249b2133a123adbe6fd7acbfd8ec8fb
2014-08-21 23:16:32 +00:00
if (!nestedInReferences) {
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
node.parentNode.insertBefore(span, node);
Make <ref>s nested in <references> editable. * Track <ref>s birthed in <references> tags and capture their HTML and set <references>.data-mw.body.html to the concatenated HTML for all those child <ref> tags. * No change in parser-tests because our test normalization code strips data-mw before comparing results. Since this patch only changes data-mw output for "References: 5. ..." test, there is no change. In any case, this patch updates the data-mw output for the test in question so that if we do modify our normalization routines, the test will still pass. Change-Id: Iaea752dec03e5e7f0d36baf3b7a9973f1b85e8a5
2013-07-24 00:15:45 +00:00
} else {
(Bug 53109) Move data-mw away from manual json attr loading * This is both faster and consistent with how we're accessing other parsoid attributes. It's also a step towards not having this data in the html output. * Changes to parserTests and the blacklist are for attribute order. * Requires upgrading domino to 1.0.18 https://github.com/fgnass/domino/pull/48 Change-Id: I1edbc260887d480adf04763b15043c374e27cceb
2014-09-08 20:10:25 +00:00
DU.storeDataParsoid( span, dataParsoid );
DU.storeDataMw( span, dataMW );
refsInReferencesHTML.push( DU.serializeNode(span), "\n" );
Moved ref-index generation to DOM post-pass * When I last fixed Cite in commit 0164b846, I didn't take it all the way through. I was assigning ref indexes in the async pipeline which is incorrect since ref-tokens and references-tokens should be processed in the same order as they show up in the input. * I now moved ref-index assignment to the DOM post processor phase where they always belonged. * Fixed references and cite reset functions to only reset a specific group, if necessary. * Pipelines processing template transclusions should propagate the extension tag if the transclusion showed up in the context of extension source so that any extension-specific constraints are respected. (Ex: ref-tags in reference-extension context are handled differently -- and this should continue to be true even when the tags show up via transcluded content) * These fixes were tested on the following snippet from 0164b846. Where this snippet was generating buggy output earlier, it is now being handled correctly. --------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> --------------------------- X1<ref name="x" /> X2<ref name="x" /> Change-Id: I838d188c90b526878a72e4baf1e54cac644aadfc
2013-04-24 19:09:08 +00:00
}
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
T88290: Keep separate content for multiple <ref>s with same name Change-Id: Iae4dcdb405ce8d233af3105e19bab2b32938ed96
2015-02-27 19:11:37 +00:00
// Keep the first content to compare multiple <ref>s with the same name.
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
if (!ref.content) {
(Bug 69827): Supported nested <ref> tags once more * Nested <ref> tag support was broken in 69b6ec4d since that patch effectively processed <refs> only on the final DOM rather than extracted <ref> information in sub-pipelines. In doing so, it broke support for <ref> in <ref> tags which are supported by {{efn}} templates and used as follows. {{efn|A clarification.{{sfn|Smith|2009|p=2}}}} The {{sfn}} tpl generates a <ref> tag inside another <ref> tag generated by the {{efn}} tpl. * This patch fixes the breakage by processing <ref> content after it is extracted (in case it is known to have nested <ref> tags). * This fixes the rendering on enwiki/Otto_I%2C_Holy_Roman_Emperor and is an improvement over when it was broken. The nested <ref> gets id 18 (just like the Cite.php handling) whereas it used to get id 1 before the breakage. * TODO: - Figure out a way to add a test for this. Change-Id: Ib82b75f66249b2133a123adbe6fd7acbfd8ec8fb
2014-08-21 23:16:32 +00:00
ref.content = content;
Forward-port Cite extension * Adapted Cite extension to use current interfaces and token formats * Improved TokenCollector Change-Id: I20419b19edd9bbad2c2abf17a2ff1411b99c0c04
2012-05-03 11:05:28 +00:00
}
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
};
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
References.prototype.insertReferencesIntoDOM = function(refsNode, refsData, refsInReferencesHTML) {
References handling: Fix bugs introduced by 213f9682 * source and group attributes were being added as HTML attrs. on the ol-node which is invalid. Plus, after refererences generation, source attr was being deleted which causes crashers in production if references output is reused from cache. * Moved both attributes to data.parsoid * Removed group="" from parserTests that I erroneously added in that previous commit. * Minor cleanup in DOMPostProcessor. Change-Id: I85f4ad0a311d653d7ba3e010c7a5db028b643e9f
2013-06-27 00:56:44 +00:00
var about = refsNode.getAttribute('about'),
Move .data off DOM nodes * For now it stores in .dataobject, freeing up .data to handle <object> elements. * Adds a test that crashes master. Bug: 57394 Change-Id: I4207d76ad9dab660e890008b2ee5014554ce52c8
2014-01-08 21:18:44 +00:00
dp = DU.getDataParsoid( refsNode ),
group = dp.group || '',
src = dp.src || '<references/>', // fall back so we don't crash
(Bug 49182) Emit data-mw for references extension as well * data-mw wasn't being emitted for references -- there was a FIXME for it. * Tested fixes on example from 3c88b310. * Removed meta-placeholder that was being emitted for <references /> tags without any refs to emit since VE might add references and this wont be valid anymore. Serializer can also handle references output without any content. So, no need for that hack anymore. Verified by testing with "<references />" input Change-Id: I3d2852f2c6a88bf22145add9b2173fd99d152775
2013-06-05 16:06:10 +00:00
// Extract ext-source for <references>..</references> usage
Enable wrapped extensions to parse and RT correctly * Temporarily hacked sanitizer to pass through typeof attribute so that mw:DOMFragment wrapper for extension tags can get to the DOM post processor and get unwrapped. * Implemented getArgDict for the extension handler since data-mw for extensions has a different form than that for templates. * Extracted common functionality into Util.js and used it in Cite.js and ExtensionHandler.js * Tested with timeline extension (test snippet below) and verified that it parses and RTs both with editMode true and false. TODO: Long overdue. Extension testing. -------- <timeline> ImageSize = width:250 height:200 PlotArea = left:40 right:10 top:10 bottom:20 TimeAxis = orientation:horizontal AlignBars = justify Colors = id:gray1 value:gray(0.9) DateFormat = yyyy Period = from:1960 till:2010 ScaleMajor = unit:year increment:10 start:1960 PlotData = bar:3000 color:gray1 width:1 from:start till:end bar:2000 color:gray1 from:start till:end bar:1000 color:gray1 from:start till:end bar:0 color:gray1 LineData = layer:front points:(48,96)(84,111) color:blue width:2 #1962 tot 1968. Inwonertal 1962: 1348 1968: 1610 points:(84,111)(100,112) color:blue width:2 #1975: 1627 points:(100,112)(128,116) color:blue width:2 #1982: 1699 points:(128,116)(160,135) color:blue width:2 #1990: 2036 points:(160,135)(196,146) color:blue width:2 #1999: 2217 points:(196,146)(228,158) color:blue width:2 #2004/5 </timeline> -------- Change-Id: Ia8d2f82e893047e2447cf809e04cc7f508f5899b
2013-06-06 00:00:41 +00:00
body = Util.extractExtBody("references", src).trim(),
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
refGroup = refsData.getRefGroup(group);
(Bug 49182) Emit data-mw for references extension as well * data-mw wasn't being emitted for references -- there was a FIXME for it. * Tested fixes on example from 3c88b310. * Removed meta-placeholder that was being emitted for <references /> tags without any refs to emit since VE might add references and this wont be valid anymore. Serializer can also handle references output without any content. So, no need for that hack anymore. Verified by testing with "<references />" input Change-Id: I3d2852f2c6a88bf22145add9b2173fd99d152775
2013-06-05 16:06:10 +00:00
(Bug 53109) Move data-mw away from manual json attr loading * This is both faster and consistent with how we're accessing other parsoid attributes. It's also a step towards not having this data in the html output. * Changes to parserTests and the blacklist are for attribute order. * Requires upgrading domino to 1.0.18 https://github.com/fgnass/domino/pull/48 Change-Id: I1edbc260887d480adf04763b15043c374e27cceb
2014-09-08 20:10:25 +00:00
var dataMW = DU.getDataMw(refsNode);
if (!Object.keys(dataMW).length) {
Make <ref>s nested in <references> editable. * Track <ref>s birthed in <references> tags and capture their HTML and set <references>.data-mw.body.html to the concatenated HTML for all those child <ref> tags. * No change in parser-tests because our test normalization code strips data-mw before comparing results. Since this patch only changes data-mw output for "References: 5. ..." test, there is no change. In any case, this patch updates the data-mw output for the test in question so that if we do modify our normalization routines, the test will still pass. Change-Id: Iaea752dec03e5e7f0d36baf3b7a9973f1b85e8a5
2013-07-24 00:15:45 +00:00
var datamwBody;
// We'll have to output data-mw.body.extsrc in
// scenarios where original wikitext was of the form:
// "<references> lot of refs here </references>"
// Ex: See [[en:Barack Obama]]
if (body.length > 0) {
datamwBody = {
'extsrc': body,
(Bug 69827): Supported nested <ref> tags once more * Nested <ref> tag support was broken in 69b6ec4d since that patch effectively processed <refs> only on the final DOM rather than extracted <ref> information in sub-pipelines. In doing so, it broke support for <ref> in <ref> tags which are supported by {{efn}} templates and used as follows. {{efn|A clarification.{{sfn|Smith|2009|p=2}}}} The {{sfn}} tpl generates a <ref> tag inside another <ref> tag generated by the {{efn}} tpl. * This patch fixes the breakage by processing <ref> content after it is extracted (in case it is known to have nested <ref> tags). * This fixes the rendering on enwiki/Otto_I%2C_Holy_Roman_Emperor and is an improvement over when it was broken. The nested <ref> gets id 18 (just like the Cite.php handling) whereas it used to get id 1 before the breakage. * TODO: - Figure out a way to add a test for this. Change-Id: Ib82b75f66249b2133a123adbe6fd7acbfd8ec8fb
2014-08-21 23:16:32 +00:00
'html': refsInReferencesHTML.join('')
Make <ref>s nested in <references> editable. * Track <ref>s birthed in <references> tags and capture their HTML and set <references>.data-mw.body.html to the concatenated HTML for all those child <ref> tags. * No change in parser-tests because our test normalization code strips data-mw before comparing results. Since this patch only changes data-mw output for "References: 5. ..." test, there is no change. In any case, this patch updates the data-mw output for the test in question so that if we do modify our normalization routines, the test will still pass. Change-Id: Iaea752dec03e5e7f0d36baf3b7a9973f1b85e8a5
2013-07-24 00:15:45 +00:00
};
}
(Bug 53109) Move data-mw away from manual json attr loading * This is both faster and consistent with how we're accessing other parsoid attributes. It's also a step towards not having this data in the html output. * Changes to parserTests and the blacklist are for attribute order. * Requires upgrading domino to 1.0.18 https://github.com/fgnass/domino/pull/48 Change-Id: I1edbc260887d480adf04763b15043c374e27cceb
2014-09-08 20:10:25 +00:00
dataMW = {
(Bug 49182) Emit data-mw for references extension as well * data-mw wasn't being emitted for references -- there was a FIXME for it. * Tested fixes on example from 3c88b310. * Removed meta-placeholder that was being emitted for <references /> tags without any refs to emit since VE might add references and this wont be valid anymore. Serializer can also handle references output without any content. So, no need for that hack anymore. Verified by testing with "<references />" input Change-Id: I3d2852f2c6a88bf22145add9b2173fd99d152775
2013-06-05 16:06:10 +00:00
'name': 'references',
Make <ref>s nested in <references> editable. * Track <ref>s birthed in <references> tags and capture their HTML and set <references>.data-mw.body.html to the concatenated HTML for all those child <ref> tags. * No change in parser-tests because our test normalization code strips data-mw before comparing results. Since this patch only changes data-mw output for "References: 5. ..." test, there is no change. In any case, this patch updates the data-mw output for the test in question so that if we do modify our normalization routines, the test will still pass. Change-Id: Iaea752dec03e5e7f0d36baf3b7a9973f1b85e8a5
2013-07-24 00:15:45 +00:00
'body': datamwBody,
(Bug 49182) Emit data-mw for references extension as well * data-mw wasn't being emitted for references -- there was a FIXME for it. * Tested fixes on example from 3c88b310. * Removed meta-placeholder that was being emitted for <references /> tags without any refs to emit since VE might add references and this wont be valid anymore. Serializer can also handle references output without any content. So, no need for that hack anymore. Verified by testing with "<references />" input Change-Id: I3d2852f2c6a88bf22145add9b2173fd99d152775
2013-06-05 16:06:10 +00:00
'attrs': {
// Dont emit empty keys
'group': group || undefined
}
(Bug 53109) Move data-mw away from manual json attr loading * This is both faster and consistent with how we're accessing other parsoid attributes. It's also a step towards not having this data in the html output. * Changes to parserTests and the blacklist are for attribute order. * Requires upgrading domino to 1.0.18 https://github.com/fgnass/domino/pull/48 Change-Id: I1edbc260887d480adf04763b15043c374e27cceb
2014-09-08 20:10:25 +00:00
};
DU.setDataMw(refsNode, dataMW);
Roundtrip {{#tag:ref|..}} properly in editMode * See example below that clarifies the problem before this patch ------------------- $ echo "{{#tag:ref|foo}} {{echo|<references />}}" | node parse --extensions ref,references --wt2wt --editMode true <ref>foo</ref> <references /> ------------------- * The problem is that references are generated after dom fragments are unpacked in the DOM which lost the mw:Transclusion typeof and data-mw that had been set on the ref and references tags. * This is a quick fix to prevent some dirty diffs with #tag:ref being reported on en-wp. Longer term, we do have a plan to use DOMFragment encapsulation for refs and references as well and splitting up references processing differently than is being done currently. Change-Id: I4186cae93b9882d367c7d4efecc092607fe17c61
2013-06-16 22:14:25 +00:00
}
Bug 50474: Remove children of references node Make sure we don't keep old references content around on re-render. Change-Id: I79ad1e5d69ebb506078fa5903d1ff78b6ff82433
2013-07-01 21:17:07 +00:00
// Remove all children from the references node
Minor: Cleanup and comment update. Change-Id: I502f838f55a0a7c046b9d5969a8e5fb6af0dbc10
2013-10-24 01:57:17 +00:00
//
// Ex: When {{Reflist}} is reused from the cache, it comes with
// a bunch of references as well. We have to remove all those cached
// references before generating fresh references.
Bug 50474: Remove children of references node Make sure we don't keep old references content around on re-render. Change-Id: I79ad1e5d69ebb506078fa5903d1ff78b6ff82433
2013-07-01 21:17:07 +00:00
while (refsNode.firstChild) {
refsNode.removeChild(refsNode.firstChild);
}
(Bug 49182) Emit data-mw for references extension as well * data-mw wasn't being emitted for references -- there was a FIXME for it. * Tested fixes on example from 3c88b310. * Removed meta-placeholder that was being emitted for <references /> tags without any refs to emit since VE might add references and this wont be valid anymore. Serializer can also handle references output without any content. So, no need for that hack anymore. Verified by testing with "<references />" input Change-Id: I3d2852f2c6a88bf22145add9b2173fd99d152775
2013-06-05 16:06:10 +00:00
if (refGroup) {
Minor code cleanup: Use forEach instead of map where appropriate * Also replaced buf with newTokens in ext.core.TemplateHandler.js Change-Id: I81e05a13c9373aa298d710b7942f9fe3cd802e63
2014-03-26 19:08:03 +00:00
refGroup.refs.forEach(refGroup.renderLine.bind(refGroup, refsNode));
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
}
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
// Remove the group from refsData
refsData.removeRefGroup(group);
A collection of small bug fixes to the grammar, Cite, the Token format converter and the HTML DOM -> WikiDom converter. The tokenizer now digests all parserTests.
2011-12-14 23:38:46 +00:00
};
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
References.prototype.insertMissingReferencesIntoDOM = function (env, refsData, node) {
T88660: Emit reflists for <ref> with no explicit <references> This patch emits a reflist for all ref groups that still have <ref>s in them at the end of the document. Currently Cite.php only does so for the default group. See also T88290. On html2wt the missing <references> are added to the wikitext, which makes the wikitext correct. Selser catches this if not part of the edit. Change tests to include an explicit <references /> tag, and add one for explcitly testing that they do get added. This last one has to be blacklisted as the new <references /> don't appear with selser. Change-Id: I79af2c34481cadbf0d68d9571928979adf559b58
2015-02-11 15:32:57 +00:00
var doc = node.ownerDocument,
self = this;
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
refsData.refGroups.forEach(function (refsValue, refsGroup) {
T88660: Emit reflists for <ref> with no explicit <references> This patch emits a reflist for all ref groups that still have <ref>s in them at the end of the document. Currently Cite.php only does so for the default group. See also T88290. On html2wt the missing <references> are added to the wikitext, which makes the wikitext correct. Selser catches this if not part of the edit. Change tests to include an explicit <references /> tag, and add one for explcitly testing that they do get added. This last one has to be blacklisted as the new <references /> don't appear with selser. Change-Id: I79af2c34481cadbf0d68d9571928979adf559b58
2015-02-11 15:32:57 +00:00
var ol = doc.createElement('ol'),
dp = DU.getDataParsoid(ol);
DU.addAttributes(ol, {
'class': 'references',
typeof: 'mw:Extension/references',
T90309: Ensure that implicitly-added <references> output have unique ids * Fixed parser test output of one test to add unique ids. Output for other parser tests modified in f528f508 still need fixing up. Left for a separate patch. Change-Id: I04546c2a590930121d960239a1954b26771e9c80
2015-02-21 01:02:45 +00:00
about: env.newAboutId()
T88660: Emit reflists for <ref> with no explicit <references> This patch emits a reflist for all ref groups that still have <ref>s in them at the end of the document. Currently Cite.php only does so for the default group. See also T88290. On html2wt the missing <references> are added to the wikitext, which makes the wikitext correct. Selser catches this if not part of the edit. Change tests to include an explicit <references /> tag, and add one for explcitly testing that they do get added. This last one has to be blacklisted as the new <references /> don't appear with selser. Change-Id: I79af2c34481cadbf0d68d9571928979adf559b58
2015-02-11 15:32:57 +00:00
});
// The new references come out of "nowhere", so to make selser work
// propertly, add a zero-sized DSR pointing to the end of the document.
dp.dsr = [env.page.src.length, env.page.src.length, 0, 0];
if (refsGroup) {
dp.group = refsGroup;
}
DU.storeDataParsoid(ol, dp);
Fix output for implicit references for improved html2wt output * <references /> was not appearing on its own line and was instead getting tacked onto previous line of wikitext output. * Change in blacklisted wt2wt test shows that the new output is better. Change-Id: Ie82401a3bc6082b733339e2456810b6b1c87529a
2015-02-17 17:48:06 +00:00
// Add a \n before the <ol> so that when serialized to wikitext,
// each <references /> tag appears on its own line.
T88660: Emit reflists for <ref> with no explicit <references> This patch emits a reflist for all ref groups that still have <ref>s in them at the end of the document. Currently Cite.php only does so for the default group. See also T88290. On html2wt the missing <references> are added to the wikitext, which makes the wikitext correct. Selser catches this if not part of the edit. Change tests to include an explicit <references /> tag, and add one for explcitly testing that they do get added. This last one has to be blacklisted as the new <references /> don't appear with selser. Change-Id: I79af2c34481cadbf0d68d9571928979adf559b58
2015-02-11 15:32:57 +00:00
node.appendChild(doc.createTextNode("\n"));
Fix output for implicit references for improved html2wt output * <references /> was not appearing on its own line and was instead getting tacked onto previous line of wikitext output. * Change in blacklisted wt2wt test shows that the new output is better. Change-Id: Ie82401a3bc6082b733339e2456810b6b1c87529a
2015-02-17 17:48:06 +00:00
node.appendChild(ol);
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
self.insertReferencesIntoDOM(ol, refsData, [""]);
T88660: Emit reflists for <ref> with no explicit <references> This patch emits a reflist for all ref groups that still have <ref>s in them at the end of the document. Currently Cite.php only does so for the default group. See also T88290. On html2wt the missing <references> are added to the wikitext, which makes the wikitext correct. Selser catches this if not part of the edit. Change tests to include an explicit <references /> tag, and add one for explcitly testing that they do get added. This last one has to be blacklisted as the new <references /> don't appear with selser. Change-Id: I79af2c34481cadbf0d68d9571928979adf559b58
2015-02-11 15:32:57 +00:00
});
};
Fix a bunch of jshint warnings. Most of these warnings are globals defined in mediawiki.parser.defines.js, which we turn into proper local variables. But there are a grab page of other use-before-def, missing semicolon, bad line break, and other minor warnings. No tricky stuff attempted in this patch, this should be safe obvious fixes only... although there is a nasty circular dependency between Util and parser.defines which had to be broken. Change-Id: I8810da3a5ec287d7ae9078463ba01472a87a492e
2013-04-22 19:51:09 +00:00
/**
* Native Parsoid implementation of the Cite extension
* that ties together <ref> and <references>
*/
var Cite = function() {
this.ref = new Ref(this);
this.references = new References(this);
};
Use TokenStreamPatcher to convert table tags outside wikitables to text * Keep track of table nesting in token stream patcher and use it to convert <td>, <tr>, and <th> tags to plain strings. * This fix is only enabled on the top-level token stream. To support this, fixed the resetState function in the parser construction code to pass in a toplevel flag which lets the token stream patcher know the context it is in. * Fixes 29 (wt2html,wt2wt,html2html,selser) tests and improves results of 1 previously blacklisted tests. The failing selser test is actually a false failure because selser is more accurate than non-selser wts. * Consolidated a few separate tests into a single test that covers all this functionality. - This new test fails wt2wt and html2wt modes because serializer uses tokenizer information which continues to return table tokens and results in <nowiki> wrappers. Bug: 66489 Bug: 66498 Change-Id: I9f42354ea9efb0f8adfc96c23760012220d00dd4
2014-06-16 17:30:25 +00:00
Cite.prototype.resetState = function(opts) {
if (opts && opts.toplevel) {
Work around Cite reset issue The Cite extension does not currently handle resetState calls in sub-pipelines, and relies on sharing a single Cite instance between all pipelines. Fixing this is a longer project, so this patch works around the issue for now by passing a flag indicating resetState calls in sub-pipelines and ignoring the call in Cite in that case. Change-Id: If3d426a5311a55d1c1530860d2b665d3681f1aa9
2014-04-30 17:02:39 +00:00
this.ref.reset();
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
this.references.reset();
Work around Cite reset issue The Cite extension does not currently handle resetState calls in sub-pipelines, and relies on sharing a single Cite instance between all pipelines. Fixing this is a longer project, so this patch works around the issue for now by passing a flag indicating resetState calls in sub-pipelines and ignoring the call in Cite in that case. Change-Id: If3d426a5311a55d1c1530860d2b665d3681f1aa9
2014-04-30 17:02:39 +00:00
}
Fix a bunch of jshint warnings. Most of these warnings are globals defined in mediawiki.parser.defines.js, which we turn into proper local variables. But there are a grab page of other use-before-def, missing semicolon, bad line break, and other minor warnings. No tricky stuff attempted in this patch, this should be safe obvious fixes only... although there is a nasty circular dependency between Util and parser.defines which had to be broken. Change-Id: I8810da3a5ec287d7ae9078463ba01472a87a492e
2013-04-22 19:51:09 +00:00
};
Improved template roundtripping based on WT ranges on DOM nodes. - Fixed several bugs in the DSR computation. - Deleted comment tokens from template output. - Eliminated TSR info from template output. - Rearranged order of DOM post passes. - Updated template encapsulation to use DSR values. - There are still a few missing pieces and scenarios that are not handled properly. Change-Id: I644914902ec1d27418138b107abe765f81178398
2012-09-17 19:46:44 +00:00
if (typeof module === "object") {
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
module.exports.Cite = Cite;
T93973: Remove state from Cite extension Keep all state in the DOM post-processor in dom.processRefs.js, so that there's no need to reset the extension and it can be safely used for multiple documents Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-03-31 16:52:01 +00:00
module.exports.ReferencesData = ReferencesData;
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
}
Reference in a new issue Copy permalink