wikimedia/mediawiki-extensions-Cite
1
0
Fork 0
mirror of https://gerrit.wikimedia.org/r/mediawiki/extensions/Cite synced 2024-11-24 23:05:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
mediawiki-extensions-Cite/js/lib/ext.Cite.js

416 lines
11 KiB
JavaScript
Raw Normal View History

Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
/* ----------------------------------------------------------------------
* This file implements <ref> and <references> extension tag handling
* natively in Parsoid.
* ---------------------------------------------------------------------- */
Deep-freeze cache returns and enable strict mode This exposed a few bugs (primarily missing var declarations), which are fixed. It would for example have caught the shallow clone bug we just fixed, and should catch similar bugs early in the future. TODO / issues this exposed: Avoid attempted tsr modification in ext.core.TemplateHandler.js:306:29. This is fixed for now by cloning the tokens, but should really be avoided by stripping the tsr before caching the tokens (anything in phase 1 will work for example). Change-Id: I6a1a07de7ac333f31da9cf0ae9ed20e5507bacba
2012-10-30 00:51:07 +00:00
"use strict";
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
var Util = require( './mediawiki.Util.js' ).Util,
Fix a bunch of jshint warnings. Most of these warnings are globals defined in mediawiki.parser.defines.js, which we turn into proper local variables. But there are a grab page of other use-before-def, missing semicolon, bad line break, and other minor warnings. No tricky stuff attempted in this patch, this should be safe obvious fixes only... although there is a nasty circular dependency between Util and parser.defines which had to be broken. Change-Id: I8810da3a5ec287d7ae9078463ba01472a87a492e
2013-04-22 19:51:09 +00:00
defines = require('./mediawiki.parser.defines.js'),
Replace jquery dependency with a minimal 'fakejquery'. The 'fakejquery' module just defines the bits we use: $.each, $.map, $.extend, $.isArray, $.isEmptyObject (and the internal functions required to implement those). Change-Id: I626b2f3a3d3a8d395d0c9679e01d3399403636dd
2013-02-15 21:25:14 +00:00
$ = require( './fakejquery' );
Fix a bunch of jshint warnings. Most of these warnings are globals defined in mediawiki.parser.defines.js, which we turn into proper local variables. But there are a grab page of other use-before-def, missing semicolon, bad line break, and other minor warnings. No tricky stuff attempted in this patch, this should be safe obvious fixes only... although there is a nasty circular dependency between Util and parser.defines which had to be broken. Change-Id: I8810da3a5ec287d7ae9078463ba01472a87a492e
2013-04-22 19:51:09 +00:00
// define some constructor shortcuts
var KV = defines.KV,
TagTk = defines.TagTk,
SelfclosingTagTk = defines.SelfclosingTagTk,
EndTagTk = defines.EndTagTk;
Forward-port Cite extension * Adapted Cite extension to use current interfaces and token formats * Improved TokenCollector Change-Id: I20419b19edd9bbad2c2abf17a2ff1411b99c0c04
2012-05-03 11:05:28 +00:00
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
/**
* Helper class used both by <ref> and <references> implementations
*/
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
function RefGroup(group) {
this.name = group || '';
this.refs = [];
this.indexByName = {};
}
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
RefGroup.prototype.add = function(refName, skipLinkback) {
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
var ref;
if (refName && this.indexByName[refName]) {
ref = this.indexByName[refName];
} else {
var n = this.refs.length,
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
refKey = (1+n) + '';
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
if (refName) {
refKey = refName + '-' + refKey;
}
ref = {
content: null,
index: n,
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
groupIndex: (1+n), // FIXME -- this seems to be wiki-specific
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
name: refName,
group: this.name,
key: refKey,
target: 'cite_note-' + refKey,
linkbacks: []
};
this.refs[n] = ref;
if (refName) {
this.indexByName[refName] = ref;
}
}
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
if (!skipLinkback) {
ref.linkbacks.push('cite_ref-' + ref.key + '-' + ref.linkbacks.length);
}
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
return ref;
};
RefGroup.prototype.renderLine = function(refsList, ref) {
var ownerDoc = refsList.ownerDocument,
arrow = ownerDoc.createTextNode('↑'),
li, a;
// Generate the li
Fix a bunch of jshint warnings. Most of these warnings are globals defined in mediawiki.parser.defines.js, which we turn into proper local variables. But there are a grab page of other use-before-def, missing semicolon, bad line break, and other minor warnings. No tricky stuff attempted in this patch, this should be safe obvious fixes only... although there is a nasty circular dependency between Util and parser.defines which had to be broken. Change-Id: I8810da3a5ec287d7ae9078463ba01472a87a492e
2013-04-22 19:51:09 +00:00
li = ownerDoc.createElement('li');
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
li.setAttribute('li', ref.target);
// Set ref content first, so the HTML gets parsed
// We then append the rest of the ref nodes before the first node
li.innerHTML = ref.content;
var contentNode = li.firstChild;
// if (!contentNode) console.warn("--empty content for: " + ref.linkbacks[0]);
// Generate leading linkbacks
if (ref.linkbacks.length === 1) {
a = ownerDoc.createElement('a');
a.setAttribute('href', '#' + ref.linkbacks[0]);
a.appendChild(arrow);
li.insertBefore(a, contentNode);
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
li.insertBefore(ownerDoc.createTextNode(' '), contentNode);
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
} else {
li.insertBefore(arrow, contentNode);
$.each(ref.linkbacks, function(i, linkback) {
a = ownerDoc.createElement('a');
a.setAttribute('data-type', 'hashlink');
a.setAttribute('href', '#' + ref.linkbacks[i]);
a.appendChild(ownerDoc.createTextNode(ref.groupIndex + '.' + i));
li.insertBefore(a, contentNode);
// Separate linkbacks with a space
li.insertBefore(ownerDoc.createTextNode(' '), contentNode);
});
}
// Add it to the ref list
refsList.appendChild(li);
};
function newRefGroup(refGroups, group) {
group = group || '';
if (!refGroups[group]) {
refGroups[group] = new RefGroup(group);
}
return refGroups[group];
}
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
// FIXME: Move out to some common helper file?
// Helper function to process extension source
function processExtSource(manager, extToken, opts) {
var extSrc = extToken.getAttribute('source'),
tagWidths = extToken.dataAttribs.tagWidths,
content = extSrc.substring(tagWidths[0], extSrc.length - tagWidths[1]);
// FIXME: Should this be specific to the extension
// or is it okay to do this unconditionally for all?
// Right now, this code is run only for ref and references,
// so not a real problem, but if this is used on other extensions,
// requires addressing.
//
// Strip white-space only lines
var wsMatch = content.match(/^((?:\s*\n)?)((?:.|\n)*)$/),
leadingWS = wsMatch[1];
// Update content to normalized form
content = wsMatch[2];
if (!content || content.length === 0) {
opts.emptyContentCB(opts.res);
} else {
// Pass an async signal since the ext-content is not processed completely.
opts.parentCB({tokens: opts.res, async: true});
// Pipeline for processing ext-content
var pipeline = manager.pipeFactory.getPipeline(
opts.pipelineType,
Util.extendProps({}, opts.pipelineOpts, {
wrapTemplates: true,
// SSS FIXME: Doesn't seem right.
// Should this be the default in all cases?
inBlockToken: true
})
);
// Set source offsets for this pipeline's content
var tsr = extToken.dataAttribs.tsr;
pipeline.setSourceOffsets(tsr[0]+tagWidths[0]+leadingWS.length, tsr[1]-tagWidths[1]);
// Set up provided callbacks
if (opts.chunkCB) {
pipeline.addListener('chunk', opts.chunkCB);
}
if (opts.endCB) {
pipeline.addListener('end', opts.endCB);
}
if (opts.documentCB) {
pipeline.addListener('document', opts.documentCB);
}
// Off the starting block ... ready, set, go!
pipeline.process(content);
}
}
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
/**
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
* Simple token transform version of the Ref extension tag
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
*
Clean up and comment the Cite extension a bit.
2011-12-13 18:45:09 +00:00
* @class
* @constructor
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
*/
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
function Ref(cite) {
this.cite = cite;
this.reset();
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
}
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
/**
* Reset state before each top-level parse -- this lets us share a pipeline
* to parse unrelated pages.
*/
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
Ref.prototype.reset = function() {
Reset the Cite extension state with the end token This should hopefully fix some spurious semantic diffs that are purely based on ref numbering, which changes if the references are not reset. Change-Id: I37b1f4cfe69538e1c5ccc81fdad3084c01151700
2012-11-16 18:03:01 +00:00
this.refGroups = {};
};
Clean up and comment the Cite extension a bit.
2011-12-13 18:45:09 +00:00
/**
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
* Handle ref tokens
Clean up and comment the Cite extension a bit.
2011-12-13 18:45:09 +00:00
*/
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
Ref.prototype.handleRef = function ( manager, pipelineOpts, refTok, cb ) {
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
var tsr = refTok.dataAttribs.tsr,
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
refOpts = $.extend({ name: null, group: null }, Util.KVtoHash(refTok.getAttribute("options"))),
group = this.refGroups[refOpts.group] || newRefGroup(this.refGroups, refOpts.group),
ref = group.add(refOpts.name, pipelineOpts.extTag === "references"),
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
linkback = ref.linkbacks[ref.linkbacks.length - 1],
bits = [];
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
if (refOpts.group) {
bits.push(refOpts.group);
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
}
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
//bits.push(Util.formatNum( ref.groupIndex ));
bits.push(ref.groupIndex);
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
var about, res;
if (pipelineOpts.extTag === "references") {
about = '';
res = [];
} else {
about = "#" + manager.env.newObjectId();
var span = new TagTk('span', [
Forward-port Cite extension * Adapted Cite extension to use current interfaces and token formats * Improved TokenCollector Change-Id: I20419b19edd9bbad2c2abf17a2ff1411b99c0c04
2012-05-03 11:05:28 +00:00
new KV('id', linkback),
new KV('class', 'reference'),
Unrelated fixes/cleanup addressing recent git-review comments. * Replaced empty (mw:Placeholder) span tags with meta-tag. * Minor code refactoring in DSR computation. * Removed hardcoding of "mwt" prefix for object ids. * Added rt-info for html tags that are not lower-case in the source wikitext. - Additional 2 wt2wt tests green. Change-Id: I9bb6a075bc60b6d78186d23000128eef6a04c787
2012-11-21 01:42:30 +00:00
new KV('about', about),
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
new KV('typeof', 'mw:Object/Ext/Ref')
], {
src: refTok.dataAttribs.src
}),
endMeta = new SelfclosingTagTk( 'meta', [
new KV( 'typeof', 'mw:Object/Ext/Ref/End' ),
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
new KV( 'about', about)
Improved template roundtripping based on WT ranges on DOM nodes. - Fixed several bugs in the DSR computation. - Deleted comment tokens from template output. - Eliminated TSR info from template output. - Rearranged order of DOM post passes. - Updated template encapsulation to use DSR values. - There are still a few missing pieces and scenarios that are not handled properly. Change-Id: I644914902ec1d27418138b107abe765f81178398
2012-09-17 19:46:44 +00:00
]);
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
if (tsr) {
span.dataAttribs.tsr = tsr;
endMeta.dataAttribs.tsr = [null, tsr[1]];
}
res = [
span,
new TagTk( 'a', [ new KV('href', '#' + ref.target) ]),
'[' + bits.join(' ') + ']',
new EndTagTk( 'a' ),
new EndTagTk( 'span' ),
endMeta
];
Improved template roundtripping based on WT ranges on DOM nodes. - Fixed several bugs in the DSR computation. - Deleted comment tokens from template output. - Eliminated TSR info from template output. - Rearranged order of DOM post passes. - Updated template encapsulation to use DSR values. - There are still a few missing pieces and scenarios that are not handled properly. Change-Id: I644914902ec1d27418138b107abe765f81178398
2012-09-17 19:46:44 +00:00
}
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
var finalCB = function(toks, content) {
toks.push(new SelfclosingTagTk( 'meta', [
new KV('typeof', 'mw:Ext/Ref/Content'),
new KV('about', about),
new KV('group', refOpts.group || ''),
new KV('name', refOpts.name || ''),
new KV('content', content || ''),
new KV('skiplinkback', pipelineOpts.extTag === "references" ? 1 : 0)
]));
RT <references> as well as </references> tag. * Fixes RT diff in en:Standing Sushi Bar. Change-Id: Ia05f9d01503e3003260a10663d1dd845a261beab
2012-11-16 23:36:07 +00:00
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
// All done!
cb({tokens: toks, async: false});
};
RT <references> as well as </references> tag. * Fixes RT diff in en:Standing Sushi Bar. Change-Id: Ia05f9d01503e3003260a10663d1dd845a261beab
2012-11-16 23:36:07 +00:00
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
processExtSource(manager, refTok, {
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
// Full pipeline for processing ref-content
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
pipelineType: 'text/x-mediawiki/full',
pipelineOpts: {
extTag: "ref",
// Always wrap templates for ref-tags
// SSS FIXME: Document why this is so
// I wasted an hour because I failed to set this flag
wrapTemplates: true
},
res: res,
parentCB: cb,
emptyContentCB: finalCB,
documentCB: function(refContentDoc) {
finalCB([], refContentDoc.body.innerHTML);
}
});
};
Added a new parsing pipeline for processing ref tokens. * Added a tokens/x-mediawiki/post-expansion sub-pipeline that is used to process ref-content. This same pipeline type could also be used to post-process tokenized content from other extensions. * Cleaned up transform handler ranks a bit. * Fixed paragraph-handler to accept initial state and also fixed it to do a complete reset on encountering EOFTk. Cite ref-content handling uses the sub-pipeline while passing the inBlockToken flag to prevent top-level p-wrapping of ref-content. * Fixes references output on en:Barack Obama and introduces lists in individual ref items. * No change in parser tests -- how do we add tests for testing extensions like cite/ref? Change-Id: I928a3a0f5f989ec28bbe73383794fe6f32599fd1
2012-11-29 20:09:06 +00:00
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
function References(cite) {
this.cite = cite;
this.reset();
}
References.prototype.reset = function() {
this.refGroups = { };
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
};
Improved template roundtripping based on WT ranges on DOM nodes. - Fixed several bugs in the DSR computation. - Deleted comment tokens from template output. - Eliminated TSR info from template output. - Rearranged order of DOM post passes. - Updated template encapsulation to use DSR values. - There are still a few missing pieces and scenarios that are not handled properly. Change-Id: I644914902ec1d27418138b107abe765f81178398
2012-09-17 19:46:44 +00:00
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
/**
* Sanitize the references tag and convert it into a meta-token
*/
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
References.prototype.handleReferences = function ( manager, pipelineOpts, refsTok, cb ) {
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
refsTok = refsTok.clone();
Handle multiply-referenced ref-tags without duplicate ref-content. * When named-ref-tags are used multiple times in a document, the Cite.onReferences function was outputting the ref-content once for each use in the references section. While this is "harmless" for regular ref-content, when the ref-content has templates (ex: {[cite .. }}), that template content is duplicated several times -- this also means that template expansions seen in the output no longer has unique ids. Occasionally, this trips up DOM passes (template encapsulation) which make the reasonable assumption that template ids are unique. It introduced an infinite loop in the code that detects template range nestings and overlaps. This is not triggered always, but only in certain scenarios. en:Euphoria (Enrique Iglesias album) was a page that sent the parser into an infinite loop. * Now fixed by outputting ref-content once. * In addition, added an error-check in the DOM post-pass that might get stuck in an infinite loop to output an error message. * No change in parser test results. * en:Euphoria (Enrique Iglesias album) now parses and rts without getting stuck in an infinite loop. Possibly other pages as well. Change-Id: I81a055999eefe3ca36dfb8a9cda3fee538ae6c65
2013-02-20 23:50:52 +00:00
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
var cite = this.cite;
Improved template roundtripping based on WT ranges on DOM nodes. - Fixed several bugs in the DSR computation. - Deleted comment tokens from template output. - Eliminated TSR info from template output. - Rearranged order of DOM post passes. - Updated template encapsulation to use DSR values. - There are still a few missing pieces and scenarios that are not handled properly. Change-Id: I644914902ec1d27418138b107abe765f81178398
2012-09-17 19:46:44 +00:00
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
// group is the only recognized option?
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
var refsOpts = Util.KVtoHash(refsTok.getAttribute("options")),
group = refsOpts.group;
Make Cite <references> group attribute handling more robust The PHP preprocessor returns a references tag with an empty group attribute, which we did not map to the null reference group. This caused the references listing to vanish when using the PHP preprocessor. This patch makes sure the attribute is converted to a string, stripped of leading/trailing whitespace and finally set to null if no string remains. No changes in parser tests, since Cite is in a separate extension. Change-Id: Ib3de375225947a78c362370be8d78504fa24def2
2012-11-27 23:13:32 +00:00
if ( group && group.constructor === Array ) {
// Array of tokens, convert to string.
group = Util.tokensToString(group);
}
if ( group ) {
// have a String, strip whitespace
group = group.replace(/^\s*(.*)\s$/, '$1');
}
// Point invalid / empty groups to null
Fixed #tag parser function. * Updated #tag parser function code to set tag attributes. This lets {{Reflist|group=n}} to pass the group=n attribute to the <references> tag. In turn, this fixes incorrect reference listing in the above template call. This then fixes crashers in en:Prophet Muhammed, en:The PS3 and such pages. - TODO: parser output for Prophet Muhammed is still broken where content from body ends up in a table that has display:none css set. * Minor code cleanup in AttributeTransformManager.process and added comment about lazy processing. Change-Id: If528a3a5c6e22d3ff691dfc6b5b2f00201a26384
2012-11-29 16:15:51 +00:00
if ( ! group ) {
Make Cite <references> group attribute handling more robust The PHP preprocessor returns a references tag with an empty group attribute, which we did not map to the null reference group. This caused the references listing to vanish when using the PHP preprocessor. This patch makes sure the attribute is converted to a string, stripped of leading/trailing whitespace and finally set to null if no string remains. No changes in parser tests, since Cite is in a separate extension. Change-Id: Ib3de375225947a78c362370be8d78504fa24def2
2012-11-27 23:13:32 +00:00
group = null;
}
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
// Emit a placeholder meta for the references token
// so that the dom post processor can generate and
// emit references at this point in the DOM.
var emitPlaceholderMeta = function() {
var placeHolder = new SelfclosingTagTk('meta',
refsTok.attribs,
refsTok.dataAttribs);
// Update properties
if (group) {
placeHolder.setAttribute('group', group);
}
placeHolder.setAttribute('typeof', 'mw:Ext/References');
placeHolder.dataAttribs.stx = undefined;
// All done!
cb({ tokens: [placeHolder], async: false });
// FIXME: This is somehow buggy -- needs investigation
// Reset refs after references token is processed
// cite.ref.resetState();
};
processExtSource(manager, refsTok, {
// Partial pipeline for processing ref-content
// Expand till stage 2 so that all embedded
// ref tags get processed
pipelineType: 'text/x-mediawiki',
pipelineOpts: {
extTag: "references",
wrapTemplates: pipelineOpts.wrapTemplates
},
res: [],
parentCB: cb,
emptyContentCB: emitPlaceholderMeta,
chunkCB: function(chunk) {
// Extract ref-content tokens and discard the rest
var res = [];
for (var i = 0, n = chunk.length; i < n; i++) {
var t = chunk[i];
if (t.constructor === SelfclosingTagTk &&
t.name === 'meta' &&
Fix crasher on :fr:126e_méridien_ouest This code is being run on: <meta property="mw:PageProp/categorydefaultsort" content="0 126"> which doesn't have a 'typeof' attribute, and so the match is crashing with a null pointer dereference. Turn the expression around. RegExp.test is more efficient, and it handles a null argument with no problems. Add start/end anchors to the regexp while we're at it. Change-Id: I11bc9fdc902a21ce0bc1c05f08d3b615d5483e2e
2013-04-24 14:50:11 +00:00
/^mw:Ext\/Ref\/Content$/.test(t.getAttribute('typeof')))
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
{
res.push(t);
}
}
// Pass along the ref toks
cb({ tokens: res, async: true });
},
endCB: emitPlaceholderMeta
});
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
};
References.prototype.extractRefFromNode = function(node) {
var group = node.getAttribute("group"),
refName = node.getAttribute("name"),
refGroup = this.refGroups[group] || newRefGroup(this.refGroups, group),
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
ref = refGroup.add(refName, node.getAttribute("skiplinkback") === "1");
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
// This effectively ignores content from later references with the same name.
// The implicit assumption is that that all those identically named refs. are
// of the form <ref name='foo' />
if (!ref.content) {
ref.content = node.getAttribute("content");
Forward-port Cite extension * Adapted Cite extension to use current interfaces and token formats * Improved TokenCollector Change-Id: I20419b19edd9bbad2c2abf17a2ff1411b99c0c04
2012-05-03 11:05:28 +00:00
}
Extension handling rewrite + cite extension refactoring. Tokenization ------------ * Fixed tokenizer to correctly parse extension tags in different contexts: with start and end tags, in self-closing tag mode, and to correctly handle scenarios when the exension end-tag is followed by a '#' (the special char used to skip over extension content). * Removed the distinction in the tokenizer between installed extensions and natively supported extension tags (<ref> and <references> for ex.). They all tokenize and get processed identically and get handled by different paths in the extension handler. * Template and TemplateArg tokens now carry tpl. transclusion source alongwith them since tsr information will not be accurate when they show up in extension contexts that in turn showed up in template context that were expanded by the php preprocessor. Ex: {{echo|<ref>{{echo|foo}}</ref>}} The tsr for the ref-tag will correspond to the template-source of the echo-template, NOT the original top-level page. So, env.page.src.substring(..) will return incorrect source for the innermost {{echo|foo}}. This fix of carrying along tpl transclusion source in the token itself eliminates this problem. Knowledge of native extensions ------------------------------ * Natively implemented extension tags (<ref> and <references>) are hardcoded in env.conf.parsoid. At some point, it would be good to have a registration mechanism for parsoid-native extensions. Extension handling ------------------ * Extracted extension handling out of the template handler into its own handler class. Right now, this class inherits from the template handler in order to be able to reuse a lot of the expansion and encapsulation functionality currently in the Template Handler. * This handler now handles extensions that are: (a) natively implemented and registered with Parsoid. (b) implemented as a PHP extension and expanded by relying on the PHP preprocessor. For (a), it uses information from env.conf.parsoid to find ext-handlers for natively implemented ext-tags. However, this can be cleaned up at some point by making available a registration mechanism. Cite/Ref/References ------------------- * Reworked the cite handler to split up ref-token processing and references token processing. * The handler now processes ref-tokens, parses content all the way to html output and encapsulates the html in an attribute of a meta-token that serves as a placeholder for where the ref-token occured. * References are handled as a DOM post-pass where these meta placeholder tokens are collected, content extracted from the attribute and spit out at the site of a references tag. The DOM walking is in DOMPostProcessor.js, but the actual processing is part of the Cite.js to keep all cite extension handling code in one place. Parser pipeline --------------- * Restructured parser pipeline recipes based on changes to Cite, TemplateHandler, and ExtensionHandler. * Added a couple functions to the parser pipeline: 1. resetState to reset state before starting a new top-level parse when pipelines are reused across top-level parses (ex: parser tests) 2. setSourceOffsets to set start/end offsets of the source being handled by the pipeline. This is required to correctly set tsr values when extension content (which is a substring of original top-level text) is parsed in its own pipeline. Other fixes ----------- * Removed env parameter from the Params object since it was not being used and seemed like unnecessary state propagation. * Removed a FIXME in DOMUtils.buildTokensFromDOM by reusing code in the tokenizer that converts "\n" in text to NlTks. * Cleanup of Util.shiftTokenTSR. * ext.util.TokenCollection is now no longer used by anything. Added a FIXME and left around in case we are able to improve tokenizing and handling of *include* tags that can eliminate the need for the messy TokenAndAttrCollector. Test results ------------ * No change in parser tests results. * Tested with a few different files. - en:BO page seems to be parsed about 10% faster than before (needs verification). - Referencs on the en:BO page seem to be more accurate than before. Change-Id: I8a095fa9fa976c7b3a2a4bd968dc9db4270b105f
2013-03-09 00:07:59 +00:00
};
References.prototype.insertReferencesIntoDOM = function(refsNode) {
var group = refsNode.getAttribute("group") || '',
refGroup = this.refGroups[group];
if (refGroup && refGroup.refs.length > 0) {
var ol = refsNode.ownerDocument.createElement('ol');
ol.setAttribute('class', 'references');
ol.setAttribute('typeof', 'mw:Object/References');
ol.setAttribute('data-parsoid', refsNode.getAttribute('data-parsoid'));
refGroup.refs.map(refGroup.renderLine.bind(refGroup, ol));
refsNode.parentNode.replaceChild(ol, refsNode);
} else {
// Not a valid references tag -- convert it to a placeholder tag that will rt as is.
refsNode.setAttribute('typeof', 'mw:Placeholder');
}
// clear refs group
Updated native <ref> and <references> tag implementations. * Updated native implementations of the <ref> and <references> tag implementations of the cite extension. * <references> tag was not being processed properly by Parsoid. This led to lost references on the BO page. This patch fixes it which fills out references and more closely matches output en:WP. * Extracted extension content processing code into a helper and reused it for both <ref> and <references> handler. - Leading ws-only lines are unconditionally stripped. Is this accurate or is this extension-specific? Given that this code is right now tied to <ref> and <references> tag, this is not yet a problem, but if made more generic, this issue has to be addressed. * PreHandler should not run when processing the refs-tag. Right now, this is a hard check in the pre-handler. Probably worth making this more generic by letting every stage in the pipeline get a chance at turning themselves on/off based on the extension being processed by the pipeline (can use the _applyToStage fn. on ParserPipeline). TO BE DONE. * <ref> extension needs to be reset after each <references> tag is processed to duplicate behavior of existing cite extension. TO BE DONE. * Updated refs group index to start at 1. * No change in parser tests. References section output on the en:Barack Obama page now more closely matches the refs output on enwp. * In addition to the en:BO page, the following wikitext was used to fix bugs and test the implementation. CMD: "node parse --extensions ref,references < /tmp/test" ---------------------------------------------- X1<ref name="x" /> X2<ref name="x" /> <references> <ref name="x">x</ref> </references> Y<ref name="y">{{echo|y}}</ref> Z<ref name="z" /> <references> {{echo|<ref name="z">z</ref>}} </references> A<ref name="a">a</ref> B<ref name="b" /> <references> {{echo|<ref name="b">b</ref>}} </references> C<ref name="c">c</ref> D<ref name="d" /> <references> <ref name="d">{{echo|d}}</ref> </references> ---------------------------------------------- Change-Id: I2d243656e9e903d8dadb55ee7c0630824c65cc01
2013-03-26 00:40:20 +00:00
if (group) {
this.refGroups[group] = undefined;
} else {
this.refGroups = {};
}
A collection of small bug fixes to the grammar, Cite, the Token format converter and the HTML DOM -> WikiDom converter. The tokenizer now digests all parserTests.
2011-12-14 23:38:46 +00:00
};
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
Fix a bunch of jshint warnings. Most of these warnings are globals defined in mediawiki.parser.defines.js, which we turn into proper local variables. But there are a grab page of other use-before-def, missing semicolon, bad line break, and other minor warnings. No tricky stuff attempted in this patch, this should be safe obvious fixes only... although there is a nasty circular dependency between Util and parser.defines which had to be broken. Change-Id: I8810da3a5ec287d7ae9078463ba01472a87a492e
2013-04-22 19:51:09 +00:00
/**
* Native Parsoid implementation of the Cite extension
* that ties together <ref> and <references>
*/
var Cite = function() {
this.ref = new Ref(this);
this.references = new References(this);
};
Cite.prototype.resetState = function() {
this.ref.reset();
this.references.reset();
};
Improved template roundtripping based on WT ranges on DOM nodes. - Fixed several bugs in the DSR computation. - Deleted comment tokens from template output. - Eliminated TSR info from template output. - Rearranged order of DOM post passes. - Updated template encapsulation to use DSR values. - There are still a few missing pieces and scenarios that are not handled properly. Change-Id: I644914902ec1d27418138b107abe765f81178398
2012-09-17 19:46:44 +00:00
if (typeof module === "object") {
Convert the Cite extension to a token stream transformer. This required a few further additions to the TokenTransformDispatcher. In particular, there is now an 'any' token match whose callbacks are executed before more specific callbacks. This is used by the Cite extension to eat all tokens between ref and /ref tags. This need is very common, so should be broken out to an intermediate layer in the future. In general, the requirements for the TokenTransformDispatcher API are now clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00
module.exports.Cite = Cite;
}
Reference in a new issue Copy permalink