* Added a tokens/x-mediawiki/post-expansion sub-pipeline that
is used to process ref-content. This same pipeline type
could also be used to post-process tokenized content from
other extensions.
* Cleaned up transform handler ranks a bit.
* Fixed paragraph-handler to accept initial state and also
fixed it to do a complete reset on encountering EOFTk.
Cite ref-content handling uses the sub-pipeline while
passing the inBlockToken flag to prevent top-level p-wrapping
of ref-content.
* Fixes references output on en:Barack Obama and introduces
lists in individual ref items.
* No change in parser tests -- how do we add tests for testing
extensions like cite/ref?
Change-Id: I928a3a0f5f989ec28bbe73383794fe6f32599fd1
* Updated #tag parser function code to set tag attributes.
This lets {{Reflist|group=n}} to pass the group=n attribute
to the <references> tag. In turn, this fixes incorrect
reference listing in the above template call. This then
fixes crashers in en:Prophet Muhammed, en:The PS3 and such
pages.
- TODO: parser output for Prophet Muhammed is still broken
where content from body ends up in a table that has
display:none css set.
* Minor code cleanup in AttributeTransformManager.process
and added comment about lazy processing.
Change-Id: If528a3a5c6e22d3ff691dfc6b5b2f00201a26384
The PHP preprocessor returns a references tag with an empty group attribute,
which we did not map to the null reference group. This caused the references
listing to vanish when using the PHP preprocessor.
This patch makes sure the attribute is converted to a string, stripped of
leading/trailing whitespace and finally set to null if no string remains.
No changes in parser tests, since Cite is in a separate extension.
Change-Id: Ib3de375225947a78c362370be8d78504fa24def2
* Replaced empty (mw:Placeholder) span tags with meta-tag.
* Minor code refactoring in DSR computation.
* Removed hardcoding of "mwt" prefix for object ids.
* Added rt-info for html tags that are not lower-case in the
source wikitext.
- Additional 2 wt2wt tests green.
Change-Id: I9bb6a075bc60b6d78186d23000128eef6a04c787
This should hopefully fix some spurious semantic diffs that are purely based
on ref numbering, which changes if the references are not reset.
Change-Id: I37b1f4cfe69538e1c5ccc81fdad3084c01151700
* Run cite handler first thing in phase 3 so that ref-content
is pulled out of the main token stream. This effectively
prevents unclosed/unbalanced tags within ref-tags from
affecting the main stream.
* No change in parser tests.
* Fixes rt diffs on the following pages:
1. en:Bandaid surgeries (zero rt diffs)
2. en:Gondi bank (zero rt diffs)
3. en:Ashland, oh (1 syntactic diff)
The reverted commit (78989e8) had fixed rt-diffs in 1. above,
hadn't changed diffs in 2. and introduced regressions in 3.
Change-Id: I6ad498d9d3711b71e7f427eea214acd10da8647b
* Setting tsr to (endTsr || [null, text.length]) was causing
serious duplication of rt-src on the following page:
en:Almirante Latorre-class battleship
This is the case for ref-tokens that come from templates
which have their TSR stripped. So, we shouldn't attempt
to set it to [null, text.length] again.
* No change in parserTests.
* No change in errors for en:L'Opéra national de Paris.
Change-Id: If6a79638c349c9a3fab42094aad51e2e476f9632
Also remove the src, as we re-set it based on DSR anyway.
No change in parserTests, but three less diffs in L'Opéra national de Paris.
Change-Id: Ia6784efe7d76c41f24a183465393bda78641a8eb
This exposed a few bugs (primarily missing var declarations), which are fixed.
It would for example have caught the shallow clone bug we just fixed, and
should catch similar bugs early in the future.
TODO / issues this exposed:
Avoid attempted tsr modification in ext.core.TemplateHandler.js:306:29. This
is fixed for now by cloning the tokens, but should really be avoided by
stripping the tsr before caching the tokens (anything in phase 1 will work for
example).
Change-Id: I6a1a07de7ac333f31da9cf0ae9ed20e5507bacba
* There was a fixme for this in ext.Cite.js which was triggering
an error in the serializer and introducing rt-diffs.
* No change in parser tests.
* Eliminates semantic error in en:Birkenhauer Island and other
pages.
Change-Id: I9fe6ea26fdf1859d751b02a288393af610be5474
- Fixed several bugs in the DSR computation.
- Deleted comment tokens from template output.
- Eliminated TSR info from template output.
- Rearranged order of DOM post passes.
- Updated template encapsulation to use DSR values.
- There are still a few missing pieces and scenarios that are
not handled properly.
Change-Id: I644914902ec1d27418138b107abe765f81178398
* Only one instance of Cite handler is created and it deals with the
final expanded token stream. So, effectively there is no information
about whether we are processing in a top-level or nested template
context. So, we instead use tsr.
Change-Id: I002f421fc78576e869cc8fbcdc386ef0abb09ca9
* Added template-wrapping information in cite-expansion with
Object/Ext/Cite RDFa type.
* Added 'ref' tag to whitelisted tags (temporary workaround to
let <ref name='blah' /> to RT correctly till we handle this
properly).
* Added support for self-closing, but non-void tags.
* Deleted Object/Template/Content type attribute since it was
not used.
* Renamed Object/Template/Attributes to ExpandedAttrs/Template
to distinguish template attribute code paths from content
code paths -- could possibly be renamed to something more
suitable later on.
Results:
- With this experimental support, RT diffs on the en:Barack_Obama
page are vastly reduced.
Change-Id: I018f03df102764d3e39442ae359b751056a38c2f
* Moved tokensToString and KVtoHash from parser.environment
to Util -- got rid of excessive warnings from tokensToString.
* Replaced used of Util.lookupKV(blah).v with Util.lookup(blah)
Change-Id: Ic48d5d0c79045b9789dc61a2b14df3fc2acf7aa5
* Nested parser functions are no longer meta-wrapped similar to
nested templates.
* HTML attributes which derive all or part of their key/value
from a template are not template-wrapped, but are marked to
be tracked via different rdfa meta attributes. The actual
tracking and addition of new meta attrs. is yet to be
implemented.
* Parser pipeline construction updated to use an options object
rather than individual option args -- used to support the two
features above.
* With this fix, we can now distinguish between the following
scenarios:
Templates:
- {{echo|[[Foo|bar]]}} -- regular wrapping of the entire link
- [[{{echo|Foo}}|bar]] -- no wrapping, but href marked for tracking
- [[Foo|{{echo|bar}}]] -- link text is wrapped.
Parser functions:
- {{uc:foo}} -- regular wrapping.
- {{uc:{{lc:{{uc:foo}}}}}} -- only the top-level parser function
is wrapped, the nested lc and uc parser functions are not.
Change-Id: I512b8de9838ad42d9255ebcd73074b3898294cd8
* /js/ for JavaScript / nodjs implementation
* /js/lib for js modules
* /js/tests for js test runners and unit tests
Change-Id: I7da1ea2a62b9312be0f1b492447340427f6a14fb
* Copied over utility methods from mediawiki.parser.environment.js
to ext.Util.js.
* Moved over utility method from mediawiki.parser.defines.js to
ext.Util.js.
* Converted Util to be a singleton object rather than an allocatable
class. There is no reason to allocate a new utility class everywhere
since this utility object has no useful state.
* Fixed up use of utility methods to use Util rather than env.
Change-Id: Ib81f96b894f6528f2ccbe36e1fd4c3d50cd1f6b7
- Added extra debug_name parameter to addTransform which is
used in addTransform to output useful trace info.
Change-Id: I160ba0c45f681149375e32ab19f97baa439b09a8
Pages titles with a wikipedia interwiki prefix now load the page from
corresponding Wikipedia. Links in a page then stay within the given language.
Note that Parsoid currently makes no effort to recognize localized namespaces,
so it won't render media files, categories etc correctly.
Change-Id: I7bc4102e81a402772ea23231170734d580ea15b9
other tokens. This is only the first half of the conversion. The next step is
to drop the type attribute on most tokens and match on the constructor in the
token transform machinery.
* TokenTransformDispatcher is now renamed to TokenTransformManager, and is
also turned into a base class
* SyncTokenTransformManager and AsyncTokenTransformManager subclass
TokenTransformManager and implement synchronous (phase 1,3) and asynchronous
(phase 2) transformation stages.
* Communication between stages uses the same chunk / end events as all the
other token stages.
* The AsyncTokenTransformManager now supports the creation of nested
AsyncTokenTransformManagers for template expansion.
The AsyncTokenTransformManager object takes on the responsibilities of a
preprocessor frame. Transforms are newly created (or potentially resurrected
from a cache), so that transforms do not have to worry about concurrency.
* The environment is pushed through to all transform managers and the
individual transforms.
This required a few further additions to the TokenTransformDispatcher. In
particular, there is now an 'any' token match whose callbacks are executed
before more specific callbacks. This is used by the Cite extension to eat all
tokens between ref and /ref tags. This need is very common, so should be
broken out to an intermediate layer in the future.
In general, the requirements for the TokenTransformDispatcher API are now
clearer, and the API should likely be cleaned up / simplified.
