Commit graph

225 commits

Author SHA1 Message Date
Subramanya Sastry 760d63d033 Improved template roundtripping based on WT ranges on DOM nodes.
- Fixed several bugs in the DSR computation.
- Deleted comment tokens from template output.
- Eliminated TSR info from template output.
- Rearranged order of DOM post passes.
- Updated template encapsulation to use DSR values.
- There are still a few missing pieces and scenarios that are
  not handled properly.

Change-Id: I644914902ec1d27418138b107abe765f81178398
2012-09-17 21:08:35 -05:00
Subramanya Sastry 0520effd3d JSHint fixes only.
Change-Id: I8ad4e283685ab460802e69e21504ddc241985f80
2012-09-12 12:06:48 -05:00
Subramanya Sastry dbd628fd87 Added getWTSource convenience method to tokens.
Change-Id: I8c99fcd4ccefbae62a1639e5738fe2d4c88f122a
2012-09-06 09:29:38 -07:00
Subramanya Sastry 67326f063a Check if tsr is present in references tokens before generating src rt-info.
* Only one instance of Cite handler is created and it deals with the
  final expanded token stream.  So, effectively there is no information
  about whether we are processing in a top-level or nested template
  context.  So, we instead use tsr.

Change-Id: I002f421fc78576e869cc8fbcdc386ef0abb09ca9
2012-09-05 19:48:08 -05:00
Subramanya Sastry d3f0007f1b Updates to the references tag rt-ing to get rid of FIXMEs.
Change-Id: I8b832925ad476fb8043f9beb1e39e6efb27ea336
2012-09-04 20:53:33 -05:00
Subramanya Sastry 23f3e0f1aa RT references extension tag.
Change-Id: Ibd1be584e17814a84154549b7bc353465734b847
2012-09-04 20:07:26 -05:00
Subramanya Sastry ca04fdc293 Experimental ref-tag support (for demoing purposes).
* Added template-wrapping information in cite-expansion with
  Object/Ext/Cite RDFa type.
* Added 'ref' tag to whitelisted tags (temporary workaround to
  let <ref name='blah' /> to RT correctly till we handle this
  properly).
* Added support for self-closing, but non-void tags.
* Deleted Object/Template/Content type attribute since it was
  not used.
* Renamed Object/Template/Attributes to ExpandedAttrs/Template
  to distinguish template attribute code paths from content
  code paths -- could possibly be renamed to something more
  suitable later on.

Results:
- With this experimental support, RT diffs on the en:Barack_Obama
  page are vastly reduced.

Change-Id: I018f03df102764d3e39442ae359b751056a38c2f
2012-09-04 18:51:29 -05:00
Subramanya Sastry f733af93c4 Code cleanup.
* Moved tokensToString and KVtoHash from parser.environment
  to Util -- got rid of excessive warnings from tokensToString.
* Replaced used of Util.lookupKV(blah).v with Util.lookup(blah)

Change-Id: Ic48d5d0c79045b9789dc61a2b14df3fc2acf7aa5
2012-08-24 15:29:53 -05:00
Subramanya Sastry 2d15258803 Fixed template meta-wrapping to handle nesting and attributes.
* Nested parser functions are no longer meta-wrapped similar to
  nested templates.
* HTML attributes which derive all or part of their key/value
  from a template are not template-wrapped, but are marked to
  be tracked via different rdfa meta attributes.  The actual
  tracking and addition of new meta attrs. is yet to be
  implemented.
* Parser pipeline construction updated to use an options object
  rather than individual option args -- used to support the two
  features above.
* With this fix, we can now distinguish between the following
  scenarios:

  Templates:
  - {{echo|[[Foo|bar]]}} -- regular wrapping of the entire link
  - [[{{echo|Foo}}|bar]] -- no wrapping, but href marked for tracking
  - [[Foo|{{echo|bar}}]] -- link text is wrapped.

  Parser functions:
  - {{uc:foo}} -- regular wrapping.
  - {{uc:{{lc:{{uc:foo}}}}}} -- only the top-level parser function
    is wrapped, the nested lc and uc parser functions are not.

Change-Id: I512b8de9838ad42d9255ebcd73074b3898294cd8
2012-08-19 15:21:36 -05:00
Gabriel Wicke 4900d85606 Reorganize Parsoid repo after split from VisualEditor
* /js/ for JavaScript / nodjs implementation
* /js/lib for js modules
* /js/tests for js test runners and unit tests

Change-Id: I7da1ea2a62b9312be0f1b492447340427f6a14fb
2012-07-30 13:10:22 -07:00
Subramanya Sastry 4ae497db1e Renamed ext.Util.js to mediawiki.Util.js
Change-Id: I909847686c8239a0b00cbaa9a0b1583826ee1487
2012-07-24 13:07:53 -05:00
Subramanya Sastry 194c7afb34 Added utility methods to ext.Util.js
* Copied over utility methods from mediawiki.parser.environment.js
  to ext.Util.js.
* Moved over utility method from mediawiki.parser.defines.js to
  ext.Util.js.
* Converted Util to be a singleton object rather than an allocatable
  class.  There is no reason to allocate a new utility class everywhere
  since this utility object has no useful state.
* Fixed up use of utility methods to use Util rather than env.

Change-Id: Ib81f96b894f6528f2ccbe36e1fd4c3d50cd1f6b7
2012-07-20 18:12:37 -05:00
Subramanya Sastry c0ab29779c In trace mode, wrap transform to output trace info
- Added extra debug_name parameter to addTransform which is
  used in addTransform to output useful trace info.

Change-Id: I160ba0c45f681149375e32ab19f97baa439b09a8
2012-07-20 18:12:37 -05:00
Gabriel Wicke 0a4f1d435b Very basic interwiki support
Pages titles with a wikipedia interwiki prefix now load the page from
corresponding Wikipedia. Links in a page then stay within the given language.

Note that Parsoid currently makes no effort to recognize localized namespaces,
so it won't render media files, categories etc correctly.

Change-Id: I7bc4102e81a402772ea23231170734d580ea15b9
2012-06-05 11:19:58 +02:00
Gabriel Wicke 89c5f4aae4 Forward-port Cite extension
* Adapted Cite extension to use current interfaces and token formats
* Improved TokenCollector

Change-Id: I20419b19edd9bbad2c2abf17a2ff1411b99c0c04
2012-05-03 13:22:01 +02:00
Gabriel Wicke 42af5277ab Token representation clean-up. Now all tokens are differentiated using
constructors instead of type attributes.
2012-03-07 20:06:54 +00:00
Gabriel Wicke 06b257089b Replace console.log with console.warn in all debug statements 2012-02-14 20:56:14 +00:00
Gabriel Wicke 142e40c563 Change token format to plain strings for text tokens, and specific objects for
other tokens. This is only the first half of the conversion. The next step is
to drop the type attribute on most tokens and match on the constructor in the
token transform machinery.
2012-02-01 16:30:43 +00:00
Gabriel Wicke 840941546b Two batteries worth of token transform manager refactoring.
* TokenTransformDispatcher is now renamed to TokenTransformManager, and is
  also turned into a base class
* SyncTokenTransformManager and AsyncTokenTransformManager subclass
  TokenTransformManager and implement synchronous (phase 1,3) and asynchronous
  (phase 2) transformation stages.
* Communication between stages uses the same chunk / end events as all the
  other token stages.
* The AsyncTokenTransformManager now supports the creation of nested
  AsyncTokenTransformManagers for template expansion.
  The AsyncTokenTransformManager object takes on the responsibilities of a
  preprocessor frame. Transforms are newly created (or potentially resurrected
  from a cache), so that transforms do not have to worry about concurrency.
* The environment is pushed through to all transform managers and the
  individual transforms.
2012-01-09 17:49:16 +00:00
Gabriel Wicke 0866116fe4 Comment out a stray console.log 2011-12-14 23:44:58 +00:00
Gabriel Wicke fc41c8887c A collection of small bug fixes to the grammar, Cite, the Token format
converter and the HTML DOM -> WikiDom converter. The tokenizer now digests all
parserTests.
2011-12-14 23:38:46 +00:00
Gabriel Wicke 730b75ca21 Add rough HTML DOM to WikiDom conversion. You can see serialized WikiDom of
parser tests using 'node parserTests.js --wikidom'.
2011-12-14 15:15:41 +00:00
Gabriel Wicke 7a6f71db2f Minor tweak to comment. 2011-12-13 18:55:44 +00:00
Gabriel Wicke 8e8362b8f0 Clean up and comment the Cite extension a bit. 2011-12-13 18:45:09 +00:00
Gabriel Wicke 0fed6f9c79 Convert the Cite extension to a token stream transformer.
This required a few further additions to the TokenTransformDispatcher. In
particular, there is now an 'any' token match whose callbacks are executed
before more specific callbacks. This is used by the Cite extension to eat all
tokens between ref and /ref tags. This need is very common, so should be
broken out to an intermediate layer in the future.

In general, the requirements for the TokenTransformDispatcher API are now
clearer, and the API should likely be cleaned up / simplified.
2011-12-13 14:48:47 +00:00