Commit graph

13 commits

Author SHA1 Message Date
Subramanya Sastry 6999246fec More fixes on the way to readable debug/trace output.
Change-Id: I3ab637b7790c359255b8bc0ad5716da12bb25884
2012-07-20 18:12:37 -05:00
Subramanya Sastry d994d1292e First pass updating debug output
Change-Id: I1e0953d7b26b962e3758dd1091e87a5361257abc
2012-07-20 18:12:37 -05:00
Gabriel Wicke e117f09362 Wikitext escaping and quite complete source range tracking
* Started to add more complete tag source range (tsr) annotations to most
  start / empty tags. These replace the old sourcePos and sourceTagPos
  annotations, and look more promising for general round-tripping than block
  source ranges (bsr). See
  http://www.mediawiki.org/wiki/User:GWicke/Parsoid_source_ranges for some
  notes on this.
* Added an escapeWikitext method in the serializer that tokenizes supposedly
  text-only content from the DOM with the tokenizer and wraps runs of returned
  non-text tokens into nowiki tags. The source corresponding to non-text
  tokens is retrieved using the tsr annotations.
* Removed old (unused) table productions to avoid confusion.
* 276 round-trip tests are passing, vs. 283 without escaping.

Known issues:
* harmless for now, can be improved later: urllinks in external link captions
  are wrapped in nowiki. Example HTML:

<a rel='mw:extLink' href="http://example.com">http://example2.com</a>

* some start-of-line syntax in wiki-syntax preformatted blocks might be
  wrapped into nowiki when that would not really be needed. Example HTML DOM:

<pre>
* foo
* bar
</pre>

Change-Id: I01c34aedd5c566614d36924add47a6a960e91987
2012-06-19 23:36:44 +02:00
Gabriel Wicke 16219ddc6d Fix up #switch a bit
* Re-establish the value-only default
* Fix value expansion

Change-Id: I32e62789b25bbe17a74c564e41e9101ad5528fb7
2012-06-01 22:15:43 +02:00
Gabriel Wicke d918fa18ac Big token transform framework overhaul part 2
* Tokens are now immutable. The progress of transformations is tracked on
  chunks instead of tokens. Tokenizer output is cached and can be directly
  returned without a need for cloning. Transforms are required to clone or
  newly create tokens they are modifying.

* Expansions per chunk are now shared between equivalent frames via a cache
  stored on the chunk itself. Equivalence of frames is not yet ideal though,
  as right now a hash tree of *unexpanded* arguments is used. This should be
  switched to a hash of the fully expanded local parameters instead.

* There is now a vastly improved maybeSyncReturn wrapper for async transforms
  that either forwards processing to the iterative transformTokens if the
  current transform is still ongoing, or manages a recursive transformation if
  needed.

* Parameters for parser functions are now wrapped in abstract Params and
  ParserValue objects, which support some handy on-demand *value* expansions.
  Keys are always expanded. Parser functions are converted to use these
  interfaces, and now properly expand their values in the correct frame.
  Making this expansion lazier is certainly possible, but would complicate
  transformTokens and other token-handling machinery. Need to investigate if
  it would really be worth it. Dead branch elimination is certainly a bigger
  win overall.

* Complex recursive asynchronous expansions should now be closer to correct
  for both the iterative (transformTokens) and recursive (maybeSyncReturn
  after transformTokens has returned) code paths.

* Performance degraded slightly. There are no micro-optimizations done yet
  and the shared expansion cache still has a low hit rate. The progress
  tracking on chunks is not yet perfect, so there are likely a lot of unneeded
  re-expansions that can be easily eliminated. There is also more debug
  tracing right now. Obama currently expands in 54 seconds on my laptop.

Change-Id: I4a603f3d3c70ca657ebda9fbb8570269f943d6b6
2012-05-15 17:05:47 +02:00
Gabriel Wicke 8368e17d6a Biggish token transform system refactoring
* All parser pipelines including tokenizer and DOM stuff are now constructed
  from a 'recipe' data structure in a ParserPipelineFactory.

* All sub-pipelines of these can now be cached

* Event registrations to a pipeline are directly forwarded to the last
  pipeline member to save relatively expensive event forwarding.

* Some APIs for on-demand expansion / format conversion of parameters from
  parser functions are added:

  param.to('tokens/expanded', cb)
  param.to('text/wiki', cb) (this does not work yet)

  All parameters are additionally wrapped into a Param object that provides
  method for positional parameter naming (.named() or conversion to a dict
  (.dict()).

* The async token transform manager is now separated from a frame object, with
  the frame holding arguments, an on-demand expansion method and loop checks.

* Only keys of template parameters are now expanded. Parser functions or
  template arguments trigger an expansion on-demand. This (unsurprisingly)
  makes a big performance difference with typical switch-heavy template
  systems.

* Return values from async transforms are no longer used in favor of plain
  callbacks. This saves the complication of having to maintain two code paths.
  A trick in transformTokens still avoids the construction of unneeded
  TokenAccumulators.

* The results of template expansions are no longer buffered.

* 301 parser tests are passing

Known issues:

* Cosmetic cleanup remains to do
* Some parser functions do not support async expansions yet, and need to be
  modified.

Change-Id: I1a7690baffbe8141cadf67270904a1b2e1df879a
2012-04-25 16:51:36 +02:00
Gabriel Wicke 2195c31abf Move link types to data-mw-rt, and support some more template tokenization
edge cases. For example, the PHP parser treats | foo | = bar | as | foo = bar |,
believe it or not ;)
2012-03-13 12:32:31 +00:00
Gabriel Wicke f02ff95aa3 Token representation clean-up. Now all tokens are differentiated using
constructors instead of type attributes.
2012-03-07 20:06:54 +00:00
Gabriel Wicke 0b8d1b0387 * Add custom toString methods for tokens to aid debugging
* Convert all attributes into strings in Sanitizer
* Use strict comparison against empty string in tokenizer
* Add very simple sitename parserfunction
* 138 tests passing
2012-02-13 17:02:23 +00:00
Gabriel Wicke c26243989e Improve toJSON handlers to include all properties 2012-02-06 19:12:29 +00:00
Gabriel Wicke 26f2026cff Add custom JSON serializers for tokens that include a type attribute 2012-02-03 13:09:01 +00:00
Gabriel Wicke 8c75aa1a7a Remove type attribute for tag tokens. 2012-02-01 18:37:48 +00:00
Gabriel Wicke 689f697a93 Push token format conversion a bit further along, and add defines that were
missing in last commit.
2012-02-01 17:03:08 +00:00