Commit graph

8 commits

Author SHA1 Message Date
Gabriel Wicke cc1afb2ad3 Support dt/dd transitions in the middle of the stack
Change-Id: I1d75caa7782d02a2c33413a078e99b17ccc4141c
2012-06-21 18:40:40 +02:00
Gabriel Wicke b3bd2ffe8d Fix definition list parsing and round-trip single vs. multi-line dt/dd
* Removed murky ' :' -> ' :' replacement in tokenizer. This breaks four
  parser tests, and should be fixed in a token stream transformer or DOM
  postprocessor. This replacement clashes with round-tripping, and is not
  terribly important visually.
* Added stx:row annotation to single-line dt/dd pairs and use it to preserve
  single-line syntax in the serializer. There is no attempt yet to support the
  addition of nested lists in an originally single-line dd. We'd need to look
  ahead in the serializer to support this. Perhaps the editor can simply drop
  data-mw in that case.
* Switched default dt/dd serialization to multi-line. This supports all nested
  lists and multiple dds.
* Don't close dls when switching from dt to dd or back in the token stream
  ListHandler.

Overall 290 round-trip tests are passing now (up from 284, some due to  ,
some due to lists). The number of passing parser tests dropped slightly from
303 to 297 (or 301/295 on weekdays other than Thursday).

Change-Id: I85ff40571833713388c6523e6a4ba2e94daa3807
2012-06-21 17:34:25 +02:00
Gabriel Wicke 2426901e5b Fix definition lists with multiple dds
Fixed a bug in the list handler for multiple dds in a definition list. Also
fixed a few JSHint warnings.

Change-Id: I3e883786698a9521347fc2a5e6420646318813a7
2012-06-20 15:34:20 +02:00
Gabriel Wicke ee2ddbd3cb Fix list handler issues
Lists interrupted by non-empty lines would not close the list properly.
Register for any token instead of just for newlines and close the list if no
listItem follows the newline.

Change-Id: I1743901e3db541bbeda78d17707db943e6ceb9b9
2012-06-04 13:38:43 +02:00
Gabriel Wicke 3414418b1f Don't eat newline tokens in the ListHandler
This fix only affects following transforms, of which there are few right now.
Also removed a stray token mutation in QuoteTransformer.

Change-Id: Id6d4adce944b06fc1a3651cfbf63fc2670125225
2012-05-16 23:14:21 +02:00
Gabriel Wicke d918fa18ac Big token transform framework overhaul part 2
* Tokens are now immutable. The progress of transformations is tracked on
  chunks instead of tokens. Tokenizer output is cached and can be directly
  returned without a need for cloning. Transforms are required to clone or
  newly create tokens they are modifying.

* Expansions per chunk are now shared between equivalent frames via a cache
  stored on the chunk itself. Equivalence of frames is not yet ideal though,
  as right now a hash tree of *unexpanded* arguments is used. This should be
  switched to a hash of the fully expanded local parameters instead.

* There is now a vastly improved maybeSyncReturn wrapper for async transforms
  that either forwards processing to the iterative transformTokens if the
  current transform is still ongoing, or manages a recursive transformation if
  needed.

* Parameters for parser functions are now wrapped in abstract Params and
  ParserValue objects, which support some handy on-demand *value* expansions.
  Keys are always expanded. Parser functions are converted to use these
  interfaces, and now properly expand their values in the correct frame.
  Making this expansion lazier is certainly possible, but would complicate
  transformTokens and other token-handling machinery. Need to investigate if
  it would really be worth it. Dead branch elimination is certainly a bigger
  win overall.

* Complex recursive asynchronous expansions should now be closer to correct
  for both the iterative (transformTokens) and recursive (maybeSyncReturn
  after transformTokens has returned) code paths.

* Performance degraded slightly. There are no micro-optimizations done yet
  and the shared expansion cache still has a low hit rate. The progress
  tracking on chunks is not yet perfect, so there are likely a lot of unneeded
  re-expansions that can be easily eliminated. There is also more debug
  tracing right now. Obama currently expands in 54 seconds on my laptop.

Change-Id: I4a603f3d3c70ca657ebda9fbb8570269f943d6b6
2012-05-15 17:05:47 +02:00
Gabriel Wicke b1bd0d73ec Don't eat end token in ListHandler, and lazier Quote handler registration
* Setting the rank on tokens is still used currently, but will be phased out
  in favor of setting it on chunks. Tokens will be immutable to allow sharing
  and caching without a need for cloning.
* Only register for newline and end tokens in QuoteTransformer when active.

Change-Id: I2c45bc7e4a105219a1404ab221eed7f242128f1e
2012-05-10 09:47:53 +02:00
Adam Wight 0a7f0b7630 List markup is created during the sync23 phase.
This makes it possible to transclude list items from a template.

Note: "5 quotes" test is broken by this patch, it appears that ListHandler
newline processing is changing some state which mysteriously affects the
QuoteTransformer.  This is ominous, hopefully there's a simple explanation...

gwicke: fix a bug in tokenizer triggered by definition lists like this:
**; foo : bar

Change-Id: I4e3a86596fe9bffcbfc4bf22895362c3bf742bad
2012-05-08 11:39:36 +02:00