wikimedia/mediawiki-extensions-VisualEditor
1
0
Fork 0
mirror of https://gerrit.wikimedia.org/r/mediawiki/extensions/VisualEditor synced 2024-11-15 18:39:52 +00:00
Code Issues Packages Projects Releases Wiki Activity
mediawiki-extensions-Visual.../modules/parser
History
Gabriel Wicke d918fa18ac Big token transform framework overhaul part 2 
* Tokens are now immutable. The progress of transformations is tracked on
  chunks instead of tokens. Tokenizer output is cached and can be directly
  returned without a need for cloning. Transforms are required to clone or
  newly create tokens they are modifying.

* Expansions per chunk are now shared between equivalent frames via a cache
  stored on the chunk itself. Equivalence of frames is not yet ideal though,
  as right now a hash tree of *unexpanded* arguments is used. This should be
  switched to a hash of the fully expanded local parameters instead.

* There is now a vastly improved maybeSyncReturn wrapper for async transforms
  that either forwards processing to the iterative transformTokens if the
  current transform is still ongoing, or manages a recursive transformation if
  needed.

* Parameters for parser functions are now wrapped in abstract Params and
  ParserValue objects, which support some handy on-demand *value* expansions.
  Keys are always expanded. Parser functions are converted to use these
  interfaces, and now properly expand their values in the correct frame.
  Making this expansion lazier is certainly possible, but would complicate
  transformTokens and other token-handling machinery. Need to investigate if
  it would really be worth it. Dead branch elimination is certainly a bigger
  win overall.

* Complex recursive asynchronous expansions should now be closer to correct
  for both the iterative (transformTokens) and recursive (maybeSyncReturn
  after transformTokens has returned) code paths.

* Performance degraded slightly. There are no micro-optimizations done yet
  and the shared expansion cache still has a low hit rate. The progress
  tracking on chunks is not yet perfect, so there are likely a lot of unneeded
  re-expansions that can be easily eliminated. There is also more debug
  tracing right now. Obama currently expands in 54 seconds on my laptop.

Change-Id: I4a603f3d3c70ca657ebda9fbb8570269f943d6b6
2012-05-15 17:05:47 +02:00
..
html5 Land big TokenTransformDispatcher and eventization refactoring. 2012-01-03 18:44:31 +00:00
ext.Cite.js Forward-port Cite extension 2012-05-03 13:22:01 +02:00
ext.cite.taghook.ref.js
ext.core.AttributeExpander.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
ext.core.BehaviorSwitchHandler.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
ext.core.LinkHandler.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
ext.core.ListHandler.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
ext.core.NoIncludeOnly.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
ext.core.ParserFunctions.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
ext.core.PostExpandParagraphHandler.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
ext.core.QuoteTransformer.js Don't eat end token in ListHandler, and lazier Quote handler registration 2012-05-10 09:47:53 +02:00
ext.core.Sanitizer.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
ext.core.TemplateHandler.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
ext.Util.js Nominate more HTML5 sectioning and heading elements for block-level treatment 2012-04-11 12:53:49 +02:00
ext.util.TokenCollector.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
mediawiki.DOMConverter.js Replace console.log with console.warn in all debug statements 2012-02-14 20:56:14 +00:00
mediawiki.DOMPostProcessor.js Biggish token transform system refactoring 2012-04-25 16:51:36 +02:00
mediawiki.HTML5TreeBuilder.node.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
mediawiki.LinearModelConverter.js Add HTML DOM -> linear model converter 2012-03-29 12:47:14 -07:00
mediawiki.parser.defines.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
mediawiki.parser.environment.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
mediawiki.parser.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
mediawiki.Title.js Add basic thumb rendering support 2012-04-09 23:04:26 +02:00
mediawiki.tokenizer.peg.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
mediawiki.TokenTransformManager.js Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
package.json Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
parse.js Fix --wikidom and --linearmodel parse.js options; retry on template fetch failures 2012-04-26 19:51:00 +02:00
pegTokenizer.pegjs.txt Big token transform framework overhaul part 2 2012-05-15 17:05:47 +02:00
README.txt As much as I have loved writing Makefiles... I've replaced its functionality with package.json, mostly so we can avoid non-node dependencies. This is one of the recommended practices. We should consider moving tests/parser into modules/parser/tests, other node projects keep all module code in one directory. 2012-04-04 11:02:58 -07:00

README.txt 

A combined Mediawiki and html parser in JavaScript running on node.js. Please
see (https://www.mediawiki.org/wiki/Future/Parser_development) for an overview
of the current implementation, and instructions on running the tests.

You might need to set the NODE_PATH environment variable,
  export NODE_PATH="node_modules"

Download the dependencies:
  npm install

Run tests:
  npm test