Commit graph

1266 commits

Author SHA1 Message Date
Gabriel Wicke 6bd7ca1e75 Misc improvements, now 196 parser tests passing.
* Add handler for post-expand paragraph wrapping on token stream, to handle
  things like comments on its own line post-expand
* Add general Util module
* Fix self-closing tag handling in HTML5 tree builder
2012-01-17 18:22:10 +00:00
Raimond Spekking 34a2f3f1aa Localisation updates for core and extension messages from translatewiki.net 2012-01-15 20:22:43 +00:00
Gabriel Wicke f4081bef08 First template expansion tests start working, and a bug fix in
DOMPostProcessor paragraph wrapper. 187 parser tests now passing.
2012-01-14 00:58:20 +00:00
Gabriel Wicke 196d704e8e Template expansion now enabled and somewhat working, but template fetching
still fails all the time.
2012-01-13 18:48:25 +00:00
Gabriel Wicke 32c9bccd7c Results of early template expansion debugging. Still disabled by default, but
getting closer.
2012-01-11 19:48:49 +00:00
Gabriel Wicke 6b6ec2933d More work towards template expansion.
* Created AttributeTokenTransformManager for generic attribute conversion, and
  removed { title, template argument {key, value} } expansion from
  TemplateHandler.
* Added caching for attribute and input sub-pipelines. Especially attribute
  pipelines would otherwise be recreated for each attribute value and key.
2012-01-11 00:05:51 +00:00
Raimond Spekking 7333f606b1 Localisation updates for core and extension messages from translatewiki.net 2012-01-10 21:07:49 +00:00
Gabriel Wicke 5ec30252f1 More token transform and pipeline setup refactoring to support template
expansion better.
2012-01-10 01:09:50 +00:00
Raimond Spekking 55bb31c2fe Localisation updates for core and extension messages from translatewiki.net 2012-01-09 21:29:15 +00:00
Gabriel Wicke 287604c422 A bit of cleanup in ParserPipeline, with better and more consistent support
for multiple input types.
2012-01-09 19:33:49 +00:00
Gabriel Wicke becf3cb7ea Add generic 'collect all tokens between delimiter tokens and call a transform
function on it' util for synchronous transformation phases. This can be used
to implement parser hooks (aka extension tags) besides other things.
2012-01-09 18:13:45 +00:00
Gabriel Wicke e99d7a2a55 Two batteries worth of token transform manager refactoring.
* TokenTransformDispatcher is now renamed to TokenTransformManager, and is
  also turned into a base class
* SyncTokenTransformManager and AsyncTokenTransformManager subclass
  TokenTransformManager and implement synchronous (phase 1,3) and asynchronous
  (phase 2) transformation stages.
* Communication between stages uses the same chunk / end events as all the
  other token stages.
* The AsyncTokenTransformManager now supports the creation of nested
  AsyncTokenTransformManagers for template expansion.
  The AsyncTokenTransformManager object takes on the responsibilities of a
  preprocessor frame. Transforms are newly created (or potentially resurrected
  from a cache), so that transforms do not have to worry about concurrency.
* The environment is pushed through to all transform managers and the
  individual transforms.
2012-01-09 17:49:16 +00:00
Raimond Spekking 534a5c60f6 Localisation updates for core and extension messages from translatewiki.net 2012-01-06 23:05:19 +00:00
Gabriel Wicke 6601c544e6 Handle default for template arg expansion, add template fetch functionality
and tweak a few minor things in the grammar and QuoteTransformer.
2012-01-06 17:19:14 +00:00
Gabriel Wicke f0c844f28f Add template expansion handler skeleton, not yet functional. Also note
improvements needed in the tokenizer template handling.
2012-01-06 14:30:55 +00:00
Raimond Spekking 80778bd0df Localisation updates for core and extension messages from translatewiki.net 2012-01-04 21:00:47 +00:00
Mark A. Hershberger 381551e039 w/s 2012-01-04 17:46:24 +00:00
Mark A. Hershberger 3a9a4cf322 re r106536 remove !transparent 2012-01-04 17:44:14 +00:00
Gabriel Wicke 2e35171fd1 Fix quote handling and tweak the whitelist a bit. 'any' token registrations
are now merged with specific registrations by rank. Not yet clear if that is a
good idea overall, need to check use cases when implementing template expansion
and other functionality.

183 parser test now passing.
2012-01-04 14:09:05 +00:00
Gabriel Wicke 6cd95fea37 Fix up constructors in EventEmitter inheritance and tweak a few more comments. 2012-01-04 12:28:41 +00:00
Gabriel Wicke e3ae9a702b Fix JSHint warnings (mostly about comment indentation) from r108012. 2012-01-04 11:06:24 +00:00
Gabriel Wicke 4c4a24f0a0 Hook up the DOMPostProcessor using events as well, and rename the subscription
methods to tell a story. Also document idea on how to dynamically configure
the pipeline depending on event registrations in comment.
2012-01-04 11:00:54 +00:00
Gabriel Wicke f0399d2ec5 Clean up comments in TokenTransformDispatcher and mark private methods with
underscore.
2012-01-04 09:48:24 +00:00
Gabriel Wicke ee79158e53 Add trailing newline in commandline parser wrapper 2012-01-04 08:42:53 +00:00
Gabriel Wicke 29362cc53c Rename ParseThingy to ParserPipeline and fix up broken WikiDom generation and
commandline runner.
2012-01-04 08:39:45 +00:00
Gabriel Wicke bd98eb4c5a Land big TokenTransformDispatcher and eventization refactoring.
The TokenTransformDispatcher now actually implements an asynchronous, phased
token transformation framework as described in
https://www.mediawiki.org/wiki/Future/Parser_development/Token_stream_transformations.

Additionally, the parser pipeline is now mostly held together using events.
The tokenizer still emits a lame single events with all tokens, as block-level
emission failed with scoping issues specific to the PEGJS parser generator.
All stages clean up when receiving the end tokens, so that the full pipeline
can be used for repeated parsing.

The QuoteTransformer is not yet 100% fixed to work with the new interface, and
the Cite extension is disabled for now pending adaptation. Bold-italic related
tests are failing currently.
2012-01-03 18:44:31 +00:00
Raimond Spekking 6d508ee687 Localisation updates for core and extension messages from translatewiki.net 2012-01-02 20:46:12 +00:00
Raimond Spekking 4d6676cf23 Localisation updates for core and extension messages from translatewiki.net 2012-01-01 20:29:00 +00:00
Neil Kandalgaonkar 9d198ecad6 when nothing to undo or redo, grey out appropriate buttons - fix bug #33112, based on patch from ashish.dubey91@gmail.com 2011-12-31 01:44:34 +00:00
Neil Kandalgaonkar 20374b5911 fix substr for IE, followup r107464 2011-12-30 21:51:03 +00:00
Raimond Spekking 5a52147307 Localisation updates for core and extension messages from translatewiki.net 2011-12-30 21:17:13 +00:00
Gabriel Wicke 8e00a72d0a Improvements to link trail handling, and two tweaks to the whitelist. 182
tests now passing. 

Link trails depend on language-dependent positive character classes in the PHP
parser. These classes all seem to disallow punctuation implicitly and list
differing plain text characters instead, so it might be possible to get away
with identifying a common class of non-trail punctuation instead. This would
help to keep the tokenizer independent of configurations, which is very
desirable for caching and simplified external parsing.
2011-12-30 12:47:06 +00:00
Gabriel Wicke 11ece76b7b Fix suffix handling for wiki links. 2011-12-30 09:35:57 +00:00
Raimond Spekking 3365260fb4 Localisation updates for core and extension messages from translatewiki.net 2011-12-29 21:54:17 +00:00
Raimond Spekking d46e07dbe3 Localisation updates for core and extension messages from translatewiki.net 2011-12-28 22:31:07 +00:00
Gabriel Wicke b3a0270d69 Remove env and load grammar in tokenizer constructor. Re-add property hack to
keep parserTests running for now. Really need a different pipeline for html
serialization or a reference to the HTML DOM.
2011-12-28 17:04:16 +00:00
Gabriel Wicke 3a63fb118e Add a few comments inline, and remove unneeded html serialization as we are
only interested in WikiDom output in this parser wrapper.
2011-12-28 13:46:52 +00:00
Neil Kandalgaonkar 8fbf36e63e put add terminal token inside tokenize method (will pull it out again for streaming interface) 2011-12-28 01:37:15 +00:00
Neil Kandalgaonkar 6103646ec8 remove need to add newline at end of input 2011-12-28 01:37:11 +00:00
Neil Kandalgaonkar 4158f82d7e refactor parser to ParseThingy in different module, can be invoked with command line utility parse.js 2011-12-28 01:37:06 +00:00
Neil Kandalgaonkar aedc6751ae made parseThingy, temp class for refactoring all thingies related to parsing 2011-12-28 01:36:58 +00:00
Neil Kandalgaonkar d91a67ba99 nodeName not defined 2011-12-28 01:36:54 +00:00
Neil Kandalgaonkar 5ff2b4d475 make peg src path outside of peg tokenizer 2011-12-28 01:36:50 +00:00
Neil Kandalgaonkar 962d1262fc create tokenizer without need to modify namespace with PEG source 2011-12-28 01:36:36 +00:00
Raimond Spekking f7dd83e07b Localisation updates for core and extension messages from translatewiki.net 2011-12-27 20:23:52 +00:00
Raimond Spekking bbef1c2f87 Localisation updates for core and extension messages from translatewiki.net 2011-12-26 21:21:55 +00:00
Siebrand Mazeland 97502c9969 Localisation updates for special pages and magic words for extensions from http://translatewiki.net.
Backwards compatibility with MediaWiki 1.15 and earlier dropped for special page aliases.
2011-12-25 22:19:29 +00:00
Raimond Spekking 833c5e1499 Localisation updates for core and extension messages from translatewiki.net 2011-12-25 20:25:45 +00:00
Raimond Spekking a7d60eb673 Localisation updates for core and extension messages from translatewiki.net 2011-12-23 21:10:51 +00:00
Raimond Spekking ad849eee21 Localisation updates for core and extension messages from translatewiki.net 2011-12-22 20:15:58 +00:00