Commit graph

195 commits

Author SHA1 Message Date
Adam Wight 0a7f0b7630 List markup is created during the sync23 phase.
This makes it possible to transclude list items from a template.

Note: "5 quotes" test is broken by this patch, it appears that ListHandler
newline processing is changing some state which mysteriously affects the
QuoteTransformer.  This is ominous, hopefully there's a simple explanation...

gwicke: fix a bug in tokenizer triggered by definition lists like this:
**; foo : bar

Change-Id: I4e3a86596fe9bffcbfc4bf22895362c3bf742bad
2012-05-08 11:39:36 +02:00
Gabriel Wicke 3be4992782 'Obama finally expands' ;) Misc fixes and documentation updates
* [[:en:Barack Obama]] can now be expanded in 77 seconds using 330MB RAM,
  while it would prevously run out of RAM after ~30 minutes. Wohoooo!
  The token transform framework rework really paid off.
* 303 parser tests are passing in the new record time of 5.5 seconds. Two more
  tests are passing since these tests expect the day of the week to be
  Thursday.  Won't be the case tomorrow.

Change-Id: I56e850838476b546df10c6a239c8c9e29a1a3136
2012-04-26 18:18:08 +02:00
Gabriel Wicke 8ff810659a Rename text/wiki and tokens/wiki to text/x-mediawiki and similar
Change-Id: I70113629f4633685cd6db3914303a15e4c79a50a
2012-04-25 20:19:43 +02:00
Gabriel Wicke 8368e17d6a Biggish token transform system refactoring
* All parser pipelines including tokenizer and DOM stuff are now constructed
  from a 'recipe' data structure in a ParserPipelineFactory.

* All sub-pipelines of these can now be cached

* Event registrations to a pipeline are directly forwarded to the last
  pipeline member to save relatively expensive event forwarding.

* Some APIs for on-demand expansion / format conversion of parameters from
  parser functions are added:

  param.to('tokens/expanded', cb)
  param.to('text/wiki', cb) (this does not work yet)

  All parameters are additionally wrapped into a Param object that provides
  method for positional parameter naming (.named() or conversion to a dict
  (.dict()).

* The async token transform manager is now separated from a frame object, with
  the frame holding arguments, an on-demand expansion method and loop checks.

* Only keys of template parameters are now expanded. Parser functions or
  template arguments trigger an expansion on-demand. This (unsurprisingly)
  makes a big performance difference with typical switch-heavy template
  systems.

* Return values from async transforms are no longer used in favor of plain
  callbacks. This saves the complication of having to maintain two code paths.
  A trick in transformTokens still avoids the construction of unneeded
  TokenAccumulators.

* The results of template expansions are no longer buffered.

* 301 parser tests are passing

Known issues:

* Cosmetic cleanup remains to do
* Some parser functions do not support async expansions yet, and need to be
  modified.

Change-Id: I1a7690baffbe8141cadf67270904a1b2e1df879a
2012-04-25 16:51:36 +02:00
Gabriel Wicke aaca5eac7d More tweaks: safesubst and image options
* Ignore safesubst for now
* Remove an unneeded whitelist entry
* Make sure the caption is not lost for thumbs (fix to last commit) and remove
  debug print

Change-Id: I243584ed0838cf7c3b4110fe9cdf869272477312
2012-04-17 11:02:52 +02:00
Gabriel Wicke afa5b95bc1 Don't work around html5 library tokenizer attribute reordering
The HTML5 parser we are using to normalize expected HTML output in parserTests
reverses the order of attributes (see
https://github.com/aredridel/html5/pull/53 for the fix). Remove whitelist
entries concerned with this and use the proper order in external image
attributes.

Change-Id: If1868cae05396a150757c85a20473ab756cbcd97
2012-04-16 17:09:06 +02:00
Catrope 7465b670e1 Add and update an offset map in DocumentNode
This has some TODOs still but I want to land it now anyway, and fix the
TODOs later.

* Add this.offsetMap which maps each linear model offset to a model tree node
* Refactor createNodesFromData()
** Rename it to buildSubtreeFromData()
** Have it build an offset map as well as a node subtree
** Have it set the root on the fake root node so that when the subtree
   is attached to the main tree later, we don't get a rippling root
   update all the way down
** Normalize the way the loop processes content, that way adding offsets
   for content is easier
* Add rebuildNodes() which uses buildSubtreeFromData() to rebuild stuff
* Use rebuildNodes() in DocumentSynchronizer
* Use pushRebuild() in TransactionProcessor
* Optimize setRoot() for the case where the root is already set correctly

Change-Id: I8b827d0823c969e671615ddd06e5f1bd70e9d54c
2012-04-13 16:46:02 -07:00
Gabriel Wicke 9913108b40 Fix fetch-parserTests (it is in path instead of fs)
Change-Id: I169502079ea2609a4f4af776b15767cf0c3ec8b5
2012-04-04 20:40:09 +02:00
Adam Wight b234edba88 As much as I have loved writing Makefiles... I've replaced its functionality with package.json, mostly so we can avoid non-node dependencies. This is one of the recommended practices. We should consider moving tests/parser into modules/parser/tests, other node projects keep all module code in one directory.
Explained in the README how to use npm to load the dependencies and run tests.  Too bad about NODE_PATH...

Don't try to find parserTests.txt in assorted places--if it isn't present, fetch from gerrit.  You can symlink from core if you're developing on both parsers, and the fetch script will not overwrite.

Use __dirname in parserTests.js to allow the script to run independent of current working directory.

Change-Id: I4c8b884e91f4fdeae385c7697aff768bdd199dd5
2012-04-04 11:02:58 -07:00
Gabriel Wicke f662690d02 Shorten data-mw-rt to data-mw and clean up whitelist
Instead of a proliferation of data-mw-* attributes, it should be easier to
stash all private / non-semantic round-trip information in a JSON object
stored in data-mw.

Change-Id: Id200a6a8789fa152f29ea530e5a24b6ee7b4b285
2012-04-02 18:12:49 +02:00
Gabriel Wicke 5ef3438ee5 Change path to parserTests from phase3 to core after switch to git.
Change-Id: Ie13f678eaa81447e98db5c8c394ab103caad8454
2012-04-02 17:10:06 +02:00
Audrey Tang d3602bb459 * Get parser tests from GitWeb, not Subversion.
Change-Id: I39f933b9e0320dc62736da07ce097ec1badec9aa
2012-03-28 23:39:01 +08:00
Catrope 7a726b0278 Add tree synchronization for replace
To handle replace operations that are not themselves consistent (these
are common, for instance when replacing an opening element in one place,
then replacing the closing element somewhere else), we process
subsequent replace operations inside the first one until things are
balanced again, then issue a single rebuild for the whole thing.

Change-Id: Ide4613f046fabfeeef383138c39e350b1b710033
2012-03-26 02:51:30 -07:00
Roan Kattouw 662633dfb3 Add a test for unwrapping and rewrapping 2012-03-14 21:02:33 +00:00
Roan Kattouw 2c43a34f74 Rewrite the rebuild action to take two ranges rather than a node and some data. 2012-03-14 21:02:31 +00:00
Roan Kattouw 37a59016e8 Break out pushAction() into separate functions for each action. This will allow me to change the rebuild action to take totally different parameters. 2012-03-14 21:02:29 +00:00
Antoine Musso f637756319 node modules required: request & jshashes 2012-03-13 15:14:18 +00:00
Roan Kattouw 16a2356e43 Add tests for list split tree sync 2012-03-13 00:14:38 +00:00
Trevor Parscal c977591886 Added test for ve.dm.DocumentSynchronizer that exercises multi-action synchronizations 2012-03-09 19:38:54 +00:00
Roan Kattouw d70aa70707 Add test for replacing a table with a list. This only works because
nesting validity isn't checked yet (lists inside lists are illegal
IIRC), but for now it tests the reversal of the order of the closing
tags nicely
2012-03-09 02:19:50 +00:00
Roan Kattouw b13d0a849d Add a check for the length of unwrapOuter, and add a test for each
exception
2012-03-09 01:44:31 +00:00
Roan Kattouw bc600b34be Make prepareWrap() use the data from the model rather than the unwrap
parameters. This fixes the case where rolling back a list unwrap would
restore the list items without their attributes
2012-03-09 01:14:41 +00:00
Roan Kattouw 3bc6b3d8c7 Add tests for unwrapping a list
This also excercises unwrapEach. One of the tests is still subtly broken
in that the attributes on the listItems aren't preserved, I'll fix that
next.
2012-03-09 00:38:35 +00:00
Roan Kattouw 5054ed320e Implement prepareWrap and add tests for it 2012-03-08 23:21:26 +00:00
Roan Kattouw 10a6ee73f4 Add tests for content replacements 2012-03-08 23:21:23 +00:00
Trevor Parscal 3ec0c07843 Fixed name of test suite to match actual class name 2012-03-08 19:37:13 +00:00
Trevor Parscal becb1daa39 Added more tests for ve.dm.DocumentSynchronizer and fixed some bugs along the way 2012-03-08 19:35:51 +00:00
Trevor Parscal 459c4fa271 Added some basic tests for resize and insert. Fixed some bugs in both of those code paths along the way. 2012-03-08 00:52:30 +00:00
Gabriel Wicke af03eb4f29 Improve generic attribute expansion before external link processing, and make
wgUploadPath configurable. Also change the hard-coded fall-back image sizes to
sensible defaults. This breaks three parser tests until image size retrieval
from the wiki is implemented.
2012-03-06 18:02:35 +00:00
Gabriel Wicke 227103e12c Accept empty table cell attribute sections, and consider percent-encoded %2525
valid. 270 tests passing.
2012-03-06 14:32:45 +00:00
Gabriel Wicke 2efcd3cd57 Reworked percent encoding handling for URIs to get closer to the 'url
construction' part of the HTML5 spec:
http://www.whatwg.org/specs/web-apps/current-work/multipage/urls.html#url-manipulation-and-creation

Removed a few whitelisted test cases that are now passing directly.

The encoding canonicalization could also be moved to the Sanitizer. Doing this
early in token stream processing however has the advantage of providing further
transformations uniform data to work with. We could even consider to move this
even further into the tokenizer.
2012-03-06 13:49:37 +00:00
Gabriel Wicke 19fe9726a2 Fix invalid external link representation. 268 tests passing. 2012-03-05 18:06:29 +00:00
Gabriel Wicke 7b0c807710 Change wikilink tokenization strategy to split on pipes. This makes it
possible to support template / template argument expansion in image options,
and causes little trouble for wikilinks. Non-image wikilinks with multiple
text pipes are quite rare in the dumps, and concatenating description tokens
with a plain '|' is quite easy. 261 parser tests passing.
2012-03-05 12:00:38 +00:00
Trevor Parscal 0e41da3340 Fixed tests that were broken by r112150. 2012-03-02 23:12:38 +00:00
Gabriel Wicke 009d7a4dea Namespaces to the rescue. 2012-03-02 15:49:05 +00:00
Gabriel Wicke fe681042c0 Collect some statistics while grepping. 2012-03-01 16:42:28 +00:00
Gabriel Wicke e0838db315 Capturing the regexp is no longer necessary, and speeds up the grepper. Also
tweaked the multi-line ISBN regexp slightly.
2012-02-29 13:02:46 +00:00
Gabriel Wicke e3deb304db Add a misc regexp file for dump grepping. 2012-02-29 11:07:17 +00:00
Gabriel Wicke 14f40aa7d5 Support capturing regexps in dumpGrepper. 2012-02-29 10:49:00 +00:00
Gabriel Wicke ebcfc2c7a1 Improve grepper documentation. 2012-02-28 14:24:37 +00:00
Gabriel Wicke b767e03449 Tweak martian regexp and grepper output format. 2012-02-28 14:11:44 +00:00
Gabriel Wicke 4806505ce4 Finish color highlighting for dump grepper / fix broken commit r112592. 2012-02-28 13:48:47 +00:00
Gabriel Wicke 7daeb34d4d Implement onlyinclude transformer. 254 tests passing. 2012-02-28 13:21:01 +00:00
Gabriel Wicke 32012c00cd Add martian-endtags regexp wrapper around dumpGrepper. 2012-02-27 16:51:20 +00:00
Gabriel Wicke 19c67c28a2 Add a simple dump grepper using DumpReader. Useful to inform parser design
decisions, and as a way to exercise the dump reader in preparation for tests
over full dumps.
2012-02-27 16:40:01 +00:00
Gabriel Wicke 21855c99cd Tweak dumpReader to work with current libxmljs and stdin 'data' events. 2012-02-27 15:46:08 +00:00
Gabriel Wicke 2e41b19af8 Green two more parser tests by implementing some parser functions. 2012-02-22 16:39:50 +00:00
Gabriel Wicke 3568dfee14 Add some support for functionhooks in test parser and parserTests.js, and
tweak a few parser functions.
2012-02-22 15:59:11 +00:00
au f1fb937b4a * Instead of sorting attributes, whitelist the one parserTest where it matters. 2012-02-20 22:26:24 +00:00
au 0ca9b00100 * Convert __patched-html-parser to .coffee.
Note that the compiled .js file (generated by "make"/"make test")
  is still under version control so folks can work on the project
  even without a running "coffee" command in PATH.

  Also updated README to mention coffee-script and "make test".
2012-02-18 18:54:12 +00:00