Commit graph

499 commits

Author SHA1 Message Date
Catrope e92db86046 Fix bug in ve.dm.Document constructor for input ending with text
If the last element of the input data array was text, the resulting text
node would have length=0 rather than the expected length value.

Change-Id: I3d089a80b8a447a12ba411b2e11c1b84f14f2959
2012-05-31 06:17:23 -07:00
Gabriel Wicke e53bc93a8e Check out old results before running tests
Change-Id: Ia56aec22194a14c94620237041c30c269ac2e56a
2012-05-30 17:37:10 +02:00
Gabriel Wicke 36084c5d93 Preserve original newlines in HTML and serialization
254 round-trip tests (up from 184) are now passing.

Also:
* tweaked runtests.sh slightly (use less -R instead of -r).
* made sure the EOFTk is preserved in phase 3 transforms

Change-Id: I1de22186bdb78e52019370e43f096877005b8f5a
2012-05-29 23:29:03 +02:00
Subramanya Sastry 8174c9dafc First attempt implementing rewriting rules on the DOM
- This is implemented as a post-processing pass.
- Might require additional checks to verify rewriteability.
- Implemented as a pair-wise tag DOM minimization strategy,
  i.e. it takes tag pairs (B, I) for ex, and attempts to
  normalize the tree just for those tag pairs.  Normalizing
  across multiple tags is implemented as pairwise rewriting
  across all pairs:  Ex:(b,i), (b,u),(i,u) for (b,i,u)
- Copied over attributes as part of rewriting, but some of the
  attributes lose their meaning on rewriting since tags are
  reordered (ex: sourcePosn, sourceTagPosn). How do we handle this?

Output examples and possible issues to fix:
   <i><b><u>biu</u></b></i><b><u>bu</u></b><u>u</u>
gets rewritten to:
   <u><b><i>biu</i>bu</b>u</u>

But, the equivalent wikitext form:
   '''''<u>biu</u>''''''''<u>bu</u>'''<u>u</u>
does not get rewritten because of parsing differences.
This wikitext gets parsed into:
   <i><b><u>biu</u>'''</b></i><u>bu<b>u</b></u>
The extra ''' token in the middle thwarts DOM rewriting.

However, a slightly different version:
   "'''''<u>biu</u>''<u>bu</u>'''<u>u</u>"
gets properly normalized to:
   <u>'''''biu''bu'''u</u>

An alternative, but fun strategy to play with is to use the following
two normalization primitives: S(wap) and M(erge).
- S rewrites T1(T2(x)) into T2(T1(x))
  (ex: <b><i>foo</i></b> ==> <i><b>foo</b></i>)
- M rewrites (T(x),T(y)) into (T(x,y)).
  (ex: <b>foo</b><b>bar</b> ==> <b>foobar</b>)

The current rewriting strategy could possibly be re-implemented as S-M
rewriting.  The problem to solve there would be to find an efficient
rewriting strategy that is guaranteed to lead to a normal form.  I may
not play with it now, but just documenting it for later (to play with
in my spare time).

This commit is just as a record of fun/experimental code where I get to
learn details of JS, wikitext, parsing, and DOM manipulation.  Next
version of this code will attempt to introduce minimal DOM restructuring
across multiple tags at once which can be more efficient.

gwicke: Removed now passing test from whitelist, and updated another whitelist
entry which is now improved.

Change-Id: Ie97bcb164eb62c34ba61aa76ba2f4c232aa713d8
2012-05-29 08:17:57 +02:00
Trevor Parscal 14862a1212 Fixed bug in isStructuralOffset which didn't consider neighboring a non-content leaf to be a structural offset
This was caused by the fact that a non-structural leaf can not have children, which makes it appear incompatible as a sibling to an arbitrary structural element (like a paragraph) but since it can not contain content we can check that instead.

Change-Id: Ie3c58b4b43f2aa6921f8f82aa82511e231207854
2012-05-25 13:19:56 -07:00
Gabriel Wicke c52c24b0cb Slightly improve formatting of web service; test commit message tweak
Change-Id: Ibac3ce3dd9aa2c4faf11eed351fea941ebf1e4b3
2012-05-25 16:36:14 +02:00
Gabriel Wicke c692bc2307 Use '/bin/sh' instead of '/bin/bash'
Bash omits the time output for some reason. We would like to keep a record of
performance too.

Change-Id: I7c435b1cf2e2f237f78a45b2819a195a240e3aa4
2012-05-25 16:19:46 +02:00
Gabriel Wicke 1ce2bc605d Add a small test runner with result archiving in git repo
Supports both -> HTML DOM and round-trip testing. Displays the diff to the
last results using less -r.

Change-Id: Ib3fbadeda3c8f4f7e3d2e6e5236a73ff7a773623
2012-05-25 14:28:53 +02:00
Inez Korczynski afe18a685a Fix methods getNearestStructuralOffset and getNearestContentOffset for case of passing offsets that are already structural or content
Change-Id: I23cd2daab9cef181b01131b3b9c6ead5ec47f9f4
2012-05-24 20:53:50 -07:00
Trevor Parscal c9dddf60da Made getRelativeOffset return -1 if no valid offset is found in document
Change-Id: If4cef5fe800b241c3efe1d27cdf01cd1ceef2248
2012-05-24 11:43:47 -07:00
Trevor Parscal 8d3044bf60 Added more test cases for the unrestricted option of getRelativeStructuralOffset and getNearestStructuralOffset
Change-Id: I1ab99f1ed050950b40dd4c34abd81a43a4099a49
2012-05-24 11:43:47 -07:00
Trevor Parscal 1ddaec8acb Added getRelativeOffset and several wrappers around it
Change-Id: Ifd10922366650165bc4c21ea8cfdd2f9b7163432
2012-05-23 17:43:24 -07:00
Trevor Parscal 541d786ced Added optional "unrestricted" argument to isStructuralOffset
Using this argument will only return true if the offset is a place you can add any element to (hence the unrestricted part of it). This is good for testing if a paragraph could potentially be inserted there.

Change-Id: I6cc91da437c52493de03eb687b28966198270fea
2012-05-23 14:56:33 -07:00
Catrope 9a1ec62f0d Remove pushInsert() and pushRemove()
The 'insert' and 'remove' operations weren't implemented in the
transaction processor and were a holdover from the old DM
implementation.

Also migrated the tests, especially those that asserted that consecutive
insert/remove operations were combined (this is no longer the case now
that they are replace operations)

Change-Id: I2379fe92b331c5316f70f4b695397da41581cce9
2012-05-23 12:45:27 -07:00
Gabriel Wicke b89f5071e5 Basic parser / serializer web service
* After installing Parsoid (sudo npm install -g in modules/parser), run 'node
  server.js' from the api directory and navigate to http://localhost:8000/ and
  follow the directions. You can start to navigate the English wikipedia at
  http://localhost:8000/Main_Page, or manually enter wikitext or HTML DOM to
  convert.
* Uses the express framework, could also use just connect
* Uses the cluster module to manage workers per-core and restart those on
  failure

Change-Id: I443f2996ed3df00826b038b7476a2f966ab0c425
2012-05-23 12:35:00 +02:00
Gabriel Wicke 39c6f42879 Link round-tripping and other improvements
* Changed RDFa for links according to
  http://www.mediawiki.org/wiki/Parsoid/RDFa_vocabulary
* Added basic support for internal/external link serialization
* Moved numbering of external links from tokenizer to LinkHandler
* Added round-tripping for generic HTML tags
* Replaced nowiki tag with <meta typeOf="mw:tag" content="nowiki"> and <meta
  typeOf="mw:tag" content="/nowiki"> for now.
* 154 round-trip tests passing (node parserTests.js --roundtrip).

Change-Id: I16c4db21b1b543ee57c73e569c83025b64664542
2012-05-22 13:36:06 +02:00
Trevor Parscal 86591e5325 Restructured static node rules
Change-Id: I7023e1f334694d59767491f1010e7c4c59c0ce54
2012-05-21 17:39:12 -07:00
Trevor Parscal 0a32898f3a Added annotation and attribute tests for transaction constructors
Change-Id: I61e5f37b1f866afc4a3b9a18cd6e7e7a08e0958f
2012-05-21 12:21:07 -07:00
Rob Moen 272f502693 Modify getAnnotationsFromOffset to account for annotated leaf nodes.
Refactor remaining annotation methods
Refactor tests to use correct annotation types

Change-Id: Ia3ce42b3d11296b4d63277e0e2a1997d23236613
2012-05-21 12:02:04 -07:00
Rob Moen 3d5da75782 Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/VisualEditor into dmrewrite 2012-05-21 11:01:16 -07:00
Gabriel Wicke fb7d5418a5 Round-trip nowiki
Change-Id: I5f7e6a43f5fdc1708ee710b2a601b20db733452c
2012-05-21 18:06:09 +02:00
Rob Moen f19c5917f2 remove hasChildren method as I was using it to debug something on
previous commit.  change traverseLeafNode test name to follow
ve2 test naming scheme

Change-Id: I8b953e3405f42cd58a80da676ac9b16d64f5e72d
2012-05-18 14:11:23 -07:00
Rob Moen 62959bec7a ported traverseLeafNodes and tests to new dm
Change-Id: I98487267d6e3efc9e0c07a02b3d5acc8f8cdb4c6
2012-05-18 13:47:14 -07:00
Subramanya Sastry 9b84e931db First pass updating parserTests to verify dom->wikitext serialization.
- Just a quick first pass updating the parserTests.js script so we can
  test DOM -> wikitext serialization (but which in effect really tests
  roundtripping).
- There is no output normalization yet which is needed for now since we
  are not yet preserving white-space.

Change-Id: Ie52058e0dc3330f852c24fa05641dced19f950e0
2012-05-18 09:51:22 +02:00
Trevor Parscal 43d62ee8a7 Refactored getMatchingAnnotations and it's tests, and added offsetContainsMatchingAnnotations
Change-Id: I9098ef8d9c7f3a4f5db112cfdcb9edb7ffd17b80
2012-05-17 14:37:29 -07:00
Rob Moen a937ad22f7 Fix getAnnotatedRange to return propper contained range.
Fixed tests

Change-Id: I305eb81644fb3c804455b4fc13bdabcce97ea471
2012-05-17 13:17:18 -07:00
Trevor Parscal 3eb31dae09 Whitespace fixes and removed some console.log statements
Change-Id: Ifcf8a496fb4b649cbf3ce07290b8f6dda60f179d
2012-05-17 11:29:50 -07:00
Trevor Parscal 08765feab3 Merged ve.dm.DocumentFragment and ve.dm.Document
* Also fixed lots of whitespace issues and some global variable introductions

Change-Id: Ia6c82ef0c3b5c4938cf72afa86da6d79e4cb385a
2012-05-17 11:23:56 -07:00
Rob Moen c2a89626d5 Rewrite getMatchingAnnotations to return a hashmap of matching anntations
in the new DM.  Change method name getAnnotationRange from offset to
getAnnotatedRangeFromOffset.  Write tests

Change-Id: I7028803065409e271ceced73e4803954d4a956dc
2012-05-17 10:46:27 -07:00
Gabriel Wicke 04fc74c76a Strip RDFa attributes in parserTests
We are adding some extra information in those, which should not make tests
fail.

Change-Id: I42cca596330252efeff5d51508f97ef1c566475b
2012-05-17 17:03:44 +02:00
Catrope bcd7431818 Remove getScope(), no longer used
Change-Id: I79e067a8e6b3eb151d3ebe1dfc3aaebe242aed39
2012-05-16 20:16:47 -07:00
Rob Moen fbaea888b9 Rewrite data model methods needed for ui tools
getAnnotationRangeFromOffset and offsetContainsAnnotation
which deprecated getAnnotationBoundaries, and getIndexOfAnnotation
write unit tests for proof

Change-Id: I6c0d4e3ca96dd569b1909cd22fce68c3a6fe382c
2012-05-16 15:55:01 -07:00
Christian Williams eefbee2262 Merge branch 'dmrewrite' of ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/VisualEditor into dmrewrite 2012-05-16 13:41:56 -07:00
Christian Williams 189e3eddbe Another test for HTMLConverter
Change-Id: I9b0fa2de44381b363fb6e612884f9911996e941b
2012-05-16 13:41:26 -07:00
Trevor Parscal 0c69370117 Added merge test for ve.dm.TransactionProcessor
Also added 2 paragraphs at the end of the example document and updated related tests accordingly

Change-Id: Idd294cbe3975d0a04069e55682a91417160e538a
2012-05-16 13:33:42 -07:00
Gabriel Wicke 542921b5a3 Removed html5 parser patch no longer needed with 0.3.8
Change-Id: Id8c23d34e8cca49a360f536e792144a85a8468a3
2012-05-16 12:06:42 +02:00
Trevor Parscal 6d0a24bec6 Greatly simplified selection and node tree comparison tests
By first summarizing a node tree or node selection and then using the normal deepEqual function we are able to take advantage of native QUnit functionality to make diagnosing a problem easier under failing conditions and reduce the verbosity of the output under passing conditions.

Change-Id: I4af5d69596cf5459aa32f61ee6d5b8355233c3df
2012-05-15 00:25:27 -07:00
Catrope cdda5e7920 Improve verbosity of selectNodes test descriptions
* Make nodeSelectionEqual() accept a desc parameter and use it for
  building descriptions
* Put the array element number in the desciption too
* Add descriptions for selectNodes tests and pass them through

Change-Id: Icd2894d11516234598cbd984cc8d88f705bfc1d6
2012-05-14 22:54:53 -07:00
Catrope 0f0c3c9939 Improvements to nodeTreeEqual():
* Don't crash if b.children has a different length than expected
* Don't crash if b.children is absent but expected
* Check if b.children is present when it's not expected
* Add assertion when neither a.children not b.children are present; this
  makes the number of assertions for leaves equal to the number of
  assertions for branches
* Check attributes too
* Accept a desc parameter, and add node paths (like
  list/listItem/paragraph/text) to the descriptions
* Use desc parameter for all calls (some tests were already using it,
  even though it didn't actually exist)

Change-Id: Ic56d27d20377e7f4fdfa038fdf4ebe00dcb3e062
2012-05-14 22:43:36 -07:00
Trevor Parscal 3b1d8098e9 Added a replace test that splits elements
Change-Id: Ifb9c42c36611b2b936a48b911e9786e2105da09c
2012-05-14 21:50:33 -07:00
Catrope 2008858830 Fix bug in selectNodes: input (0,0) returned []
This was because the while loop was never entered as end >= left was
true from the start. Convert the while loop to a do-while loop to make
sure it runs at least once

Change-Id: I9c6436a7b296e65a36b8301095b6edd00507d321
2012-05-14 21:41:08 -07:00
Rob Moen 30a671fcea Rename some tests
Change-Id: Ie0c50023a17fead64eb28e6184f70a32b626eaff
2012-05-14 17:24:11 -07:00
Rob Moen 6e8bdde910 Add tests for getAnnotationsFromRange
Change-Id: I51aebb54d3d65c3160ac48e9ee971183fb3885bb
2012-05-14 17:22:16 -07:00
Trevor Parscal 14a4c7b780 Fixed annotation processing to support leaf nodes and detect more errors
Also added lots of tests

Change-Id: I45fe4fb569d9050d7d79f71f777d82f5b34fdd11
2012-05-14 17:16:30 -07:00
Trevor Parscal 94613589e0 Removed more trailing whitespace in multi-line comment blocks
Continues cleanup work done in e0de881a8a

Change-Id: I7165a8f8c4b468793d936fe695de92335c106271
2012-05-14 17:16:30 -07:00
Rob Moen ca48a4a8a1 Merge branch 'dmrewrite' of ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/VisualEditor into dmrewrite 2012-05-14 16:44:08 -07:00
Rob Moen d2fc95fa52 Optimize getAnnotationsFrom dramatically by less object manipulation
Now returning an empty array when a non annotated character is found
in the range.  No longer looping through each annotation, simply
comparing to previous characters annotations and trimming differences.
Write additional test.

Change-Id: I41d2422a931a74325693edca409aed6d5da20ba8
2012-05-14 16:40:49 -07:00
Trevor Parscal e0de881a8a Removed trailing whitespace in multi-line comment blocks
JSHint and various git tools complain about this, so let's just do away with them altogether

Change-Id: I731866bd21f1ee0088fb0a71df3abf92692aca23
2012-05-14 15:05:09 -07:00
Trevor Parscal 5708c0e02f Added transparent 1px image to make unit tests not show HTTP errors
Change-Id: I800e98ae6c47121ec652c34bd678e54e7efcb77c
2012-05-14 15:04:06 -07:00
Trevor Parscal aa9148c9ee Refactored ve.dm.TransactionProcessor tests to use data-providers
Change-Id: I0535f3ac56b6037712d731c71fc68c56ae7ea614
2012-05-14 14:32:34 -07:00
Rob Moen b8302c715f Rewrite getAnnotationsFromOffset to work with new Linear Model.
Properly init variables in getAnnotationsFromRange
Polish up tests.

Change-Id: Idc231998ce27e3a1c64363e862da22c837ea2ca2
2012-05-14 10:47:48 -07:00
Rob Moen e05863b0dc use proper assertion in getAnnotationsFromOffset
Change-Id: I41b5cfa3628d5a2ed6d3f59d157ebe372585059c
2012-05-14 09:58:30 -07:00
Trevor Parscal 6321821c86 Rewrote tests for ve.dm.Transaction to use a data provider
Change-Id: I7ecf09c627b96b4c84824152bd178fb36c1ce348
2012-05-11 16:49:31 -07:00
Rob Moen 7d8e06c78c Fix up wrapping on getAnnotationsFromRange test
Change-Id: I793cd33b88f7c7f3a17d6fbfb80218438aa878af
2012-05-11 16:33:38 -07:00
Rob Moen bd9a0b6fba Rewrite DocumentFragment getAnnotationsFromRange method
Write unit tests for getAnnotationsFromRange

Change-Id: I82ff15e97378c07e5c555a72231da4161a72993c
2012-05-11 16:14:46 -07:00
Catrope 5b9a2645a6 Port getScope() from the old VE code and add tests for it
Change-Id: I8c34fed919e60fa1b8cb6a13cb9ca98f3e695421
2012-05-11 16:03:46 -07:00
Christian Williams 13f26682da Removing data from annotations when empty, more tests for HTMLConverter
Change-Id: I3c3e234e9392e4afdcb4c8bc748aaa74a1fb30ca
2012-05-11 14:07:20 -07:00
Trevor Parscal 9cdfb84f9b Added tests to getRelativeContentOffset
Also fixed a typo in the implementation

Change-Id: I86855795009e34d42a3a5857addc5ccb6366130f
2012-05-11 12:14:29 -07:00
Trevor Parscal ae7b0c822b Whitespace changes
Change-Id: I9ae27bee1207282d0f07d9a05584628cb8061683
2012-05-11 11:37:45 -07:00
Rob Moen 11a6ff67a7 Remove broken test, need to refactor
Change-Id: I7963454032eceec73cf5a5d71f4d3a173627bfc3
2012-05-11 10:33:52 -07:00
Rob Moen 469a55bde4 Merge branch 'dmrewrite' of ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/VisualEditor into dmrewrite 2012-05-11 10:31:12 -07:00
Rob Moen ced54d06a9 Write better test for DocumentFragment getAnnotationsFromOffset
Fix errors in example test data
Remove whitespace

Change-Id: Ibb2b5c0851d6abac9974035753b95354b3c8dd5e
2012-05-11 10:29:09 -07:00
Catrope 6652f7573b Add indexInNode to selectNodes() output
This is for the case where we have a zero-length range in between two
siblings, and we need to know what index that corresponds to in order to
be able to insert nodes there (rebuildNodes() will use it for this
purpose)

Change-Id: I357d1cd665667a76f955a10b8d9d2810976cdbd7
2012-05-10 20:20:42 -07:00
Catrope 7eeb6c7cac Throw an exception when DocumentFragment gets unbalanced input
Change-Id: Ie891bd7ea4d9e9b1c84e7a0390f1af39c0e55fd4
2012-05-10 20:20:42 -07:00
Trevor Parscal 1475100a22 Ported isContentOffset, isStructuralOffset, isElementData, containsElementData and getRelativeContentOffset
Tests for all but getRelativeContentOffset are done too

Change-Id: Iee953647ec89e13259ae2a402c50c809f93bc655
2012-05-10 17:20:57 -07:00
Christian Williams 081ec31ccd Merge branch 'dmrewrite' of ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/VisualEditor into dmrewrite 2012-05-10 16:58:47 -07:00
Christian Williams d1dbb32238 Fixing updateDomWrapper test
Change-Id: I93f5ceb96a9212652aa93eeebaf91ad1c5704ae5
2012-05-10 16:58:29 -07:00
Catrope 8834145f9e Fix nodeSelectionEqual() so it doesn't explode when a and b have unequal lengths
Change-Id: Idb84bc1238576c711a201cdbf05f63a7b603a819
2012-05-10 16:42:58 -07:00
Catrope 9b6402801a Fix the selectNodes() bug that Inez reported
Selecting a zero-length range at the start or end of a text node
(e.g. (1,1)) would return the text node's parent instead of the text
node

Change-Id: I7fe089bf66b93185dd3415eff53aa7e04e3ffdb2
2012-05-10 16:42:58 -07:00
Christian Williams 6e7c02a126 Merge branch 'dmrewrite' of ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/VisualEditor into dmrewrite 2012-05-10 16:29:37 -07:00
Christian Williams 07b4534a4c Basic test for HTMLConverter
Change-Id: Ie3fa3e036b92052305288c64edca8f404afded72
2012-05-10 16:28:45 -07:00
Catrope 13b79d309c Add 'index' and 'nodeRange' to selectNodes() output
Change-Id: I38712545b713cc24663ebec3fe3e5c8741b4fad7
2012-05-10 15:03:23 -07:00
Catrope e137659722 Fix spelling of 'equivalent'
Change-Id: I0a444059c187ab5608aeca85a85c147ad1d69de6
2012-05-10 15:02:11 -07:00
Trevor Parscal 7168b4e041 Removed duplicate node rules from tests
These methods are wrappers for the model methods where the rules are actually being applied

Change-Id: I08be359010fa9fa81cd3a8aa5152083f92dc0b9c
2012-05-10 14:57:52 -07:00
Trevor Parscal 8d07a4e08c Fixed test so it doesn't break when nodes add classes in their constructors
Change-Id: Ib2f7bcedcf61a9b98da0e3e889f309f271483064
2012-05-10 14:55:13 -07:00
Trevor Parscal 1798a478e7 Massive refactor of node trees, generalizing lots of previously dm only stuff
Change-Id: I9dbb7094f86338ea2e725ad29f66e9e4ca2b52ed
2012-05-09 21:11:09 -07:00
Trevor Parscal b4478216be Added tests for document synchronizer
Change-Id: If114e3fc962598f7ea8f7f8ec8975a3b0d2d83e6
2012-05-09 18:31:09 -07:00
Trevor Parscal 2747ff05c5 Added basic test for ve.dm.DocumentSychronizer
Change-Id: Ieea96aaf0477b4d9ba7e59f5cf6f20f4b867a419
2012-05-09 17:13:55 -07:00
Rob Moen d417c571a4 * Ported getAnnotationsFromOffset into DocumentFragment
* Wrote unit test for getAnnotationsFromOffset

Change-Id: Ie88b15ac6c16838b2d2bf9a2d83ca2952f5df817
2012-05-09 16:31:56 -07:00
Trevor Parscal 89a2b1bc40 Fixed naming in test moved from ve.dm.DocumentFragment to ve.dm.Document
Change-Id: I6843e452b0222632d1d244568bd75f3b229b4954
2012-05-09 15:30:18 -07:00
Trevor Parscal c017d7421c Removed use of expected in conjunction with ve.dm.example.*Equal methods
Change-Id: I48108f3fac7dda32344e105b7c52483cd2a84ef9
2012-05-09 15:28:06 -07:00
Catrope 22c88f77c3 Add a 'siblings' mode to selectNodes(), and add tests for it
Change-Id: Ib3fd3ee5aba7aab7c26193a2a9c48612f624862a
2012-05-08 18:31:50 -07:00
Adam Wight 0a7f0b7630 List markup is created during the sync23 phase.
This makes it possible to transclude list items from a template.

Note: "5 quotes" test is broken by this patch, it appears that ListHandler
newline processing is changing some state which mysteriously affects the
QuoteTransformer.  This is ominous, hopefully there's a simple explanation...

gwicke: fix a bug in tokenizer triggered by definition lists like this:
**; foo : bar

Change-Id: I4e3a86596fe9bffcbfc4bf22895362c3bf742bad
2012-05-08 11:39:36 +02:00
Trevor Parscal 123c64cbd5 Added more tests for selectNodes
Change-Id: Ibd9672fb9b6ce682905d20f9346fada6fb47b380
2012-05-07 19:03:59 -07:00
Catrope 4d9299b808 Fix bugs in selectNodes(), tests pass now
Also fix one instance of bad expected data in a selectNodes test

Change-Id: Ie226b237cf537143bce2f6b1e531b9067b140cb4
2012-05-07 17:03:42 -07:00
Catrope 32c65cdaa8 Change 'leaf' to 'leaves' and document it
Change-Id: I09d5995d3d9bf82670802f0efe849f40be2a18a3
2012-05-07 17:03:42 -07:00
Catrope 770775eec0 Fix rebuildNodes tests, were messing up ve.dm.example
Fixed by operating on a copy of the data instead

Change-Id: Ia2607d94f07b8d43e8c98ae8a94c34df90081508
2012-05-07 17:03:42 -07:00
Trevor Parscal 4bae026190 Added additional test to rebuildNodes, now testing rebuilding after a change
Change-Id: I669457389bf742c100dec5205b972f6d1e14ef44
2012-05-07 16:25:39 -07:00
Trevor Parscal 811110d7c6 Simplified selectNode tests
Change-Id: I878fd430b24661fc11c953b3303e395cdd617006
2012-05-07 16:18:24 -07:00
Trevor Parscal 88a05b5606 Added another falling selectNodes test
Change-Id: I1f18b0af956c98a5facaf12a88fd94ca76e50a65
2012-05-07 16:05:40 -07:00
Trevor Parscal e1f8fef228 Changed selectNodes test to desired behavior (which fails atm)
Change-Id: I79eb5caaa3e2193e2309d37dc1d0ca67c2602836
2012-05-07 15:50:11 -07:00
Trevor Parscal f4fcf9663e Added documentation and tests for ve.dm.Document.selectNodes
Also:
* Broke node lookup method out into ve.dm.example.lookupNode
* Added ve.dm.example.nodeSelectionEqual

Change-Id: I437cdd3f3154d10782f03f78df6d5c457ecfc845
2012-05-07 15:15:21 -07:00
Trevor Parscal cbb868577e Whitespace and syntax fixes
Change-Id: I5a553272c1faf1e83a5e8be8ecf7911a11ec27a8
2012-05-07 13:40:09 -07:00
Catrope 4427122f90 Merge branch 'dmrewrite' of ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/VisualEditor into dmrewrite 2012-05-07 12:00:59 -07:00
Catrope 93d6adb5f2 Add an isWrapped property and make getOuterLength use it
This gets rid of the length == outerLength-2 hack in getDataFromNode()
and will make it easier to implement similar logic in selectNodes()

Change-Id: I1294350b67ca3eefde2b7fe9fea0bc6d8b90f772
2012-05-07 12:00:07 -07:00
Trevor Parscal 39b3755310 Line-length fixes
Change-Id: I5258ad2e550313a95ae5b7bba951992db2db966c
2012-05-07 11:45:31 -07:00
Trevor Parscal e34ec4b479 Added canHaveChildren and canHaveGrandchildren to all nodes
Change-Id: Ibe53292f7269736750e03da49e5bb5913c0124ab
2012-05-04 17:50:54 -07:00
Trevor Parscal 1d74945dfc Fixes for Roan's crazy half-baked commit
Change-Id: Ic104debc3a941e4cf8e4dca19633dcad04a0fe0a
2012-05-04 16:13:54 -07:00
Trevor Parscal 0433717b29 Added attribute add and remove tests and cleaned some things up
Change-Id: If339e0b59a8cdd90a151128138cec85b902681a8
2012-05-04 15:50:50 -07:00
Trevor Parscal bce509195b Removed unused attributes in example data
Change-Id: I0ce52456113e6efc5d879041270e06fd0b6bdebe
2012-05-04 15:50:32 -07:00
Rob Moen d42e738636 - Optimize surface model and write unit tests
Change-Id: I8de24c840c5aa2d4232e6e2cc5f72cbd30be96b4
2012-05-04 15:29:47 -07:00
Trevor Parscal dc27d8acb4 Refactored dom wrapper functionality in ce nodes
* Moved implementation of getting and updating a DOM wrapper to ve.ce.BranchNode
* Updated ve.ce.BranchNode tests
* Renamed ve.NodeFactory.createNode to ve.NodeFactory.create
* Added  ve.NodeFactory.lookup which gets the constructor of a type
* Added attribute pass-through to ve.dm.BranchNodeStub

Change-Id: I8f5b7d3d3ae616cc5f39828b24b655163d782ae5
2012-05-04 13:31:14 -07:00
Trevor Parscal 9887dbd96f Split Broke DefinitionListNode out of ListNode and DefinitionListItemNode out of ListItemNode
* Makes it simpler in the linear model because we don't have to use style: "item" for regular list items and style: "definition" for definition lists
* Enforces correct nesting through existing node rules systems
* Updates tests accordingly

Change-Id: I64d80af938e325f1961226505bdc386bb35ccdda
2012-05-04 11:56:32 -07:00
Catrope 11a3b6886b Implement basic replace processor
* Implement basic TransactionProcessor.replace(), only does content
  replacements
* Add ve.dm.Document.containsElementData()
* Fix bug in attribute()
* Write basic tests for attribute() and replace()

Change-Id: Ie9c22aec3f2631be5b0bd66790408ad283565491
2012-05-04 11:09:10 -07:00
Trevor Parscal a9b26c8b55 Removed recursive rendering calls in favor of 100% event driven updates
Change-Id: I636222518e4cbeddb1c02901b79a9dcb09eeb952
2012-05-03 17:19:01 -07:00
Trevor Parscal 458ba0b7d5 Added tests for getModel, getParent, attach and detach on ve.ce.Node
Change-Id: I064b2a13cbfe7a7ebc11af83fb7744ebbad39da1
2012-05-03 17:06:23 -07:00
Trevor Parscal f738e75caa Fixed mistakes in the linear and tree example models
Change-Id: Id19ac7b2efb5c5f3147a4abf5e62fab3044b5fd7
2012-05-03 14:54:27 -07:00
Rob Moen 8518e1cc1d Merge branch 'dmrewrite' of ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/VisualEditor into dmrewrite 2012-05-03 13:12:36 -07:00
Trevor Parscal b1b9fb5133 Added tests for getDataFromNode
Change-Id: Ic762832dcdda3bdaf92dbd989659235fdff73596
2012-05-03 13:11:36 -07:00
Rob Moen 424e58ab1d - write tests for ve.dm.BranchNode setDocument
Change-Id: I7aa2dc0ff5a9b7b7f190da29ac62a5e9c7817dbc
2012-05-03 13:08:37 -07:00
Trevor Parscal 71f36aa52b Added tests for getOffsetFromNode
Change-Id: I4dc06e9ea3b634fbd5034df721162098441ea368
2012-05-03 11:52:17 -07:00
Inez Korczynski a3b32b6615 Big refactoring to TextNode to make it work with new format of storing
annotations.

Change-Id: If84fcfdd04ff42e9b2449e1c8186d64fec4fb1ac
2012-05-02 22:28:57 -07:00
Inez Korczynski 9f11f437d2 Merge branch 'dmrewrite' of ssh://review/mediawiki/extensions/VisualEditor into dmrewrite 2012-05-02 19:29:49 -07:00
Inez Korczynski d81b972cdf Very dirty but working version of getHtml method for TextNode (with bunch
of tests that are passing now).

Change-Id: Ie76bd59bc0a5e10e2a3b3e014b2d0e4701fcc368
2012-05-02 19:29:03 -07:00
Trevor Parscal 64d56971a9 Removing unused code
Change-Id: I23213531eeef95110d0a3d1e2476449c580b55b7
2012-05-02 17:05:22 -07:00
Trevor Parscal 6a60c1f60b Removed console.log
Change-Id: I7cd2e1cf5e1cf7a2e9864090cfab391faba10997
2012-05-02 16:46:36 -07:00
Trevor Parscal b7b2fc1179 Made render for ce leaf nodes abstract and added a test for it
Change-Id: Iee228d5f1a18a5c9618680fa81ab132a85472fa6
2012-05-02 15:59:43 -07:00
Trevor Parscal 16e12debdc Added ve.getHash and ported ve.Json
Change-Id: I4a5d86426902e0473cdb0f98c3d121eed607f93c
2012-05-02 15:23:01 -07:00
Trevor Parscal 6792762285 Added insertion and removal tests to ve.ce.BranchNode.onSplice
Change-Id: Ic4fc51f3f8f869b03e9ae0fced88c74d2ddf4388
2012-05-02 14:59:07 -07:00
Trevor Parscal d97864a92e Added removal test for ve.ce.BranchNode.onSplice
Change-Id: I54b622fa05ce02965a2ae48fc78958b6408bca4d
2012-05-02 14:50:28 -07:00
Inez Korczynski dd6094842e Created a test file for ve.ce.TextNode and fixed some minor issues.
Change-Id: Ifbf6b7ba2a27eedf6d5d4f6cb06b98c38eb52c5b
2012-05-02 14:18:02 -07:00
Trevor Parscal fd28fff50f Added test for replaceDomWrapper and onSplice in ve.ce.BranchNode
* Also fixed calls to addListenerMethod
* Also routed adding children in the constructor of ve.dm.BranchNode to the splice method
* Renamed types of ve.dm stub nodes to avoid collisions (since we have to register ce nodes by the same names for them to be generated by onSplice)

Change-Id: Ia2e75cf0a62186cc0e214683feb25c619590318a
2012-05-02 13:58:50 -07:00
Trevor Parscal 8bd0d385eb Added test for whitespacePattern, getDomText and getDomHash in ce
* Also fixed some naming typos in the units under test

Change-Id: I17f753a122f94dd16f7783f60b5c0eb1c6d3e480
2012-05-02 12:54:42 -07:00
Trevor Parscal f1505a0e62 Merge remote-tracking branch 'origin/master' into dmrewrite
Change-Id: I43a3cd999ec0a5d3d05ee1e5bb71c450d485bc01
2012-05-02 10:17:44 -07:00
Trevor Parscal cac3c6efd0 Merge branch 'dmrewrite' of ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/VisualEditor into dmrewrite 2012-04-30 19:39:02 -07:00
Trevor Parscal 370070dfa2 Lots of ce porting, got text nodes fully supported
* Fixed constructor of ve.ce.BranchNode which was calling the wrong method to perform an onSplice and with the wrong arguments
* Removed/renamed events emitted from ve.ce.BranchNode.onSplice
* Reintroduced .$ to all ce nodes
* Ported over functionality for DOM node type variance used in headings, lists and list items
* Moved the old ve.ce.Content guts to ve.ce.TextNode
* Added getOffsetFromNode and getDataFromNode to ve.dm.DocumentFragment
* Added setDocument and getDocument to dm nodes

Change-Id: I185423ba2f1a858dde562cb2f5bc3852aec930db
2012-04-30 19:38:42 -07:00
Catrope d9cd1392fc Add TransactionProcessor tests, and fix TP to make them pass
* Add .process() and its helpers from ve1
* Fix applyAnnotations()
** Use data[i] rather than data[j]
** Don't add empty annotation objects for no reason
** Remove empty annotation objects
*** I thought this wasn't needed, but it is needed for clean rollbacks
* Remove unused second parameter in applyAnnotations() call

Change-Id: Ia338f62d2eaf2a76f8ef653eead05bc44757a122
2012-04-30 19:06:20 -07:00
Trevor Parscal 7a0e086879 Fixed test name to match method name
Change-Id: I06a4e63c5032b74253bcc3a1cd4e36f48d3f424a
2012-04-30 17:41:24 -07:00
Trevor Parscal 35af704173 Added initial ce nodes
Change-Id: Ibdcb4502a8fdc5b1a6c22ae0e9c42c86b3a7d4dc
2012-04-30 17:36:22 -07:00
Trevor Parscal de0debb866 Added skeleton tests for ce nodes
Change-Id: I1f5988b31fdef1bf5c5f9266d0bfa8da91d97c3a
2012-04-30 17:04:58 -07:00
Trevor Parscal c2d4a2d928 Added basic ve.ce nodes
* Also removed beforeSplice and afterSplice in favor of just plain splice which is the same as afterSplice used to be - beforeSplice was never used and it was making things more complex looking than needed

Change-Id: Icbbc57eac73a2a206ba35409ab57b3d1a49ab1a5
2012-04-30 16:58:41 -07:00
Trevor Parscal aaa322642b Added ve.ce.NodeFactory and tests
Also added tests for initialization of factories at ve.ce.factory and ve.dm.factory

Change-Id: Ic6ac74aab86ecdfd4f094d9bb1fa16de930387b7
2012-04-30 16:36:02 -07:00
Trevor Parscal a667425874 Moved canHaveChildren and canHaveGrandchildren back to ve.NodeFactory
This is generally useful information, not only for dm

Change-Id: I9511467285e9594b4a7aa659bf65bb0417da25a1
2012-04-30 16:25:44 -07:00
Trevor Parscal 2cee2adb6d Split node factory into dm specific implementation
* Added support for asking if a given node type can have children or grandchildren and what types of nodes can be it's parent or child
* Removed canHaveChildren methods from leaf and branch nodes and converted use of them to depend on factory to read static rules from constructor lookup by type

Change-Id: I9769f95647066576416bacb791c4b68dd0285b35
2012-04-30 15:42:36 -07:00
Trevor Parscal b1d0ed052d Added more canHaveChildren tests
These are needed to make sure the base classes behave correctly, the ones in sub classes are needed to make sure the prototypes of the base class are correctly inherited

Change-Id: I334cc3ce1c4c0ce2eed23c79e8877332a953f7c3
2012-04-30 14:43:02 -07:00
Trevor Parscal 93390847bc Added getOuterLength tests for text and document nodes
Neither of these nodes have elements around them, so they must override the default behavior of ve.dm.Node

Change-Id: I19c02c210bfc04b6e2ee1a37b8890e84a236eee3
2012-04-30 13:48:20 -07:00
Trevor Parscal 0d4fb6fdb5 Got rebuildNodes tests working
* Moved node tree assertion to ve.dm.example
* Added rebuildNodes test
* Fixed some typos in rebuildNodes

Change-Id: I4853ded4b062aaa3758435093368bc23667ca3bf
2012-04-30 13:42:32 -07:00
Trevor Parscal 8ce7fd0c85 Fixed length calculations and added tests for node tree
Image nodes are leafs, so providing an empty array to their children/length "contents" constructor argument ends up setting the numeric length to [], which casts to an empty string in arithmetic, causing all further calculations to be concatenations instead of additions.

Change-Id: I40e1ea2295f6095318bc4c24185cadfdfb684557
2012-04-30 13:23:46 -07:00
Trevor Parscal a1846c8f47 Fixed node tree example
* Indexes for attribute references were off (the comments that were previously wrong before were relied on and cause this to also be wrong)
* Tree should start with a document node

Change-Id: Ia8e4faa4bcb25db797ff97f6e798ba253adfe535
2012-04-30 12:37:08 -07:00
Trevor Parscal 4d03be0301 Added comments and tests for canHaveChildren
Change-Id: I0b9538a89cba4c36d1a8af7395476b9612d18637
2012-04-30 11:57:45 -07:00
Trevor Parscal 822e873775 Fixed numbering in comments in example data
Change-Id: I009f1f32294cce3f154ac3c88e1fa6b2a50a7d1e
2012-04-30 11:43:01 -07:00
Trevor Parscal 489794c89c Renamed rootNode to documentNode and added tests
Within ve.dm.DocumentFragment it makes more sense to call the root node (which is always a document node) a document node, especially since there may be a different node used as a root.

This commit also adds test for getDocumentNode and getNodeFromOffset which uses the offset map.

Change-Id: Ic4609233cedc41f7e5a5f8fdb0e6178652c95554
2012-04-30 11:37:48 -07:00
Trevor Parscal a774b3dbf6 Added test for getOffset map
And fixed ve.dm.DocumentFragment constructor to generate a correct offset map which creates references to branch nodes only

Change-Id: If9e515be0c63d272bfed9bf4da625a48edd36f48
2012-04-27 17:16:29 -07:00
Trevor Parscal fb9f6e0a3b Fixed test module name typo
Change-Id: Ie65c26ec27fc8d8a9d792d41b0dd7deffbd59171
2012-04-27 15:10:43 -07:00
Trevor Parscal 44fe109f14 Added test data and fixed test suite links
Change-Id: Idb5de70b58c525a67f16b21f7adc53214af9b486
2012-04-27 14:59:52 -07:00
Trevor Parscal d4a99f2d26 Removed "prototype." prefix from test names
Change-Id: I9d8993c95782b700d0b8b45b253a3c4c689381a5
2012-04-27 11:03:08 -07:00
Gabriel Wicke 3be4992782 'Obama finally expands' ;) Misc fixes and documentation updates
* [[:en:Barack Obama]] can now be expanded in 77 seconds using 330MB RAM,
  while it would prevously run out of RAM after ~30 minutes. Wohoooo!
  The token transform framework rework really paid off.
* 303 parser tests are passing in the new record time of 5.5 seconds. Two more
  tests are passing since these tests expect the day of the week to be
  Thursday.  Won't be the case tomorrow.

Change-Id: I56e850838476b546df10c6a239c8c9e29a1a3136
2012-04-26 18:18:08 +02:00
Gabriel Wicke 8ff810659a Rename text/wiki and tokens/wiki to text/x-mediawiki and similar
Change-Id: I70113629f4633685cd6db3914303a15e4c79a50a
2012-04-25 20:19:43 +02:00
Gabriel Wicke 8368e17d6a Biggish token transform system refactoring
* All parser pipelines including tokenizer and DOM stuff are now constructed
  from a 'recipe' data structure in a ParserPipelineFactory.

* All sub-pipelines of these can now be cached

* Event registrations to a pipeline are directly forwarded to the last
  pipeline member to save relatively expensive event forwarding.

* Some APIs for on-demand expansion / format conversion of parameters from
  parser functions are added:

  param.to('tokens/expanded', cb)
  param.to('text/wiki', cb) (this does not work yet)

  All parameters are additionally wrapped into a Param object that provides
  method for positional parameter naming (.named() or conversion to a dict
  (.dict()).

* The async token transform manager is now separated from a frame object, with
  the frame holding arguments, an on-demand expansion method and loop checks.

* Only keys of template parameters are now expanded. Parser functions or
  template arguments trigger an expansion on-demand. This (unsurprisingly)
  makes a big performance difference with typical switch-heavy template
  systems.

* Return values from async transforms are no longer used in favor of plain
  callbacks. This saves the complication of having to maintain two code paths.
  A trick in transformTokens still avoids the construction of unneeded
  TokenAccumulators.

* The results of template expansions are no longer buffered.

* 301 parser tests are passing

Known issues:

* Cosmetic cleanup remains to do
* Some parser functions do not support async expansions yet, and need to be
  modified.

Change-Id: I1a7690baffbe8141cadf67270904a1b2e1df879a
2012-04-25 16:51:36 +02:00
Catrope 69df3eefbc Implement ve.NodeFactory and add tests
Change-Id: I34fdf24c0099072fe5f7178400abbc323be975d4
2012-04-23 11:46:30 -07:00
Trevor Parscal 3d6391419d Added more nodes and removed canHave[Grandc|C]hildren methods
Replacing them with static members on each node type

Change-Id: I455debf880bef4e280eea072364f5f57308ec2b1
2012-04-20 16:34:47 -07:00
Catrope f9fd9ea66b Add tests for ve.dm.Transaction
These are currently broken because pushReplace() doesn't update the
lengthDifference

Change-Id: If0b611b7228c54ed15551514e773865be343e63a
2012-04-20 11:29:30 -07:00
Trevor Parscal b004b22241 Added tests for all exceptions
We are now checking for the exception messages as well.

Change-Id: I3a306ce9fe82afe6fd1e46a2e4da4d0a70952688
2012-04-19 18:05:48 -07:00
Trevor Parscal 9a3784301b Improved test coverage for ve.dm.BranchNode.splice and ve.dm.TwigNode.splice
* Changed splice to check all elements about to be inserted are allowed before inserting any of them so that catching an exception leaves you in a sane state
* Fixed the order of execution of parent class constructors in ve.dm.LeafNode and ve.dm.TwigNode so that canHaveChildren and canHaveGrandchildren produce correct values and added tests to ensure these methods are correctly inherited in subclasses
* Added tests that check for exceptions when adding nodes that can have children to nodes that can not have grandchildren
* Added test that check for events being emitted before and after splicing, including that beforeSplice should be emitted even in cases where a splice fails and throws an exception because the nodes are incompatible (but afterSplice is not called in this case) since beforeSplice might modify the nodes in some way before the compatibility tests are run

Change-Id: Id12aea995a42c26ff63a74ae3d31f2bf455759e3
2012-04-19 17:45:58 -07:00
Trevor Parscal f081c0932a A few fix ups for fd49e8d
* Moved getParent and getRoot from ve.dm.Node back to ve.Node
* Fixed use of getElementLength that should have been changed to getOuterLength, but was changed to getLength (oops)

Change-Id: Ibe5b855aef533dcd493f762a8a02c6a11ce6e7de
2012-04-19 16:47:40 -07:00
Trevor Parscal fd49e8df32 Added more tests for ve.*Node and ve.dm.*Node classes
In this commit several methods (child node add/remove and parent/root modification) were also moved to ve.dm.BranchNode ve.dm.Node respectively. ve.Node and ve.BranchNode are immutable. ve.dm.Node and ve.dm.BranchNode are mutable. Other subclasses of ve.Node and ve.BranchNode should implement functionality to mimic changes made to a data model.

Change-Id: Ia9ff78764f8f50f99fc8f9f9593657c0a0bf287e
2012-04-19 16:03:59 -07:00
Trevor Parscal c9ce7dbffe Added some basic coverage for ve.*Node classes
* prototype.canHaveChildren
* prototype.canHaveGrandchildren
* prototype.getType
* prototype.getParent
* prototype.getRoot
* prototype.setRoot
* prototype.attach
* prototype.detach

Change-Id: I920f7c9504e467f4818df537608760165c28d432
2012-04-19 14:43:58 -07:00
Trevor Parscal b16ed2b12d Setup tests, which are still empty
Had to fix a few namespace typos too

Change-Id: I3ebdc418f374bd3e151c516e1c0cfe85398772f0
2012-04-19 14:17:59 -07:00
Gabriel Wicke aaca5eac7d More tweaks: safesubst and image options
* Ignore safesubst for now
* Remove an unneeded whitelist entry
* Make sure the caption is not lost for thumbs (fix to last commit) and remove
  debug print

Change-Id: I243584ed0838cf7c3b4110fe9cdf869272477312
2012-04-17 11:02:52 +02:00
Gabriel Wicke afa5b95bc1 Don't work around html5 library tokenizer attribute reordering
The HTML5 parser we are using to normalize expected HTML output in parserTests
reverses the order of attributes (see
https://github.com/aredridel/html5/pull/53 for the fix). Remove whitelist
entries concerned with this and use the proper order in external image
attributes.

Change-Id: If1868cae05396a150757c85a20473ab756cbcd97
2012-04-16 17:09:06 +02:00
Catrope 7465b670e1 Add and update an offset map in DocumentNode
This has some TODOs still but I want to land it now anyway, and fix the
TODOs later.

* Add this.offsetMap which maps each linear model offset to a model tree node
* Refactor createNodesFromData()
** Rename it to buildSubtreeFromData()
** Have it build an offset map as well as a node subtree
** Have it set the root on the fake root node so that when the subtree
   is attached to the main tree later, we don't get a rippling root
   update all the way down
** Normalize the way the loop processes content, that way adding offsets
   for content is easier
* Add rebuildNodes() which uses buildSubtreeFromData() to rebuild stuff
* Use rebuildNodes() in DocumentSynchronizer
* Use pushRebuild() in TransactionProcessor
* Optimize setRoot() for the case where the root is already set correctly

Change-Id: I8b827d0823c969e671615ddd06e5f1bd70e9d54c
2012-04-13 16:46:02 -07:00
Gabriel Wicke 9913108b40 Fix fetch-parserTests (it is in path instead of fs)
Change-Id: I169502079ea2609a4f4af776b15767cf0c3ec8b5
2012-04-04 20:40:09 +02:00
Adam Wight b234edba88 As much as I have loved writing Makefiles... I've replaced its functionality with package.json, mostly so we can avoid non-node dependencies. This is one of the recommended practices. We should consider moving tests/parser into modules/parser/tests, other node projects keep all module code in one directory.
Explained in the README how to use npm to load the dependencies and run tests.  Too bad about NODE_PATH...

Don't try to find parserTests.txt in assorted places--if it isn't present, fetch from gerrit.  You can symlink from core if you're developing on both parsers, and the fetch script will not overwrite.

Use __dirname in parserTests.js to allow the script to run independent of current working directory.

Change-Id: I4c8b884e91f4fdeae385c7697aff768bdd199dd5
2012-04-04 11:02:58 -07:00
Gabriel Wicke f662690d02 Shorten data-mw-rt to data-mw and clean up whitelist
Instead of a proliferation of data-mw-* attributes, it should be easier to
stash all private / non-semantic round-trip information in a JSON object
stored in data-mw.

Change-Id: Id200a6a8789fa152f29ea530e5a24b6ee7b4b285
2012-04-02 18:12:49 +02:00
Gabriel Wicke 5ef3438ee5 Change path to parserTests from phase3 to core after switch to git.
Change-Id: Ie13f678eaa81447e98db5c8c394ab103caad8454
2012-04-02 17:10:06 +02:00
Audrey Tang d3602bb459 * Get parser tests from GitWeb, not Subversion.
Change-Id: I39f933b9e0320dc62736da07ce097ec1badec9aa
2012-03-28 23:39:01 +08:00
Catrope 7a726b0278 Add tree synchronization for replace
To handle replace operations that are not themselves consistent (these
are common, for instance when replacing an opening element in one place,
then replacing the closing element somewhere else), we process
subsequent replace operations inside the first one until things are
balanced again, then issue a single rebuild for the whole thing.

Change-Id: Ide4613f046fabfeeef383138c39e350b1b710033
2012-03-26 02:51:30 -07:00
Roan Kattouw 662633dfb3 Add a test for unwrapping and rewrapping 2012-03-14 21:02:33 +00:00
Roan Kattouw 2c43a34f74 Rewrite the rebuild action to take two ranges rather than a node and some data. 2012-03-14 21:02:31 +00:00
Roan Kattouw 37a59016e8 Break out pushAction() into separate functions for each action. This will allow me to change the rebuild action to take totally different parameters. 2012-03-14 21:02:29 +00:00
Antoine Musso f637756319 node modules required: request & jshashes 2012-03-13 15:14:18 +00:00
Roan Kattouw 16a2356e43 Add tests for list split tree sync 2012-03-13 00:14:38 +00:00
Trevor Parscal c977591886 Added test for ve.dm.DocumentSynchronizer that exercises multi-action synchronizations 2012-03-09 19:38:54 +00:00
Roan Kattouw d70aa70707 Add test for replacing a table with a list. This only works because
nesting validity isn't checked yet (lists inside lists are illegal
IIRC), but for now it tests the reversal of the order of the closing
tags nicely
2012-03-09 02:19:50 +00:00
Roan Kattouw b13d0a849d Add a check for the length of unwrapOuter, and add a test for each
exception
2012-03-09 01:44:31 +00:00
Roan Kattouw bc600b34be Make prepareWrap() use the data from the model rather than the unwrap
parameters. This fixes the case where rolling back a list unwrap would
restore the list items without their attributes
2012-03-09 01:14:41 +00:00
Roan Kattouw 3bc6b3d8c7 Add tests for unwrapping a list
This also excercises unwrapEach. One of the tests is still subtly broken
in that the attributes on the listItems aren't preserved, I'll fix that
next.
2012-03-09 00:38:35 +00:00
Roan Kattouw 5054ed320e Implement prepareWrap and add tests for it 2012-03-08 23:21:26 +00:00
Roan Kattouw 10a6ee73f4 Add tests for content replacements 2012-03-08 23:21:23 +00:00
Trevor Parscal 3ec0c07843 Fixed name of test suite to match actual class name 2012-03-08 19:37:13 +00:00
Trevor Parscal becb1daa39 Added more tests for ve.dm.DocumentSynchronizer and fixed some bugs along the way 2012-03-08 19:35:51 +00:00
Trevor Parscal 459c4fa271 Added some basic tests for resize and insert. Fixed some bugs in both of those code paths along the way. 2012-03-08 00:52:30 +00:00
Gabriel Wicke af03eb4f29 Improve generic attribute expansion before external link processing, and make
wgUploadPath configurable. Also change the hard-coded fall-back image sizes to
sensible defaults. This breaks three parser tests until image size retrieval
from the wiki is implemented.
2012-03-06 18:02:35 +00:00
Gabriel Wicke 227103e12c Accept empty table cell attribute sections, and consider percent-encoded %2525
valid. 270 tests passing.
2012-03-06 14:32:45 +00:00
Gabriel Wicke 2efcd3cd57 Reworked percent encoding handling for URIs to get closer to the 'url
construction' part of the HTML5 spec:
http://www.whatwg.org/specs/web-apps/current-work/multipage/urls.html#url-manipulation-and-creation

Removed a few whitelisted test cases that are now passing directly.

The encoding canonicalization could also be moved to the Sanitizer. Doing this
early in token stream processing however has the advantage of providing further
transformations uniform data to work with. We could even consider to move this
even further into the tokenizer.
2012-03-06 13:49:37 +00:00
Gabriel Wicke 19fe9726a2 Fix invalid external link representation. 268 tests passing. 2012-03-05 18:06:29 +00:00
Gabriel Wicke 7b0c807710 Change wikilink tokenization strategy to split on pipes. This makes it
possible to support template / template argument expansion in image options,
and causes little trouble for wikilinks. Non-image wikilinks with multiple
text pipes are quite rare in the dumps, and concatenating description tokens
with a plain '|' is quite easy. 261 parser tests passing.
2012-03-05 12:00:38 +00:00
Trevor Parscal 0e41da3340 Fixed tests that were broken by r112150. 2012-03-02 23:12:38 +00:00
Gabriel Wicke 009d7a4dea Namespaces to the rescue. 2012-03-02 15:49:05 +00:00
Gabriel Wicke fe681042c0 Collect some statistics while grepping. 2012-03-01 16:42:28 +00:00
Gabriel Wicke e0838db315 Capturing the regexp is no longer necessary, and speeds up the grepper. Also
tweaked the multi-line ISBN regexp slightly.
2012-02-29 13:02:46 +00:00
Gabriel Wicke e3deb304db Add a misc regexp file for dump grepping. 2012-02-29 11:07:17 +00:00
Gabriel Wicke 14f40aa7d5 Support capturing regexps in dumpGrepper. 2012-02-29 10:49:00 +00:00
Gabriel Wicke ebcfc2c7a1 Improve grepper documentation. 2012-02-28 14:24:37 +00:00
Gabriel Wicke b767e03449 Tweak martian regexp and grepper output format. 2012-02-28 14:11:44 +00:00
Gabriel Wicke 4806505ce4 Finish color highlighting for dump grepper / fix broken commit r112592. 2012-02-28 13:48:47 +00:00
Gabriel Wicke 7daeb34d4d Implement onlyinclude transformer. 254 tests passing. 2012-02-28 13:21:01 +00:00
Gabriel Wicke 32012c00cd Add martian-endtags regexp wrapper around dumpGrepper. 2012-02-27 16:51:20 +00:00
Gabriel Wicke 19c67c28a2 Add a simple dump grepper using DumpReader. Useful to inform parser design
decisions, and as a way to exercise the dump reader in preparation for tests
over full dumps.
2012-02-27 16:40:01 +00:00
Gabriel Wicke 21855c99cd Tweak dumpReader to work with current libxmljs and stdin 'data' events. 2012-02-27 15:46:08 +00:00