Commit graph

389 commits

Author SHA1 Message Date
Subramanya Sastry bf0f5d1b7e Minor code cleanup
Change-Id: Ic5d99b6c483841310b0c295c1c30246f907455b4
2012-06-13 13:47:26 -05:00
Subramanya Sastry 23ec054013 Fixed round-tripping of interwiki links.
Change-Id: If0427b9865b3e9cf8c0ad0b4efaebc9f9f7fb865
2012-06-13 13:39:18 -05:00
Subramanya Sastry 445780b4d3 Revert default tokenization result from null to ''
* As part of an earlier fix, I had changed default value of 'res'
  to null instead of ''.  But, this was potentially buggy because
  the previous check was (res !== '') which could be triggered
  by return values of handlers.  By changing the check to null,
  I was effectively changing the code paths for those handlers that
  returned ''.

Change-Id: I2302023be7422ce4fb384ff5a50fe53fa7732855
2012-06-13 11:53:05 -05:00
Subramanya Sastry cfe94eed1f Minor code refactoring
Change-Id: Iec3cb4d83d16174371f0b1f3f23b1056aeed458e
2012-06-13 09:46:34 -05:00
Subramanya Sastry f882a65153 Fix serialization of <pre> tags
Change-Id: I7ae95e7ec06167d0c1bfdaba3d0c67d941043299
2012-06-12 13:54:35 -05:00
Subramanya Sastry 727c2119bb Refactored serializeToken method and added special-case handling of
paragraphs in lists.

* We need to look at other special-case handling requirements of
  html tags in lists (and other contexts like tables).

Change-Id: I84b8402d90a186c9075c2d45263c94377312927a
2012-06-11 17:55:41 -05:00
Gabriel Wicke 1ca586e5f1 Improve interwiki config a bit
* Moved wikipedia default prefixes to environment
* Added 'addInterwiki' method
* Adjusted link handling normalizeTitle to reflect this

Change-Id: If5b2314cc36346b6da8649ed410457a612d80a22
2012-06-07 12:30:16 +02:00
Gabriel Wicke 2fa5baabbb Make it easier to configure the default wiki, and add support for mediawiki.org
* mw:Foo now loads pages from mediawiki.org
* The default prefix still is 'en'. You can switch this to 'mw' in ParserService.js.

Change-Id: I1208667e6114bd711b7988a8b3adb32ffab70969
2012-06-07 11:50:40 +02:00
Subramanya Sastry b665a2558f Fixed bugs handing/transforming quotes
- Three bugs that were messing up quote transformations.
- Now, the following cases are handled properly:

  * ''foo'''
  * '''foo''
  * ''foo''''
  * ''''foo''

  These tests (and other quote tests) have to be added to core parser
  tests file.

- One more parser test green.

Change-Id: I4f93e8910639f546bfc9304becab17d26d5529de
2012-06-07 01:37:45 -05:00
Gabriel Wicke 350e700d8f Add core-upgrade
Change-Id: I5ad0955e8272d376f009f89461bed310978b25e4
2012-06-06 15:58:17 +02:00
Gabriel Wicke a146fcb8ad Improve the handling of newlines for round-tripping
An improvement, but there still are some extra newlines inserted after
paragraphs. Example input:

-------

Foo:
{|
|foo
|}
-------

Extra newlines are inserted after the Foo: and the foo in the table. They are
not fed as tokens or text to the tree builder, so there is likely a bug in the
html5 library or JSDom.

Change-Id: I83eb6180e3cd1c4e7f9b15b31d339e1d32bccd3f
2012-06-06 10:17:03 +02:00
Gabriel Wicke 59fc634cce Update patched html5 library to version 0.3.8
Change-Id: I321d9a58ea1af33842a606fc8706938093a8330f
2012-06-06 10:17:03 +02:00
Subramanya Sastry fe6f289486 Merge changes I5d98c704,Ib8d3de75
* changes:
  A few tweaks to link round-tripping
  Use word diff if --color is enabled
2012-06-05 16:04:23 +00:00
Subramanya Sastry b095db4303 Simpler implementation of flatten.
* Possibly more efficient under heavy GC load -- untested.
* No change in time and memory use for single file parsing.

Change-Id: Id2f3f65cc0e5f38ed968bbda60b97e46523e700e
2012-06-05 10:47:46 -05:00
Gabriel Wicke dc3168cf6d A few tweaks to link round-tripping
* Moved the tail attribute to the second attribute (a bit cleaner)
* Disallowed newlines in the tail production
* Improved the selection of round-tripped href vs. generated content vs. href
  in the serializer
* renamed state.linkTail to state.dropTail

Change-Id: I5d98c704b6ea566011e22237786f8da17548570f
2012-06-05 17:26:27 +02:00
Gabriel Wicke d16032ae9a Track html syntax in block_tag production
Change-Id: If560523644f007485809762f12216e08fb3c3ed3
2012-06-05 12:39:56 +02:00
Gabriel Wicke cc96ff4f5e Very basic interwiki support
Pages titles with a wikipedia interwiki prefix now load the page from
corresponding Wikipedia. Links in a page then stay within the given language.

Note that Parsoid currently makes no effort to recognize localized namespaces,
so it won't render media files, categories etc correctly.

Change-Id: I7bc4102e81a402772ea23231170734d580ea15b9
2012-06-05 11:19:58 +02:00
Gabriel Wicke 92f753a365 Pre and link target improvements
* Don't explicitly add the newline in the pre, as we preserve newline tokens
  now. This avoids doubling of newlines when round-tripping.
* Use the sHref attribute even if the href contains spaces.

Change-Id: I8bec8fbfd6a7836bf2e5eec20869a0edd95c93b6
2012-06-04 14:03:05 +02:00
Gabriel Wicke ee2ddbd3cb Fix list handler issues
Lists interrupted by non-empty lines would not close the list properly.
Register for any token instead of just for newlines and close the list if no
listItem follows the newline.

Change-Id: I1743901e3db541bbeda78d17707db943e6ceb9b9
2012-06-04 13:38:43 +02:00
Gabriel Wicke f821eac102 Optionally round-trip sHref in data-mw
If the href would not denormalize, add a copy of the original href in data-mw
and use it to preserve non-conventional capitalization etc.

Change-Id: Ifef50eec7343b0e6b0ba66b6d19a8a3e8c9f8001
2012-06-04 12:28:05 +02:00
Gabriel Wicke e0809209ec Don't set the data-mw attribute if the object is actually empty.
Change-Id: I984f1b44bba67d7a9f1a709738d14c0ee02f69a9
2012-06-04 12:26:03 +02:00
Gabriel Wicke 2774e5aa6c Actually replace all underscores in wikilink target
Change-Id: I633f8d6e4f639aff90fd456600376b7c6515fd50
2012-06-04 11:48:59 +02:00
Gabriel Wicke 3f2c72f920 Fix padleft / padright (mis)use as substr
Change-Id: I0645e11c8ef8b550ad35300d1904788940fc748a
2012-06-04 11:30:45 +02:00
Gabriel Wicke 4533c274ca Fix a crasher in the serializer
A tail containing regexp syntax (a ? in [[:en:Main Page]]) would crash the
serializer. Use substr instead.

Change-Id: I8519aec9c07dfe31893d676b1c936a42d2af74a0
2012-06-04 00:00:54 +02:00
Gabriel Wicke 31522d3d49 Add ApiRequest
Change-Id: I5f2a1cb65223a68f10bc63903000248efca05586
2012-06-02 16:52:51 +02:00
Gabriel Wicke 63abd57fc8 Improve newline-before-paragraph round-tripping support
Change-Id: I9176a97f9695018650d9a63b89514c07e0d6be90
2012-06-02 16:39:33 +02:00
Gabriel Wicke d3975a8d03 Very basic round-trip test mode for the API
Returns both the resulting wikitext and the diff with the original input.

Change-Id: Iad25039beb054a84e1ad51ffa9fee924db49c60b
2012-06-02 16:20:54 +02:00
Gabriel Wicke 74135b295f Some more switch fixes
Change-Id: If1a6086348c45a73a941bc8e6728ef75d002be50
2012-06-02 15:04:20 +02:00
Subramanya Sastry 8f216af2f5 Handle link tails properly.
- Added a tail json attribute for wikiLinks
- During serialization, this attribute is used to strip the tail from
  the link target and render it after the link

  [[hen]]s ==> <a ... data-mw="{gc:1, tail: 's'}" ...>hens</a>
           ==> [[hen]]s

- 2 more roundtrip tests green

Change-Id: I84f3dabaf0271f7a67641a00148467daa8310eb0
2012-06-01 23:41:10 -05:00
Subramanya Sastry 413fc5e043 Fixed bug serializing wikilinks with implicit link text.
* Simple fix but greens 10 more roundtrip tests.

Change-Id: I7f82d788a10bd83e0e3215568c2168081c332c50
2012-06-01 17:25:21 -05:00
Gabriel Wicke 16219ddc6d Fix up #switch a bit
* Re-establish the value-only default
* Fix value expansion

Change-Id: I32e62789b25bbe17a74c564e41e9101ad5528fb7
2012-06-01 22:15:43 +02:00
Gabriel Wicke e2301813ed Merge "Tokenizer backtracking cache bug fix and memory savings" 2012-06-01 12:06:00 +00:00
GWicke befd223476 Merge "First pass implementing a general tag minimization routine" 2012-06-01 11:15:48 +00:00
Gabriel Wicke ece2b0f810 Tokenizer backtracking cache bug fix and memory savings
* The state of syntax stops is now properly included in the cache key for the
  tokenizer-internal backtracking cache. This fixes some mis-parses when
  re-parsing a bit of text with different flags.
* Clear the backtracking cache after each toplevelblock. This drops the peak
  memory usage when expanding [[:en:Barack Obama]] from ~380M to ~110M.

Change-Id: Icdb879cae5907e4595903dd6acba2e686e8c2e4b
2012-06-01 12:53:49 +02:00
Subramanya Sastry 1c80e2d7f0 First pass implementing a general tag minimization routine
* This routine attempts to rewrite the DOM to maximize tag overlap
  and thus minimize tag uses.

* This takes as input a set of tags which participate in the
  minimization.

* Tested on the following example
  <b><i><u><s>BIUS</s></u></i></b><b><i><s>BIS</s></i></b><b><u><s>BUS</s></u></b><u><i>UI</i></u>
  with multiple combinations of the 2^4 possible variations of i,b,u,s
  tags: [], ['i','b','u','s'], ['i'], ['b','s'], ['i','b','u']

  - But, I am not fully sure if this implements the right behavior when
    only a subset of inline tags are provided.  Needs discussion and tweaking
	 as necessary.

* Also tested on few others:
  <b>B</b><b><i>BI</i></b><b><i><u>BIU</u></i></b><b><i><u><s>BIUS</s></u></i></b>
  <s><i><b>SIB</s></i></b><s><i><u>SIU</u></i></s><i><u>IU</u></i><i>I</i>

* The previous pairwise tag rewriting version fails on several of these
  examples, so this new version is a definite improvement.

* No change in parserTests run (203 passing before and after).

* Possible improvements that could/should be undertaken:
  - get rid of useless/idempotent add/remove of nodes that don't change
    the DOM.
  - ensure that node attributes post-restructuring are correct.

Change-Id: Ib4a8b39583fa96a2be880a77021ca81cefa06484
2012-05-31 12:10:28 -05:00
Gabriel Wicke 4ea6b8e2be Revert part of last template syntax tweak
Change-Id: I084e1210577f80c3b96020d57cfa5c68eb5d139b
2012-05-31 12:02:42 +02:00
Gabriel Wicke c5d7e01944 Another tokenizer robustness improvement
This patch fixes a tokenizer syntax error encountered on
[[:en:Template:JacksonvilleWikiProject-Member]] and [[:en:Template:Infobox
former country]] by allowing optional whitespace before start-of-line template
syntax.

Change-Id: Ic214a731de58bf766e51f23d5e24ea2ce6788f58
2012-05-30 18:38:23 +02:00
Gabriel Wicke a133768781 Don't eat '}}' in generic attributes and similar productions
This fixes some syntax errors, at least one in Template:Geobox.

Change-Id: I32338febe25d0833c1d9bc4de293cd15b4cbb7be
2012-05-30 17:37:10 +02:00
Gabriel Wicke 36084c5d93 Preserve original newlines in HTML and serialization
254 round-trip tests (up from 184) are now passing.

Also:
* tweaked runtests.sh slightly (use less -R instead of -r).
* made sure the EOFTk is preserved in phase 3 transforms

Change-Id: I1de22186bdb78e52019370e43f096877005b8f5a
2012-05-29 23:29:03 +02:00
Subramanya Sastry 8174c9dafc First attempt implementing rewriting rules on the DOM
- This is implemented as a post-processing pass.
- Might require additional checks to verify rewriteability.
- Implemented as a pair-wise tag DOM minimization strategy,
  i.e. it takes tag pairs (B, I) for ex, and attempts to
  normalize the tree just for those tag pairs.  Normalizing
  across multiple tags is implemented as pairwise rewriting
  across all pairs:  Ex:(b,i), (b,u),(i,u) for (b,i,u)
- Copied over attributes as part of rewriting, but some of the
  attributes lose their meaning on rewriting since tags are
  reordered (ex: sourcePosn, sourceTagPosn). How do we handle this?

Output examples and possible issues to fix:
   <i><b><u>biu</u></b></i><b><u>bu</u></b><u>u</u>
gets rewritten to:
   <u><b><i>biu</i>bu</b>u</u>

But, the equivalent wikitext form:
   '''''<u>biu</u>''''''''<u>bu</u>'''<u>u</u>
does not get rewritten because of parsing differences.
This wikitext gets parsed into:
   <i><b><u>biu</u>'''</b></i><u>bu<b>u</b></u>
The extra ''' token in the middle thwarts DOM rewriting.

However, a slightly different version:
   "'''''<u>biu</u>''<u>bu</u>'''<u>u</u>"
gets properly normalized to:
   <u>'''''biu''bu'''u</u>

An alternative, but fun strategy to play with is to use the following
two normalization primitives: S(wap) and M(erge).
- S rewrites T1(T2(x)) into T2(T1(x))
  (ex: <b><i>foo</i></b> ==> <i><b>foo</b></i>)
- M rewrites (T(x),T(y)) into (T(x,y)).
  (ex: <b>foo</b><b>bar</b> ==> <b>foobar</b>)

The current rewriting strategy could possibly be re-implemented as S-M
rewriting.  The problem to solve there would be to find an efficient
rewriting strategy that is guaranteed to lead to a normal form.  I may
not play with it now, but just documenting it for later (to play with
in my spare time).

This commit is just as a record of fun/experimental code where I get to
learn details of JS, wikitext, parsing, and DOM manipulation.  Next
version of this code will attempt to introduce minimal DOM restructuring
across multiple tags at once which can be more efficient.

gwicke: Removed now passing test from whitelist, and updated another whitelist
entry which is now improved.

Change-Id: Ie97bcb164eb62c34ba61aa76ba2f4c232aa713d8
2012-05-29 08:17:57 +02:00
Gabriel Wicke b2adee0ae7 Basic rt support for indent pre variant
* Added a generic stx_v 'syntax variant' round-trip attribute
* For pre, use stx:'html' vs. no syntax annotation. This might not be 100%
  safe for arbitrary html input, so we might want to flip this to stx:'wiki'
  later.
* 181 round-trip tests passing

Change-Id: If6080917a3a7c069066db3db60efe59b1f6c28d8
2012-05-25 18:55:38 +02:00
Gabriel Wicke a31ccaabe4 Support definition lists with empty definition
Change-Id: I81c39a7e49f2ea7ce32cdd3600caeb5eb9f50d84
2012-05-25 15:40:32 +02:00
Gabriel Wicke 06b51b1f3f Properly round-trip dd/dt; 178 round-trip tests passing.
Need to track variable whitespace before elements to make some more tests
pass.

Change-Id: Ia86535d6f352e2ffe7965547cd506b0dbb6dfba2
2012-05-25 13:59:55 +02:00
Gabriel Wicke 6f62878c78 Resolve subpage links, and remove hack for H: titles
Change-Id: I6c9c64179274e5c1641a3b127ac3b273a3c5254e
2012-05-24 17:57:41 +02:00
Gabriel Wicke dc61f313a2 Notes on missing parser functions, more error reporting tweaks
Change-Id: Ib6ce60cf1b55671a6ff57aa47edb5787ec3aefea
2012-05-24 17:31:26 +02:00
Gabriel Wicke cc10aab54f Add self alias
Change-Id: I47682f407da6b554179611c7d0f63f882ab5a871
2012-05-24 17:16:35 +02:00
Gabriel Wicke 13ae7cda11 A few (partly hackish) improvements
* Very basic support attribute key-value pairs emitted from templates
* Add TALKPAGENAME stub implementation
* Only show 'no revisions' message for top-level pages

Change-Id: I4b4ac0c7b2c0531ac4b39f0f49f4217302576ab9
2012-05-24 16:30:26 +02:00
Gabriel Wicke 3e0e11b1d0 Sanity check for tokens being an array
Change-Id: Ia4e4071e1469c31e3b320d854500938bb0245f82
2012-05-24 14:35:58 +02:00
Gabriel Wicke 93ce7453f0 Fake fullpagename et al a bit better
Change-Id: I85ddf9e88e5f8ac274f371bea0879600997001e4
2012-05-24 11:05:31 +02:00
Gabriel Wicke cdd1eca42d Fix non-existing revision error reporting
Change-Id: I6b8687bcde98b92d9d6217a738a177db279fd006
2012-05-24 10:50:47 +02:00