Commit graph

1328 commits

Author SHA1 Message Date
Subramanya Sastry 23ec054013 Fixed round-tripping of interwiki links.
Change-Id: If0427b9865b3e9cf8c0ad0b4efaebc9f9f7fb865
2012-06-13 13:39:18 -05:00
Subramanya Sastry 445780b4d3 Revert default tokenization result from null to ''
* As part of an earlier fix, I had changed default value of 'res'
  to null instead of ''.  But, this was potentially buggy because
  the previous check was (res !== '') which could be triggered
  by return values of handlers.  By changing the check to null,
  I was effectively changing the code paths for those handlers that
  returned ''.

Change-Id: I2302023be7422ce4fb384ff5a50fe53fa7732855
2012-06-13 11:53:05 -05:00
Subramanya Sastry cfe94eed1f Minor code refactoring
Change-Id: Iec3cb4d83d16174371f0b1f3f23b1056aeed458e
2012-06-13 09:46:34 -05:00
Catrope 6bf79475e4 Add suggested development configuration in comments
Change-Id: I3ec8c5326faced6d4b5c878f26f37a281b03bd95
2012-06-12 19:19:48 -07:00
Subramanya Sastry e00cfdbc16 Merge "Remove a few entries we now care about from the whitelist" 2012-06-12 18:57:17 +00:00
Subramanya Sastry f882a65153 Fix serialization of <pre> tags
Change-Id: I7ae95e7ec06167d0c1bfdaba3d0c67d941043299
2012-06-12 13:54:35 -05:00
Subramanya Sastry 727c2119bb Refactored serializeToken method and added special-case handling of
paragraphs in lists.

* We need to look at other special-case handling requirements of
  html tags in lists (and other contexts like tables).

Change-Id: I84b8402d90a186c9075c2d45263c94377312927a
2012-06-11 17:55:41 -05:00
Translation updater bot e605878f46 Localisation updates from http://translatewiki.net.
Change-Id: I632b12009951716606f71530485ffc7ef377213d
2012-06-11 14:31:40 +00:00
Trevor Parscal ea73773854 Added node_modules and node error log file to git ignore
Change-Id: I9006d1e4bfd266fb470fe9143995c5dda112ac43
2012-06-10 23:56:50 -07:00
Translation updater bot 81d53403a8 Localisation updates from http://translatewiki.net.
Change-Id: I29672237acf0ab18963bdd46702b53c675e00b4c
2012-06-07 19:15:00 +00:00
Gabriel Wicke 3f61dc9821 Link talk page separately
Change-Id: Ib839f619e7e14ccf0ef698fc2e780ef4b0d65505
2012-06-07 13:42:05 +02:00
Gabriel Wicke 3549a16085 Add a 'report issue' link below round-trip results
Change-Id: I5e3a785a328af0debcf83dc2038b5e5417fa5158
2012-06-07 13:37:40 +02:00
Gabriel Wicke bec7fb2f8c Mention citations as not round-tripping
Change-Id: I57e25f6f4072bae2f5681b8611e98f899875d1e2
2012-06-07 13:18:44 +02:00
Gabriel Wicke 76cca063ba Add hint on where to support issues in web service entry page
* Explain what we are currently interested in and link to
  :mw:Talk:Parsoid/Todo.

Change-Id: I747c6ee8a021a7a73ec91b73281c1c679a00da8f
2012-06-07 13:16:05 +02:00
Gabriel Wicke 1ca586e5f1 Improve interwiki config a bit
* Moved wikipedia default prefixes to environment
* Added 'addInterwiki' method
* Adjusted link handling normalizeTitle to reflect this

Change-Id: If5b2314cc36346b6da8649ed410457a612d80a22
2012-06-07 12:30:16 +02:00
Gabriel Wicke 2fa5baabbb Make it easier to configure the default wiki, and add support for mediawiki.org
* mw:Foo now loads pages from mediawiki.org
* The default prefix still is 'en'. You can switch this to 'mw' in ParserService.js.

Change-Id: I1208667e6114bd711b7988a8b3adb32ffab70969
2012-06-07 11:50:40 +02:00
Gabriel Wicke b49102281f Remove a few entries we now care about from the whitelist
They are mostly about whitespace, but there is also a debatable quote test
that outputs an empty bold element at the end of the line. We should perhaps
strip this empty bold in the QuoteTransformer, as the preservation of an empty
bold tag in round-tripping does not seem to be too useful.

Change-Id: I1d8f3ebabcd9f6249e5170de420ba52e8aea22ca
2012-06-07 10:04:20 +02:00
Subramanya Sastry b665a2558f Fixed bugs handing/transforming quotes
- Three bugs that were messing up quote transformations.
- Now, the following cases are handled properly:

  * ''foo'''
  * '''foo''
  * ''foo''''
  * ''''foo''

  These tests (and other quote tests) have to be added to core parser
  tests file.

- One more parser test green.

Change-Id: I4f93e8910639f546bfc9304becab17d26d5529de
2012-06-07 01:37:45 -05:00
Translation updater bot 42daebe50a Localisation updates from http://translatewiki.net.
Change-Id: Ieb79571c97e1158414ecccbc8d5e984382f2cce5
2012-06-06 20:19:14 +00:00
Gabriel Wicke 413df0c471 Strip \r from form input- we normalize everything to Unix
Change-Id: I5cd255e1a7ab9958f120fad408362e6f709e4b91
2012-06-06 19:26:29 +02:00
Gabriel Wicke 47204c4ca0 Use diffChars instead of diffWords, as the former misses some changes
The improved merge algorithm now makes diffChars output more palatable. Things
could still be improved by collecting single-character 'neutral' changes in a
block of 'add' changes and converting them to adds / removes.

Change-Id: I8439e8acab4360c08b89d9ce8a6b8523e7a0a210
2012-06-06 18:36:28 +02:00
Subramanya Sastry f8221b128b Used a more robust heuristic for merging consecutive diffs
- Check if consecutive diffs are separate by 1 word in addition
  to max 3 chars.  This takes care of diffs introduced by template diffs
  separated by the template name and creates a clean single diff.

Change-Id: I9181d2ed9a07bee6ca5d5ebd6ddea84f7e2cecac
2012-06-06 11:01:47 -05:00
Gabriel Wicke 2bc066b42d Up the diff merge size heuristic a bit and always use the same algorithm
Change-Id: I707c8a55ed1758cdd591d2fc95e03a360c8e76d1
2012-06-06 17:46:25 +02:00
Gabriel Wicke bc1a77a812 Make modified newlines visible by replacing empty lines with a space
Change-Id: If7b811245e0d01a7a147ab54c3801fc1754730a9
2012-06-06 17:11:29 +02:00
Gabriel Wicke 1876d785a7 Swap ins/del in the diff
Change-Id: Id336d713d1767a4b7859b158f2c2ddf9adc11cfb
2012-06-06 16:02:54 +02:00
Gabriel Wicke 350e700d8f Add core-upgrade
Change-Id: I5ad0955e8272d376f009f89461bed310978b25e4
2012-06-06 15:58:17 +02:00
Gabriel Wicke d0a0454ada Merge "Improve the handling of newlines for round-tripping" 2012-06-06 13:54:04 +00:00
Gabriel Wicke aee35f627d Merge "Update patched html5 library to version 0.3.8" 2012-06-06 13:53:37 +00:00
Gabriel Wicke a146fcb8ad Improve the handling of newlines for round-tripping
An improvement, but there still are some extra newlines inserted after
paragraphs. Example input:

-------

Foo:
{|
|foo
|}
-------

Extra newlines are inserted after the Foo: and the foo in the table. They are
not fed as tokens or text to the tree builder, so there is likely a bug in the
html5 library or JSDom.

Change-Id: I83eb6180e3cd1c4e7f9b15b31d339e1d32bccd3f
2012-06-06 10:17:03 +02:00
Gabriel Wicke 59fc634cce Update patched html5 library to version 0.3.8
Change-Id: I321d9a58ea1af33842a606fc8706938093a8330f
2012-06-06 10:17:03 +02:00
Subramanya Sastry bff08b799e Improvement to the refineDiffs function to improve diff quality.
* Attempt to accumulate consecutive add-delete pairs
  with "short text" separating the pairs.  This is equivalent to
  the <b><i> ... </i></b> minimization to expand range of
  <b> and <i> tags, except there is no optimal solution except
  as determined by heuristics ("short text": <= 2 chars).

Change-Id: I408e318c315eba18aac4051ed84d77e3e092d497
2012-06-06 00:08:00 -05:00
Subramanya Sastry fe6f289486 Merge changes I5d98c704,Ib8d3de75
* changes:
  A few tweaks to link round-tripping
  Use word diff if --color is enabled
2012-06-05 16:04:23 +00:00
Subramanya Sastry b095db4303 Simpler implementation of flatten.
* Possibly more efficient under heavy GC load -- untested.
* No change in time and memory use for single file parsing.

Change-Id: Id2f3f65cc0e5f38ed968bbda60b97e46523e700e
2012-06-05 10:47:46 -05:00
Gabriel Wicke dc3168cf6d A few tweaks to link round-tripping
* Moved the tail attribute to the second attribute (a bit cleaner)
* Disallowed newlines in the tail production
* Improved the selection of round-tripped href vs. generated content vs. href
  in the serializer
* renamed state.linkTail to state.dropTail

Change-Id: I5d98c704b6ea566011e22237786f8da17548570f
2012-06-05 17:26:27 +02:00
Gabriel Wicke 0f9d939b00 Use word diff if --color is enabled
Change-Id: Ib8d3de75ac306974abfdaca22bfc7b69bc62891d
2012-06-05 16:10:13 +02:00
Gabriel Wicke d16032ae9a Track html syntax in block_tag production
Change-Id: If560523644f007485809762f12216e08fb3c3ed3
2012-06-05 12:39:56 +02:00
Gabriel Wicke c1d8270bdb Fix wgScriptPath in round-trip mode without interwiki
Change-Id: I7cc80b7be1afffc586a2ea45d21303e9ba07c0d4
2012-06-05 12:11:45 +02:00
Gabriel Wicke 3346aed86e Support interwiki links, and some cleanup
Change-Id: I205c53a03f5230e3ef9100487f4934f97bdc179a
2012-06-05 12:05:33 +02:00
Gabriel Wicke cc96ff4f5e Very basic interwiki support
Pages titles with a wikipedia interwiki prefix now load the page from
corresponding Wikipedia. Links in a page then stay within the given language.

Note that Parsoid currently makes no effort to recognize localized namespaces,
so it won't render media files, categories etc correctly.

Change-Id: I7bc4102e81a402772ea23231170734d580ea15b9
2012-06-05 11:19:58 +02:00
Gabriel Wicke 92f753a365 Pre and link target improvements
* Don't explicitly add the newline in the pre, as we preserve newline tokens
  now. This avoids doubling of newlines when round-tripping.
* Use the sHref attribute even if the href contains spaces.

Change-Id: I8bec8fbfd6a7836bf2e5eec20869a0edd95c93b6
2012-06-04 14:03:05 +02:00
Gabriel Wicke ee2ddbd3cb Fix list handler issues
Lists interrupted by non-empty lines would not close the list properly.
Register for any token instead of just for newlines and close the list if no
listItem follows the newline.

Change-Id: I1743901e3db541bbeda78d17707db943e6ceb9b9
2012-06-04 13:38:43 +02:00
Gabriel Wicke f821eac102 Optionally round-trip sHref in data-mw
If the href would not denormalize, add a copy of the original href in data-mw
and use it to preserve non-conventional capitalization etc.

Change-Id: Ifef50eec7343b0e6b0ba66b6d19a8a3e8c9f8001
2012-06-04 12:28:05 +02:00
Gabriel Wicke e0809209ec Don't set the data-mw attribute if the object is actually empty.
Change-Id: I984f1b44bba67d7a9f1a709738d14c0ee02f69a9
2012-06-04 12:26:03 +02:00
Gabriel Wicke 2774e5aa6c Actually replace all underscores in wikilink target
Change-Id: I633f8d6e4f639aff90fd456600376b7c6515fd50
2012-06-04 11:48:59 +02:00
Gabriel Wicke 3f2c72f920 Fix padleft / padright (mis)use as substr
Change-Id: I0645e11c8ef8b550ad35300d1904788940fc748a
2012-06-04 11:30:45 +02:00
Gabriel Wicke 0eabd2c67e Add round-trip form and split out rt diffing
Change-Id: I3bc8ad7f273937ce6c767b8d7bbccdc86cbd93b4
2012-06-04 10:49:59 +02:00
Gabriel Wicke 99c98d6c56 Diff refinement fixes
Change-Id: I11c69de0fdcd636ccd11cd0b6cb16c5acdb188b3
2012-06-04 10:16:05 +02:00
Gabriel Wicke d2602c47a6 Switch back to word-based diff
The char-based diff looked good in some pages, but yielded terrible results in
others. The word-based algo is more consistent overall.

Change-Id: I7f2d40315ad96df037c2d9a1d50739e3d21b6c81
2012-06-04 00:02:49 +02:00
Gabriel Wicke 4533c274ca Fix a crasher in the serializer
A tail containing regexp syntax (a ? in [[:en:Main Page]]) would crash the
serializer. Use substr instead.

Change-Id: I8519aec9c07dfe31893d676b1c936a42d2af74a0
2012-06-04 00:00:54 +02:00
Gabriel Wicke d01581c380 Create a 'refinement diff' algorithm
The word or char-based algorithm does not scale well beyond 5k chars or so. We
now perform a line-based diff and then continue to diff the line differences
using the char-based algorithm. This gives a char-based diff even for bigger
inputs.

Change-Id: Iec87ca56540060e4df2859ba54c992e7ff5cfe10
2012-06-03 23:46:57 +02:00