This solves lots of issues in the integration work, and also makes it much easier to extend this class to integrate it into other skins
Change-Id: I3b3c5b22a5664e6cf37e429cc0ac3be2e75b630f
* tokens are not modified any more (they are supposed to be immutable)
* handler info is now split in start / end objects and potentially a 'make'
method; added more flags to govern the newline behavior of different tags
* added a generic singleLine mode for single-line syntactical environments
* switched the web service to line-based diffs to avoid issues when diffing
the round-trip results of [[:en:Programming language]]
* 280 round-trip tests are passing now
Change-Id: I74b8ffbf69643c5d6e5ec852ec58e680c9018901
HTML5 defines space characters as [ \r\n\t\f] in
http://www.whatwg.org/specs/web-apps/current-work/multipage/common-microsyntaxes.html#space-character.
It treats these specially in a few contexts. As an example, the foster
parenting algorithm does not apply to space characters.
As a result, this change fixes the round-tripping of spaces between table
tags, which were previously moved before the table.
Change-Id: I32ab29275a9f824fc66d8286638eb42748cfc9a5
from Parsoid HTML output as well as VE HTML output. There are still
some newline related failures from parser tests that needs fixing, but
this is getting close. So committing for now so other eyes can make the
bugs shallow :).
Change-Id: Ia6a218ee9fb3e18fe0573c89ff3a4236779e1e64
- Check if href for links has the wgScriptPath prefix before
attempting to strip it from the href.
Change-Id: I844151ef7317476668d1306b96a2aec5a56fd0f1
- Something like this:
<ul><li>1</li><li>2<ul><li>2.1</li><li>2.2<ul><li>2.2.1</li><li>2.2.2</li></ul></li><li>2.3</li></ul></li><li>3</li></ul>
now serializes properly to:
*1
*2
**2.1
**2.2
***2.2.1
***2.2.2
**2.3
*3
So does this form which is what the above wikitext parses to:
<ul><li>1
</li><li>2
<ul><li>2.1
</li><li>2.2
<ul><li>2.2.1
</li><li>2.2.2
</li></ul></li><li>2.3
</li></ul></li><li>3
</li></ul>
- Lists (and nested lists) are not entirely newline-insensitive.
They still depend on newlines *between* lists. The opening
<ul> tag for non-nested lists should always start on a new line.
So, for example,
<ul><li>foo</li></ul><ul><li>bar</li></ul>
will serialize to:
*foo
*bar
which is incorrect. But,
<ul><li>foo</li></ul>
<ul><li>bar</li></ul>
will correctly serialize to:
*foo
*bar
Change-Id: I13a0290368574865957bcf57aebab488fbbb7026
- More pieces are now simplified and all(?) newline handling
is now centralized in the serializeToken function.
- This commit fixes bugs in rt-ing some code snippets
----------
Ex 1: foo<p>bar</p>baz
----------
- This commit fixes bugs serializing VE generated html
----------
Ex 2: <p>foo</p><pre>bar</pre> ==> foo\n bar
----------
- But, this round of fixes introduces RT failures for certain
code examples in parserTests.txt. In all these failing cases,
inline text/html is embedded within a generated <p> tag during
parsing. If these generated <p> tags can have a "gc:1" attribute
added to them, we can properly serialize them to the original
form.
----------
Ex 3: foo<pre>bar</pre>
Parsed HTML: <p>foo</p><pre>bar</pre>
----------
Note how this parsed HTML is identical to what the VE outputs
in Example 2 above. So, without the gc:1 attribute, we now
have conflicting requirements on the example same HTML.
This increases confidence in the correctness of my commit here.
Change-Id: I86beadec91c445a7f8a6d36a639b406697daa0a2
- Eliminated newline handling from several places in code and
mostly isolated it to serializeToken thus simplifying newline
handling logic.
- Fixing some bugs in the process: # of green roundtrip tests
went up by 5 (294 --> 299) but actually introduced failures on
a few originally succeeding tests (additional leading/trailing
newlines on the entire test output).
- Added bonus: made list serializing (mostly) insensitive to
newlines between tags. So, all the following DOM serialize
identically to the following wikitext:
*foo
*bar
----------
<ul><li>foo</li><li>bar</li></ul>
----------
<ul>
<li>foo</li>
<li>bar</li>
</ul>
----------
<ul>
<li>
foo
</li>
<li>
bar</li>
</ul>
----------
Change-Id: I76be56c4b2789039dff5f47de4659746882e45d6
HTML5 defines space characters as [ \r\n\t\f] in
http://www.whatwg.org/specs/web-apps/current-work/multipage/common-microsyntaxes.html#space-character.
It treats these specially in a few contexts. As an example, the foster
parenting algorithm does not apply to space characters.
As a result, this change fixes the round-tripping of spaces between table
tags, which were previously moved before the table.
Change-Id: I32ab29275a9f824fc66d8286638eb42748cfc9a5
from Parsoid HTML output as well as VE HTML output. There are still
some newline related failures from parser tests that needs fixing, but
this is getting close. So committing for now so other eyes can make the
bugs shallow :).
Change-Id: Ia6a218ee9fb3e18fe0573c89ff3a4236779e1e64
* Parsoid outputs bare newlines after a heading unless it's followed by
a <p>, so strip leading and trailing newlines in all bare text
* Adding a leading newline in <p>s is only needed if preceded by a
heading, don't add it otherwise
* Headings need a bare newline after them unless followed by a <p>
* Headings also need a bare newline before them if preceded by a <pre>
Change-Id: Ib02f800b26453541604e920fbb3845c51cdc6dea
This strips certain newlines added by Parsoid so they don't end up in
the linear model, and puts them back in on the way out so Parsoid
doesn't freak out and produce invalid wikitext
Change-Id: I256aaded4229c915868dc868ec6eaa1a73e00be1
I know this code is still being worked on but I felt like I should put
this in anyway, it might save the person working on it some work
Change-Id: I1535399b3798cd8de2fc5334cd1eac64b71e8821
This is needed because there are onTransact event handlers that use the
selection and expect it to be up-to-date. The previous behavior caused a
bug when pressing backspace at the end of the document, because the old
selection (at the end) was invalid in the context of the updated
document.
Change-Id: I159e37894d14d437f46495604c14804c0a13e84e
Empty annotation objects are unexpected by the rest of the data model
and cause weird breakage in the converter, resulting in inserted text
being in its own paragraph
Change-Id: I63de37c3c5e19ac650e7c7f2d1a0bfab21d45da9
I forgot to adjust a range based on this.cursor for this.adjustment .
This indirectly caused Rob to get an exception when trying to wrap
the last node in the document, because the unadjusted range was past the
end of the document.
Change-Id: If9d5b76568fae25ba2c0f405f1c4fcdd8d879e4f
- Check if href for links has the wgScriptPath prefix before
attempting to strip it from the href.
Change-Id: I844151ef7317476668d1306b96a2aec5a56fd0f1
This fixed a bug where an <h2> element with HTML attributes would be
converted to a 'heading' element with those HTML attributes but without
the 'level' attribute, which indirectly caused an exception somewhere in
ve.ce
Change-Id: I8bf32ff0d8e0f9d016b2abc6cb31824df05bdfc2
- Something like this:
<ul><li>1</li><li>2<ul><li>2.1</li><li>2.2<ul><li>2.2.1</li><li>2.2.2</li></ul></li><li>2.3</li></ul></li><li>3</li></ul>
now serializes properly to:
*1
*2
**2.1
**2.2
***2.2.1
***2.2.2
**2.3
*3
So does this form which is what the above wikitext parses to:
<ul><li>1
</li><li>2
<ul><li>2.1
</li><li>2.2
<ul><li>2.2.1
</li><li>2.2.2
</li></ul></li><li>2.3
</li></ul></li><li>3
</li></ul>
- Lists (and nested lists) are not entirely newline-insensitive.
They still depend on newlines *between* lists. The opening
<ul> tag for non-nested lists should always start on a new line.
So, for example,
<ul><li>foo</li></ul><ul><li>bar</li></ul>
will serialize to:
*foo
*bar
which is incorrect. But,
<ul><li>foo</li></ul>
<ul><li>bar</li></ul>
will correctly serialize to:
*foo
*bar
Change-Id: I13a0290368574865957bcf57aebab488fbbb7026
- More pieces are now simplified and all(?) newline handling
is now centralized in the serializeToken function.
- This commit fixes bugs in rt-ing some code snippets
----------
Ex 1: foo<p>bar</p>baz
----------
- This commit fixes bugs serializing VE generated html
----------
Ex 2: <p>foo</p><pre>bar</pre> ==> foo\n bar
----------
- But, this round of fixes introduces RT failures for certain
code examples in parserTests.txt. In all these failing cases,
inline text/html is embedded within a generated <p> tag during
parsing. If these generated <p> tags can have a "gc:1" attribute
added to them, we can properly serialize them to the original
form.
----------
Ex 3: foo<pre>bar</pre>
Parsed HTML: <p>foo</p><pre>bar</pre>
----------
Note how this parsed HTML is identical to what the VE outputs
in Example 2 above. So, without the gc:1 attribute, we now
have conflicting requirements on the example same HTML.
This increases confidence in the correctness of my commit here.
Change-Id: I86beadec91c445a7f8a6d36a639b406697daa0a2
Previously, data-mw-gc (generated content) elements were unconditionally
converted to alienInline nodes, and unrecognized elements were
unconditionally converted to alienBlock nodes. This is wrong and
produced weird results when I started experimenting with <code> tags.
Instead, I made both gc and unknown element trigger alienation, but the
decision of whether we generate an alienInline or an alienBlock node is
separate and is based only on whether we're inside a content node.
Change-Id: I12335337c3fa60c725ae7bcfbfb52a1dda153fb5
- Eliminated newline handling from several places in code and
mostly isolated it to serializeToken thus simplifying newline
handling logic.
- Fixing some bugs in the process: # of green roundtrip tests
went up by 5 (294 --> 299) but actually introduced failures on
a few originally succeeding tests (additional leading/trailing
newlines on the entire test output).
- Added bonus: made list serializing (mostly) insensitive to
newlines between tags. So, all the following DOM serialize
identically to the following wikitext:
*foo
*bar
----------
<ul><li>foo</li><li>bar</li></ul>
----------
<ul>
<li>foo</li>
<li>bar</li>
</ul>
----------
<ul>
<li>
foo
</li>
<li>
bar</li>
</ul>
----------
Change-Id: I76be56c4b2789039dff5f47de4659746882e45d6
Parsoid ignores sHref when converting back to wikitext, so we have to
set the href attribute to "/$title"
Change-Id: I1068116c0be72197619d0df3b4d1231a3879fa14
And made it not start on it's own, but be started by ve.Surface - this makes it so it's not polling in the unit tests, for instance
Change-Id: I940df04d392fd134d18847949efe0e2232328323
* As part of an earlier fix, I had changed default value of 'res'
to null instead of ''. But, this was potentially buggy because
the previous check was (res !== '') which could be triggered
by return values of handlers. By changing the check to null,
I was effectively changing the code paths for those handlers that
returned ''.
Change-Id: I2302023be7422ce4fb384ff5a50fe53fa7732855
Fix Inspector bug which prevented applying a link annotation to data
already containing annotations.
Change-Id: I6f315d50805c8c71f2155f955ea5674a7ce98656
This was causing some issues when you started typing there, and it's not clear that the white text is really the best way to go anyways
Change-Id: I8a9d6571ea204603729e96b7ff77184279a31a95
The offset map was broken from the start because it wasn't updated when
adjusting the length of a text node, and if we fix that bug it's
doubtful whether the costs of updating the offset map outweigh the
benefits, especially considering that adjusting the length of a text
node is something we do for almost every keypress. If it turns out
having an offset map does make sense, we can always reintroduce it
later.
Change-Id: I59e8bc154f7d07aa1bab2f473c13ff466d0e463f
paragraphs in lists.
* We need to look at other special-case handling requirements of
html tags in lists (and other contexts like tables).
Change-Id: I84b8402d90a186c9075c2d45263c94377312927a
Parsoid outputs rel="mw:wikiLink" or rel="mw:extLink", so we convert that to link/wikiLink and wiki/extLink respectively.
Also preserve the data-mw attribute; we probably need to do this more generally but this'll do for now.
Change-Id: I32e570bffa5a73a733a120d52cfd8b75d3191e02
Adding a 'style' attribute which is set to either 'data' or 'header'
This breaks even more tests because of missing style attributes
Change-Id: I0a75d8c1578b4414eeae8c484f6c4d6f8a59472a
When a text node was closed (either by encountering a non-text node or by reaching the end of the document), the fixupStack would not be processed. This led to attributes being dropped when nodes were split because of text being inserted into them
Change-Id: I41f6d20e0c1bfc8d8689b7e6325e724dd8156ab1
* Add code to handle elements and annotations
* Drop support for aliens from getDomElementFromDataElement() and move it into getDomFromData()
* Implement getDomElementFromDataAnnotation()
* Document a few functions
Change-Id: Ic6a418cbf9d7d1ad96299d7d3633970a876c6103
it does not include Alien nodes and Images - because we are not going to
support them for June release).
Change-Id: I229e4b5f2881714252699f23aef164655fa8bcf6
* Moved wikipedia default prefixes to environment
* Added 'addInterwiki' method
* Adjusted link handling normalizeTitle to reflect this
Change-Id: If5b2314cc36346b6da8649ed410457a612d80a22
* mw:Foo now loads pages from mediawiki.org
* The default prefix still is 'en'. You can switch this to 'mw' in ParserService.js.
Change-Id: I1208667e6114bd711b7988a8b3adb32ffab70969
- Three bugs that were messing up quote transformations.
- Now, the following cases are handled properly:
* ''foo'''
* '''foo''
* ''foo''''
* ''''foo''
These tests (and other quote tests) have to be added to core parser
tests file.
- One more parser test green.
Change-Id: I4f93e8910639f546bfc9304becab17d26d5529de
Created new document method to determine if a specific annotation
object is inside an annotation array.
Change-Id: Id645929cbf31030b8b0fcacb8dfb36e61aaad129
This fixes a bug where the second replace operation in a transaction
would cause the rebuild of the wrong range, or the adjustment of the
wrong text node.
Change-Id: I9b1c68d84999d538fe10bb193f4dfdd694121d2a
This is needed to make the results of certain transactions' tree sync
round-trip cleanly through the ve.dm.Document constructor
Change-Id: I2ab0758ec6bd7afba5b6645c7330f9fa2d45205d
new static method looks for annotation in annotation object.
ve2 Cleanup on annotate method and surface model.
Partially revive UI tools by exchaning old method usage
for ve2 methods.
Change-Id: Id0ac58330292d76801bbcf1d71a919b493f8ab9e
An improvement, but there still are some extra newlines inserted after
paragraphs. Example input:
-------
Foo:
{|
|foo
|}
-------
Extra newlines are inserted after the Foo: and the foo in the table. They are
not fed as tokens or text to the tree builder, so there is likely a bug in the
html5 library or JSDom.
Change-Id: I83eb6180e3cd1c4e7f9b15b31d339e1d32bccd3f
* Possibly more efficient under heavy GC load -- untested.
* No change in time and memory use for single file parsing.
Change-Id: Id2f3f65cc0e5f38ed968bbda60b97e46523e700e
* Moved the tail attribute to the second attribute (a bit cleaner)
* Disallowed newlines in the tail production
* Improved the selection of round-tripped href vs. generated content vs. href
in the serializer
* renamed state.linkTail to state.dropTail
Change-Id: I5d98c704b6ea566011e22237786f8da17548570f
Pages titles with a wikipedia interwiki prefix now load the page from
corresponding Wikipedia. Links in a page then stay within the given language.
Note that Parsoid currently makes no effort to recognize localized namespaces,
so it won't render media files, categories etc correctly.
Change-Id: I7bc4102e81a402772ea23231170734d580ea15b9
Functional changes (fixes):
* Make writeElement() also update parentNode and parentType for openings
* Also add to fixupStack when opening a wrapper for a text node
Non-functional changes (cleanup&docs):
* Document all variables at the beginning of the function
* Group variables according to where/how they're used
* Move expectedType into writeElement()
* Kill node, duplicates parentNode unnecessarily
* Kill paragraphOpened, was misnamed and unnecessary
* Rename closedElements to reopenElements
Change-Id: Ie5b4e4f30b267943048fdc170accb29139039192
* Push entire elements onto openingStack rather than type strings
* When closing an element, build a clone of the opening and push it onto
closedElements, then insert that clone when reopening the element
Change-Id: I8b0fb44394aed6c471dc6dacaab03e44c2333733
* Don't explicitly add the newline in the pre, as we preserve newline tokens
now. This avoids doubling of newlines when round-tripping.
* Use the sHref attribute even if the href contains spaces.
Change-Id: I8bec8fbfd6a7836bf2e5eec20869a0edd95c93b6
Lists interrupted by non-empty lines would not close the list properly.
Register for any token instead of just for newlines and close the list if no
listItem follows the newline.
Change-Id: I1743901e3db541bbeda78d17707db943e6ceb9b9
If the href would not denormalize, add a copy of the original href in data-mw
and use it to preserve non-conventional capitalization etc.
Change-Id: Ifef50eec7343b0e6b0ba66b6d19a8a3e8c9f8001
A tail containing regexp syntax (a ? in [[:en:Main Page]]) would crash the
serializer. Use substr instead.
Change-Id: I8519aec9c07dfe31893d676b1c936a42d2af74a0
- Added a tail json attribute for wikiLinks
- During serialization, this attribute is used to strip the tail from
the link target and render it after the link
[[hen]]s ==> <a ... data-mw="{gc:1, tail: 's'}" ...>hens</a>
==> [[hen]]s
- 2 more roundtrip tests green
Change-Id: I84f3dabaf0271f7a67641a00148467daa8310eb0
This allows us to check the watchlist checkbox on save dialog.
Added watchlist toggling to ve save api.
Added some i18n messages to core integration.
Change-Id: Ibed8edb2c59ad49e1738c937c3bea518238d0845
* The state of syntax stops is now properly included in the cache key for the
tokenizer-internal backtracking cache. This fixes some mis-parses when
re-parsing a bit of text with different flags.
* Clear the backtracking cache after each toplevelblock. This drops the peak
memory usage when expanding [[:en:Barack Obama]] from ~380M to ~110M.
Change-Id: Icdb879cae5907e4595903dd6acba2e686e8c2e4b
