Commit graph

141 commits

Author SHA1 Message Date
Arlo Breault 131718460b Add a function to create the references node uniformly
Change-Id: I58fd7ea7c692b1a38dab160083d3d9a529527fb0
2018-01-10 00:22:00 +00:00
Subramanya Sastry 0ec01e4ca6 Linter: Fix crasher in Cite - don't assume well-formedness
Change-Id: Ie5da61934b91f115b0eea995e05de6c231d77bdb
2018-01-08 16:05:40 -06:00
Arlo Breault 03bf14b003 Restrict nested refs not coming from templates
This better matches the behaviour of the php extension.  An instructive
example is,

<ref>123 {{#tag:ref|321}}</ref>

{{#tag:ref|456 <ref>654</ref>}}

The php implementation only parses the ref contents when producing the
output for reference tags, at which point, nested refs will be ignored,
as in the former case.  The "321" is dropped.

The latter is special, since the extension tag takes precedence over the
parser function, the inner ref will already have been processed when the
outer is added to the stack, and hence the nesting is permitted.  This
is why the inner ref precedes the outer in the references list (has a
lower number).  Unfortunately, Parsoid doesn't yet get that ordering
right.

Change-Id: Ieb0e418cca634605c2a9f1487139b15095f38d81
2018-01-08 19:49:47 +00:00
Subramanya Sastry 288feddbc6 Linter: Use extension mechanism to provide ext-specific lint handers
* <ref> content ends up in references section. The original <ref>
  site only has a backlink to that references section.

* Given this, when the references tag itself comes from a template,
  this throws off the linter attribution code. It assigns the
  references template's DSR to the lints from the <ref> tags.

* A clean way of dealing with this would be to treat extension
  content as its own independent document (which we are already doing
  currently in our wt2html code path) and asking the extension to process
  its content while giving it the handler for linting an individual node.

* This patch implements special handlers for <ref> and <references> content
  in the Cite extension and registers them.

* Move cite-specific lint handling code to the Cite extension.

* New mocha tests spec this behavior.

Change-Id: Ib57f0303605a73408333e133f8051be0b8d45d69
2017-12-21 17:43:37 -06:00
C. Scott Ananian 228325cb09 Update Parsoid to generate modern HTML5 IDs w/ legacy fallback
The citation portion contains fixes for []-encoding in attributes,
based on Iec3439f76ecc2a3543b30b35f8735c92b0cfb711.

Fix some other no- or double-encoding issues with HTML entities in the
process, based on I88e8e2077e6f5eec2b232391f7818370894a62dc
(T103714/T104196).

Unrelated parsertest fix which needed an id fix -- incorrect id
added by the section-tag patches exposed here.

Bug: T152540
Bug: T103714
Change-Id: I12b2a148f7170d20bd9aacd3b5b8ee1965859592
2017-11-30 16:14:33 -06:00
Arlo Breault d188af60d0 Fix processing ref in ref
* Follow up to f7594328

 * Fixes the regression found in rt:

   node bin/roundtrip-test.js --domain ru.wikipedia.org "Феодосия"

 * Simple test case (do we really not have one!?):

   test <ref>haha{{#tag:ref|ok}}</ref>

Change-Id: Ie0a53a8e885a6d94769034cce4ef432773635842
2017-10-26 21:35:01 -04:00
Arlo Breault f75943282f Avoid serialize/parse of mw:dom-fragment-token content
* The contents of "mw:dom-fragment-token"s was being serialized
   after processing to the DOM and stored on the token to be
   shuttled through tree building.  Only to be reparsed in the
   unpacking phase.

 * Here we store a pointer to the contents in a fragment map.

 * Doing less work results in a performance improvement, though
   only slightly because the content still needs to be adopted
   by the main document.

Change-Id: Ia0aec7de469101a2a93342ea89daac0f0e73cf1a
2017-10-25 22:41:18 -04:00
Arlo Breault 84c80a7ed2 T133334: Ref marker in caption in data-mw
* Replace ref markers instead of waiting for cleanup to remove them
   since that doesn't happen on embedded html.

Change-Id: Ied746f025a0ac7f14d922aff6640fef3aa4b55b0
2017-10-26 00:12:27 +00:00
Arlo Breault 27799ea9b5 There's only one refInRefProcessor
* Avoid the unnecessary binding.

Change-Id: I80e558a86eb7197ce39a07062499ec58f170a3af
2017-10-26 00:06:20 +00:00
Arlo Breault 23db74cd9c Remove unnecessary serialize/parse of references content
* Also, change the input parameter of buildDOMFragmentTokens
   to uniformly accepts a <body>, rather than a doc or string,
   so that we can pass it nodes from our dummy document.

Change-Id: I4bb44573fe7203277d51e804a4a6423100a34f03
2017-10-25 23:52:19 +00:00
C. Scott Ananian fc4967ac5f Remove unnecessary calls to Node#childNodes.length
These calls force us to allocate a backing array for childNodes,
deoptimizing a linked list representation.  Use the standard
Node#hasChildNodes() method instead, which can be efficiently
implemented without allocating a backing array.

Change-Id: I1706bfa15263564bc981d689947835a3d0d4a68f
2017-10-24 23:04:21 +00:00
Arlo Breault 3aa2e1a106 Bump version to 0.8.0 for release
Change-Id: I2279cbd8d56d3cd097a086b4f52741234a014346
2017-10-23 17:37:34 -04:00
Arlo Breault 0dba785946 T176425: Transfer about group to reference wrapper
* Follow up to 74acc71e

Change-Id: Ia403bf88dd48702dbbbf53a247ea2714243a0acc
2017-09-25 12:59:46 -04:00
Arlo Breault ab0c40f496 Fix crasher from empty references list
* See on /en.wikipedia.org/v3/page/html/Manuela_Maleeva/800363347

 * Folow up to 74acc71e

Change-Id: Iae89d4b0073dafce5fe4c28dc94fd6d5779d71c6
2017-09-13 22:35:37 +00:00
Arlo Breault 74acc71e98 T159894: Add support for Cite's responsive parameter
* Ports commit 04c3ad01 in core's Cite extension.

 * Until $wgCiteResponsiveReferences is exported, a number of wikis
   which have enabled this won't be getting the right default.
   https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L14945-L14970

   However, it looks like enwiki's {{reflist}} explicitly sets the
   parameter either way, so not a bad start.

 * The "ext.cite.styles" css resource is added from core's Cite
   extension 05cb5cc1, since that's where the responsive css lives.

 * The blacklist changes for existing tests are because we now only
   serialize the children of the div wrappers.  Those tests probably
   deserve Parsoid specific sections..

Depends-On: I2404999ab11b5cf7b740ae43696c4676ab1b6d22
Change-Id: I8f9277b3ecb253e0bee7ee55eef7af4935821527
2017-09-13 21:24:52 +00:00
Arlo Breault ff09c1667d Add 'references' class to reflist node
* Matches the core Cite extension commit cec36fed

 * This change means we can get rid of some of wiki specific styling in,
   https://github.com/wikimedia/integration-visualdiff/blob/master/lib/parsoid.custom_styles.yaml

Change-Id: I763b8086d057ea0cac8ba8fd4c9f4f4279766fed
2017-09-11 15:13:55 -04:00
Arlo Breault 6e5b366ff6 [eslint] Enforce lines-around-directive
* Done with --fix

Change-Id: Ie9b34c69615493870427be5f318a14f31a361d6a
2017-04-27 16:05:00 -04:00
Arlo Breault d13cfa4f71 [eslint] Use eslint-config-node-services
* Lots of overrides here that we should either fix or be explicit about
   wanting to diverge and maybe talk about upstreaming the change.

Change-Id: I927dd325e49ef72ccbe5e8b926f7984b82dd0f2e
2017-04-27 16:04:53 -04:00
Arlo Breault 863e53ff8e T163744: Switch to eslint
* .eslintrc was generated from .jshintrc and .jscsrc using polyjuice
   and some manual tweaking.

 * Forced to follow the convention in eslint #4174

 * However, in a follow up, we'll use wikimedia/eslint-config-node-services

Change-Id: I2500a6520a5c9f41d5333e937c151228aec88be0
2017-04-27 14:00:51 -04:00
Arlo Breault dea21633ba T64270: Support video and audio content
* Spec'd at https://www.mediawiki.org/wiki/Specs/HTML/1.3.0#Audio.2FVideo_.28Proposal.29

Depends-On: Icb26ffb802150c1d1f8c6e0aab05b25e503077e2
Change-Id: I53c6fa8d862c152bb1c8dd3ae96f4ee464a60d32
2017-04-21 20:44:44 +00:00
Subramanya Sastry 4b242825cf Update HISTORY.md + bump version numbers to 0.7.1 for deb release
Change-Id: I32cf92c566dc7d501ec0b28552ef6a4fbb879d82
2017-04-05 11:13:30 -05:00
Subramanya Sastry d4140cb46f Update HISTORY.md + bump version numbers to 0.7.0 for deb release
Change-Id: I46189fae2dc0b01b473c471a8a62d2f6204f3692
2017-04-03 17:25:07 -05:00
Subramanya Sastry 2368fba08f Escape cite ids with Sanitizer.escapeId
* Without this, refs like <ref name=":0"> won't generate the same
  links that the PHP parser generates.

* Updated an existing test to add the :0 key that require escapeId
  to be encoded properly.

Change-Id: I69e5f16ccf64bd1c9cf05bdea7a379e679d36b1a
2016-12-20 15:07:07 -06:00
Subramanya Sastry 18c965dec0 T102134: Fix cite hrefs to render properly
* With Parsoid's base href pointing to the wiki, plain #-fragment
  links won't resolve properly. Add the page title to the href
  for the links to start resolving properly again.

* Updated parser tests accordingly.

Change-Id: I280c41a0382bd2acd82cc586212695aa3b920171
2016-12-20 15:07:07 -06:00
jenkins-bot 0b1277434f Merge "T104662: Allow nested ref tags only in templates" 2016-12-12 15:15:10 +00:00
Arlo Breault ea599bb074 Let extensions supply the pp tracing name
* They won't all have tags, like the linter.

Change-Id: Idf309a82bf229219d98d693302c305726fa25fcd
2016-12-09 19:03:35 -08:00
Arlo Breault 439bc3af2d T104662: Allow nested ref tags only in templates
* To support the #tag parser function.

 * An aside, #tag is nutty,

     {{#tag:nowiki|test<ref>haha</ref>}}

Change-Id: I0ab93f6ae959b30a887e967952c904ef0400b189
2016-12-08 10:33:00 -08:00
Subramanya Sastry a16445812f Bump version to 0.6.1 (0.6.0 deb pkg was botched)
Change-Id: I601f31c79d8c2361ca33a6d0e6d6196c1b6c5f0f
2016-11-14 16:44:35 -06:00
Subramanya Sastry c8a0ec200a Bump version to 0.6.0 for new deb release + update HISTORY.md
* Updated HISTORY file based on Parsoid deployment logs and
  added only the most pertinent entries.

Change-Id: If07bc75fcb09507cae30fc82d7d89b8d87a4c69b
2016-11-07 16:30:16 -06:00
Arlo Breault 8c60ef0f79 Let native extensions add stylesheets
Change-Id: Ib418544236ddf2d9a075934d72712734d90b4537
2016-09-12 17:09:05 -07:00
Arlo Breault e9383e0877 Use tagWidths for content in Util.getArgInfo
* Use nestedRefsHTML.length to determine when we need a dataMw.body,
   the regexps there were unnecessary.

 * Pulled out of https://gerrit.wikimedia.org/r/#/c/264026/

Change-Id: I4602594e468322c8b6b8653eee33047ef9af9ebc
2016-08-31 13:33:03 -07:00
Arlo Breault 419db6b794 Bump version after release
Change-Id: Iaed8279cb06c0c7b55940b1e4aed3f8b21262e79
2016-06-14 11:18:15 -07:00
Arlo Breault 53c9863382 Keep the data-* attributes at the edges of the DOM
* FIXME: stop skipping jsapi tests

 * DU.serializeNode is renamed to DU.toXML, to avoid confusion with
   wikitext serialization (and a name clash when grepping).

 * DU.serializeToXML is renamed to DU.ppToXML, to distinguish it both in
   name and use from DU.toXML.  This is a special xml serializer that's
   meant to be used in Parsoid's DOM post-processing phase.  It's aware
   of a node's .dataobject and automatically transfers that to json
   stringified attributes before serializing a node.

 * DU.ppToDOM is added as helper for loading attributes when parsing
   html, as in DU.parseHTML.  It's the converse to DU.ppToXML

 * Diff markers are no longer stored as json stringified attributes on
   nodes while serializing.  Only when dom dumping, and even then only
   on clones.

 * Once T100856 is fixed, we can look into discarding data-parsoid from
   html that's stored in data-mw (ie. captions, references, etc.)

 * Filed T133334 to note that some of the html we're stuffing in data-mw
   needs further processing, since we're storing ref markers.

Bug: T91700
Change-Id: Ia1a60951e6292b5eb073eedca0b79938094809d2
2016-04-28 18:49:22 -04:00
C. Scott Ananian 8a5316e0c6 Add autoload mechanism for user extension code.
This parallels the autoload mechanism in mediawiki core.  In fact, small
wikis can use the same extensions directory for both core and Parsoid.

We also export a very basic "extension API" for external extensions
and convert our current "native extensions" to use this to show how it
is to be done.

Bug: T133320
Change-Id: I8e05d5bfdff873f28a58dead68aaca0e4823cf32
2016-04-26 22:25:59 -04:00
Arlo Breault 87296241d5 Rename DU.serializeChilden to DU.serializeToXML
* And use it where appropriate.  This has the effect of reducing the
   dom size because of the use of smartQuote (see all the parserTests /
   blacklist changes).

 * The serializeChildren name clashes with the one in the
   serializeState, causing confusion and making grepping harder.

 * Bonus: removed some dsr info in parserTests and fixed a html2wt typo
   that was prevented a test from running.

Change-Id: I287f1efd4afe06b158844e0de9f16494d4a89f93
2016-04-12 14:01:04 -07:00
Arlo Breault 126ed00d40 Suppress errors raised when getting debugging info
* An example of an href from eswiki/Kate_Gosselin?oldid=90347467
   is <a href="#cite_note-13.3F_It_Can't_Be!-3">
   which domino should do a better job validating, instead of throwing
   "Object [object Object] has no method '''".

Change-Id: Ie4bd702d7eee23ad021f2ad31b47cfbada947cd9
2016-04-08 19:36:01 -07:00
Arlo Breault df00a42c5e Fix log type
* Broken in ae45ffe9

Change-Id: Ife2a42e8a805e68fb4bfe83076e39b64be6dbf71
2016-03-30 18:22:07 -07:00
Subramanya Sastry 53117cef54 T113331: Move auto-generated refs flag from data-parsoid to data-mw
VisualEditor requires this info and presumably other clients might
and this is, in reality, semantic information about wikitext.

The html2html failure is because:
* even when the references html has been autogenerated,
  in non-rt-testing html2wt mode, we always generate the
  <references /> tag.
* we moved the flag from data-parsoid to data-mw but ignore changes
  to data-parsoid while comparing test output.
* So, in html -> wt mode, <references /> is always generated
  => data-mw.autoGenerated flag doesn't show up in the wt -> html
     phase of the (html -> wt -> html) test
  => the html2html test fails.

Change-Id: I8e79f2a436a1ca276b9351228a3d8f02d7ebd0c6
2016-03-24 23:17:34 -05:00
Arlo Breault 3b44e7e544 Remove extsrc bloat from references data-mw
* This hasn't been used since 2013! See 46ddbf3f

Change-Id: Iea382829f38788541b3bc4f8595a0d6bf600c167
2016-02-05 18:26:33 -08:00
Arlo Breault ae45ffe95e Move extension content serialization to cite
* See the follow up for why references extsrc is irrelevant.

Change-Id: I1db0f3c3f3e85706125418bb431aa40c17a97407
2016-02-05 18:26:22 -08:00
Arlo Breault 41756cb70b Flatten ext/
Change-Id: I083dc0d7c5ab37c8f0f5c051e82e4705c891812d
2016-01-13 15:34:40 -08:00
Arlo Breault 1c10f71b47 Move processRefs.js to ext/cite/
Change-Id: I10ab294900f090dcdb85b205d9eb0472fd3e9157
2015-10-28 15:31:53 -07:00
Subramanya Sastry 05bf16d3eb T115665: Reorg parsoid repo
* First pass moving around files into different directories.

* Renamed files to remove unnecessary prefixes or align the name
  closer to what the file contains.

* Added temporary soft links to bin/parse.js and bin/roundtrip-test.js
  in the tests/ directory since jenkins jobs seems to have hardcoded
  refs to those paths.

* Deleted:
  - a couple of stale scripts in tests/ that are no longer relevant.
  - a couple of state scripts in api/ that didn't look relevant.
  - swagger spec since it was incomplete, stale, and unmaintained.

Change-Id: I97c30467b190b417eec9e750238704330ae91137
2015-10-27 12:50:47 -05:00
Subramanya Sastry 381ceda257 T93580: Handle <ref>s in inline image captions
Inline images don't display their captions and don't have a
DOM structure for them. They are hidden away in the data-mw
attribute.

So far, we weren't handling <ref>s embedded in these inline
image captions -- both causing rendering diffs in the references
section and crashing the serializer when these nodes were
encountered.

This patch fixes that and also adds parser tests.
The html2html failure added to the blacklist is because of
about-id mismatches in the embedded html. We need smarter
normalization in parserTests.js to remove these false failures.

There is also a long comment about generalizing how this kind
of scenario is handled. Worth tackling that in a separate patch.
In order to simplify and generalize the adhoc nature of
trace output that is currently in place, I had been pondering
creating a DOMProcessor class with state to do that, and
looks like something like will help with this as well.
For this patch, I am leaving behind the special case handling.
Adding state and generalizing how embedded HTML is processed
should be done separately since that requires more thinking
and experimentation.

enwiki/High_Laver?oldid=659441291 and hiwiki/मुक्केबाज़ी?oldid=2689792
now RT without crashers.

Change-Id: I39854c7b5b3e8d7cce84b1b4e05213185f8cccb0
2015-09-04 15:07:41 -05:00
Subramanya Sastry 37206e2a12 Save data-attribs in DOMs of nested refs
* 4ea8dbd8 exposed a bug handling of nested <refs> (done via
  templates). It caused crashers in rt-testing of a few pages.
  Ex: enwiki/António de Oliveira Salazar?oldid=676623209

* Reproducible with the following wikitext:
-------------------------------------
x <ref>y {{sfn|Kay|1970|pp=123}}</ref>
<references />
-------------------------------------

* This patch fixes the problem by making sure data-mw and
  data-parsoid of the DOM for the nested ref are saved before
  serialization to a string.

* Removed the saveHandler from dom.t.unpackDOMFragments.js and
  reused the simpler dom-walk code added as part of this patch.

* Also updated comments in ext.Cite.js

* Hard to introduce nested <ref> in parser tests, so no
  new tests for this scenario.

Change-Id: I2298bbe87ccddd87f307d206d77d78fcfb0d8a75
2015-08-18 13:43:50 -05:00
Arlo Breault 7b901bc99b [jscs] Enforce requireTrailingComma
Change-Id: Ib8dde69fec221b970c80483b25c370c3c7067105
2015-07-28 14:58:33 -07:00
Arlo Breault 3f8115937a [jscs] Enforce disallowSpacesInsideParentheses
* Change generated with `jscs -x .`

 * Well, except in the tokenizer.

Change-Id: Ia516d0d150255c764e5c023274741c520cc9dce7
2015-07-23 12:06:38 -07:00
Marc Ordinas i Llopis 7661d25fdb Use CSS to style Cite references
Output a default style of reference numbers and text for <ref>s and
<references>, use CSS content generation in supported browsers to render
per-language style.

Right now it provides pixel-perfect renditions of refs in enwiki and
eswiki, and initial support for fawiki (which can't be tested with current
phantomjs). The CSS is loaded from the new module 'ext.cite.style',
implemented in the Cite.php extension.

Further changes to Cite.php will be necessary to change it to use this
same CSS instead of system messages.

Test results (and two blacklisted results) changed due to new HTML
structure.

Related to bugs T51538, T45235 and T73803.

Bug: T86782
Change-Id: I21fbbd3247bf7801e5ef9bd5312f95777f4dd6ef
2015-06-29 15:59:32 -05:00
Arlo Breault c74d64c2d7 Provide section-offsets for immediate children of <body>
* Changes xmlserializer.serializeToString and DU.serializeNode to return
   an object with the str and, if options.captureOffsets, the html offsets
   of top-level nodes.

 * Added mocha tests to capture expectations about offsets computation.

Bug: T96279
Change-Id: Id74988d3ef39078fbfea72359884b75290da041b
2015-05-14 11:50:16 -05:00
C. Scott Ananian 4ca968a95c Remove vestigial reset() methods in Cite.
Cite is now stateless and should remain so!

Change-Id: Ib90cb1da8cb809f6fec9410ff6fb00c32e276dcf
2015-04-30 16:34:34 -04:00
Arlo Breault 4c4f5ce368 [jscs] Enforce disallowMultipleVarDecl
* Continues on 3311936a5e15c2080d8040dd605e58fa33407e60 and
   05a8947fef30e6fc4519b822b0931d965d57efa8

 * With some whitespace cleanup.

Change-Id: I32109c6fd35439914df4df06432376d6a968268b
2015-04-22 11:18:38 -07:00
Arlo Breault 61053dbcc7 [jscs] Enforce requireSpaceAfterBinaryOperators
* This was auto-corrected with jscs -x .

Change-Id: Idee8a562ae2c29cc5c661532b016220be3c107aa
2015-04-08 00:07:38 -07:00
Arlo Breault 4c9ad094d9 [jscs] Enforce disallowSpacesInFunction
* This was auto-corrected with jscs -x .

Change-Id: I7110a2eb946ae91ded268d73320a9e8194ca2ab1
2015-04-07 23:54:59 -07:00
Arlo Breault 49b3a70b39 [jscs] Enforce requireSpaceBeforeBinaryOperators
* This was auto-corrected with jscs -x .

Change-Id: I13df3896e8130743eb31581e80d4c7909e48768d
2015-04-07 23:28:41 -07:00
Marc Ordinas i Llopis e060ef8fe3 T93973: Remove state from Cite extension
Keep all state in the DOM post-processor in dom.processRefs.js, so
that there's no need to reset the extension and it can be safely
used for multiple documents

Change-Id: Iacf7e3003929f74b76774d3e765c792991700e51
2015-04-01 11:22:33 +02:00
Subramanya Sastry 9e29edb03f Bug fix in processRefs DOM pass refactoring
Change-Id: Icd4ce9919de4ba50e5186b8ec3b1923ae9eeb18e
2015-03-26 11:48:09 -05:00
Subramanya Sastry 4040f5d160 Code restructuring of processRefs DOM pass.
* This lets us do reset state in the right place where it belongs.

Change-Id: I542d521733c6c1fb6d93f269604e256df7f6492d
2015-03-25 17:45:15 -05:00
Marc Ordinas i Llopis c1b71a69ab T63165: Ensure reference index is reset at the end of document
Once all the missing references have been added, ensure that the
the reference index is reset to 0.

Also related to T93715.

Change-Id: I00a2b2da9116602d2eb00e782d4832cf580e4b59
2015-03-25 11:06:38 -07:00
Marc Ordinas i Llopis dbb4e4b028 T88290: Keep separate content for multiple <ref>s with same name
Change-Id: Iae4dcdb405ce8d233af3105e19bab2b32938ed96
2015-02-27 11:40:04 -08:00
Marc Ordinas i Llopis 5a0abc13b9 T88290: Only output <ref> contents in <references>
Instead of outputting a <ref>'s HTML in both data-mw and in
<references>, output it only in the later and point to it from
data-mw.body.id.

Also preserve data-parsoid for <ref>s text in <references>, as now
that's the only representation of it.

To correctly do html2wt when there are <ref>s inside <references>
we need access to the main document DOM when serializing, so also
ensure that env.page.dom is correctly set (it was only set in v2
before).

Updated tests results and blacklist (some tests now pass).

Change-Id: I0fa7ad692585af19136909bfec39db9868b137c5
2015-02-27 03:14:53 -08:00
Subramanya Sastry 6e6d3481b6 T90309: Ensure that implicitly-added <references> output have unique ids
* Fixed parser test output of one test to add unique ids. Output for
  other parser tests modified in f528f508 still need fixing up.
  Left for a separate patch.

Change-Id: I04546c2a590930121d960239a1954b26771e9c80
2015-02-20 19:02:45 -06:00
Subramanya Sastry e3bf708b77 Fix output for implicit references for improved html2wt output
* <references /> was not appearing on its own line and was
  instead getting tacked onto previous line of wikitext output.
* Change in blacklisted wt2wt test shows that the new output
  is better.

Change-Id: Ie82401a3bc6082b733339e2456810b6b1c87529a
2015-02-17 17:51:07 +00:00
Marc Ordinas i Llopis f528f508e8 T88660: Emit reflists for <ref> with no explicit <references>
This patch emits a reflist for all ref groups that still have
<ref>s in them at the end of the document. Currently Cite.php only
does so for the default group. See also T88290.

On html2wt the missing <references> are added to the wikitext,
which makes the wikitext correct. Selser catches this if not part
of the edit.

Change tests to include an explicit <references /> tag, and add
one for explcitly testing that they do get added. This last one
has to be blacklisted as the new <references /> don't appear with
selser.

Change-Id: I79af2c34481cadbf0d68d9571928979adf559b58
2015-02-16 03:16:18 -08:00
Marc Ordinas i Llopis cd8e50acb2 [Cite] Generate the same ids for <ref>s and notes as Cite.php
One particular case is that Cite.php considers equal a name and
its encoding, i.e. "a & b" === "a &amp; b". Added a new test for
this case, but blacklisted it on html2wt, wt2wt and html2html due
to a different problem with how Parsoid encodes entities. This
will be investigated separately, as a simple fix could break
unrelated cases.

Also updated tests and blacklist to the new ids.

Change-Id: I87637a1dc812a3a8f29327b9e6c0040b22a651c4
2015-01-07 11:27:07 +01:00
Marc Ordinas i Llopis 3c82fb7787 T58916: Parse extension parameters as plain text
Also encode cite ids properly as now they can contain arbitrary
text. Change in blacklist due to this.

TODO: Investigate if it would be better to do this directly in
the tokenizer.

Change-Id: Ic112124e90d256d73a351d0d57fe3c7546fa065f
2014-12-22 14:02:05 +00:00
Arlo Breault 9e6055d547 Fix failures from v2 endpoint
* Although this resolves the crashes, I'm unsatisfied with it as a
   proper fix to the underlying issue. There are many places throughout
   the codebase where we serialize and then parse document fragments
   that should be instrumented to store and unpack data-* attributes.

Bug: T76518
Change-Id: Idca1b0a37ec924a71cb51160d000c7de9717d422
2014-12-02 16:47:32 -08:00
Subramanya Sastry 8694562ee8 (Bug 57252) Generate HTML5-compliant cite id/about attr values
* Currently, this mimics Cite.php behavior where "a b" and "a_b"
  are considered identical ids.

* Added new parser test.
* Fixed output of another test.
* Fixed section name of a commented out test.

Change-Id: I0c51404c3e659bbddfe9a8909aa6a109d368b762
2014-10-24 17:59:17 -05:00
Arlo Breault 90c5303678 (Bug 53109) Move data-mw away from manual json attr loading
* This is both faster and consistent with how we're accessing other
   parsoid attributes. It's also a step towards not having this data in
   the html output.

 * Changes to parserTests and the blacklist are for attribute order.

 * Requires upgrading domino to 1.0.18
   https://github.com/fgnass/domino/pull/48

Change-Id: I1edbc260887d480adf04763b15043c374e27cceb
2014-09-27 08:53:22 -07:00
Subramanya Sastry 6c503e8973 (Bug 64901) Fix paragraph-wrapping to match PHP parser + Tidy combo
General changes
---------------
* Replaced the hacky 'inBlockNode' parser pipeline option with
  a cleaner 'noPWrapping' option that suppresses paragraph wrapping
  in sub-pipelines (ex: recursive link content, ref tags, attribute
  content, etc.).

Changes to wt2html pipeline
---------------------------
* Fixed paragraph-wrapping code to ensure that there are no bare
  text nodes left behind, but without removing the line-based block-tag
  influences on p-wrapping. Some simplifications as well.

  TODO: There are still some discrepancies around <blockquote>
  p-wrapping behavior. These will be investigated and addressed
  in a future patch.

* Fixed foster parenting code to ensure that fostered content is
  added in p-tags where necessary rather than span-tags.

Changes to html2wt/selser pipeline
----------------------------------
* Fixed DOMDiff to tag mw:DiffMarker nodes with a is-block-node
  attribute when the deleted node is a block node. This is used
  during selective serialization to discard original separators
  between adjacent p-nodes if either of their neighbors is a
  deleted block node.

* Fixed serialization to account for changes to p-wrapping.
  - Updated tag handlers for the <p> tag.
  - Updated separator handling to deal with deleted block tags
    and their influence on separators around adjacent p-tags.
  - Updated selser output code to test whether a deleted block
    tag forces nowiki escaping on unedited content from adjacent
    p-tags.

Changes to parser tests / test setup
------------------------------------
* Tweaked selser test generation to ensure that text nodes are always
  inserted in p-wrappers where necessary.

* Updated parser test output for several tests to introduce p-tags
  instead of span-tags or missing p-tags, add html/parsoid section,
  or in one case, add missing HTML output.

Parser Test Result changes
--------------------------
Newly passing
- 12 wt2html
- 1 wt2wt
- 3 html2html
- 3 html2wt

Newly failing
- 1 html2wt

  "3. Leading whitespace in indent-pre suppressing contexts should not be escaped"
  This is just normalization of output where multiple HTML forms
  serialize to the same wikitext with a newline difference. It is not
  worth the complexity to fix this.

- 1 wt2wt
  ""Trailing newlines in a deep dom-subtree that ends a wikitext line"
  This is again normalization during serialization where an extra
  unnecessary newline is introduced.

- A bunch of selser test changes.
  182 +add, 188 -add => 6 fewer selser failures

  - That is a lot of changes to sift through, and I didn't look at every
    one of those, but a number of changes seem to be harmless, and just
    a change to previously "failing" tests.

  - "Media link with nasty text" test seems to have a lot of selser
    changes, but the HTML generated by Parsoid seems to be "buggy" with
    interesting DSR values as well. That needs investigation separately.

  - "HTML nested bullet list, closed tags (bug 5497) [[3,3,4,[0,1,4],3]]"
    has seen a degradation where a dirty diff got introduced.
    Haven't investigated carefully why that is so.

Change-Id: Ia9c9950717120fbcd03abfe4e09168e787669ac4
2014-09-16 11:59:55 -05:00
Subramanya Sastry f5e7f667d5 (Bug 69827): Supported nested <ref> tags once more
* Nested <ref> tag support was broken in 69b6ec4d since that patch
  effectively processed <refs> only on the final DOM rather than
  extracted <ref> information in sub-pipelines. In doing so, it
  broke support for <ref> in <ref> tags which are supported by {{efn}}
  templates and used as follows.

  {{efn|A clarification.{{sfn|Smith|2009|p=2}}}}

  The {{sfn}} tpl generates a <ref> tag inside another <ref> tag
  generated by the {{efn}} tpl.

* This patch fixes the breakage by processing <ref> content after it
  is extracted (in case it is known to have nested <ref> tags).

* This fixes the rendering on enwiki/Otto_I%2C_Holy_Roman_Emperor
  and is an improvement over when it was broken. The nested <ref>
  gets id 18 (just like the Cite.php handling) whereas it used to
  get id 1 before the breakage.

* TODO:
  - Figure out a way to add a test for this.

Change-Id: Ib82b75f66249b2133a123adbe6fd7acbfd8ec8fb
2014-08-22 13:04:50 -05:00
Subramanya Sastry 69b6ec4d4d (Bug 67237): Fix citation numbering issue
* Process <ref> and <references> tag on the top-level DOM only
  and ignore the generateRefs pass when processing other content.

* This required a few fixes:
  - ensure that DOMPostProcessor knows about the top-level.
  - ensure that DOMVisitor knows about the top-level.
  - cleanup pass leaves behind the ref-marker metas from DOMs from
    non top-level content.
  - process nested references content.

* One of the references tests had incorrect parsed output. That test
  has been updated to reflect the correct output from this patch.

* Barack Obama seems to now have the correct numbering on references.

Change-Id: I5465721d2fc715f2168f267e773a446bc37d198b
2014-07-01 22:48:13 -05:00
Subramanya Sastry e59aaecbcf Use TokenStreamPatcher to convert table tags outside wikitables to text
* Keep track of table nesting in token stream patcher and use it to
  convert <td>, <tr>, and <th> tags to plain strings.

* This fix is only enabled on the top-level token stream.
  To support this, fixed the resetState function in the parser
  construction code to pass in a toplevel flag which lets the
  token stream patcher know the context it is in.

* Fixes 29 (wt2html,wt2wt,html2html,selser) tests and improves
  results of 1 previously blacklisted tests. The failing selser
  test is actually a false failure because selser is more accurate
  than non-selser wts.

* Consolidated a few separate tests into a single test that covers
  all this functionality.
  - This new test fails wt2wt and html2wt modes because serializer
    uses tokenizer information which continues to return table tokens
    and results in <nowiki> wrappers.

Bug: 66489
Bug: 66498
Change-Id: I9f42354ea9efb0f8adfc96c23760012220d00dd4
2014-06-24 12:33:55 -05:00
Gabriel Wicke 7db2d118dc Work around Cite reset issue
The Cite extension does not currently handle resetState calls in
sub-pipelines, and relies on sharing a single Cite instance between all
pipelines. Fixing this is a longer project, so this patch works around the
issue for now by passing a flag indicating resetState calls in sub-pipelines
and ignoring the call in Cite in that case.

Change-Id: If3d426a5311a55d1c1530860d2b665d3681f1aa9
2014-04-30 10:02:39 -07:00
Arlo Breault 8ed953ee0f Temp fix for handling non-string ref attribute vals
* Entities in ref name weren't expected

 * Fixes the crash from arwiki:تأثير_الدمعة_السوداء

 * Makes use of the fix from de3642b8dd4a804ac654f2943a900496f2c8b3f3

Bug: 63790
Change-Id: Icb8781b4d9decc5a8b115d0b11def4d18f5d5025
2014-04-15 14:27:12 -07:00
Arlo Breault a28fbbbac7 Only pass document.body to the post-processors
* With some other cleanup pulled out of the external ids patch.

Change-Id: I8cee994079579d922da97a4c1c793540bfd8d1b1
2014-04-03 16:05:36 -07:00
Subramanya Sastry d99166010c Minor code cleanup: Use forEach instead of map where appropriate
* Also replaced buf with newTokens in ext.core.TemplateHandler.js

Change-Id: I81e05a13c9373aa298d710b7942f9fe3cd802e63
2014-03-26 16:48:18 -05:00
Subramanya Sastry e8351a4f07 Cite: Parse <references> content all the way to DOM.
* Thus far, <references> tag content was being parsed to
  stage 2 and merged into main pipeline. This patch takes
  this all the way to DOM. This required some tweaks to
  handling of <ref>s nested inside <references>.

* Fixed up a buggy parser test in the bargain -- the old parsoid
  result was buggy as well. I verified output in the enwp
  sandbox.

Change-Id: Iff6c528066b71ce1b00dd769910a04ee66623340
2014-03-11 10:06:53 -05:00
Marc Ordinas i Llopis 99b75249f1 <references> handler should respect inTemplate pipeline option
Change-Id: Ibfc1ea97aef8681bfd3cf04dba0aa32824c076f0
2014-03-07 18:39:28 +01:00
Subramanya Sastry 5445cfcd92 DSR computation: Handle <ref> tags nested in <references>
* The current fix is a hack to fix dsr issues right away.
  Meanwhile, will invesitigate a fix that will involve processing
  <references> in its own subpipeline and persisting state into
  the top-level page.

* Fixes a known selser failure from bug 62025.

Change-Id: I0f80d68e927f500939a44af401cc73c07e24721f
2014-02-28 15:56:19 -06:00
Subramanya Sastry 742b5d4453 Renamed function name & reorder args for clarity + added docs.
* Renamed buildDOMFragmentForTokenStream --> buildDOMFragmentTokens
  and made env. the first arg.

* Added documentation to buildDOMFragmentTokens and encapsulateHTML

Change-Id: I7eccfd3f4dc5b4b91d20d1d24d98ec514df6dfbc
2014-02-21 14:37:10 -06:00
Subramanya Sastry 7a949a4eb5 Code cleanup: Pipeline utilities now take env and parent frame.
* Removed manager and passed in env and parent-frame to all
  utilities that process content in new pipelines.

* Added more documentation to mediawiki.Util.js.

* Renamed processAttributeToDOM to a more appropriate name.

* Added pipelineFactory property to env and used that to
  construct parsing pipelines everywhere.

Change-Id: Ic612e5630d19d4e3f5d6388bc5cd117d337fd799
2014-02-12 01:21:15 +00:00
Arlo Breault a0cbd6e4de Remove token caching and cloning
Bug: 51203
Change-Id: Ia377d5878ac048bab15dbd42187330bee653c6eb
2014-02-11 00:47:58 +00:00
Subramanya Sastry 34c33103bf Use Map.delete(..) to remove an entry from a map.
* Fixes crasher on frwiki:Mundos_opuestos_2 with oldid 99313179

Change-Id: Ide5a9594ebc7622bb7ff5fc3d6eac09a0f0e1642
2014-02-03 20:10:54 +00:00
Subramanya Sastry fc7985f5ca Added a flag to DOM-fragment unpacking to update fragment DSR
* This patch adds a flag to DOM fragment unpacking to update
  fragment DSR in cases where the fragment needs it (references
  block + reused template/extension/images content). In other
  cases of dom-fragment use, the DSR should not be updated.

  Ex: All cases where fragments are used to implement parsing
  scopes (all A-tag content currently) since DSR computation
  is set up with offsets in the top-level source.

  This is not just an optimization, but a correctness issue
  since the fragment unwrapper always sets the fragment DSR
  on the first node which would be incorrect in scenarios where
  the fragment DOM has multiple top-level nodes.

* Parser test runs now have better DSR values in certain cases.

Change-Id: If1f5bf98dab246a3c8a1869b38335e90268cb5c5
2014-01-30 18:19:10 -06:00
Gabriel Wicke 32a1718133 A few small performance tweaks
* Use string concatenation instead of Array.push & join for accumulators:
  http://stackoverflow.com/questions/7299010/why-is-string-concatenation-faster-than-array-join

Overall parser tests might be slightly faster, but the improvement seems to be
within the margin of error.

Change-Id: I5d9e4191a79253fa808c9b0a26052470e8613a79
2014-01-30 11:14:55 -08:00
C. Scott Ananian 301b9541b5 Uniformly use Array.isArray to test whether an object is an array.
This reads better than manually testing the constructor, and often
leads to terser code since we don't have to check whether the argument
is an non-null object before querying the constructor field.

Change-Id: I53ec87d6e80d658aa3d26dc2b613dc6c58e2d026
2014-01-27 17:28:11 -05:00
C. Scott Ananian b4fd3bef6f Use es6 methods instead of fakejquery.
In particular, use `Array.isArray` instead of `$.isArray`, and
`Object.assign` instead of `$.extend`.  `Object.assign` operates only on own
properties, so use `Object.create` on the prototype where necessary to
get inherited properties.  `Object.assign` does a simple assignment and
is appropriate in most places, but be careful if we ever install
getters/setters on a prototype.

Implement `Util.clone()` from scratch to have a jquery-compatible deep
clone operation.  In particular, this needs to ignore objects which
aren't "plain objects", so we don't try to clone DOM nodes.  Our
definition of a "plain object" is compatible with jquery/zepto.js, and
is thus something of a hack.  We should eventually replace this with a
`console.assert()` and remove/rewrite the places where we try to clone
objects which contain DOM trees and other cruft.

Change-Id: I88c8fe41a9be84c167d5a0ea1187fd258f077968
2014-01-27 17:21:28 -05:00
Arlo Breault eced96cc2c Move .data off DOM nodes
* For now it stores in .dataobject, freeing up .data to
   handle <object> elements.

 * Adds a test that crashes master.

Bug: 57394
Change-Id: I4207d76ad9dab660e890008b2ee5014554ce52c8
2014-01-15 16:59:18 -06:00
Arlo Breault d2ba2d73e7 Properly bump reference anchors
* Adds an index of all the references on a page in order to avoid
   repeating attrs when multiple <references /> tags are present.

 * Update tests to reflect the new behaviour.

Bug: 59782
Change-Id: Ia44bf59a9304788aca170041d3b85f53557151fc
2014-01-13 12:09:54 -05:00
Arlo Breault a372d97939 Use maps where wanting
Comments seemed to indicate so.

Change-Id: I63d0524d134d8be166403cfa9d6400cecc1749ed
2014-01-09 22:33:57 -05:00
Gabriel Wicke 4cfad3a771 Clean up repo for Parsoid service only
* remove non-JS code (PHP, cpp, misc)
* move js code to the root dir

Change-Id: Ibdf1c0ea46f128adc91238b145c46382c6f4d550
2013-12-13 14:53:24 -08:00