Commit graph

231 commits

Author SHA1 Message Date
C. Scott Ananian e82f60f62d Turn on eslint jsdoc/check-tag-names rule.
Change-Id: Ifd9a5319171684015463d6b5759ed2308bff1a5b
2018-02-27 18:10:04 +00:00
C. Scott Ananian 6668c45947 Improve jsdoc
Lots of doc comment tweaks to better match JSDoc standards.  Also use
the latest version of jsdoc-wmf-theme.

Change-Id: I9bf4f22b292b9cff0ad926f31005bccc7c89cbdb
2018-02-22 14:16:58 -05:00
Arlo Breault ab3f062098 Upgrade eslint
Then changes here were made by running `npm run eslint-fix`

Change-Id: Ifb623af610cadc4200b34b52af41c9fa29856b6b
2018-02-17 09:08:32 -05:00
C. Scott Ananian fd719e02e2 Use Promise.async/yield in lib/ext/Cite
Change-Id: I08458b0083de8d9aec9673af7f08c4b2e25112e5
2018-02-16 23:21:09 +00:00
C. Scott Ananian 8713e6cb55 Replace .bind(this) and var self = this; with arrow functions.
Arrow functions don't redefine `this`; they are supported in node >= 4.

http://node.green/#ES2015-functions-arrow-functions

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions

Change-Id: I03790eaa790b00194d137eef397371d63f242ac1
2018-02-09 14:56:48 -05:00
Arlo Breault c60d8f6078 Rethink responsive references wrappers
Emit them earlier (before post-processing) so that attributes don't need
to be migrated.

Bug: T181408
Change-Id: I2b5c7ff552b3322be74f79a81936c41d58fecabc
2018-02-01 17:53:30 +00:00
Arlo Breault 04aa4beb5b Fix infinite recursion from linting named ref cycles
On pages like,
http://localhost:8000/fr.wikipedia.org/v3/page/html/Krzysztof_Charamsa/142052535
http://localhost:8000/es.wikipedia.org/v3/page/html/Taxonom%C3%ADa/104913080

Bug: T185267
Change-Id: I156cce384873609a65b659360667323b8f0a497b
2018-01-22 18:43:11 -05:00
sbailey 4eaddbb890 Fix for T45094 replaces <span> with <sup> for references
This change was prompted by a request to follow the PHP Cite extensions
lead in using <sup> for linkbacks. Also, using superscript for
notations/citations is semantically appropriate and follows style guide
conventions.

Change-Id: I7c83d12dd900682799c124ddae1a8689969d5e8c
2018-01-16 18:41:25 +00:00
Arlo Breault 131718460b Add a function to create the references node uniformly
Change-Id: I58fd7ea7c692b1a38dab160083d3d9a529527fb0
2018-01-10 00:22:00 +00:00
Subramanya Sastry 0ec01e4ca6 Linter: Fix crasher in Cite - don't assume well-formedness
Change-Id: Ie5da61934b91f115b0eea995e05de6c231d77bdb
2018-01-08 16:05:40 -06:00
Arlo Breault 03bf14b003 Restrict nested refs not coming from templates
This better matches the behaviour of the php extension.  An instructive
example is,

<ref>123 {{#tag:ref|321}}</ref>

{{#tag:ref|456 <ref>654</ref>}}

The php implementation only parses the ref contents when producing the
output for reference tags, at which point, nested refs will be ignored,
as in the former case.  The "321" is dropped.

The latter is special, since the extension tag takes precedence over the
parser function, the inner ref will already have been processed when the
outer is added to the stack, and hence the nesting is permitted.  This
is why the inner ref precedes the outer in the references list (has a
lower number).  Unfortunately, Parsoid doesn't yet get that ordering
right.

Change-Id: Ieb0e418cca634605c2a9f1487139b15095f38d81
2018-01-08 19:49:47 +00:00
Subramanya Sastry 288feddbc6 Linter: Use extension mechanism to provide ext-specific lint handers
* <ref> content ends up in references section. The original <ref>
  site only has a backlink to that references section.

* Given this, when the references tag itself comes from a template,
  this throws off the linter attribution code. It assigns the
  references template's DSR to the lints from the <ref> tags.

* A clean way of dealing with this would be to treat extension
  content as its own independent document (which we are already doing
  currently in our wt2html code path) and asking the extension to process
  its content while giving it the handler for linting an individual node.

* This patch implements special handlers for <ref> and <references> content
  in the Cite extension and registers them.

* Move cite-specific lint handling code to the Cite extension.

* New mocha tests spec this behavior.

Change-Id: Ib57f0303605a73408333e133f8051be0b8d45d69
2017-12-21 17:43:37 -06:00
C. Scott Ananian 228325cb09 Update Parsoid to generate modern HTML5 IDs w/ legacy fallback
The citation portion contains fixes for []-encoding in attributes,
based on Iec3439f76ecc2a3543b30b35f8735c92b0cfb711.

Fix some other no- or double-encoding issues with HTML entities in the
process, based on I88e8e2077e6f5eec2b232391f7818370894a62dc
(T103714/T104196).

Unrelated parsertest fix which needed an id fix -- incorrect id
added by the section-tag patches exposed here.

Bug: T152540
Bug: T103714
Change-Id: I12b2a148f7170d20bd9aacd3b5b8ee1965859592
2017-11-30 16:14:33 -06:00
Arlo Breault d188af60d0 Fix processing ref in ref
* Follow up to f7594328

 * Fixes the regression found in rt:

   node bin/roundtrip-test.js --domain ru.wikipedia.org "Феодосия"

 * Simple test case (do we really not have one!?):

   test <ref>haha{{#tag:ref|ok}}</ref>

Change-Id: Ie0a53a8e885a6d94769034cce4ef432773635842
2017-10-26 21:35:01 -04:00
Arlo Breault f75943282f Avoid serialize/parse of mw:dom-fragment-token content
* The contents of "mw:dom-fragment-token"s was being serialized
   after processing to the DOM and stored on the token to be
   shuttled through tree building.  Only to be reparsed in the
   unpacking phase.

 * Here we store a pointer to the contents in a fragment map.

 * Doing less work results in a performance improvement, though
   only slightly because the content still needs to be adopted
   by the main document.

Change-Id: Ia0aec7de469101a2a93342ea89daac0f0e73cf1a
2017-10-25 22:41:18 -04:00
Arlo Breault 84c80a7ed2 T133334: Ref marker in caption in data-mw
* Replace ref markers instead of waiting for cleanup to remove them
   since that doesn't happen on embedded html.

Change-Id: Ied746f025a0ac7f14d922aff6640fef3aa4b55b0
2017-10-26 00:12:27 +00:00
Arlo Breault 27799ea9b5 There's only one refInRefProcessor
* Avoid the unnecessary binding.

Change-Id: I80e558a86eb7197ce39a07062499ec58f170a3af
2017-10-26 00:06:20 +00:00
Arlo Breault 23db74cd9c Remove unnecessary serialize/parse of references content
* Also, change the input parameter of buildDOMFragmentTokens
   to uniformly accepts a <body>, rather than a doc or string,
   so that we can pass it nodes from our dummy document.

Change-Id: I4bb44573fe7203277d51e804a4a6423100a34f03
2017-10-25 23:52:19 +00:00
C. Scott Ananian fc4967ac5f Remove unnecessary calls to Node#childNodes.length
These calls force us to allocate a backing array for childNodes,
deoptimizing a linked list representation.  Use the standard
Node#hasChildNodes() method instead, which can be efficiently
implemented without allocating a backing array.

Change-Id: I1706bfa15263564bc981d689947835a3d0d4a68f
2017-10-24 23:04:21 +00:00
Arlo Breault 3aa2e1a106 Bump version to 0.8.0 for release
Change-Id: I2279cbd8d56d3cd097a086b4f52741234a014346
2017-10-23 17:37:34 -04:00
Arlo Breault 0dba785946 T176425: Transfer about group to reference wrapper
* Follow up to 74acc71e

Change-Id: Ia403bf88dd48702dbbbf53a247ea2714243a0acc
2017-09-25 12:59:46 -04:00
Arlo Breault ab0c40f496 Fix crasher from empty references list
* See on /en.wikipedia.org/v3/page/html/Manuela_Maleeva/800363347

 * Folow up to 74acc71e

Change-Id: Iae89d4b0073dafce5fe4c28dc94fd6d5779d71c6
2017-09-13 22:35:37 +00:00
Arlo Breault 74acc71e98 T159894: Add support for Cite's responsive parameter
* Ports commit 04c3ad01 in core's Cite extension.

 * Until $wgCiteResponsiveReferences is exported, a number of wikis
   which have enabled this won't be getting the right default.
   https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L14945-L14970

   However, it looks like enwiki's {{reflist}} explicitly sets the
   parameter either way, so not a bad start.

 * The "ext.cite.styles" css resource is added from core's Cite
   extension 05cb5cc1, since that's where the responsive css lives.

 * The blacklist changes for existing tests are because we now only
   serialize the children of the div wrappers.  Those tests probably
   deserve Parsoid specific sections..

Depends-On: I2404999ab11b5cf7b740ae43696c4676ab1b6d22
Change-Id: I8f9277b3ecb253e0bee7ee55eef7af4935821527
2017-09-13 21:24:52 +00:00
Arlo Breault ff09c1667d Add 'references' class to reflist node
* Matches the core Cite extension commit cec36fed

 * This change means we can get rid of some of wiki specific styling in,
   https://github.com/wikimedia/integration-visualdiff/blob/master/lib/parsoid.custom_styles.yaml

Change-Id: I763b8086d057ea0cac8ba8fd4c9f4f4279766fed
2017-09-11 15:13:55 -04:00
Arlo Breault 6e5b366ff6 [eslint] Enforce lines-around-directive
* Done with --fix

Change-Id: Ie9b34c69615493870427be5f318a14f31a361d6a
2017-04-27 16:05:00 -04:00
Arlo Breault d13cfa4f71 [eslint] Use eslint-config-node-services
* Lots of overrides here that we should either fix or be explicit about
   wanting to diverge and maybe talk about upstreaming the change.

Change-Id: I927dd325e49ef72ccbe5e8b926f7984b82dd0f2e
2017-04-27 16:04:53 -04:00
Arlo Breault 863e53ff8e T163744: Switch to eslint
* .eslintrc was generated from .jshintrc and .jscsrc using polyjuice
   and some manual tweaking.

 * Forced to follow the convention in eslint #4174

 * However, in a follow up, we'll use wikimedia/eslint-config-node-services

Change-Id: I2500a6520a5c9f41d5333e937c151228aec88be0
2017-04-27 14:00:51 -04:00
Arlo Breault dea21633ba T64270: Support video and audio content
* Spec'd at https://www.mediawiki.org/wiki/Specs/HTML/1.3.0#Audio.2FVideo_.28Proposal.29

Depends-On: Icb26ffb802150c1d1f8c6e0aab05b25e503077e2
Change-Id: I53c6fa8d862c152bb1c8dd3ae96f4ee464a60d32
2017-04-21 20:44:44 +00:00
Subramanya Sastry 4b242825cf Update HISTORY.md + bump version numbers to 0.7.1 for deb release
Change-Id: I32cf92c566dc7d501ec0b28552ef6a4fbb879d82
2017-04-05 11:13:30 -05:00
Subramanya Sastry d4140cb46f Update HISTORY.md + bump version numbers to 0.7.0 for deb release
Change-Id: I46189fae2dc0b01b473c471a8a62d2f6204f3692
2017-04-03 17:25:07 -05:00
Subramanya Sastry 2368fba08f Escape cite ids with Sanitizer.escapeId
* Without this, refs like <ref name=":0"> won't generate the same
  links that the PHP parser generates.

* Updated an existing test to add the :0 key that require escapeId
  to be encoded properly.

Change-Id: I69e5f16ccf64bd1c9cf05bdea7a379e679d36b1a
2016-12-20 15:07:07 -06:00
Subramanya Sastry 18c965dec0 T102134: Fix cite hrefs to render properly
* With Parsoid's base href pointing to the wiki, plain #-fragment
  links won't resolve properly. Add the page title to the href
  for the links to start resolving properly again.

* Updated parser tests accordingly.

Change-Id: I280c41a0382bd2acd82cc586212695aa3b920171
2016-12-20 15:07:07 -06:00
jenkins-bot 0b1277434f Merge "T104662: Allow nested ref tags only in templates" 2016-12-12 15:15:10 +00:00
Arlo Breault ea599bb074 Let extensions supply the pp tracing name
* They won't all have tags, like the linter.

Change-Id: Idf309a82bf229219d98d693302c305726fa25fcd
2016-12-09 19:03:35 -08:00
Arlo Breault 439bc3af2d T104662: Allow nested ref tags only in templates
* To support the #tag parser function.

 * An aside, #tag is nutty,

     {{#tag:nowiki|test<ref>haha</ref>}}

Change-Id: I0ab93f6ae959b30a887e967952c904ef0400b189
2016-12-08 10:33:00 -08:00
Subramanya Sastry a16445812f Bump version to 0.6.1 (0.6.0 deb pkg was botched)
Change-Id: I601f31c79d8c2361ca33a6d0e6d6196c1b6c5f0f
2016-11-14 16:44:35 -06:00
Subramanya Sastry c8a0ec200a Bump version to 0.6.0 for new deb release + update HISTORY.md
* Updated HISTORY file based on Parsoid deployment logs and
  added only the most pertinent entries.

Change-Id: If07bc75fcb09507cae30fc82d7d89b8d87a4c69b
2016-11-07 16:30:16 -06:00
Arlo Breault 8c60ef0f79 Let native extensions add stylesheets
Change-Id: Ib418544236ddf2d9a075934d72712734d90b4537
2016-09-12 17:09:05 -07:00
Arlo Breault e9383e0877 Use tagWidths for content in Util.getArgInfo
* Use nestedRefsHTML.length to determine when we need a dataMw.body,
   the regexps there were unnecessary.

 * Pulled out of https://gerrit.wikimedia.org/r/#/c/264026/

Change-Id: I4602594e468322c8b6b8653eee33047ef9af9ebc
2016-08-31 13:33:03 -07:00
Arlo Breault 419db6b794 Bump version after release
Change-Id: Iaed8279cb06c0c7b55940b1e4aed3f8b21262e79
2016-06-14 11:18:15 -07:00
Arlo Breault 53c9863382 Keep the data-* attributes at the edges of the DOM
* FIXME: stop skipping jsapi tests

 * DU.serializeNode is renamed to DU.toXML, to avoid confusion with
   wikitext serialization (and a name clash when grepping).

 * DU.serializeToXML is renamed to DU.ppToXML, to distinguish it both in
   name and use from DU.toXML.  This is a special xml serializer that's
   meant to be used in Parsoid's DOM post-processing phase.  It's aware
   of a node's .dataobject and automatically transfers that to json
   stringified attributes before serializing a node.

 * DU.ppToDOM is added as helper for loading attributes when parsing
   html, as in DU.parseHTML.  It's the converse to DU.ppToXML

 * Diff markers are no longer stored as json stringified attributes on
   nodes while serializing.  Only when dom dumping, and even then only
   on clones.

 * Once T100856 is fixed, we can look into discarding data-parsoid from
   html that's stored in data-mw (ie. captions, references, etc.)

 * Filed T133334 to note that some of the html we're stuffing in data-mw
   needs further processing, since we're storing ref markers.

Bug: T91700
Change-Id: Ia1a60951e6292b5eb073eedca0b79938094809d2
2016-04-28 18:49:22 -04:00
C. Scott Ananian 8a5316e0c6 Add autoload mechanism for user extension code.
This parallels the autoload mechanism in mediawiki core.  In fact, small
wikis can use the same extensions directory for both core and Parsoid.

We also export a very basic "extension API" for external extensions
and convert our current "native extensions" to use this to show how it
is to be done.

Bug: T133320
Change-Id: I8e05d5bfdff873f28a58dead68aaca0e4823cf32
2016-04-26 22:25:59 -04:00
Arlo Breault 87296241d5 Rename DU.serializeChilden to DU.serializeToXML
* And use it where appropriate.  This has the effect of reducing the
   dom size because of the use of smartQuote (see all the parserTests /
   blacklist changes).

 * The serializeChildren name clashes with the one in the
   serializeState, causing confusion and making grepping harder.

 * Bonus: removed some dsr info in parserTests and fixed a html2wt typo
   that was prevented a test from running.

Change-Id: I287f1efd4afe06b158844e0de9f16494d4a89f93
2016-04-12 14:01:04 -07:00
Arlo Breault 126ed00d40 Suppress errors raised when getting debugging info
* An example of an href from eswiki/Kate_Gosselin?oldid=90347467
   is <a href="#cite_note-13.3F_It_Can't_Be!-3">
   which domino should do a better job validating, instead of throwing
   "Object [object Object] has no method '''".

Change-Id: Ie4bd702d7eee23ad021f2ad31b47cfbada947cd9
2016-04-08 19:36:01 -07:00
Arlo Breault df00a42c5e Fix log type
* Broken in ae45ffe9

Change-Id: Ife2a42e8a805e68fb4bfe83076e39b64be6dbf71
2016-03-30 18:22:07 -07:00
Subramanya Sastry 53117cef54 T113331: Move auto-generated refs flag from data-parsoid to data-mw
VisualEditor requires this info and presumably other clients might
and this is, in reality, semantic information about wikitext.

The html2html failure is because:
* even when the references html has been autogenerated,
  in non-rt-testing html2wt mode, we always generate the
  <references /> tag.
* we moved the flag from data-parsoid to data-mw but ignore changes
  to data-parsoid while comparing test output.
* So, in html -> wt mode, <references /> is always generated
  => data-mw.autoGenerated flag doesn't show up in the wt -> html
     phase of the (html -> wt -> html) test
  => the html2html test fails.

Change-Id: I8e79f2a436a1ca276b9351228a3d8f02d7ebd0c6
2016-03-24 23:17:34 -05:00
Arlo Breault 3b44e7e544 Remove extsrc bloat from references data-mw
* This hasn't been used since 2013! See 46ddbf3f

Change-Id: Iea382829f38788541b3bc4f8595a0d6bf600c167
2016-02-05 18:26:33 -08:00
Arlo Breault ae45ffe95e Move extension content serialization to cite
* See the follow up for why references extsrc is irrelevant.

Change-Id: I1db0f3c3f3e85706125418bb431aa40c17a97407
2016-02-05 18:26:22 -08:00
Arlo Breault 41756cb70b Flatten ext/
Change-Id: I083dc0d7c5ab37c8f0f5c051e82e4705c891812d
2016-01-13 15:34:40 -08:00
Arlo Breault 1c10f71b47 Move processRefs.js to ext/cite/
Change-Id: I10ab294900f090dcdb85b205d9eb0472fd3e9157
2015-10-28 15:31:53 -07:00