Commit graph

66 commits

Author SHA1 Message Date
Subramanya Sastry 8a9908abbc Remove .bind() usages in native extension code.
Bug: T205336
Change-Id: I544205d7f32f755c0d0616af404a68ea40f4d2f5
2018-12-07 03:23:31 +00:00
Arlo Breault c1b003553d Move applying about ids from fragment wrappers to extension content
It's only the extension content that needs the about ids.  This matches
b241ac6 where span wrapping was only applied to extension content, and
goes hand-in-hand.  The one necessitates the other.

Fragment wrappers don't need abouts since they don't have siblings and
are represented as a single node.

When we get to unpacking, it's now clear that if the wrapper has an
about id, it came from template encapsulation and would need to be
applied to the top-level nodes in the fragment.  This being true of
all fragment, it lets use get rid of the `isForeignContent` option to
`encapsulateExpansionHTML`, since that no longer serves to distinguish
anything.

Change-Id: Id9faae3d4e3c8771c2de1fb42ba62a7d92d76673
2018-12-06 23:35:07 +00:00
Subramanya Sastry cee1086f35 Update HISTORY.md + bump version numbers to 0.10.0 for deb release
Change-Id: I0be0c80a3f47ba3b1c2f94016bd9304de99388d4
2018-12-05 12:09:26 -05:00
Subramanya Sastry 7bee635f21 Split utils/DOMUtils.js into separate functional units
* DOMUtils was perhaps the biggest kitchen sink utility class we had
  with different classes of utilities.

* The split reflects a clear dependence hierarchy:
  DOMUtils -> DOMDataUtils -> WTUtils -> ContentUtils

  This also seems to reflect a bit in the use patterns.
  Content helpers are not used as much in html2wt.

* DOMUtils now only has DOM utilities that are independent of Parsoid
  and could be useful in another project.

* Moved a couple helpers into WTSUtils.js since they are very narrowly
  scoped to html2wt functionality and are unlikely to be every useful
  elsewhere.

* Moved diffing related utils to html2wt/DiffUtils.js
  - There is still a dependency in development / debug mode when
    doms have to be dumped. I couldn't think of an easy way of
    removing this dependency.

  - But otherwise, DiffUtils is scoped to html2wt use cases only.

* One more circular dependency eliminated.

* All tests pass.

Bug: T208360
Bug: T205333
Change-Id: I522e8b5c7d726706f994386282476102fe35e91e
2018-12-03 14:57:40 -06:00
Arlo Breault 878f6c7937 Unnecessary passing along of "responsive" attribute
These are already set on the first encapsulation node by toDOM.

Change-Id: I1eefdaf94e28222fc44d0be9d017eeab2f70c592
2018-12-03 03:54:26 +00:00
Arlo Breault 26984c0294 Allow extensions to set more options to encapsulateExpansionHTML
The nowiki extension will want to override `isForeignContent` so that
about attributes are omitted on the contents.

Change-Id: Iea28d7a66a90d516226ba9dce4518be6b996d18e
2018-11-29 20:28:54 -05:00
Subramanya Sastry b4d7a158cd Cite: Import Sanitizer from extapi
Change-Id: Ic4cb95d2fbe678641b4496d38295ec8e06269d28
2018-11-21 20:24:48 +00:00
Arlo Breault e644753c6b Introduce a parseTokenContentsToDOM helper
It calls out to a slimmed down parseWikitextToDOM, since only a subset
of that functionality is needed in some extensions, like gallery.

Change-Id: I72e29c4b285fc8cae0207966cc2eddd0fe3ebcf3
2018-11-19 12:20:43 -05:00
Subramanya Sastry 678effaf0e Extension API: Add a html2wtPreprocessor DOM processor stub
* This patch doesn't do any actual work beyond adding the framework
  in place for HTML preprocessing before actual serialization.

Change-Id: I2505f5937657813480abef8ec5da9d765a953e29
2018-11-07 17:22:31 +00:00
Subramanya Sastry a944352878 Extracted token-related helpers from Util to TokenUtils
* Things to investigate as followups:
  - Should this be part of the Token base class in parser.defines.js?
  - Can we remove nowiki & cite usage of these helpers? Why do they
    need to know about (parser internals) tokens?

Bug: T208360
Change-Id: Ib962b4acf9534852240fcb083ce67d696a997a83
2018-10-30 18:05:24 -05:00
Subramanya Sastry e1a6c9a953 Rename parsing pipeline options to something more semantic
* noPwrapping and noPre existed to reproduce the symptomatic
  effects of other behavior / semantics in the parsing pipeline.

  One of that semantic source was the fact that the content
  being parsed in their own pipeline was being using in inline
  contexts and p-wrapping and indent-pre generation doesn't
  make sense in those contexts.

  The other semantic source was the fact that when the content
  being parsed in their own pipeline was embedded "inPHPBlock"
  in the PHP Parser's doBlockLevels sense, i.e. inside a block
  element that existed on the same wikitext source line.

* This patch introduces inPHPBlock and inlineContext as two
  new parsing pipeline options and uses one or the other or both
  where appropriate.

* No change in parser test results as expected.

Change-Id: I2c625456c6593e49b6a247fb9b2f2e3d6021155b
2018-10-02 23:05:53 +00:00
Subramanya Sastry 499c2c4564 Remove more Cite leakage from html2wt code
* This removes Cite leakage in html2wt code by adding PHP parser
  side-effect leakage into the Cite extension configs.
  This is a somewhat temporary solution which we hope to address
  better as part of ongoing cleanup.

* After 4 years, 'a quick hack for now' introduced in 3301393515f
  has been addressed.

* No change in parser test passes.

Change-Id: Ic5f8c160d4eedcc4236bce5f095344bd879188cd
2018-10-02 23:00:15 +00:00
Subramanya Sastry 635be2e906 Tweak extension config a bit more
* Move the name property to the top level

Change-Id: Id8730f35e40ad9f01d250f29ef3c2545fd4fd612
2018-08-30 19:00:35 +00:00
Subramanya Sastry bf0a743c2d Document more extension-API related FIXMEs
* Cite-specific code has leaked in html2wt
* Other possible FIXMEs in Cite about generalizing or
  adding new extension API functions.

Change-Id: I30712a3af81edfce0760f4a45ab694c798fb294a
2018-08-29 21:48:43 +00:00
Subramanya Sastry 62cd9789f2 Tweak extension DOM config for extensibility
* Make the dom part of extension registration / config more general
  to allow for other kinds of dom processors.

* For example, we will probably be letting extensions define DOM
  handlers that lets us take a document's subtree and convert it to
  self-contained top-level document.

Change-Id: I0ddfe3bc677041cedf63e79faa9c5af939953419
2018-08-23 19:39:16 +00:00
Arlo Breault 93371d4dee Support directionality for reference
Ports commit 1ca27aa from extensions/Cite

Bug: T198221
Depends-On: Iaf84966e37cea730c9eca07c19a555971ffeadf3
Change-Id: I28c8efa6fd5cb06220af4399aa26f3d01b8d3761
2018-08-23 14:36:04 -04:00
Arlo Breault a118efbbd7 Ensure the ref.cachedHtml isn't being regenerated needlessly
Change-Id: If319ad282ed7dd437dce400417e4555291a2dc38
2018-08-16 19:34:04 -04:00
Arlo Breault 9928fe4044 Separate which source of data attributes are used in extractRefFromNode
Follow up to 53ae5aa

It's somewhat unclear which attributes belong on the representation for
the fragment, so don't rely on them unnecessarily.

Change-Id: Ib5ba0110291f0c02bd703482813b9567dab63cd6
2018-08-16 19:33:22 -04:00
Subramanya Sastry 8fa5bc3376 Enable extensions to distinguish between <ext /> and <ext></ext>
* This now lets Cite distinguish between <ref></ref> and <ref />
  Seen in improvement in blacklisted output of a parser test.

Bug: T130224
Change-Id: If9511498fe8f6d091f8a725b51810eb452db95de
2018-08-14 21:47:43 +00:00
Subramanya Sastry 1a2b7e796f Remove Cite-specific flag from parser pipeline options
* Use a generic extTagOpts that is opaque to the core parser
  but the individual extension can inspect.

Change-Id: I4d1331604828d583b820084e00af68232ec767f8
2018-08-09 02:00:03 +00:00
Subramanya Sastry d8f4a31efe Handle <references></references> without crashing
* Fixes rt regressions on svwiki:Havre and a bunch of other titles.

Change-Id: I1d31ccf1045ba5a4fef9377101ad4e90a534c1e9
2018-08-09 02:00:01 +00:00
Subramanya Sastry 53ae5aac06 Cite: Get rid of tokenHandler that relies on Parsoid internals
* This still exposes and exploits some Parsoid internals, but
  that can be fixed in the next round of updates.

* Cite (and any other extension) that want to manage fragments
  on their own vs. running the default fragment unpacking routine
  can now specify that they don't want the content unwrapped.

* Changes to parser tests are to rearrange the attrs and body
  attributes which switched positions in data-mw.

* The blacklist changes show that there has actually been an
  improvement in test failures.

Change-Id: I1e1a651e8f2d6d9456bb5849b0bce1f8a87c4446
2018-08-06 21:40:52 +00:00
Subramanya Sastry 116d3c3744 Bump Parsoid release version to 0.9.0
Change-Id: Icd71628513380d7a7fe4fb88fa7e26150ed0e7ef
2018-03-22 13:29:06 -05:00
C. Scott Ananian 4fbcb99cc6 Improve jsdoc summaries.
The summary is the first sentence of the method/class description.  In
some cases we're missing a period so our "summary" runs on and on.

Many of these were fixed up automatically by the
`jsdoc/require-description-complete-sentence` eslint rule, but
it has a few bugs yet (especially
https://github.com/gajus/eslint-plugin-jsdoc/issues/62 ) so we
can't yet turn it on by default.

Change-Id: I5163ed2c52b21090ce018573bb3ff06d50620d73
2018-02-27 22:41:39 +00:00
C. Scott Ananian e82f60f62d Turn on eslint jsdoc/check-tag-names rule.
Change-Id: Ifd9a5319171684015463d6b5759ed2308bff1a5b
2018-02-27 18:10:04 +00:00
C. Scott Ananian 6668c45947 Improve jsdoc
Lots of doc comment tweaks to better match JSDoc standards.  Also use
the latest version of jsdoc-wmf-theme.

Change-Id: I9bf4f22b292b9cff0ad926f31005bccc7c89cbdb
2018-02-22 14:16:58 -05:00
Arlo Breault ab3f062098 Upgrade eslint
Then changes here were made by running `npm run eslint-fix`

Change-Id: Ifb623af610cadc4200b34b52af41c9fa29856b6b
2018-02-17 09:08:32 -05:00
C. Scott Ananian fd719e02e2 Use Promise.async/yield in lib/ext/Cite
Change-Id: I08458b0083de8d9aec9673af7f08c4b2e25112e5
2018-02-16 23:21:09 +00:00
C. Scott Ananian 8713e6cb55 Replace .bind(this) and var self = this; with arrow functions.
Arrow functions don't redefine `this`; they are supported in node >= 4.

http://node.green/#ES2015-functions-arrow-functions

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/Arrow_functions

Change-Id: I03790eaa790b00194d137eef397371d63f242ac1
2018-02-09 14:56:48 -05:00
Arlo Breault c60d8f6078 Rethink responsive references wrappers
Emit them earlier (before post-processing) so that attributes don't need
to be migrated.

Bug: T181408
Change-Id: I2b5c7ff552b3322be74f79a81936c41d58fecabc
2018-02-01 17:53:30 +00:00
Arlo Breault 04aa4beb5b Fix infinite recursion from linting named ref cycles
On pages like,
http://localhost:8000/fr.wikipedia.org/v3/page/html/Krzysztof_Charamsa/142052535
http://localhost:8000/es.wikipedia.org/v3/page/html/Taxonom%C3%ADa/104913080

Bug: T185267
Change-Id: I156cce384873609a65b659360667323b8f0a497b
2018-01-22 18:43:11 -05:00
sbailey 4eaddbb890 Fix for T45094 replaces <span> with <sup> for references
This change was prompted by a request to follow the PHP Cite extensions
lead in using <sup> for linkbacks. Also, using superscript for
notations/citations is semantically appropriate and follows style guide
conventions.

Change-Id: I7c83d12dd900682799c124ddae1a8689969d5e8c
2018-01-16 18:41:25 +00:00
Arlo Breault 131718460b Add a function to create the references node uniformly
Change-Id: I58fd7ea7c692b1a38dab160083d3d9a529527fb0
2018-01-10 00:22:00 +00:00
Subramanya Sastry 0ec01e4ca6 Linter: Fix crasher in Cite - don't assume well-formedness
Change-Id: Ie5da61934b91f115b0eea995e05de6c231d77bdb
2018-01-08 16:05:40 -06:00
Arlo Breault 03bf14b003 Restrict nested refs not coming from templates
This better matches the behaviour of the php extension.  An instructive
example is,

<ref>123 {{#tag:ref|321}}</ref>

{{#tag:ref|456 <ref>654</ref>}}

The php implementation only parses the ref contents when producing the
output for reference tags, at which point, nested refs will be ignored,
as in the former case.  The "321" is dropped.

The latter is special, since the extension tag takes precedence over the
parser function, the inner ref will already have been processed when the
outer is added to the stack, and hence the nesting is permitted.  This
is why the inner ref precedes the outer in the references list (has a
lower number).  Unfortunately, Parsoid doesn't yet get that ordering
right.

Change-Id: Ieb0e418cca634605c2a9f1487139b15095f38d81
2018-01-08 19:49:47 +00:00
Subramanya Sastry 288feddbc6 Linter: Use extension mechanism to provide ext-specific lint handers
* <ref> content ends up in references section. The original <ref>
  site only has a backlink to that references section.

* Given this, when the references tag itself comes from a template,
  this throws off the linter attribution code. It assigns the
  references template's DSR to the lints from the <ref> tags.

* A clean way of dealing with this would be to treat extension
  content as its own independent document (which we are already doing
  currently in our wt2html code path) and asking the extension to process
  its content while giving it the handler for linting an individual node.

* This patch implements special handlers for <ref> and <references> content
  in the Cite extension and registers them.

* Move cite-specific lint handling code to the Cite extension.

* New mocha tests spec this behavior.

Change-Id: Ib57f0303605a73408333e133f8051be0b8d45d69
2017-12-21 17:43:37 -06:00
C. Scott Ananian 228325cb09 Update Parsoid to generate modern HTML5 IDs w/ legacy fallback
The citation portion contains fixes for []-encoding in attributes,
based on Iec3439f76ecc2a3543b30b35f8735c92b0cfb711.

Fix some other no- or double-encoding issues with HTML entities in the
process, based on I88e8e2077e6f5eec2b232391f7818370894a62dc
(T103714/T104196).

Unrelated parsertest fix which needed an id fix -- incorrect id
added by the section-tag patches exposed here.

Bug: T152540
Bug: T103714
Change-Id: I12b2a148f7170d20bd9aacd3b5b8ee1965859592
2017-11-30 16:14:33 -06:00
Arlo Breault d188af60d0 Fix processing ref in ref
* Follow up to f7594328

 * Fixes the regression found in rt:

   node bin/roundtrip-test.js --domain ru.wikipedia.org "Феодосия"

 * Simple test case (do we really not have one!?):

   test <ref>haha{{#tag:ref|ok}}</ref>

Change-Id: Ie0a53a8e885a6d94769034cce4ef432773635842
2017-10-26 21:35:01 -04:00
Arlo Breault f75943282f Avoid serialize/parse of mw:dom-fragment-token content
* The contents of "mw:dom-fragment-token"s was being serialized
   after processing to the DOM and stored on the token to be
   shuttled through tree building.  Only to be reparsed in the
   unpacking phase.

 * Here we store a pointer to the contents in a fragment map.

 * Doing less work results in a performance improvement, though
   only slightly because the content still needs to be adopted
   by the main document.

Change-Id: Ia0aec7de469101a2a93342ea89daac0f0e73cf1a
2017-10-25 22:41:18 -04:00
Arlo Breault 84c80a7ed2 T133334: Ref marker in caption in data-mw
* Replace ref markers instead of waiting for cleanup to remove them
   since that doesn't happen on embedded html.

Change-Id: Ied746f025a0ac7f14d922aff6640fef3aa4b55b0
2017-10-26 00:12:27 +00:00
Arlo Breault 27799ea9b5 There's only one refInRefProcessor
* Avoid the unnecessary binding.

Change-Id: I80e558a86eb7197ce39a07062499ec58f170a3af
2017-10-26 00:06:20 +00:00
Arlo Breault 23db74cd9c Remove unnecessary serialize/parse of references content
* Also, change the input parameter of buildDOMFragmentTokens
   to uniformly accepts a <body>, rather than a doc or string,
   so that we can pass it nodes from our dummy document.

Change-Id: I4bb44573fe7203277d51e804a4a6423100a34f03
2017-10-25 23:52:19 +00:00
C. Scott Ananian fc4967ac5f Remove unnecessary calls to Node#childNodes.length
These calls force us to allocate a backing array for childNodes,
deoptimizing a linked list representation.  Use the standard
Node#hasChildNodes() method instead, which can be efficiently
implemented without allocating a backing array.

Change-Id: I1706bfa15263564bc981d689947835a3d0d4a68f
2017-10-24 23:04:21 +00:00
Arlo Breault 3aa2e1a106 Bump version to 0.8.0 for release
Change-Id: I2279cbd8d56d3cd097a086b4f52741234a014346
2017-10-23 17:37:34 -04:00
Arlo Breault 0dba785946 T176425: Transfer about group to reference wrapper
* Follow up to 74acc71e

Change-Id: Ia403bf88dd48702dbbbf53a247ea2714243a0acc
2017-09-25 12:59:46 -04:00
Arlo Breault ab0c40f496 Fix crasher from empty references list
* See on /en.wikipedia.org/v3/page/html/Manuela_Maleeva/800363347

 * Folow up to 74acc71e

Change-Id: Iae89d4b0073dafce5fe4c28dc94fd6d5779d71c6
2017-09-13 22:35:37 +00:00
Arlo Breault 74acc71e98 T159894: Add support for Cite's responsive parameter
* Ports commit 04c3ad01 in core's Cite extension.

 * Until $wgCiteResponsiveReferences is exported, a number of wikis
   which have enabled this won't be getting the right default.
   https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L14945-L14970

   However, it looks like enwiki's {{reflist}} explicitly sets the
   parameter either way, so not a bad start.

 * The "ext.cite.styles" css resource is added from core's Cite
   extension 05cb5cc1, since that's where the responsive css lives.

 * The blacklist changes for existing tests are because we now only
   serialize the children of the div wrappers.  Those tests probably
   deserve Parsoid specific sections..

Depends-On: I2404999ab11b5cf7b740ae43696c4676ab1b6d22
Change-Id: I8f9277b3ecb253e0bee7ee55eef7af4935821527
2017-09-13 21:24:52 +00:00
Arlo Breault ff09c1667d Add 'references' class to reflist node
* Matches the core Cite extension commit cec36fed

 * This change means we can get rid of some of wiki specific styling in,
   https://github.com/wikimedia/integration-visualdiff/blob/master/lib/parsoid.custom_styles.yaml

Change-Id: I763b8086d057ea0cac8ba8fd4c9f4f4279766fed
2017-09-11 15:13:55 -04:00
Arlo Breault 6e5b366ff6 [eslint] Enforce lines-around-directive
* Done with --fix

Change-Id: Ie9b34c69615493870427be5f318a14f31a361d6a
2017-04-27 16:05:00 -04:00
Arlo Breault d13cfa4f71 [eslint] Use eslint-config-node-services
* Lots of overrides here that we should either fix or be explicit about
   wanting to diverge and maybe talk about upstreaming the change.

Change-Id: I927dd325e49ef72ccbe5e8b926f7984b82dd0f2e
2017-04-27 16:04:53 -04:00