In PHP, DOMNode#getAttribute() return '' if the attribute is not present,
not null. Audit our uses and try to either explicitly use `|| ''` (which
will ensure that PHP behaves the same was as JS) or use `hasAttribute`
to explicitly test for the presence of the attribute.
Changes have also been ported to PHP from JS.
Also added src/Wt2Html/PP/Processors/AddMediaInfo.php which was missing.
Change-Id: Ie1ae1df88e4fca70daf97b6f720f28014ebc99ed
This applies the JS changes from the following recently-merged patches:
6679c3bf Protect data-object-id attribute
d4e76d5b Fix new linter category to enable code work with templates
e567db8d Tweak storeDataAttribs to suppress DOM nodes in data-parsoid.tmp
16603953 Fix setting dsr on body for genTest
3a84a9dd Fix stashing data attributes for mw:StartTag
22c4a19a Remove redundant dataParsoid call
ed7b0ba0 Fix crasher in newly added linter category
505a357b Linter.js: Add new function to detect the use of links in links
8885b20e Move redlink updating into lib/parse.js
ccfce23d templatedepth is either an int or false
6d1571bd Move language conversion work into lib/parse.js
5a89c7de Avoid serialize/parse of data attributes when treebuilding
021d9958 Rename `document.env` to `document.bag`
c03ba494 Use XMLSerializer on both PHP & JS side in the DOM pass test script
e0c3cca9 Use env.createDocument in lib/api/apiUtils.js
550d3d71 Use a bag-on-the-side implementation for node data
f8de8b25 Add bin/inspectTokenizer.js
db704eea Add ability to splice a PHP transformer into the pipeline
a8be3ad6 Fix crasher in cite extension from accessing data after it's stored
2874f200 Simplify and clean up stops usage
6368265d Add some strategic isElt guards
5ae9553f DRY out transform test runners + tweak genTest to enable that
b0f2adc6 Assert that the .dataobject isn't touched after storing attrs on a node
1ce6a98d Skip separators when looking for the next th/td
Change-Id: I6a66ecb061e7ee7ed53feba1895dd315d9324715
The basic idea here is to generate the media structure in the token
stream using a stuffed span with a redlink, as in T169975, and
augmenting the nodes on the DOM once the media info has been fetched.
A redlink is justified as the canonical representation of the media
elements before info is fetched because it's the fallback if fetching
fails and the media type is unknown until the info is retrieved.
Most options are stored in data-mw until the media type is fetched and
it's determined that they're applicable. This is a bit of a reversion
of how things were done before where inapplicable options were removed
post-facto.
For consistency and styling's sake, figcaptions are now always added to
block figures.
The pass has to be run before generating headings anchor, since that
depends on the text content (ie. redlinks). This rearranges things in
the post-processor and adds another pass.
The post-processing pass to add media info is run on subpipelines as
well as the top level so that the media info is present in cases where
we embed HTML in data-mw (which is currently skipped by the top level
only passes, except for the cite extension, which has special handling,
see T214994) and to avoid an additional post-processing pass for the
gallery extension, which scales media of packed galleries. This comes
at the cost of making additional queries for each pipeline and requires
the add media pass to be idempotent.
Filed T214241 for figuring out what to do about data-mw info being
clobbered by template annotations.
The newly failing blacklisted tests are from roundtripping media options
in galleries, which requires a general refactor for support. See the
FIXMEs added there.
Performance should be expected to regress by the amount of work we're
able to overlap in the async phase of the pipeline while the media info
is being fetched. Considering a lot of that work is caught up waiting
for the batch to return (other async requests are found in the same
batch), this doesn't turn out to be much in practice in the average
case.
Bug: T153080
Bug: T169975
Change-Id: I856ee962b70cef1f8d49652396ea5264e11a8ade
* This initialization lets us do a git log --follow and follow
git history for that file across the js -> php port boundary.
This works because git uses content hashes for objects and
the copied code in the new .php file will have the same content
hash as the .js file.
* The following directories were skipped
- ./lib/config/baseconfig
- ./tests
* The following JS files were skipped
- ./lib/utils/promise.js
- ./lib/config/wmf.sitematrix.json
- ./tools/sync-baseconfig.js
- ./tools/sync-parserTests.js
- ./tools/fetch_ve_nowiki_edits.js
- ./tools/fetch-parserTests.txt.js
- ./tools/fetch-wmf-sitematrix.js
- ./tools/compare.linter.results.js
- ./tools/fetch-revision-data.js
- ./tools/fetch-wt.js
- ./tools/regression-testing.js
- ./tools/build-langconv-fst.js
- ./bin/server.js
Change-Id: I0b22057c23b72795aebbd66e3abcb627c6858ef3
* Extensions only need config info (env) and potentially abstract
parsing context (frame).
* Added some FIXMEs about potential future improvements.
Change-Id: Ib4cec4a77ecb96c855798eeb06f7742c5efb0729
It's only the extension content that needs the about ids. This matches
b241ac6 where span wrapping was only applied to extension content, and
goes hand-in-hand. The one necessitates the other.
Fragment wrappers don't need abouts since they don't have siblings and
are represented as a single node.
When we get to unpacking, it's now clear that if the wrapper has an
about id, it came from template encapsulation and would need to be
applied to the top-level nodes in the fragment. This being true of
all fragment, it lets use get rid of the `isForeignContent` option to
`encapsulateExpansionHTML`, since that no longer serves to distinguish
anything.
Change-Id: Id9faae3d4e3c8771c2de1fb42ba62a7d92d76673
* DOMUtils was perhaps the biggest kitchen sink utility class we had
with different classes of utilities.
* The split reflects a clear dependence hierarchy:
DOMUtils -> DOMDataUtils -> WTUtils -> ContentUtils
This also seems to reflect a bit in the use patterns.
Content helpers are not used as much in html2wt.
* DOMUtils now only has DOM utilities that are independent of Parsoid
and could be useful in another project.
* Moved a couple helpers into WTSUtils.js since they are very narrowly
scoped to html2wt functionality and are unlikely to be every useful
elsewhere.
* Moved diffing related utils to html2wt/DiffUtils.js
- There is still a dependency in development / debug mode when
doms have to be dumped. I couldn't think of an easy way of
removing this dependency.
- But otherwise, DiffUtils is scoped to html2wt use cases only.
* One more circular dependency eliminated.
* All tests pass.
Bug: T208360
Bug: T205333
Change-Id: I522e8b5c7d726706f994386282476102fe35e91e
The nowiki extension will want to override `isForeignContent` so that
about attributes are omitted on the contents.
Change-Id: Iea28d7a66a90d516226ba9dce4518be6b996d18e
It calls out to a slimmed down parseWikitextToDOM, since only a subset
of that functionality is needed in some extensions, like gallery.
Change-Id: I72e29c4b285fc8cae0207966cc2eddd0fe3ebcf3
* This patch doesn't do any actual work beyond adding the framework
in place for HTML preprocessing before actual serialization.
Change-Id: I2505f5937657813480abef8ec5da9d765a953e29
* Things to investigate as followups:
- Should this be part of the Token base class in parser.defines.js?
- Can we remove nowiki & cite usage of these helpers? Why do they
need to know about (parser internals) tokens?
Bug: T208360
Change-Id: Ib962b4acf9534852240fcb083ce67d696a997a83
* noPwrapping and noPre existed to reproduce the symptomatic
effects of other behavior / semantics in the parsing pipeline.
One of that semantic source was the fact that the content
being parsed in their own pipeline was being using in inline
contexts and p-wrapping and indent-pre generation doesn't
make sense in those contexts.
The other semantic source was the fact that when the content
being parsed in their own pipeline was embedded "inPHPBlock"
in the PHP Parser's doBlockLevels sense, i.e. inside a block
element that existed on the same wikitext source line.
* This patch introduces inPHPBlock and inlineContext as two
new parsing pipeline options and uses one or the other or both
where appropriate.
* No change in parser test results as expected.
Change-Id: I2c625456c6593e49b6a247fb9b2f2e3d6021155b
* This removes Cite leakage in html2wt code by adding PHP parser
side-effect leakage into the Cite extension configs.
This is a somewhat temporary solution which we hope to address
better as part of ongoing cleanup.
* After 4 years, 'a quick hack for now' introduced in 3301393515f
has been addressed.
* No change in parser test passes.
Change-Id: Ic5f8c160d4eedcc4236bce5f095344bd879188cd
* Cite-specific code has leaked in html2wt
* Other possible FIXMEs in Cite about generalizing or
adding new extension API functions.
Change-Id: I30712a3af81edfce0760f4a45ab694c798fb294a
* Make the dom part of extension registration / config more general
to allow for other kinds of dom processors.
* For example, we will probably be letting extensions define DOM
handlers that lets us take a document's subtree and convert it to
self-contained top-level document.
Change-Id: I0ddfe3bc677041cedf63e79faa9c5af939953419
Follow up to 53ae5aa
It's somewhat unclear which attributes belong on the representation for
the fragment, so don't rely on them unnecessarily.
Change-Id: Ib5ba0110291f0c02bd703482813b9567dab63cd6
* This now lets Cite distinguish between <ref></ref> and <ref />
Seen in improvement in blacklisted output of a parser test.
Bug: T130224
Change-Id: If9511498fe8f6d091f8a725b51810eb452db95de
* Use a generic extTagOpts that is opaque to the core parser
but the individual extension can inspect.
Change-Id: I4d1331604828d583b820084e00af68232ec767f8
* This still exposes and exploits some Parsoid internals, but
that can be fixed in the next round of updates.
* Cite (and any other extension) that want to manage fragments
on their own vs. running the default fragment unpacking routine
can now specify that they don't want the content unwrapped.
* Changes to parser tests are to rearrange the attrs and body
attributes which switched positions in data-mw.
* The blacklist changes show that there has actually been an
improvement in test failures.
Change-Id: I1e1a651e8f2d6d9456bb5849b0bce1f8a87c4446
The summary is the first sentence of the method/class description. In
some cases we're missing a period so our "summary" runs on and on.
Many of these were fixed up automatically by the
`jsdoc/require-description-complete-sentence` eslint rule, but
it has a few bugs yet (especially
https://github.com/gajus/eslint-plugin-jsdoc/issues/62 ) so we
can't yet turn it on by default.
Change-Id: I5163ed2c52b21090ce018573bb3ff06d50620d73
Lots of doc comment tweaks to better match JSDoc standards. Also use
the latest version of jsdoc-wmf-theme.
Change-Id: I9bf4f22b292b9cff0ad926f31005bccc7c89cbdb
Emit them earlier (before post-processing) so that attributes don't need
to be migrated.
Bug: T181408
Change-Id: I2b5c7ff552b3322be74f79a81936c41d58fecabc
This change was prompted by a request to follow the PHP Cite extensions
lead in using <sup> for linkbacks. Also, using superscript for
notations/citations is semantically appropriate and follows style guide
conventions.
Change-Id: I7c83d12dd900682799c124ddae1a8689969d5e8c
This better matches the behaviour of the php extension. An instructive
example is,
<ref>123 {{#tag:ref|321}}</ref>
{{#tag:ref|456 <ref>654</ref>}}
The php implementation only parses the ref contents when producing the
output for reference tags, at which point, nested refs will be ignored,
as in the former case. The "321" is dropped.
The latter is special, since the extension tag takes precedence over the
parser function, the inner ref will already have been processed when the
outer is added to the stack, and hence the nesting is permitted. This
is why the inner ref precedes the outer in the references list (has a
lower number). Unfortunately, Parsoid doesn't yet get that ordering
right.
Change-Id: Ieb0e418cca634605c2a9f1487139b15095f38d81
* <ref> content ends up in references section. The original <ref>
site only has a backlink to that references section.
* Given this, when the references tag itself comes from a template,
this throws off the linter attribution code. It assigns the
references template's DSR to the lints from the <ref> tags.
* A clean way of dealing with this would be to treat extension
content as its own independent document (which we are already doing
currently in our wt2html code path) and asking the extension to process
its content while giving it the handler for linting an individual node.
* This patch implements special handlers for <ref> and <references> content
in the Cite extension and registers them.
* Move cite-specific lint handling code to the Cite extension.
* New mocha tests spec this behavior.
Change-Id: Ib57f0303605a73408333e133f8051be0b8d45d69
The citation portion contains fixes for []-encoding in attributes,
based on Iec3439f76ecc2a3543b30b35f8735c92b0cfb711.
Fix some other no- or double-encoding issues with HTML entities in the
process, based on I88e8e2077e6f5eec2b232391f7818370894a62dc
(T103714/T104196).
Unrelated parsertest fix which needed an id fix -- incorrect id
added by the section-tag patches exposed here.
Bug: T152540
Bug: T103714
Change-Id: I12b2a148f7170d20bd9aacd3b5b8ee1965859592
* Follow up to f7594328
* Fixes the regression found in rt:
node bin/roundtrip-test.js --domain ru.wikipedia.org "Феодосия"
* Simple test case (do we really not have one!?):
test <ref>haha{{#tag:ref|ok}}</ref>
Change-Id: Ie0a53a8e885a6d94769034cce4ef432773635842
* The contents of "mw:dom-fragment-token"s was being serialized
after processing to the DOM and stored on the token to be
shuttled through tree building. Only to be reparsed in the
unpacking phase.
* Here we store a pointer to the contents in a fragment map.
* Doing less work results in a performance improvement, though
only slightly because the content still needs to be adopted
by the main document.
Change-Id: Ia0aec7de469101a2a93342ea89daac0f0e73cf1a
* Replace ref markers instead of waiting for cleanup to remove them
since that doesn't happen on embedded html.
Change-Id: Ied746f025a0ac7f14d922aff6640fef3aa4b55b0
