The `typeof` attribute is a space-separated list. Use consistent methods
to test for membership in that list.
Try to set a good example by using ::addTypeOf() in general to set the
value of the 'typeof' attribute, except when we are transferring values
from one node to another.
Move the *TypeOf method to DOMUtils from DOMDataUtils, eliminating some
duplication. Create Wikimedia\Parsoid\Ext\DOMUtils so that extensions
can use these methods as well.
Change-Id: Ib2ef827ef1cf5e31a1ef0a5034cdd3d9a0212bdb
Since we aren't excluding the other two, MissingDocumentationPublic and
MissingDocumentationProtected.
The stubs could be useful if we ever expanded on what these functions do
and doxygen probably gets the information from here instead of the type
hints?
Change-Id: Ie18c4f00ceca8f06b9c0f0a3359cb4077892f97d
This will be called by the ExtensionRegistry in core for extensions
that are Parsoid-compatible. The set of registered extensions is part
of the SiteConfig, which bundles all the configuration for a particular
Parsoid instance.
In addition, renamed some classes to make things clearer:
`ExtensionModule` is the thing which is registered; it bundles a
number of `ExtensionTagHandler`s, `ContentModelHandler`s, and DOM
processors (which don't have a proper interface yet). There are a set
of core handlers, which include wikitext, JSON, and a few extension
tags (<pre>, <nowiki>, <gallery>).
Change-Id: Iadbeb378bacb09264a4b1d3ee430a914eec23e48
* We only had a htmlToWikitext API method whereas we have been
trying to stay in DOM land all along. With this change, extensions
can use the intuitive domToWikitext method when they are dealing
with DOM nodes.
* Renamed WTS's serializeHTML method to htmlToWikitext and added
a domToWikitext method there as well which ParsoidExtensionAPI uses.
* Turns out that <ref>s were converting DOM to HTML and then using
the htmlToWikitext method. I switched it use the domToWikitext
method. However, turns out WTS requires a <body> element for its
top-level method!
For now, while we figure out if that can be changed to be more
lenient, added an internal DOM -> HTML conversion in the
domToWikitext method. When we fix WTS, this DOM -> HTML -> DOM
roundtrip can be eliminated.
Bug: T242746
Change-Id: I340d5a363e0d1b8ed6d0ffb0234315e6d9523a76
This is cleaner and less prone to subtle errors since it forces
extension developers to explicitly choose the more performant version.
Bug: T242746
Change-Id: Ia25bc3ae261b43dba97d369940065254faacdd80
Instead of 'fragmentOptions' and 'html2wt' for extension tags,
embed them as 'wt2html' and 'html2wt' components of an 'options'
property.
Bug: T242746
Change-Id: I4cf32a70ec76a415a98b68eef548206f8b917168
* Added DOMDataUtils, WTUtils, and Util for use by extension
developers. These classes might acquire more functionality in the
future based on usage and need.
* Added PHPUtils for a single helper to work around a GC bug in PHP.
Once we move on to a newer version of PHP where this is fixed, we
can get rid of this class and helper.
* These classes proxy the various helpers used by currently ported
extensions. For reasons of coherency, the set of helpers in these
classes are a superset of what the extensions use.
* Updated references to the other helpers to use these classes
* Since DOMUtils or DOMCompat are not Parsoid-centric, it feels safe
to provide extensions direct access to those utils classes. We could
consider moving DOMUtils to the Core/Utils namespace if appropriate.
* In one case, I replaced the escapeNowikiTags helper that the Nowiki
"extension" used with an inlined preg_replace.
* TODO: Add unit tests to ensure these utils don't break!
Bug: T242746
Change-Id: I9e733f4ddd6fca8ce13c2957a7d0065d80f7ae9a
* The functionality looks effectively identical to inlineContext
and everywhere inPHPBlock was inspected, inlineContext was also
being inspected.
* Cite's use of this flag is a hack to get desired bacward compatible
behavior but that is a hack no matter what we call the flag.
Change-Id: I3c62590b9bfda224897bb85b18d96c072f3d74ef
* At this point, DSR is a first-class Parsoid concept and
extensions will need to use this as well. So, make it part
of the Core/ namespace to capture high-level concepts that
might be used outside Parsoid itself.
* Move ParsoidExtensionApi to the Ext directory since that is
where it best belongs.
Change-Id: If824c4af9e2f8d658f1cb726cbd837222b60790d
* Added API method to let content-model extensions to add metadata
to <head>.
* The title API methods seem legitimate
* But, the newAboutId helper is suspect -- currently only needed
by Cite. Explore if we can eliminate the need for this helper.
* This eliminates a few more Env use sites from extensions.
Bug: T242746
Change-Id: I0e982d4be173f7d49df19467fbf49c11d428e650
* Cite (or other extensions) don't need to explicitly load/store
data attributes from html attributes to/from the data bag held
separately from the DOM.
Bug: T242746
Change-Id: I4a52be2b06ccfe53d0cf81987af12a1d139fef4c
* Presumably, extensions would benefit from having access to the
wiki config via SiteConfig.
* Yet to figure out if extensions need access to the page config.
* But, with this change, extensions don't need $env when all they
need is access to the wiki and page config.
Bug: T242746
Change-Id: I88736f882f185ee9376b73f7e4bb0b2bd318bb1a
* This seems to work and also will make the job of keeping extensions
free of DOM state easier.
Arlo clarifies that this wasn't necessary since f7594328 and could
have been cleaned up there.
Change-Id: I96edaa5b2743f1ce0d8596acfdc59035491541cb
* $env was unused in extension DOM post processors. So get rid of
that since we are already in the process of removing $env access
to extensions.
* html2wtPreProcessor is currently unimplemented but there is WIP
code in Parsoid/JS that can be revived at a later point. No need
to pass $env here as well.
* In both cases, pass $extApi so they can access any necessary
helpers or state provided by that API object.
Bug: T242746
Change-Id: I1d1544af817d03e01a569e6aeaeed0d6c3058fc0
* Proxy all accesses to the santiizer via appropriately named methods
in the ParsoidExtensionApi interface
Bug: T242746
Change-Id: I9d3d98639bb98b4abe404139786517591323d61d
* Remove use of $env from ReferencesData and RefGroup by
providing high-level helpers in ParsoidExtensionAPI.
- Given a fragment id, provide helpers to fetch fragment DOM
or fragment HTML
- Fetch the URI for the current page (being parsed)
* There is still a lot of subtle knowledge Cite has about
how data-parsoid and data-mw attributes are held off to the
side in a bag and all the pp* and load/store manipulation
of those attributes. It would be an interesting exercise
to purge this implementation of those notions OR figure out
high-level concepts that we document as being part of Parsoid
reality that we'll forever support.
Bug: T242746
Change-Id: I29ff154f2f17123b9756dfd2f3b422f0b30222b1
* In this patch, toDOM, fromDOM, and DOM postprocessor extension
methods all get a ParsoidExtensionAPI object. These API objects
are constructed at the appropriate times in the wt2html and html2wt
pipelines.
* Got rid of direct references to SerializerState from fromDOM
methods in extensions.
* Exposed generic serialization and wikitext escaping methods
in ParsoidExtensionAPI for extensions to leverage. The implementation
of these methods is partial and only supports current usage
of extensions in Parsoid's repo. This will need to be fully
fleshed out going forward.
* Stopped exposing wt2html options in toto and provided more specific
convenience methods.
* Reduced direct access to the Env object in a few more places.
* Cite has code to inspect embedded HTML in data attributes of a node.
Moved this code out of Cite into ParsoidExtensionAPI which reduces
knowledge that extensions need. Unlike the other cleanups, this one
is more of a convenience method since this code only requires
knowledge of a publicly published spec. But, nevertheless an useful
cleanup since it simplifies Cite's complexity just a bit.
* More followup work is needed.
- before/after methods should be eliminated in favour of a config flag
that implements the inline/block layout option. Once this is done,
extensions will no longer need direct access to the SerializerState
internal object.
- Env exposure should be reduced.
- Provide access to Sanitizer via ParsoidExtensionAPI instead of
needing extensions to directly import it.
- It should be possible to eliminate the need for extensions to know
about DSR / DSR-shifting and do it automatically via some high-level
conceptual flag.
- It might also be possible to infer source offsets directly via args
instead of passing that explicitly.
- Should we provide a convenience helper class with access to all the
src/Utils/* methods?
Bug: T242746
Change-Id: I7ffb5aa52a84854a9d363a0e8f1ce650241f1c41
It is about time to implement a generic fix for T214994 which will
capture the Cite-specific scenario we are tackling here in bits
and pieces. We have fragments of a T214994 solution spread in
Utils/ and here while tackling shiftDSR and convertOffsets
for Parsoid/PHP. So, someone needs to take a look at all these
pieces and implement the necessary abstraction.
As part of that, maybe we also need to revisit our embedded HTML
data-* representation so there is a uniform way to inspect these
attributes rather than every semantic element (template, extensions,
lang variants, media) choosing their own custom embedding scheme
which makes discoverability harder and error-prone.
Change-Id: Ifd19d2d8d20dbd0dda0fa1cc338a07afb37c4213
This managed to catch a few bugs where there is likely coverage gaps.
The regenerated plugin list again comments out a few that generated too
much noise but may be worthwhile follow ups.
Change-Id: I746abd0f1406b4b7ffa497afdad3939fe437c25d
* This is an instance of a bigger issue that we need to look closely
when we are integrating Parsoid with core parser.
Bug: T235656
Change-Id: I3d652727293461c7968e83be8994ba0572bae8e4
* Though this doesn't immediately affect anything, it just makes
usage a bit more consistent.
* A followup patch that fixes gaps in shiftDSR code will now be
able to reference the html property as dmw->body->html to match
html2wt usage.
Change-Id: I9dfcd9d40205f6e64e139bf3f75a322915af3232
* When a page is missing an explicit <references /> tag, we insert
an implicit <references /> tag and assign it a zero-width DSR with
a starting offset equal to the length of the page. However, now that
we have byte offsets, that should have been strlen, not mb_strlen.
This was causing incorrect DSR assignment on this implicit tag
and causing trailing newline selser diffs on these pages.
* Debugged on this reduced test case: "* a – b <ref>x</ref>\n\nc\n\n"
and comparing selser trace and then DSR offsets on the DOMs.
Change-Id: I8aebf307197935259df78251fb4a26c593f29603
Tim Starling has indicated in couple different places that end(..)
is not preferred and he had implemented a private version of lastItem
in the PEG grammar code whereas PHPUtils::lastItem was recommending
use of end(..).
In this patch, I moved the implementation from the grammar to PHPUtils
and replaced end(..) with PHPUtils::lastItem in a number of places in
the codebase. We should discuss whether we want to use this helper
everywhere.
Resolved a couple of PORT-FIXMEs in the bargain.
Change-Id: I837f2a98003df8ab7dbdf9af045e17bdd6e27799
* We could potentially also exclude regexps for node name checks
* A few additions looks like could potentially have caused subtle
failures in edge cases.
* Unrelated changes: Used # regexp terminator in a number of regexps
to eliminate escaping of / character.
Bug: T231980
Change-Id: Ie2451349684c248d93e064e3e7009d0d2d60acf3
The extension handler expects a `null` for this, `false` is an
indication not to use the toDOM handler.
php bin/parserTests.php tests/citeParserTests.txt --filter "CircularRef" --wt2html
Change-Id: I849a9aca1133f8a793c9d77e05f192a6af5d78f9
Introduce PHPUtils::arrayToObject() which duplicates the array before
converting it to an object. Workaround for
https://bugs.php.net/bug.php?id=78379
Bug: T228346
Change-Id: If9ef35e9e5183117025bc9cd705b695f270aa244
Two classic PHP errors
- string '0' is a falsy value in if conditions and so we need an
explicit === '' check for empty strings
- arrays need to be passed by reference to capture modifications
in callers.
Change-Id: I07d0e39c44a923ac1faeb2de01433e951c3de914
* Don't unconditionally run fromHTML and before handlers without
checking if we have a native extension handler.
* Remove unintential implementation of `before` in Ref.php
* Hybrid tests for Cite now passes again.
* Discovered while running native parser tests and isolated those
crashers to this.
Change-Id: I45b48b595a5aee2b8b8d00b4ebcf73a5ea7bc8a3
Restructure ExtensionTag as an abstract class with default do-nothing
implementations of all methods. So instead of Translate and LST
not implementing ExtensionTag::toDOM, they inherit the default
implementation which returns false, and that has the same effect.
The intention is to move SerialHandler::fromHTML and SerialHandler::before
into this framework as well. Every "optional" method should have a default
implementation in the base class which returns false.
Change-Id: I0ad5c714601c0cf0b5189d4d282c67c6b53fc760
Passing srcOffsets which don't actually correspond to actual regions of
the source wikitext cause problems in the token offset conversion code.
Instead, parse the wikitext as itself, then adjust the TSRs in the DOM
tree.
Since Gallery isn't ported to PHP (yet), update the
automatically-generated Gallery/index.php. The newly-added
ContentUtils::shiftDSR() was ported, however.
Change-Id: I28f3d3398930733ae2bcf9759e49c45f93bc7190
* Now that we are in sync land, we don't need to catch exceptions
and log error messages at multiple places. Let them bubble up
to the top.
* I noticed this was actually getting in the way of debugging because
with $env->log unimplemented, I was only getting very generic
failures instead of the root cause that was being suppressed
and unlogged.
* There are still a couple of places where we have generic Exception
catching in place where it does make sense currently. For example,
we aren't interested in what caused a templatedata fetch to fail.
We simply fall back to regular serialization - the rationale here is
that it is better to emit a transclusion without the preferred formatting
(but syntactically correct) instead of losing the edit altogether.
* Minor unrelated fix in Cite/Ref.php: Use !isset() instead of empty()
Change-Id: Iebff6f37dcd8278185c4a74b72a99b528efa20ff
* Source offset fixes: followup to 31d356a5
- there were instances of $tsr[0] and $tsr[1] that hadn't
been converted over to $tsr->start, $tsr->end
- removed dead code
* Cite fixes: followup ot 005176a3
- Fixes array / object mixups
- Bug fix
* html2wt/WikiLinkHandler fixes
- Protect access to missing properties in data-mw opt list
* Other assorted fixes
- Added missing typehints and improved doc types
- Simplified some code patterns
- Cast extension attributes to object since that ends up
in data-mw which is a stdclass object.
Change-Id: Idd04b0d3819be3660823047a90330fd1213388cf
* All wt2wt, html2wt, and all but one html2html tests pass in
hybrid mode when entire html2wt code is run in PHP
Set "Serializer: true" in the html2wt section of phpconfig.yaml
* The single failing html2html test is a <gallery> test which is
presumably related to the unported <gallery> extension code, but
not sure. Not investigating it now.
* Update Parsoid Extension API to provide access to extension source
without exposing internals.
Change-Id: I6d6e21ad2324acfc4306b32c9055d6c088708c48
* unwrapFragment had a somewhat unusual behavior which could be
a source of bugs while reasoning with it.
If undefined, it's default value is true which is contrary
to how we think of undefined.
* Flipping the polarity of the flag to sealFragment makes the
semantics easier to reason with and where !empty(..) applies
more naturally to it.
Change-Id: Ia50cba345f37e815e5f5f95abb452c8eefcf9011
In PHP, DOMNode#getAttribute() return '' if the attribute is not present,
not null. Audit our uses and try to either explicitly use `|| ''` (which
will ensure that PHP behaves the same was as JS) or use `hasAttribute`
to explicitly test for the presence of the attribute.
Changes have also been ported to PHP from JS.
Also added src/Wt2Html/PP/Processors/AddMediaInfo.php which was missing.
Change-Id: Ie1ae1df88e4fca70daf97b6f720f28014ebc99ed
This applies the JS changes from the following recently-merged patches:
6679c3bf Protect data-object-id attribute
d4e76d5b Fix new linter category to enable code work with templates
e567db8d Tweak storeDataAttribs to suppress DOM nodes in data-parsoid.tmp
16603953 Fix setting dsr on body for genTest
3a84a9dd Fix stashing data attributes for mw:StartTag
22c4a19a Remove redundant dataParsoid call
ed7b0ba0 Fix crasher in newly added linter category
505a357b Linter.js: Add new function to detect the use of links in links
8885b20e Move redlink updating into lib/parse.js
ccfce23d templatedepth is either an int or false
6d1571bd Move language conversion work into lib/parse.js
5a89c7de Avoid serialize/parse of data attributes when treebuilding
021d9958 Rename `document.env` to `document.bag`
c03ba494 Use XMLSerializer on both PHP & JS side in the DOM pass test script
e0c3cca9 Use env.createDocument in lib/api/apiUtils.js
550d3d71 Use a bag-on-the-side implementation for node data
f8de8b25 Add bin/inspectTokenizer.js
db704eea Add ability to splice a PHP transformer into the pipeline
a8be3ad6 Fix crasher in cite extension from accessing data after it's stored
2874f200 Simplify and clean up stops usage
6368265d Add some strategic isElt guards
5ae9553f DRY out transform test runners + tweak genTest to enable that
b0f2adc6 Assert that the .dataobject isn't touched after storing attrs on a node
1ce6a98d Skip separators when looking for the next th/td
Change-Id: I6a66ecb061e7ee7ed53feba1895dd315d9324715
* This initialization lets us do a git log --follow and follow
git history for that file across the js -> php port boundary.
This works because git uses content hashes for objects and
the copied code in the new .php file will have the same content
hash as the .js file.
* The following directories were skipped
- ./lib/config/baseconfig
- ./tests
* The following JS files were skipped
- ./lib/utils/promise.js
- ./lib/config/wmf.sitematrix.json
- ./tools/sync-baseconfig.js
- ./tools/sync-parserTests.js
- ./tools/fetch_ve_nowiki_edits.js
- ./tools/fetch-parserTests.txt.js
- ./tools/fetch-wmf-sitematrix.js
- ./tools/compare.linter.results.js
- ./tools/fetch-revision-data.js
- ./tools/fetch-wt.js
- ./tools/regression-testing.js
- ./tools/build-langconv-fst.js
- ./bin/server.js
Change-Id: I0b22057c23b72795aebbd66e3abcb627c6858ef3
