Also, FIXMEs for a follow up to 4438a72 that's exposed by this test.
Nested refs in references aren't getting marked up for the
"cite_error_ref_no_text" errors, where applicable.
Change-Id: Ie6e461571402a96e47d3df26585d9a40f1038891
* This lets us expand the range of available contexts in the future
without needing API changes.
* This patch only touches extension and extension API code. Parsoid
internal code can be changed independently.
Change-Id: I51d4c2120a31efb6dbb409926f8f8dad61f4dcc3
* Detects grouped and named refs that fail to define content.
* Uses group and name ref list tracking info to back patch
'mw:Error' and i18n error key string into the data-mw
section of all instances of named refs that all fail to
define content.
* The failures for test References: 7b is because selser is
arguable smarter than wt2wt. The newline before the references
list has been randomly deleted but selser manages to restore it
from source. wt2wt doesn't put the references tag on a line by
itself, even though it asks for block format, because it isn't
a new list - (these comments are from Arlo's review)
* Added test: "References: 7b. Multiple references tags some with
errors..." to ensure that refs with and without content errors
grouped and named do not cross references section boundaries.
Bug: T51538
Change-Id: I884fc337165506c5abbef18bcd5a5fca015786d2
Mediawiki prefers to use an object factory pattern when creating objects.
Use ObjectFactory consistently when creating objects specified using the
extension API. Thanks to the 'allowClassName' option to
ObjectFactory::getObjectFromSpec(), this is mostly consistent with
previous practice. Note that the string class name is short for:
[ 'class' => Foo::class ]
and so we've chosen to rename the 'class' property in the extension tag
configuration so that we have (in long form):
[ 'name' => 'Cite', 'handler' => [ 'class' => Cite::class ] ]
instead of nesting two keys named 'class' in a row. (And besides, the
content isn't really a 'class' any more, it's an "object factory
specification".)
SiteConfig::registerExtensionModule() can now take *either* an object
factory specification for an ExtensionModule object (including a bare
class-string) *or* the contents of the configuration array that
would be returned by ExtensionModule::getConfig(), in which case it
creates an anonymous ExtensionModule object for you. It's expected
that the latter will be preferred in extension.json, but we use the
former for our internal extension implementations at the moment.
Finally, call SiteConfig::registerExtensionModule() on the results
of ExtensionRegistery::getInstance()->getAttribute('ParsoidModules')
when running in integrated mode. This allows you to register your
extension with a clause such as the following in your extension.json:
(simple case, naming a class which implements ExtensionModule)
{
"name": "JsonExtension",
"manifest_version": 2,
...
"ParsoidModules": [ "Wikimedia\\Parsoid\\Ext\\JSON" ]
}
(complex case, putting the configuration array into extension.json)
{
"name": "Cite",
"manifest_version": 2,
...
"ParsoidModules": [
{
"name": "Cite",
"domProcessors": [
"Wikimedia\\Parsoid\\Ext\\Cite\\RefProcessor",
],
"tags": [
{
"name": "ref",
"handler": "Wikimedia\\Parsoid\\Ext\\Cite\\Ref",
"options": {
"wt2html": { "sealFragment": true }
},
},
{
"name": "references",
"handler": "Wikimedia\\Parsoid\\Ext\\Cite\\References",
"options": {
"html2wt": { "format": "block" }
},
}
],
"styles": [
"ext.cite.style",
"ext.cite.styles"
]
}
]
}
The syntax above, with `ParsoidModules` as a top-level attribute, requires
I6c74938883376ec17f3790678b435585083a440f in core. However, with or without
that patch, the following also works:
{
...
"attributes": {
"Parsoid": {
"Modules": [ ... ]
}
}
}
Bug: T133320
Change-Id: I20f641a1ff032a6da3549b01dfaf8f4cf1eb5071
The `typeof` attribute is a space-separated list. Use consistent methods
to test for membership in that list.
Try to set a good example by using ::addTypeOf() in general to set the
value of the 'typeof' attribute, except when we are transferring values
from one node to another.
Move the *TypeOf method to DOMUtils from DOMDataUtils, eliminating some
duplication. Create Wikimedia\Parsoid\Ext\DOMUtils so that extensions
can use these methods as well.
Change-Id: Ib2ef827ef1cf5e31a1ef0a5034cdd3d9a0212bdb
Since we aren't excluding the other two, MissingDocumentationPublic and
MissingDocumentationProtected.
The stubs could be useful if we ever expanded on what these functions do
and doxygen probably gets the information from here instead of the type
hints?
Change-Id: Ie18c4f00ceca8f06b9c0f0a3359cb4077892f97d
This will be called by the ExtensionRegistry in core for extensions
that are Parsoid-compatible. The set of registered extensions is part
of the SiteConfig, which bundles all the configuration for a particular
Parsoid instance.
In addition, renamed some classes to make things clearer:
`ExtensionModule` is the thing which is registered; it bundles a
number of `ExtensionTagHandler`s, `ContentModelHandler`s, and DOM
processors (which don't have a proper interface yet). There are a set
of core handlers, which include wikitext, JSON, and a few extension
tags (<pre>, <nowiki>, <gallery>).
Change-Id: Iadbeb378bacb09264a4b1d3ee430a914eec23e48
* We only had a htmlToWikitext API method whereas we have been
trying to stay in DOM land all along. With this change, extensions
can use the intuitive domToWikitext method when they are dealing
with DOM nodes.
* Renamed WTS's serializeHTML method to htmlToWikitext and added
a domToWikitext method there as well which ParsoidExtensionAPI uses.
* Turns out that <ref>s were converting DOM to HTML and then using
the htmlToWikitext method. I switched it use the domToWikitext
method. However, turns out WTS requires a <body> element for its
top-level method!
For now, while we figure out if that can be changed to be more
lenient, added an internal DOM -> HTML conversion in the
domToWikitext method. When we fix WTS, this DOM -> HTML -> DOM
roundtrip can be eliminated.
Bug: T242746
Change-Id: I340d5a363e0d1b8ed6d0ffb0234315e6d9523a76
This is cleaner and less prone to subtle errors since it forces
extension developers to explicitly choose the more performant version.
Bug: T242746
Change-Id: Ia25bc3ae261b43dba97d369940065254faacdd80
Instead of 'fragmentOptions' and 'html2wt' for extension tags,
embed them as 'wt2html' and 'html2wt' components of an 'options'
property.
Bug: T242746
Change-Id: I4cf32a70ec76a415a98b68eef548206f8b917168
* Added DOMDataUtils, WTUtils, and Util for use by extension
developers. These classes might acquire more functionality in the
future based on usage and need.
* Added PHPUtils for a single helper to work around a GC bug in PHP.
Once we move on to a newer version of PHP where this is fixed, we
can get rid of this class and helper.
* These classes proxy the various helpers used by currently ported
extensions. For reasons of coherency, the set of helpers in these
classes are a superset of what the extensions use.
* Updated references to the other helpers to use these classes
* Since DOMUtils or DOMCompat are not Parsoid-centric, it feels safe
to provide extensions direct access to those utils classes. We could
consider moving DOMUtils to the Core/Utils namespace if appropriate.
* In one case, I replaced the escapeNowikiTags helper that the Nowiki
"extension" used with an inlined preg_replace.
* TODO: Add unit tests to ensure these utils don't break!
Bug: T242746
Change-Id: I9e733f4ddd6fca8ce13c2957a7d0065d80f7ae9a
* The functionality looks effectively identical to inlineContext
and everywhere inPHPBlock was inspected, inlineContext was also
being inspected.
* Cite's use of this flag is a hack to get desired bacward compatible
behavior but that is a hack no matter what we call the flag.
Change-Id: I3c62590b9bfda224897bb85b18d96c072f3d74ef
* At this point, DSR is a first-class Parsoid concept and
extensions will need to use this as well. So, make it part
of the Core/ namespace to capture high-level concepts that
might be used outside Parsoid itself.
* Move ParsoidExtensionApi to the Ext directory since that is
where it best belongs.
Change-Id: If824c4af9e2f8d658f1cb726cbd837222b60790d
* Added API method to let content-model extensions to add metadata
to <head>.
* The title API methods seem legitimate
* But, the newAboutId helper is suspect -- currently only needed
by Cite. Explore if we can eliminate the need for this helper.
* This eliminates a few more Env use sites from extensions.
Bug: T242746
Change-Id: I0e982d4be173f7d49df19467fbf49c11d428e650
* Cite (or other extensions) don't need to explicitly load/store
data attributes from html attributes to/from the data bag held
separately from the DOM.
Bug: T242746
Change-Id: I4a52be2b06ccfe53d0cf81987af12a1d139fef4c
* Presumably, extensions would benefit from having access to the
wiki config via SiteConfig.
* Yet to figure out if extensions need access to the page config.
* But, with this change, extensions don't need $env when all they
need is access to the wiki and page config.
Bug: T242746
Change-Id: I88736f882f185ee9376b73f7e4bb0b2bd318bb1a
* This seems to work and also will make the job of keeping extensions
free of DOM state easier.
Arlo clarifies that this wasn't necessary since f7594328 and could
have been cleaned up there.
Change-Id: I96edaa5b2743f1ce0d8596acfdc59035491541cb
* $env was unused in extension DOM post processors. So get rid of
that since we are already in the process of removing $env access
to extensions.
* html2wtPreProcessor is currently unimplemented but there is WIP
code in Parsoid/JS that can be revived at a later point. No need
to pass $env here as well.
* In both cases, pass $extApi so they can access any necessary
helpers or state provided by that API object.
Bug: T242746
Change-Id: I1d1544af817d03e01a569e6aeaeed0d6c3058fc0
* Proxy all accesses to the santiizer via appropriately named methods
in the ParsoidExtensionApi interface
Bug: T242746
Change-Id: I9d3d98639bb98b4abe404139786517591323d61d
* Remove use of $env from ReferencesData and RefGroup by
providing high-level helpers in ParsoidExtensionAPI.
- Given a fragment id, provide helpers to fetch fragment DOM
or fragment HTML
- Fetch the URI for the current page (being parsed)
* There is still a lot of subtle knowledge Cite has about
how data-parsoid and data-mw attributes are held off to the
side in a bag and all the pp* and load/store manipulation
of those attributes. It would be an interesting exercise
to purge this implementation of those notions OR figure out
high-level concepts that we document as being part of Parsoid
reality that we'll forever support.
Bug: T242746
Change-Id: I29ff154f2f17123b9756dfd2f3b422f0b30222b1
* In this patch, toDOM, fromDOM, and DOM postprocessor extension
methods all get a ParsoidExtensionAPI object. These API objects
are constructed at the appropriate times in the wt2html and html2wt
pipelines.
* Got rid of direct references to SerializerState from fromDOM
methods in extensions.
* Exposed generic serialization and wikitext escaping methods
in ParsoidExtensionAPI for extensions to leverage. The implementation
of these methods is partial and only supports current usage
of extensions in Parsoid's repo. This will need to be fully
fleshed out going forward.
* Stopped exposing wt2html options in toto and provided more specific
convenience methods.
* Reduced direct access to the Env object in a few more places.
* Cite has code to inspect embedded HTML in data attributes of a node.
Moved this code out of Cite into ParsoidExtensionAPI which reduces
knowledge that extensions need. Unlike the other cleanups, this one
is more of a convenience method since this code only requires
knowledge of a publicly published spec. But, nevertheless an useful
cleanup since it simplifies Cite's complexity just a bit.
* More followup work is needed.
- before/after methods should be eliminated in favour of a config flag
that implements the inline/block layout option. Once this is done,
extensions will no longer need direct access to the SerializerState
internal object.
- Env exposure should be reduced.
- Provide access to Sanitizer via ParsoidExtensionAPI instead of
needing extensions to directly import it.
- It should be possible to eliminate the need for extensions to know
about DSR / DSR-shifting and do it automatically via some high-level
conceptual flag.
- It might also be possible to infer source offsets directly via args
instead of passing that explicitly.
- Should we provide a convenience helper class with access to all the
src/Utils/* methods?
Bug: T242746
Change-Id: I7ffb5aa52a84854a9d363a0e8f1ce650241f1c41
It is about time to implement a generic fix for T214994 which will
capture the Cite-specific scenario we are tackling here in bits
and pieces. We have fragments of a T214994 solution spread in
Utils/ and here while tackling shiftDSR and convertOffsets
for Parsoid/PHP. So, someone needs to take a look at all these
pieces and implement the necessary abstraction.
As part of that, maybe we also need to revisit our embedded HTML
data-* representation so there is a uniform way to inspect these
attributes rather than every semantic element (template, extensions,
lang variants, media) choosing their own custom embedding scheme
which makes discoverability harder and error-prone.
Change-Id: Ifd19d2d8d20dbd0dda0fa1cc338a07afb37c4213
This managed to catch a few bugs where there is likely coverage gaps.
The regenerated plugin list again comments out a few that generated too
much noise but may be worthwhile follow ups.
Change-Id: I746abd0f1406b4b7ffa497afdad3939fe437c25d
* This is an instance of a bigger issue that we need to look closely
when we are integrating Parsoid with core parser.
Bug: T235656
Change-Id: I3d652727293461c7968e83be8994ba0572bae8e4
* Though this doesn't immediately affect anything, it just makes
usage a bit more consistent.
* A followup patch that fixes gaps in shiftDSR code will now be
able to reference the html property as dmw->body->html to match
html2wt usage.
Change-Id: I9dfcd9d40205f6e64e139bf3f75a322915af3232
* When a page is missing an explicit <references /> tag, we insert
an implicit <references /> tag and assign it a zero-width DSR with
a starting offset equal to the length of the page. However, now that
we have byte offsets, that should have been strlen, not mb_strlen.
This was causing incorrect DSR assignment on this implicit tag
and causing trailing newline selser diffs on these pages.
* Debugged on this reduced test case: "* a – b <ref>x</ref>\n\nc\n\n"
and comparing selser trace and then DSR offsets on the DOMs.
Change-Id: I8aebf307197935259df78251fb4a26c593f29603
Tim Starling has indicated in couple different places that end(..)
is not preferred and he had implemented a private version of lastItem
in the PEG grammar code whereas PHPUtils::lastItem was recommending
use of end(..).
In this patch, I moved the implementation from the grammar to PHPUtils
and replaced end(..) with PHPUtils::lastItem in a number of places in
the codebase. We should discuss whether we want to use this helper
everywhere.
Resolved a couple of PORT-FIXMEs in the bargain.
Change-Id: I837f2a98003df8ab7dbdf9af045e17bdd6e27799
* We could potentially also exclude regexps for node name checks
* A few additions looks like could potentially have caused subtle
failures in edge cases.
* Unrelated changes: Used # regexp terminator in a number of regexps
to eliminate escaping of / character.
Bug: T231980
Change-Id: Ie2451349684c248d93e064e3e7009d0d2d60acf3
The extension handler expects a `null` for this, `false` is an
indication not to use the toDOM handler.
php bin/parserTests.php tests/citeParserTests.txt --filter "CircularRef" --wt2html
Change-Id: I849a9aca1133f8a793c9d77e05f192a6af5d78f9
