The description in T179082 suggests that by using one document for the
entire parse, we'd probably see some performance gains from not having
to import nodes when we get to the top level pipeline and we'd avoid the
validation errors from 19a9c3c.
However, the spec seems to suggest creating a new document when parsing
an HTML fragment,
https://html.spec.whatwg.org/#html-fragment-parsing-algorithm
And, indeed, domino implements it that way,
12a5f67136/lib/htmlelts.js (L84-L96)
So, the request in T217705 may be a little misguided.
What then is this patch good for? In T221790 the ask is that
sub-pipelines produce DocumentFragment which make for cleaner interfaces
and less confusion when migrating children.
The general outline here is that a document is created when the
environment is constructed that gives us the 1-1 correspondence.
Sub-pipelines do create their own documents for the purpose of tree
building, as in the fragment parsing algorithm, but are then immediately
imported to DocumentFragments to be used for the rest of the
post-processing passes.
Bug: T221790
Bug: T179082
Bug: T217705
Change-Id: Idf856d4e071d742ca38486c8ab402e39b3c8949f
* Remove the id's from follow refs because they were
duplicating the same key value erroneously and also
did not provide useful info. Fixed all tests accordingly.
* Added FIXME which refers to a new Phab ticket about
removing the code which adds style = display-none
that will be moved to CSS at some point.
Bug: T262986
Change-Id: Ib59f5eec951aa83a02357de865df8ab3dd8d2f67
These refs get a `style="display: none;"` since they're
not intended to be user visible.
Follow refs with errors conform to the proposed spec in T251842
Bug: T51538
Change-Id: Ie4ea28e7f9afde24614874bb4b8e07c5cabafa12
* Interim state commit with experimental code.
* Updates to citeParserTests.txt to check now valid follow
functionality and newly passing tests.
* Added to follow refs, <sup style="display: none;" about=...
to suppress display of hidden sups needed for VE to use
in editing follow refs.
* Added code to implemented follow functionality and catch
invalid usage.
Bug: T51538
Change-Id: Ic3ac8237fd2c490cfaf2fe799759742f72f10686
* Bug fix for accessing undefined extsrc member variable in edge
case. See T260082 for deeper explanation of the WT that caused a
case where empty flag is not set and extsrc is also missing, but
since either case including extsrc being unset indicates no
content, this additional check is safe for now.
Bug: T259676
Change-Id: I20750c6977883668c83bdae78fbeb171f899e1ab
* The latter feels more readable and intuitive.
* The option defaults to true.
* To make for simpler code, I ensured that the option value is always
set before it is accessed.
* But, the typeof value still uses the "/sealed/" qualifier.
Alternatively, I could use "/packed/" if we want to adhere to the
config value name more closely.
* Tangentially related changes:
- made getWrapperTokens a private method since it is only used
internal to PipelineUtils.
- remove default value for $opts in encapsulateExpansionHTML since
the value is always passed in everywhere.
Change-Id: I86c4e5adf11e3151f51f2623e5ed85282a2e1298
This was done with a custom sniff in,
MediaWiki/Sniffs/Commenting/FunctionCommentSniff.php
`$singleType === 'null' && count( $explodedType ) === 2`
since there's some ambiguity with,
`what|type|null`
but also a case like the following is left out,
`string[]|null`
Change-Id: I1bd50a4486d7ef4974280b476fd03d3ee53232b3
Alternative to I6ea271a5d5c7b12a13bb12a682c39bcfd7b1f116
We can follow this up by passing the ExtensionTag to the
ExtensionTagHandler constructor.
Change-Id: I5b1b191bc85968ad617eb3ebcdd7721c55006af2
* Bug fixes to accurately match core cite use of
cite_error_ref_no_key error and adjusted citeTests to match.
Bug: T51538
Change-Id: I3ae5300a5f86decebb7e67c5ea57c0c15677fbcc
* named refs which attempt to redefine the content are flagged with
an error, but not follow on named refs that leave content blank
or repeats the original content.
* Fixed cite tests affected by this change to include !! html/parsoid
sections.
Change-Id: I6832603c523a0465a6cc08f68c9ca79499331cd7
Keep the contents in the references and serialize at the end, which
allows us to mark up the errors there.
This is a follow up to 4438a72 which resolves the FIXMEs added in
8cb34b6.
Change-Id: Ia5b5cdbd0e9f3b5c558b8bbc5eb4b9955f4922c9
Also, FIXMEs for a follow up to 4438a72 that's exposed by this test.
Nested refs in references aren't getting marked up for the
"cite_error_ref_no_text" errors, where applicable.
Change-Id: Ie6e461571402a96e47d3df26585d9a40f1038891
* This lets us expand the range of available contexts in the future
without needing API changes.
* This patch only touches extension and extension API code. Parsoid
internal code can be changed independently.
Change-Id: I51d4c2120a31efb6dbb409926f8f8dad61f4dcc3
* Detects grouped and named refs that fail to define content.
* Uses group and name ref list tracking info to back patch
'mw:Error' and i18n error key string into the data-mw
section of all instances of named refs that all fail to
define content.
* The failures for test References: 7b is because selser is
arguable smarter than wt2wt. The newline before the references
list has been randomly deleted but selser manages to restore it
from source. wt2wt doesn't put the references tag on a line by
itself, even though it asks for block format, because it isn't
a new list - (these comments are from Arlo's review)
* Added test: "References: 7b. Multiple references tags some with
errors..." to ensure that refs with and without content errors
grouped and named do not cross references section boundaries.
Bug: T51538
Change-Id: I884fc337165506c5abbef18bcd5a5fca015786d2
Mediawiki prefers to use an object factory pattern when creating objects.
Use ObjectFactory consistently when creating objects specified using the
extension API. Thanks to the 'allowClassName' option to
ObjectFactory::getObjectFromSpec(), this is mostly consistent with
previous practice. Note that the string class name is short for:
[ 'class' => Foo::class ]
and so we've chosen to rename the 'class' property in the extension tag
configuration so that we have (in long form):
[ 'name' => 'Cite', 'handler' => [ 'class' => Cite::class ] ]
instead of nesting two keys named 'class' in a row. (And besides, the
content isn't really a 'class' any more, it's an "object factory
specification".)
SiteConfig::registerExtensionModule() can now take *either* an object
factory specification for an ExtensionModule object (including a bare
class-string) *or* the contents of the configuration array that
would be returned by ExtensionModule::getConfig(), in which case it
creates an anonymous ExtensionModule object for you. It's expected
that the latter will be preferred in extension.json, but we use the
former for our internal extension implementations at the moment.
Finally, call SiteConfig::registerExtensionModule() on the results
of ExtensionRegistery::getInstance()->getAttribute('ParsoidModules')
when running in integrated mode. This allows you to register your
extension with a clause such as the following in your extension.json:
(simple case, naming a class which implements ExtensionModule)
{
"name": "JsonExtension",
"manifest_version": 2,
...
"ParsoidModules": [ "Wikimedia\\Parsoid\\Ext\\JSON" ]
}
(complex case, putting the configuration array into extension.json)
{
"name": "Cite",
"manifest_version": 2,
...
"ParsoidModules": [
{
"name": "Cite",
"domProcessors": [
"Wikimedia\\Parsoid\\Ext\\Cite\\RefProcessor",
],
"tags": [
{
"name": "ref",
"handler": "Wikimedia\\Parsoid\\Ext\\Cite\\Ref",
"options": {
"wt2html": { "sealFragment": true }
},
},
{
"name": "references",
"handler": "Wikimedia\\Parsoid\\Ext\\Cite\\References",
"options": {
"html2wt": { "format": "block" }
},
}
],
"styles": [
"ext.cite.style",
"ext.cite.styles"
]
}
]
}
The syntax above, with `ParsoidModules` as a top-level attribute, requires
I6c74938883376ec17f3790678b435585083a440f in core. However, with or without
that patch, the following also works:
{
...
"attributes": {
"Parsoid": {
"Modules": [ ... ]
}
}
}
Bug: T133320
Change-Id: I20f641a1ff032a6da3549b01dfaf8f4cf1eb5071
The `typeof` attribute is a space-separated list. Use consistent methods
to test for membership in that list.
Try to set a good example by using ::addTypeOf() in general to set the
value of the 'typeof' attribute, except when we are transferring values
from one node to another.
Move the *TypeOf method to DOMUtils from DOMDataUtils, eliminating some
duplication. Create Wikimedia\Parsoid\Ext\DOMUtils so that extensions
can use these methods as well.
Change-Id: Ib2ef827ef1cf5e31a1ef0a5034cdd3d9a0212bdb
Since we aren't excluding the other two, MissingDocumentationPublic and
MissingDocumentationProtected.
The stubs could be useful if we ever expanded on what these functions do
and doxygen probably gets the information from here instead of the type
hints?
Change-Id: Ie18c4f00ceca8f06b9c0f0a3359cb4077892f97d
This will be called by the ExtensionRegistry in core for extensions
that are Parsoid-compatible. The set of registered extensions is part
of the SiteConfig, which bundles all the configuration for a particular
Parsoid instance.
In addition, renamed some classes to make things clearer:
`ExtensionModule` is the thing which is registered; it bundles a
number of `ExtensionTagHandler`s, `ContentModelHandler`s, and DOM
processors (which don't have a proper interface yet). There are a set
of core handlers, which include wikitext, JSON, and a few extension
tags (<pre>, <nowiki>, <gallery>).
Change-Id: Iadbeb378bacb09264a4b1d3ee430a914eec23e48
* We only had a htmlToWikitext API method whereas we have been
trying to stay in DOM land all along. With this change, extensions
can use the intuitive domToWikitext method when they are dealing
with DOM nodes.
* Renamed WTS's serializeHTML method to htmlToWikitext and added
a domToWikitext method there as well which ParsoidExtensionAPI uses.
* Turns out that <ref>s were converting DOM to HTML and then using
the htmlToWikitext method. I switched it use the domToWikitext
method. However, turns out WTS requires a <body> element for its
top-level method!
For now, while we figure out if that can be changed to be more
lenient, added an internal DOM -> HTML conversion in the
domToWikitext method. When we fix WTS, this DOM -> HTML -> DOM
roundtrip can be eliminated.
Bug: T242746
Change-Id: I340d5a363e0d1b8ed6d0ffb0234315e6d9523a76
This is cleaner and less prone to subtle errors since it forces
extension developers to explicitly choose the more performant version.
Bug: T242746
Change-Id: Ia25bc3ae261b43dba97d369940065254faacdd80
Instead of 'fragmentOptions' and 'html2wt' for extension tags,
embed them as 'wt2html' and 'html2wt' components of an 'options'
property.
Bug: T242746
Change-Id: I4cf32a70ec76a415a98b68eef548206f8b917168
* Added DOMDataUtils, WTUtils, and Util for use by extension
developers. These classes might acquire more functionality in the
future based on usage and need.
* Added PHPUtils for a single helper to work around a GC bug in PHP.
Once we move on to a newer version of PHP where this is fixed, we
can get rid of this class and helper.
* These classes proxy the various helpers used by currently ported
extensions. For reasons of coherency, the set of helpers in these
classes are a superset of what the extensions use.
* Updated references to the other helpers to use these classes
* Since DOMUtils or DOMCompat are not Parsoid-centric, it feels safe
to provide extensions direct access to those utils classes. We could
consider moving DOMUtils to the Core/Utils namespace if appropriate.
* In one case, I replaced the escapeNowikiTags helper that the Nowiki
"extension" used with an inlined preg_replace.
* TODO: Add unit tests to ensure these utils don't break!
Bug: T242746
Change-Id: I9e733f4ddd6fca8ce13c2957a7d0065d80f7ae9a
* The functionality looks effectively identical to inlineContext
and everywhere inPHPBlock was inspected, inlineContext was also
being inspected.
* Cite's use of this flag is a hack to get desired bacward compatible
behavior but that is a hack no matter what we call the flag.
Change-Id: I3c62590b9bfda224897bb85b18d96c072f3d74ef
* At this point, DSR is a first-class Parsoid concept and
extensions will need to use this as well. So, make it part
of the Core/ namespace to capture high-level concepts that
might be used outside Parsoid itself.
* Move ParsoidExtensionApi to the Ext directory since that is
where it best belongs.
Change-Id: If824c4af9e2f8d658f1cb726cbd837222b60790d
* Added API method to let content-model extensions to add metadata
to <head>.
* The title API methods seem legitimate
* But, the newAboutId helper is suspect -- currently only needed
by Cite. Explore if we can eliminate the need for this helper.
* This eliminates a few more Env use sites from extensions.
Bug: T242746
Change-Id: I0e982d4be173f7d49df19467fbf49c11d428e650
* Cite (or other extensions) don't need to explicitly load/store
data attributes from html attributes to/from the data bag held
separately from the DOM.
Bug: T242746
Change-Id: I4a52be2b06ccfe53d0cf81987af12a1d139fef4c
* Presumably, extensions would benefit from having access to the
wiki config via SiteConfig.
* Yet to figure out if extensions need access to the page config.
* But, with this change, extensions don't need $env when all they
need is access to the wiki and page config.
Bug: T242746
Change-Id: I88736f882f185ee9376b73f7e4bb0b2bd318bb1a
* This seems to work and also will make the job of keeping extensions
free of DOM state easier.
Arlo clarifies that this wasn't necessary since f7594328 and could
have been cleaned up there.
Change-Id: I96edaa5b2743f1ce0d8596acfdc59035491541cb
* $env was unused in extension DOM post processors. So get rid of
that since we are already in the process of removing $env access
to extensions.
* html2wtPreProcessor is currently unimplemented but there is WIP
code in Parsoid/JS that can be revived at a later point. No need
to pass $env here as well.
* In both cases, pass $extApi so they can access any necessary
helpers or state provided by that API object.
Bug: T242746
Change-Id: I1d1544af817d03e01a569e6aeaeed0d6c3058fc0
* Proxy all accesses to the santiizer via appropriately named methods
in the ParsoidExtensionApi interface
Bug: T242746
Change-Id: I9d3d98639bb98b4abe404139786517591323d61d
* Remove use of $env from ReferencesData and RefGroup by
providing high-level helpers in ParsoidExtensionAPI.
- Given a fragment id, provide helpers to fetch fragment DOM
or fragment HTML
- Fetch the URI for the current page (being parsed)
* There is still a lot of subtle knowledge Cite has about
how data-parsoid and data-mw attributes are held off to the
side in a bag and all the pp* and load/store manipulation
of those attributes. It would be an interesting exercise
to purge this implementation of those notions OR figure out
high-level concepts that we document as being part of Parsoid
reality that we'll forever support.
Bug: T242746
Change-Id: I29ff154f2f17123b9756dfd2f3b422f0b30222b1
* In this patch, toDOM, fromDOM, and DOM postprocessor extension
methods all get a ParsoidExtensionAPI object. These API objects
are constructed at the appropriate times in the wt2html and html2wt
pipelines.
* Got rid of direct references to SerializerState from fromDOM
methods in extensions.
* Exposed generic serialization and wikitext escaping methods
in ParsoidExtensionAPI for extensions to leverage. The implementation
of these methods is partial and only supports current usage
of extensions in Parsoid's repo. This will need to be fully
fleshed out going forward.
* Stopped exposing wt2html options in toto and provided more specific
convenience methods.
* Reduced direct access to the Env object in a few more places.
* Cite has code to inspect embedded HTML in data attributes of a node.
Moved this code out of Cite into ParsoidExtensionAPI which reduces
knowledge that extensions need. Unlike the other cleanups, this one
is more of a convenience method since this code only requires
knowledge of a publicly published spec. But, nevertheless an useful
cleanup since it simplifies Cite's complexity just a bit.
* More followup work is needed.
- before/after methods should be eliminated in favour of a config flag
that implements the inline/block layout option. Once this is done,
extensions will no longer need direct access to the SerializerState
internal object.
- Env exposure should be reduced.
- Provide access to Sanitizer via ParsoidExtensionAPI instead of
needing extensions to directly import it.
- It should be possible to eliminate the need for extensions to know
about DSR / DSR-shifting and do it automatically via some high-level
conceptual flag.
- It might also be possible to infer source offsets directly via args
instead of passing that explicitly.
- Should we provide a convenience helper class with access to all the
src/Utils/* methods?
Bug: T242746
Change-Id: I7ffb5aa52a84854a9d363a0e8f1ce650241f1c41
