Use the '--json' flag to get Pygments to output its list of supported
lexers in a machine-readable format. Support for this flag was added (at
our request) to Pygments and included in the 2.11 release[1].
Tested by running updateLexerList.php and confirming empty diff.
[1]: https://github.com/pygments/pygments/issues/1437
Change-Id: I0f1d7fceca9034e6034bafa6a8dd312b99d379d1
Changes:
<https://github.com/pygments/pygments/blob/2.8.0/CHANGES#L10-L150>
The 2.8.0 release contains fixes to existing lexers, as well as the
following new lexers:
amdgpu, cddl, futhark, graphviz/dot, markdown
Bug: T274741
Change-Id: I84c32591a06aac5e1afe46dab1f80bb53d981bb3
Changes:
<https://github.com/pygments/pygments/blob/2.7.2/CHANGES#L15-L158>
The 2.7.2 release contains fixes to existing lexers, as well as the
following new lexers:
arrow, bare, devicetree, dmesg, dts, execline, fstar, gd, gdscript,
ipython, ipython2, ipython3, ipythonconsole, kmsg, llvm-mir,
llvm-mir-body, miniscript, mosel, ms, peg, pointless, promql, psysh,
raku, reason, reasonml, ride, sieve, singularity, tid, tnt, usd, usda,
webidl, and yang
To verify the integrity of the pygmentize binary, you can run
'create_pygmentize_bundle' and verify that the same pygmentize file is
created.
Change-Id: I0be3c1c204237b4a8873063e282be7ab8655be9d
We originally started using symfony/process because kzykhys/pygments
depended upon it. But that library was unmaintained and became broken,
so we stopped using it, and just used symfony/process directly.
At the time, the main reason in favor of symfony/process was that it
could pass stdin to pygments, while Shell\Command couldn't - but it can
now (T182463)! On top of that, there are downsides, like not respecting
the default MediaWiki shell limits, being incompatible with core's
firejail support, and requiring an external composer dependency.
Note that because Shell::command() will enforce MediaWiki's normal
limits, it's possible that some large pages may no longer render with
syntax highlighting if they pass those limits.
Bug: T182467
Bug: T181771
Change-Id: Ie1cb72b7eb17d943f79ecae4d94a2110546ef039
This makes installation easier for local development.
Per https://www.mediawiki.org/wiki/Manual:External_libraries
I was supicious about whether 'use KzykHys\Pygments\Pygments;'
works in a file if the namespace/class itself is only loaded
later, but it works fine.
Ref T103890.
Change-Id: Ia1a577f555cd8ab09969dec219316bc1e76769f4
It previously supported more values, but since the switch to Pygments
these additional values are no longer supported. It now always
outputs <pre> or <code>, not e.g. <div>.
Change-Id: I406eee0eeeb3b9aaec3fa464ffdf6e200c868b84
Include Pygments 2.0.2 as an executable zip bundle. Also include a script to
automate the process of creating such bundles and to make it reproducible and
verifiable.
Change-Id: I67e6f804e493f065311164c610dc541a5779654e
GeSHi is unmaintained, lacks support for many popular modern languages, and
suffers from deep architectural flaws, chief among them the inconsistent
tokenization of different languages, each of which requires a custom
stylesheet.
Pygments is a well-maintained alternative. It is, by my count, the most popular
syntax highlighting library around. It is BSD-licensed, actively maintained,
and is widely used in PHP projects.
To keep this easy to review, this change does not include update for l10n
files, and it does not delete the geshi/ directory. I will do those in a
separate patch.
The chief change between this and the previous implementation is that errors
result in the code block not being highlighted, as opposed to not being printed
at all, having been replaced by an angry red error message. I think that is the
right user experience. If you go to StackOverflow or GitHub and try to mark up
your code block as being written in some language that their highlighter
doesn't know about, you don't get an error message -- the code simply doesn't
get highlighted.
Because we don't recursively load dependencies for extensions, to test this,
you will need to create a composer.local.json in $IP and add:
{
"extra": {
"merge-plugin": {
"include": [
"extensions/SyntaxHighlight_GeSHi/composer.json"
]
}
}
}
Then run `composer update`.
Bug: T85794
Change-Id: I07446ec9893fae3d1e394f435d3d95cf8be6bc33
Store the list of supported languages in SyntaxHighlight_GeSHi.langs.php, which
is auto-generated via a maintenance script, updateLanguageList.php.
Change-Id: Ie0be7c42fa6716555c3e03e3f28734d7e0302664
(...modules of unusual size.)
Enabling full syntax highlighting for very long Lua modules can produce DOMs
that have hundreds of thousands of elements and cause browsers to lock up.
I took a count of spans by class (which amounts to a count of tokens by type)
of https://en.wiktionary.org/wiki/Module:languages and came up with:
sy0: 62545 (symbols)
br0: 61952 (brackets)
st0: 39291 (strings)
kw3: 7746 (keywords)
kw1: 3
kw2: 2
co2: 2
co1: 2
nu0: 1
------ ------
Total: 171544
GeSHi allows you to disable highlighting for a particular token type (see
<http://qbnz.com/highlighter/geshi-doc.html#disabling-lexics>) which like a
good way of handling this issue.
Disabling symbols (set_symbols_highlighting(false)) removes both sy0 and br0
elements from the DOM (about 124k elements in the case of Module:languages),
with about 47k elements remaining on the page. This is enough to make Chromium
responsive on my laptop (2.3ghz i5, 8 GB RAM), but it's still noticeably
sluggish. Adding 'set_string_highlighting(false);' removes another 40k elements
from the rendered output, and the resulting DOM is quite zippy at 8k elements.
Proposed solution: disable symbols highlighting when >100 kB; disable strings
highlighting too when >200 kB.
Change-Id: I90c645f9d03bbdc135058a3717a463dec40aa77d
