2014-11-19 00:32:46 +00:00
|
|
|
<?php
|
|
|
|
/**
|
Highlight using Pygments rather than Geshi
GeSHi is unmaintained, lacks support for many popular modern languages, and
suffers from deep architectural flaws, chief among them the inconsistent
tokenization of different languages, each of which requires a custom
stylesheet.
Pygments is a well-maintained alternative. It is, by my count, the most popular
syntax highlighting library around. It is BSD-licensed, actively maintained,
and is widely used in PHP projects.
To keep this easy to review, this change does not include update for l10n
files, and it does not delete the geshi/ directory. I will do those in a
separate patch.
The chief change between this and the previous implementation is that errors
result in the code block not being highlighted, as opposed to not being printed
at all, having been replaced by an angry red error message. I think that is the
right user experience. If you go to StackOverflow or GitHub and try to mark up
your code block as being written in some language that their highlighter
doesn't know about, you don't get an error message -- the code simply doesn't
get highlighted.
Because we don't recursively load dependencies for extensions, to test this,
you will need to create a composer.local.json in $IP and add:
{
"extra": {
"merge-plugin": {
"include": [
"extensions/SyntaxHighlight_GeSHi/composer.json"
]
}
}
}
Then run `composer update`.
Bug: T85794
Change-Id: I07446ec9893fae3d1e394f435d3d95cf8be6bc33
2015-06-16 03:52:58 +00:00
|
|
|
* Script to update list of supported lexers.
|
2014-11-19 00:32:46 +00:00
|
|
|
*
|
|
|
|
* This program is free software; you can redistribute it and/or modify
|
|
|
|
* it under the terms of the GNU General Public License as published by
|
|
|
|
* the Free Software Foundation; either version 2 of the License, or
|
|
|
|
* (at your option) any later version.
|
|
|
|
*
|
|
|
|
* This program is distributed in the hope that it will be useful,
|
|
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
* GNU General Public License for more details.
|
|
|
|
*
|
|
|
|
* You should have received a copy of the GNU General Public License along
|
|
|
|
* with this program; if not, write to the Free Software Foundation, Inc.,
|
|
|
|
* 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
|
|
|
|
* http://www.gnu.org/copyleft/gpl.html
|
|
|
|
*
|
|
|
|
* @file
|
|
|
|
* @author Ori Livneh <ori@wikimedia.org>
|
|
|
|
* @ingroup Maintenance
|
|
|
|
*/
|
|
|
|
|
Drop symfony/process dependency, use Shell\Command instead
We originally started using symfony/process because kzykhys/pygments
depended upon it. But that library was unmaintained and became broken,
so we stopped using it, and just used symfony/process directly.
At the time, the main reason in favor of symfony/process was that it
could pass stdin to pygments, while Shell\Command couldn't - but it can
now (T182463)! On top of that, there are downsides, like not respecting
the default MediaWiki shell limits, being incompatible with core's
firejail support, and requiring an external composer dependency.
Note that because Shell::command() will enforce MediaWiki's normal
limits, it's possible that some large pages may no longer render with
syntax highlighting if they pass those limits.
Bug: T182467
Bug: T181771
Change-Id: Ie1cb72b7eb17d943f79ecae4d94a2110546ef039
2017-12-08 20:57:08 +00:00
|
|
|
use MediaWiki\Shell\Shell;
|
2018-08-18 05:33:35 +00:00
|
|
|
use Wikimedia\StaticArrayWriter;
|
Highlight using Pygments rather than Geshi
GeSHi is unmaintained, lacks support for many popular modern languages, and
suffers from deep architectural flaws, chief among them the inconsistent
tokenization of different languages, each of which requires a custom
stylesheet.
Pygments is a well-maintained alternative. It is, by my count, the most popular
syntax highlighting library around. It is BSD-licensed, actively maintained,
and is widely used in PHP projects.
To keep this easy to review, this change does not include update for l10n
files, and it does not delete the geshi/ directory. I will do those in a
separate patch.
The chief change between this and the previous implementation is that errors
result in the code block not being highlighted, as opposed to not being printed
at all, having been replaced by an angry red error message. I think that is the
right user experience. If you go to StackOverflow or GitHub and try to mark up
your code block as being written in some language that their highlighter
doesn't know about, you don't get an error message -- the code simply doesn't
get highlighted.
Because we don't recursively load dependencies for extensions, to test this,
you will need to create a composer.local.json in $IP and add:
{
"extra": {
"merge-plugin": {
"include": [
"extensions/SyntaxHighlight_GeSHi/composer.json"
]
}
}
}
Then run `composer update`.
Bug: T85794
Change-Id: I07446ec9893fae3d1e394f435d3d95cf8be6bc33
2015-06-16 03:52:58 +00:00
|
|
|
|
2014-11-19 00:32:46 +00:00
|
|
|
$IP = getenv( 'MW_INSTALL_PATH' ) ?: __DIR__ . '/../../..';
|
|
|
|
|
|
|
|
require_once "$IP/maintenance/Maintenance.php";
|
|
|
|
|
2017-12-10 00:32:49 +00:00
|
|
|
class UpdateLexerList extends Maintenance {
|
2014-11-19 00:32:46 +00:00
|
|
|
public function __construct() {
|
|
|
|
parent::__construct();
|
2018-01-28 16:33:37 +00:00
|
|
|
|
|
|
|
$this->requireExtension( 'SyntaxHighlight' );
|
Highlight using Pygments rather than Geshi
GeSHi is unmaintained, lacks support for many popular modern languages, and
suffers from deep architectural flaws, chief among them the inconsistent
tokenization of different languages, each of which requires a custom
stylesheet.
Pygments is a well-maintained alternative. It is, by my count, the most popular
syntax highlighting library around. It is BSD-licensed, actively maintained,
and is widely used in PHP projects.
To keep this easy to review, this change does not include update for l10n
files, and it does not delete the geshi/ directory. I will do those in a
separate patch.
The chief change between this and the previous implementation is that errors
result in the code block not being highlighted, as opposed to not being printed
at all, having been replaced by an angry red error message. I think that is the
right user experience. If you go to StackOverflow or GitHub and try to mark up
your code block as being written in some language that their highlighter
doesn't know about, you don't get an error message -- the code simply doesn't
get highlighted.
Because we don't recursively load dependencies for extensions, to test this,
you will need to create a composer.local.json in $IP and add:
{
"extra": {
"merge-plugin": {
"include": [
"extensions/SyntaxHighlight_GeSHi/composer.json"
]
}
}
}
Then run `composer update`.
Bug: T85794
Change-Id: I07446ec9893fae3d1e394f435d3d95cf8be6bc33
2015-06-16 03:52:58 +00:00
|
|
|
$this->addDescription( 'Update list of lexers supported by SyntaxHighlight_GeSHi' );
|
2014-11-19 00:32:46 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
public function execute() {
|
2018-07-29 04:29:30 +00:00
|
|
|
$header = 'Generated by ' . basename( __FILE__ );
|
Highlight using Pygments rather than Geshi
GeSHi is unmaintained, lacks support for many popular modern languages, and
suffers from deep architectural flaws, chief among them the inconsistent
tokenization of different languages, each of which requires a custom
stylesheet.
Pygments is a well-maintained alternative. It is, by my count, the most popular
syntax highlighting library around. It is BSD-licensed, actively maintained,
and is widely used in PHP projects.
To keep this easy to review, this change does not include update for l10n
files, and it does not delete the geshi/ directory. I will do those in a
separate patch.
The chief change between this and the previous implementation is that errors
result in the code block not being highlighted, as opposed to not being printed
at all, having been replaced by an angry red error message. I think that is the
right user experience. If you go to StackOverflow or GitHub and try to mark up
your code block as being written in some language that their highlighter
doesn't know about, you don't get an error message -- the code simply doesn't
get highlighted.
Because we don't recursively load dependencies for extensions, to test this,
you will need to create a composer.local.json in $IP and add:
{
"extra": {
"merge-plugin": {
"include": [
"extensions/SyntaxHighlight_GeSHi/composer.json"
]
}
}
}
Then run `composer update`.
Bug: T85794
Change-Id: I07446ec9893fae3d1e394f435d3d95cf8be6bc33
2015-06-16 03:52:58 +00:00
|
|
|
|
2017-06-15 16:16:11 +00:00
|
|
|
$lexers = [];
|
2015-12-02 21:21:11 +00:00
|
|
|
|
Drop symfony/process dependency, use Shell\Command instead
We originally started using symfony/process because kzykhys/pygments
depended upon it. But that library was unmaintained and became broken,
so we stopped using it, and just used symfony/process directly.
At the time, the main reason in favor of symfony/process was that it
could pass stdin to pygments, while Shell\Command couldn't - but it can
now (T182463)! On top of that, there are downsides, like not respecting
the default MediaWiki shell limits, being incompatible with core's
firejail support, and requiring an external composer dependency.
Note that because Shell::command() will enforce MediaWiki's normal
limits, it's possible that some large pages may no longer render with
syntax highlighting if they pass those limits.
Bug: T182467
Bug: T181771
Change-Id: Ie1cb72b7eb17d943f79ecae4d94a2110546ef039
2017-12-08 20:57:08 +00:00
|
|
|
$result = Shell::command(
|
|
|
|
SyntaxHighlight::getPygmentizePath(),
|
|
|
|
'-L', 'lexer'
|
2017-12-08 21:03:35 +00:00
|
|
|
)
|
|
|
|
->restrict( Shell::RESTRICT_DEFAULT | Shell::NO_NETWORK )
|
|
|
|
->execute();
|
2015-12-02 21:21:11 +00:00
|
|
|
|
Drop symfony/process dependency, use Shell\Command instead
We originally started using symfony/process because kzykhys/pygments
depended upon it. But that library was unmaintained and became broken,
so we stopped using it, and just used symfony/process directly.
At the time, the main reason in favor of symfony/process was that it
could pass stdin to pygments, while Shell\Command couldn't - but it can
now (T182463)! On top of that, there are downsides, like not respecting
the default MediaWiki shell limits, being incompatible with core's
firejail support, and requiring an external composer dependency.
Note that because Shell::command() will enforce MediaWiki's normal
limits, it's possible that some large pages may no longer render with
syntax highlighting if they pass those limits.
Bug: T182467
Bug: T181771
Change-Id: Ie1cb72b7eb17d943f79ecae4d94a2110546ef039
2017-12-08 20:57:08 +00:00
|
|
|
if ( $result->getExitCode() != 0 ) {
|
2020-05-29 16:42:48 +00:00
|
|
|
$this->fatalError( 'Non-zero exit code: ' . $result->getStderr() );
|
2015-12-02 21:21:11 +00:00
|
|
|
}
|
|
|
|
|
Drop symfony/process dependency, use Shell\Command instead
We originally started using symfony/process because kzykhys/pygments
depended upon it. But that library was unmaintained and became broken,
so we stopped using it, and just used symfony/process directly.
At the time, the main reason in favor of symfony/process was that it
could pass stdin to pygments, while Shell\Command couldn't - but it can
now (T182463)! On top of that, there are downsides, like not respecting
the default MediaWiki shell limits, being incompatible with core's
firejail support, and requiring an external composer dependency.
Note that because Shell::command() will enforce MediaWiki's normal
limits, it's possible that some large pages may no longer render with
syntax highlighting if they pass those limits.
Bug: T182467
Bug: T181771
Change-Id: Ie1cb72b7eb17d943f79ecae4d94a2110546ef039
2017-12-08 20:57:08 +00:00
|
|
|
$output = $result->getStdout();
|
2015-12-02 21:21:11 +00:00
|
|
|
foreach ( explode( "\n", $output ) as $line ) {
|
|
|
|
if ( substr( $line, 0, 1 ) === '*' ) {
|
2022-11-21 05:46:55 +00:00
|
|
|
$newLexers = explode( ', ', trim( $line, "* :\r\n" ) );
|
2021-03-10 18:20:16 +00:00
|
|
|
// Skip internal, unnamed lexers
|
|
|
|
if ( $newLexers[0] !== '' ) {
|
|
|
|
$lexers = array_merge( $lexers, $newLexers );
|
|
|
|
}
|
2015-12-02 21:21:11 +00:00
|
|
|
}
|
|
|
|
}
|
2016-02-04 01:28:49 +00:00
|
|
|
$lexers = array_unique( $lexers );
|
Highlight using Pygments rather than Geshi
GeSHi is unmaintained, lacks support for many popular modern languages, and
suffers from deep architectural flaws, chief among them the inconsistent
tokenization of different languages, each of which requires a custom
stylesheet.
Pygments is a well-maintained alternative. It is, by my count, the most popular
syntax highlighting library around. It is BSD-licensed, actively maintained,
and is widely used in PHP projects.
To keep this easy to review, this change does not include update for l10n
files, and it does not delete the geshi/ directory. I will do those in a
separate patch.
The chief change between this and the previous implementation is that errors
result in the code block not being highlighted, as opposed to not being printed
at all, having been replaced by an angry red error message. I think that is the
right user experience. If you go to StackOverflow or GitHub and try to mark up
your code block as being written in some language that their highlighter
doesn't know about, you don't get an error message -- the code simply doesn't
get highlighted.
Because we don't recursively load dependencies for extensions, to test this,
you will need to create a composer.local.json in $IP and add:
{
"extra": {
"merge-plugin": {
"include": [
"extensions/SyntaxHighlight_GeSHi/composer.json"
]
}
}
}
Then run `composer update`.
Bug: T85794
Change-Id: I07446ec9893fae3d1e394f435d3d95cf8be6bc33
2015-06-16 03:52:58 +00:00
|
|
|
sort( $lexers );
|
2018-07-29 04:29:30 +00:00
|
|
|
$data = [];
|
|
|
|
foreach ( $lexers as $lexer ) {
|
|
|
|
$data[$lexer] = true;
|
|
|
|
}
|
Highlight using Pygments rather than Geshi
GeSHi is unmaintained, lacks support for many popular modern languages, and
suffers from deep architectural flaws, chief among them the inconsistent
tokenization of different languages, each of which requires a custom
stylesheet.
Pygments is a well-maintained alternative. It is, by my count, the most popular
syntax highlighting library around. It is BSD-licensed, actively maintained,
and is widely used in PHP projects.
To keep this easy to review, this change does not include update for l10n
files, and it does not delete the geshi/ directory. I will do those in a
separate patch.
The chief change between this and the previous implementation is that errors
result in the code block not being highlighted, as opposed to not being printed
at all, having been replaced by an angry red error message. I think that is the
right user experience. If you go to StackOverflow or GitHub and try to mark up
your code block as being written in some language that their highlighter
doesn't know about, you don't get an error message -- the code simply doesn't
get highlighted.
Because we don't recursively load dependencies for extensions, to test this,
you will need to create a composer.local.json in $IP and add:
{
"extra": {
"merge-plugin": {
"include": [
"extensions/SyntaxHighlight_GeSHi/composer.json"
]
}
}
}
Then run `composer update`.
Bug: T85794
Change-Id: I07446ec9893fae3d1e394f435d3d95cf8be6bc33
2015-06-16 03:52:58 +00:00
|
|
|
|
2018-08-18 05:33:35 +00:00
|
|
|
$writer = new StaticArrayWriter();
|
|
|
|
$code = $writer->create( $data, $header );
|
Highlight using Pygments rather than Geshi
GeSHi is unmaintained, lacks support for many popular modern languages, and
suffers from deep architectural flaws, chief among them the inconsistent
tokenization of different languages, each of which requires a custom
stylesheet.
Pygments is a well-maintained alternative. It is, by my count, the most popular
syntax highlighting library around. It is BSD-licensed, actively maintained,
and is widely used in PHP projects.
To keep this easy to review, this change does not include update for l10n
files, and it does not delete the geshi/ directory. I will do those in a
separate patch.
The chief change between this and the previous implementation is that errors
result in the code block not being highlighted, as opposed to not being printed
at all, having been replaced by an angry red error message. I think that is the
right user experience. If you go to StackOverflow or GitHub and try to mark up
your code block as being written in some language that their highlighter
doesn't know about, you don't get an error message -- the code simply doesn't
get highlighted.
Because we don't recursively load dependencies for extensions, to test this,
you will need to create a composer.local.json in $IP and add:
{
"extra": {
"merge-plugin": {
"include": [
"extensions/SyntaxHighlight_GeSHi/composer.json"
]
}
}
}
Then run `composer update`.
Bug: T85794
Change-Id: I07446ec9893fae3d1e394f435d3d95cf8be6bc33
2015-06-16 03:52:58 +00:00
|
|
|
|
2017-05-10 14:18:57 +00:00
|
|
|
file_put_contents( __DIR__ . '/../SyntaxHighlight.lexers.php', $code );
|
|
|
|
$this->output( "Updated language list written to SyntaxHighlight.lexers.php\n" );
|
2014-11-19 00:32:46 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-03-03 01:22:47 +00:00
|
|
|
$maintClass = UpdateLexerList::class;
|
2017-05-10 14:18:57 +00:00
|
|
|
require_once RUN_MAINTENANCE_IF_MAIN;
|