wikimedia/mediawiki-extensions-VisualEditor
1
0
Fork 0
mirror of https://gerrit.wikimedia.org/r/mediawiki/extensions/VisualEditor synced 2024-11-25 06:46:26 +00:00
Code Issues Packages Projects Releases Wiki Activity
mediawiki-extensions-Visual.../modules/parser/parse.js

159 lines
4.5 KiB
JavaScript
Raw Normal View History

refactor parser to ParseThingy in different module, can be invoked with command line utility parse.js
2011-12-28 01:37:06 +00:00
/**
A few minor documentation updates.
2012-03-07 18:42:26 +00:00
* Command line parse utility.
refactor parser to ParseThingy in different module, can be invoked with command line utility parse.js
2011-12-28 01:37:06 +00:00
* Read from STDIN, write to STDOUT.
Added html2wt command-line option to parse.js * A quick way to see how a html fragment serializes * Minor JSHint tweaks. Change-Id: I901398ec15700905c53de6f39cde93400b2f964a
2012-07-24 15:30:44 +00:00
*
Collapse all requests per template into a single outstanding request using an event-emitting TemplateRequest object and a request queue.
2012-01-20 02:36:18 +00:00
* @author Neil Kandalgaonkar <neilk@wikimedia.org>
* @author Gabriel Wicke <gwicke@wikimedia.org>
refactor parser to ParseThingy in different module, can be invoked with command line utility parse.js
2011-12-28 01:37:06 +00:00
*/
Biggish token transform system refactoring * All parser pipelines including tokenizer and DOM stuff are now constructed from a 'recipe' data structure in a ParserPipelineFactory. * All sub-pipelines of these can now be cached * Event registrations to a pipeline are directly forwarded to the last pipeline member to save relatively expensive event forwarding. * Some APIs for on-demand expansion / format conversion of parameters from parser functions are added: param.to('tokens/expanded', cb) param.to('text/wiki', cb) (this does not work yet) All parameters are additionally wrapped into a Param object that provides method for positional parameter naming (.named() or conversion to a dict (.dict()). * The async token transform manager is now separated from a frame object, with the frame holding arguments, an on-demand expansion method and loop checks. * Only keys of template parameters are now expanded. Parser functions or template arguments trigger an expansion on-demand. This (unsurprisingly) makes a big performance difference with typical switch-heavy template systems. * Return values from async transforms are no longer used in favor of plain callbacks. This saves the complication of having to maintain two code paths. A trick in transformTokens still avoids the construction of unneeded TokenAccumulators. * The results of template expansions are no longer buffered. * 301 parser tests are passing Known issues: * Cosmetic cleanup remains to do * Some parser functions do not support async expansions yet, and need to be modified. Change-Id: I1a7690baffbe8141cadf67270904a1b2e1df879a
2012-04-25 14:35:59 +00:00
var ParserPipelineFactory = require('./mediawiki.parser.js').ParserPipelineFactory,
Add some commandline switches to parse.js. Supports switching on/off debug mode and a selection of html/WikiDom serialization.
2012-01-21 22:42:54 +00:00
ParserEnv = require('./mediawiki.parser.environment.js').MWParserEnvironment,
Fix --wikidom and --linearmodel parse.js options; retry on template fetch failures Change-Id: I444397936fd87971fe085df4b467089367e9ffa6
2012-04-26 17:51:00 +00:00
ConvertDOMToLM = require('./mediawiki.LinearModelConverter.js').ConvertDOMToLM,
DOMConverter = require('./mediawiki.DOMConverter.js').DOMConverter,
Simplistic wikitext round-tripping with parse.js --wikitext Lists are a bit tricky, as nested lists are not wrapped in a separate list item. Should work now though. Change-Id: I2e5f29f6afa6bdd2d5e5c0c5d019b70c611b73d1
2012-05-16 14:53:18 +00:00
WikitextSerializer = require('./mediawiki.WikitextSerializer.js').WikitextSerializer,
Added html2wt command-line option to parse.js * A quick way to see how a html fragment serializes * Minor JSHint tweaks. Change-Id: I901398ec15700905c53de6f39cde93400b2f964a
2012-07-24 15:30:44 +00:00
optimist = require('optimist'),
html5 = require('html5');
refactor parser to ParseThingy in different module, can be invoked with command line utility parse.js
2011-12-28 01:37:06 +00:00
Add --wikidom flag to parse.js Also remove unused import of DOMConverter Change-Id: I1eabe6bf9935970c1f049681b52e867a510ea77a
2012-04-06 01:30:18 +00:00
( function() {
Add HTML DOM -> linear model converter Also, in ParserPipeline: * Import the LM converter and expose it through getLinearModel() * Fix getWikiDom() to actually work (still unused) In parse.js: * Add --help option that prints usage information (was unreachable) * Add --linearmodel option to output linear model JSON instead of HTML Change-Id: Ic534e03ff40a7c9117bb63f0c635a4213d5e3406
2012-03-26 19:40:56 +00:00
var opts = optimist.usage( 'Usage: echo wikitext | $0', {
'help': {
description: 'Show this message',
'boolean': true,
'default': false
},
'linearmodel': {
description: 'Output linear model data instead of HTML',
'boolean': true,
'default': false
},
Add --wikidom flag to parse.js Also remove unused import of DOMConverter Change-Id: I1eabe6bf9935970c1f049681b52e867a510ea77a
2012-04-06 01:30:18 +00:00
'wikidom': {
description: 'Output WikiDOM instead of HTML',
'boolean': true,
'default': false
},
Added html2wt command-line option to parse.js * A quick way to see how a html fragment serializes * Minor JSHint tweaks. Change-Id: I901398ec15700905c53de6f39cde93400b2f964a
2012-07-24 15:30:44 +00:00
'html2wt': {
description: 'Convert input HTML to Wikitext',
'boolean': true,
'default': false
},
Simplistic wikitext round-tripping with parse.js --wikitext Lists are a bit tricky, as nested lists are not wrapped in a separate list item. Should work now though. Change-Id: I2e5f29f6afa6bdd2d5e5c0c5d019b70c611b73d1
2012-05-16 14:53:18 +00:00
'wikitext': {
description: 'Output WikiText instead of HTML',
'boolean': true,
'default': false
},
Add some commandline switches to parse.js. Supports switching on/off debug mode and a selection of html/WikiDom serialization.
2012-01-21 22:42:54 +00:00
'debug': {
description: 'Debug mode',
'boolean': true,
'default': false
A few parser function fixes and maximum template expansion in environment config.
2012-01-22 19:32:28 +00:00
},
A few more debug helpers including a --trace mode for light debugging. Some improvements to parser functions on the way to support the cite extensions. Preparation for generic template and template arg in attribute support. 222 parser tests now passing.
2012-01-31 16:50:16 +00:00
'trace': {
description: 'Trace mode (light debugging), implied by --debug',
'boolean': true,
'default': false
},
A few parser function fixes and maximum template expansion in environment config.
2012-01-22 19:32:28 +00:00
'maxdepth': {
description: 'Maximum expansion depth',
'boolean': false,
'default': 40
first pass at an API method that returns wikidom. Shells out to node. Some issues with XML API result formatting but works fine in JSON
2012-01-31 00:02:48 +00:00
},
Add basic thumb rendering support * DOM based on Wikia's thumb output: HTML5, clean caption without magnify icon. * basic RDFa annotations, but most options additionally in data-mw object- might want to move more (or all?) of those into RDFa data using meta tags. * no support yet for framed or other formats, image scaling etc * also tweaked some config options in the environment Change-Id: Ie461fcdce060cfc2dec65cc057709ae650ef3368
2012-04-09 20:58:55 +00:00
'wgScript': {
first pass at an API method that returns wikidom. Shells out to node. Some issues with XML API result formatting but works fine in JSON
2012-01-31 00:02:48 +00:00
description: 'http path to remote API, e.g. http://wiki.sample.com/w',
'boolean': false,
'default': 'http://en.wikipedia.org/w'
},
Add basic thumb rendering support * DOM based on Wikia's thumb output: HTML5, clean caption without magnify icon. * basic RDFa annotations, but most options additionally in data-mw object- might want to move more (or all?) of those into RDFa data using meta tags. * no support yet for framed or other formats, image scaling etc * also tweaked some config options in the environment Change-Id: Ie461fcdce060cfc2dec65cc057709ae650ef3368
2012-04-09 20:58:55 +00:00
'wgScriptPath': {
description: 'http path to remote web interface, e.g. http://wiki.sample.com/wiki',
'boolean': false,
Very basic interwiki support Pages titles with a wikipedia interwiki prefix now load the page from corresponding Wikipedia. Links in a page then stay within the given language. Note that Parsoid currently makes no effort to recognize localized namespaces, so it won't render media files, categories etc correctly. Change-Id: I7bc4102e81a402772ea23231170734d580ea15b9
2012-06-05 09:19:58 +00:00
'default': 'http://en.wikipedia.org/wiki/'
Add basic thumb rendering support * DOM based on Wikia's thumb output: HTML5, clean caption without magnify icon. * basic RDFa annotations, but most options additionally in data-mw object- might want to move more (or all?) of those into RDFa data using meta tags. * no support yet for framed or other formats, image scaling etc * also tweaked some config options in the environment Change-Id: Ie461fcdce060cfc2dec65cc057709ae650ef3368
2012-04-09 20:58:55 +00:00
},
first pass at an API method that returns wikidom. Shells out to node. Some issues with XML API result formatting but works fine in JSON
2012-01-31 00:02:48 +00:00
'wgScriptExtension': {
description: 'Extension for PHP files on remote API server, if any. Include the period, e.g. ".php"',
'boolean': false,
'default': '.php'
},
'fetchTemplates': {
description: 'Whether to fetch included templates recursively',
'boolean': true,
'default': true
Fixes to template expansion / token transform managers, 296 tests passing. * Convert isNoInclude logic to positive isInclude throughout and set it properly on attribute pipelines. Also don't cache non-include pipelines. * Add a --pagename parameter to parse.js, which sets the page name in the environment. This is then returned by {{PAGENAME}}. Not the final solution, but useful for taxobox testing as taxons are selected based on PAGENAME. * Add rudimentary pagenamebase parser function Change-Id: If9c0be4c255200d0f2a30f02e5619437b4fd8f12
2012-04-11 14:34:27 +00:00
},
'pagename': {
description: 'The page name, returned for {{PAGENAME}}.',
'boolean': false,
'default': 'Main page'
Add some commandline switches to parse.js. Supports switching on/off debug mode and a selection of html/WikiDom serialization.
2012-01-21 22:42:54 +00:00
}
Add HTML DOM -> linear model converter Also, in ParserPipeline: * Import the LM converter and expose it through getLinearModel() * Fix getWikiDom() to actually work (still unused) In parse.js: * Add --help option that prints usage information (was unreachable) * Add --linearmodel option to output linear model JSON instead of HTML Change-Id: Ic534e03ff40a7c9117bb63f0c635a4213d5e3406
2012-03-26 19:40:56 +00:00
});
var argv = opts.argv;
if ( argv.help ) {
optimist.showHelp();
return;
}
refactor parser to ParseThingy in different module, can be invoked with command line utility parse.js
2011-12-28 01:37:06 +00:00
Added html2wt command-line option to parse.js * A quick way to see how a html fragment serializes * Minor JSHint tweaks. Change-Id: I901398ec15700905c53de6f39cde93400b2f964a
2012-07-24 15:30:44 +00:00
var env = new ParserEnv( {
Collapse all requests per template into a single outstanding request using an event-emitting TemplateRequest object and a request queue.
2012-01-20 02:36:18 +00:00
// fetch templates from enwiki by default..
Add basic thumb rendering support * DOM based on Wikia's thumb output: HTML5, clean caption without magnify icon. * basic RDFa annotations, but most options additionally in data-mw object- might want to move more (or all?) of those into RDFa data using meta tags. * no support yet for framed or other formats, image scaling etc * also tweaked some config options in the environment Change-Id: Ie461fcdce060cfc2dec65cc057709ae650ef3368
2012-04-09 20:58:55 +00:00
wgScript: argv.wgScript,
first pass at an API method that returns wikidom. Shells out to node. Some issues with XML API result formatting but works fine in JSON
2012-01-31 00:02:48 +00:00
wgScriptPath: argv.wgScriptPath,
wgScriptExtension: argv.wgScriptExtension,
Improve generic attribute expansion before external link processing, and make wgUploadPath configurable. Also change the hard-coded fall-back image sizes to sensible defaults. This breaks three parser tests until image size retrieval from the wiki is implemented.
2012-03-06 18:02:35 +00:00
// XXX: add options for this!
A few fixes to parser functions and template expansion. Trim whitespace off template arguments, let the last duplicate key win and fake pagenamee slightly better.
2012-03-08 11:44:37 +00:00
wgUploadPath: 'http://upload.wikimedia.org/wikipedia/commons',
first pass at an API method that returns wikidom. Shells out to node. Some issues with XML API result formatting but works fine in JSON
2012-01-31 00:02:48 +00:00
fetchTemplates: argv.fetchTemplates,
Added html2wt command-line option to parse.js * A quick way to see how a html fragment serializes * Minor JSHint tweaks. Change-Id: I901398ec15700905c53de6f39cde93400b2f964a
2012-07-24 15:30:44 +00:00
// enable/disable debug output using this switch
A few parser function fixes and maximum template expansion in environment config.
2012-01-22 19:32:28 +00:00
debug: argv.debug,
A few more debug helpers including a --trace mode for light debugging. Some improvements to parser functions on the way to support the cite extensions. Preparation for generic template and template arg in attribute support. 222 parser tests now passing.
2012-01-31 16:50:16 +00:00
trace: argv.trace,
Fixes to template expansion / token transform managers, 296 tests passing. * Convert isNoInclude logic to positive isInclude throughout and set it properly on attribute pipelines. Also don't cache non-include pipelines. * Add a --pagename parameter to parse.js, which sets the page name in the environment. This is then returned by {{PAGENAME}}. Not the final solution, but useful for taxobox testing as taxons are selected based on PAGENAME. * Add rudimentary pagenamebase parser function Change-Id: If9c0be4c255200d0f2a30f02e5619437b4fd8f12
2012-04-11 14:34:27 +00:00
maxDepth: argv.maxdepth,
pageName: argv.pagename
Biggish token transform system refactoring * All parser pipelines including tokenizer and DOM stuff are now constructed from a 'recipe' data structure in a ParserPipelineFactory. * All sub-pipelines of these can now be cached * Event registrations to a pipeline are directly forwarded to the last pipeline member to save relatively expensive event forwarding. * Some APIs for on-demand expansion / format conversion of parameters from parser functions are added: param.to('tokens/expanded', cb) param.to('text/wiki', cb) (this does not work yet) All parameters are additionally wrapped into a Param object that provides method for positional parameter naming (.named() or conversion to a dict (.dict()). * The async token transform manager is now separated from a frame object, with the frame holding arguments, an on-demand expansion method and loop checks. * Only keys of template parameters are now expanded. Parser functions or template arguments trigger an expansion on-demand. This (unsurprisingly) makes a big performance difference with typical switch-heavy template systems. * Return values from async transforms are no longer used in favor of plain callbacks. This saves the complication of having to maintain two code paths. A trick in transformTokens still avoids the construction of unneeded TokenAccumulators. * The results of template expansions are no longer buffered. * 301 parser tests are passing Known issues: * Cosmetic cleanup remains to do * Some parser functions do not support async expansions yet, and need to be modified. Change-Id: I1a7690baffbe8141cadf67270904a1b2e1df879a
2012-04-25 14:35:59 +00:00
} );
refactor parser to ParseThingy in different module, can be invoked with command line utility parse.js
2011-12-28 01:37:06 +00:00
process.stdin.resume();
process.stdin.setEncoding('utf8');
var inputChunks = [];
process.stdin.on( 'data', function( chunk ) {
inputChunks.push( chunk );
} );
Added html2wt command-line option to parse.js * A quick way to see how a html fragment serializes * Minor JSHint tweaks. Change-Id: I901398ec15700905c53de6f39cde93400b2f964a
2012-07-24 15:30:44 +00:00
process.stdin.on( 'end', function() {
refactor parser to ParseThingy in different module, can be invoked with command line utility parse.js
2011-12-28 01:37:06 +00:00
var input = inputChunks.join('');
Added html2wt command-line option to parse.js * A quick way to see how a html fragment serializes * Minor JSHint tweaks. Change-Id: I901398ec15700905c53de6f39cde93400b2f964a
2012-07-24 15:30:44 +00:00
if (argv.html2wt) {
var p = new html5.Parser();
p.parse('<html><body>' + input.replace(/\r/g, '') + '</body></html>');
var content = p.tree.document.childNodes[0].childNodes[1];
var stdout = process.stdout;
new WikitextSerializer({env: env}).serializeDOM(content, stdout.write.bind(stdout));
Eventify parser tests and parse.js commandline wrapper to actuallly allow async template fetching. Async expansion is not yet fully debugged, but at least the preconditions for that are now there.
2012-01-18 23:46:01 +00:00
// add a trailing newline for shell user's benefit
Added html2wt command-line option to parse.js * A quick way to see how a html fragment serializes * Minor JSHint tweaks. Change-Id: I901398ec15700905c53de6f39cde93400b2f964a
2012-07-24 15:30:44 +00:00
stdout.write( "\n" );
Eventify parser tests and parse.js commandline wrapper to actuallly allow async template fetching. Async expansion is not yet fully debugged, but at least the preconditions for that are now there.
2012-01-18 23:46:01 +00:00
process.exit(0);
Added html2wt command-line option to parse.js * A quick way to see how a html fragment serializes * Minor JSHint tweaks. Change-Id: I901398ec15700905c53de6f39cde93400b2f964a
2012-07-24 15:30:44 +00:00
} else {
var parserPipelineFactory = new ParserPipelineFactory( env );
var parser = parserPipelineFactory.makePipeline( 'text/x-mediawiki/full' );
parser.on('document', function ( document ) {
// Print out the html
if ( argv.linearmodel ) {
process.stdout.write(
JSON.stringify( ConvertDOMToLM( document.body ), null, 2 ) );
} else if ( argv.wikidom ) {
process.stdout.write(
JSON.stringify(
new DOMConverter().HTMLtoWiki( document.body ),
null,
2
));
} else if ( argv.wikitext ) {
new WikitextSerializer({env: env}).serializeDOM( document.body,
process.stdout.write.bind( process.stdout ) );
} else {
process.stdout.write( document.body.innerHTML );
}
// add a trailing newline for shell user's benefit
process.stdout.write( "\n" );
process.exit(0);
});
// Kick off the pipeline by feeding the input into the parser pipeline
parser.process( input );
}
refactor parser to ParseThingy in different module, can be invoked with command line utility parse.js
2011-12-28 01:37:06 +00:00
} );
} )();
Reference in a new issue Copy permalink