Commit graph

257 commits

Author SHA1 Message Date
jenkins-bot 6f7349dcc8 Merge "Add mw.text.unstripNoWiki, mw.text.killMarkers, fix mw.text.unstrip" 2014-12-01 21:42:17 +00:00
Thiemo Mättig 55fe0b69d0 Add missing visibility keywords everywhere
Change-Id: I270d1dd9b6545e15398c2f8b8e9ae533844cc998
2014-11-14 10:10:23 +01:00
Thiemo Mättig 61af67b819 Fix visibility of setUp/tearDown
Required for Idf44d31.

Change-Id: If6d270549290bed2d1c7617da0fedbd385f3e96c
2014-11-12 12:28:58 +01:00
Brad Jorsch e5564cf942 Add mw.text.unstripNoWiki, mw.text.killMarkers, fix mw.text.unstrip
mw.text.unstrip is too broad, it's allowing for unstripping things that
cause problems when unstripped (e.g. bug 61268). Since the original
request was only for unstripping <nowiki>, let's add a function that
does only that.

We should also add an interface to StripState::killMarkers(), instead of
requiring everyone to roll their own work-alike.

Then, to fix the bug, we can make mw.text.unstrip be the combination of
the two. This is the most like the original behavior of mw.text.unstrip
(removes all strip markers, replacing them with text where applicable)
without causing issues.

Bug: 61268
Change-Id: I3a151fd678b365d629b71b4f1cb0d5d284b98555
2014-11-05 12:32:35 -05:00
Brad Jorsch df38a296bf Allow for dynamically-loaded PHP libraries
Scribunto currently supports libraries with PHP callbacks that are
loaded on startup, and pure-Lua libraries that may be loaded from the
module with require().

This change allows for libraries with PHP callbacks to also be loaded
with require().

Change-Id: Ibdc1f4ef51b1c8644c3d4c98d57755b5c06447a5
2014-10-03 09:27:23 -04:00
Jackmcbarn 634f75f53e Don't escape the delete character
Escaping the delete character breaks strip markers, so don't do it.

Bug: 68011
Change-Id: Ica97c898209c59c0084bf700d891b28603f79dd1
2014-09-21 22:59:52 -04:00
Jackmcbarn c5959b56e7 Display the full error message in wikitext
Instead of just displaying "Script error" in wikitext, display the actual
error string. Currently, many users don't see meaningful error messages
generated by modules, and instead just assume that "Script error" means
the module itself has a bug.

Bug: 71038
Change-Id: Ie5358049975352e0fef92088a06b97ad94717000
2014-09-19 12:11:38 +00:00
Jackmcbarn b970046f2e Don't output a semicolon at the end of CSS
It's not necessary, it makes the output bigger, and some pages have enough
elements with CSS that it does make an actual difference.

Change-Id: I80d471899c7e04a8a4876c205198a8c0d0b1f281
2014-09-10 19:08:34 -04:00
jenkins-bot 96508a3215 Merge "Output &nbsp; instead of &#nbsp;" 2014-09-08 20:37:05 +00:00
Jackmcbarn f5894a6a9f Output &nbsp; instead of &#nbsp;
Bug: 70475
Change-Id: I19aeceaa1eed17be4a128acd7fb50a9c8b40cf12
2014-09-08 16:06:34 -04:00
Jackmcbarn ece8cc5e82 Only set indexOffset when the function name is indexed
If a function name contains an equals sign, it doesn't increase the index,
so indexOffset=1 makes Lua see arguments as starting with 0. This change
fixes that, so that indexOffset is only set to 1 if the function name has
an index.

Change-Id: Ifa2dfad4d2f7228c3fd5721ccfc4c235576b663d
2014-09-06 15:42:13 -04:00
addshore e29c8eda63 Use public instead of var in classes
Change-Id: I0c059515cbf0fa8ef3846526a989f1a322fcd8ae
2014-08-17 22:11:26 +01:00
Jackmcbarn fd9ecb9cbe Expose cascading protection directly to Lua
Add a way to fetch cascading protection information from Lua without
needing to call the CASCADINGSOURCES parser function.

Change-Id: I1b3ac18af11d3066f78d27b31da8d6709a6a2631
2014-08-13 12:34:47 -04:00
Brad Jorsch 0367e9bddd Fix deceptively-simple pattern in pure-Lua ustring
The pure-Lua ustring pattern matching functions short-circuit to the
much faster string library when the pattern would match the same against
the raw bytes.

A pattern like "[^a-z]" can match a partial UTF-8 character when applied
bytewise, and so must be detected as unsafe.

Let's also directly test the pure-Lua module, instead of me having to
comment out lines in Scribunto_LuaUstringLibrary::register() whenever I
want to test them.

Change-Id: I91ed3374aadfea379b9db2e13b4248ab20df509e
2014-08-10 01:18:18 +00:00
Jackmcbarn 40b8bd2caa Add comments and remove trailing whitespace
Clean up trailing whitespace from all of our code, and add comments
indicating that apparently unused variables are ScopedCallbacks.

Change-Id: I8e5997797cc7b1c64c5351ec112a18f30edc8fef
2014-07-07 14:46:59 -04:00
Brad Jorsch 85aca87e94 Improve mw.getCurrentFrame handling
Two similar bugs are handled here:
* mw.getCurrentFrame() doesn't work when the module is loaded (only when
  a function is called), which breaks os.date and os.time at module
  scope since I59ad364d.
* mw.getCurrentFrame() gives access to frame args from inside
  mw.loadData, which allows for data leakage between #invokes.

Bug: 67498
Bug: 65687
Change-Id: I82dde43e2601b59c03c6ed4b9365829c40a953a5
2014-07-07 13:11:20 -04:00
Brad Jorsch 631feb8a3d Allow for skipping Lua tests
From the PHP subclass of Scribunto_LuaEngineTestBase, Lua tests run by
Scribunto_LuaEngineTestBase::testLua() may be skipped by adding an entry
for the test name to $this->skipTests.

From the Lua code run by Scribunto_LuaEngineTestBase::testLua(), tests
may be skipped by calling the "markTestSkipped" function exported by the
TestFramework module.

Also, use this new mechanism to skip certain mw.language tests if
Extension:CLDR or a similar extension is not available.

Bug: 67343
Change-Id: I4f0b15073a84bf2f9d8a5d905c3c960941dd2b71
2014-07-01 10:48:36 -04:00
Jackmcbarn 8d1d5ac84c Fix strange mw.html errors with numeric arguments
Some functions in mw.html accept numbers as arguments, but later fail when
constructing the string. This disallows numbers in attribute names, since
they aren't valid anyway, and fixes the remainder of the cases to properly
build the string.

Bug: 67201
Change-Id: Ie7bcbb9d8df580dd8793681f78a8b0719d8a287a
2014-06-27 14:41:42 -04:00
Brad Jorsch bf39827980 mw.ustring functions should accept numbers where string functions do
Lua's string functions tend to auto-convert numbers to strings. We
should do the same in mw.ustring.

Bug: 67201
Change-Id: Icd3c5e93bac19dafd78d737ec9b315daba9f1729
2014-06-27 12:31:04 -04:00
Brad Jorsch ccab415701 Don't cache volatile wikitext
Certain wikitext, such as that containing Cite.php <ref> or <references>
or the #tag versions of the same, should not be cached. This uses the
isVolatile method added to PPFrame in I95b3cf87 to avoid caching the
preprocessed output of such wikitext from frame:preprocess and similar
methods.

Bug: 46815
Change-Id: I1084f87fd863eb22f2f3f3d3ff308b24e20a08ef
2014-06-26 22:03:35 -04:00
Jackmcbarn 780d8e1ec7 Set TTLs on outputs containing times
When os.date, os.time, or mw.language:formatDate are called, set the
appropriate TTL on the output. This needs I412febf3 in core to function at
all, and I3f5a80aa in core to function with formatDate.

Change-Id: I59ad364d502fc247500d94c5606516ad9f98a24d
2014-06-23 15:52:17 +00:00
Jackmcbarn d1030989bc Allow passing nils to mw.html
Rather than calling error() when nils get passed to mw.html methods,
either remove whatever it was that the nil would go to (if that makes
sense), or just do nothing. The seemingly inconsistent use of "not x" and
"x ~= nil" is to allow any falsey value where it wouldn't be ambiguous
(such as class names), but not where it could be (such as attribute values).

Bug: 62982
Change-Id: I76773abbb4394aa9bb8c8a08445e019cade3b2bf
2014-06-19 11:40:39 -04:00
Jackmcbarn 1201618b7d Keep modules' export tables inside Lua
When tables are passed from Lua to PHP, their metatables are lost. Because
of this, they need to be kept inside of Lua to allow the __index
metamethod to return a method to be called by #invoke.

Bug: 64141
Change-Id: I0840bc12b25dee72828ec97d2b205812e4929f2b
2014-05-26 20:01:18 -04:00
Brad Jorsch 35ee461a0b Improve frame:callParserFunction argument validation
From wikitext, $parser->callParserFunction() will always get an array of
strings with at least an element [0]. Let's match this from Scribunto:
stringify numbers, and require that [0] (although in Lua it'll be [1]).

Also fix an old broken unit test.

Bug: 63597
Change-Id: Ie7ac34ae4bce70cec455d90c3f02a658644f6866
2014-05-13 05:14:07 +00:00
aude bb0f10b751 Make LuaEngineTestBase compatible with phpunit 4.1
PHPUnit_Framework_TestSuite::isPublicTestMethod has
been removed from phpunit 4.1.

Change-Id: I191b5f460fbef122d8e916c8f661f09083b33970
2014-05-08 07:11:14 +02:00
Jackmcbarn d74c81c233 Add name to nosuchfunction and nosuchmodule errors
When displaying a nosuchfunction or nosuchmodule error, include the name
of the nonexistent function or module.

Change-Id: I17fc2c68dc8267302a82eee3cb2c5df9b5a3c46c
2014-04-13 21:29:05 -04:00
Brad Jorsch 708d87c181 Prevent passing information between #invokes using math.random
It's possible to pass information between multiple #invokes on a page by
having the first call math.randomseed with one of a set of known seeds
and then having the second examine the output from math.random to
determine which of those known seeds was used.

Prevent that by calling math.randomseed( 1 ) when invoking (see the bug
for details on why that seed). But avoid doing so if e.g. a
frame:expandTemplate() call results in a recursive invoke.

Bug: 62291
Change-Id: Id01cb63eca52ced29bf4efebc38beb9f159b7b0e
2014-04-01 13:46:09 -04:00
Brad Jorsch 29452c9bef mw.language: Fix exceptions and add tests
Various methods are throwing exceptions when passed invalid language
codes. Those need to be caught.

And we should really add unit tests for the mw.language library, too.
Doing so exposed another bug (in lang:gender), which is also fixed here.

Bug: 62242
Change-Id: Ib7d257cbb1ce179c510273526910d6ac5f3cac5d
2014-03-09 02:56:06 +00:00
Brad Jorsch 84d96e843c LuaStandalone: Release functions when no longer referenced
The LuaStandalone interpreter needs to keep a mapping from integers
returned to PHP to the corresponding function. But if it never releases
these functions when PHP no longer has any reference to them, it can
result in Lua running out of memory if a module with a large number of
functions is invoked many times in one page.

The fix here is to track which function ids are referenced from PHP, and
periodically send the list to Lua so it can remove any that are no
longer used from its cache.

This also takes care of another issue where having multiple interpreter
instances and passing function objects from one into another could call
the wrong function in Lua.

Bug: 51886
Change-Id: I4f15841051f7748d1d6df24080949e5cbd88f217
2014-02-25 22:59:05 +00:00
Brad Jorsch ed54cd7032 Fix PHP implementation of mw.ustring %W
Incorrect regex is making it match basically everything.

Bug: 60908
Change-Id: Ic1c5a740c6b825bb7dfacf684d0c7fa905b135ee
2014-02-10 00:45:46 +00:00
Jackmcbarn 2b06e83816 Remove message formats other than plain
Message formats other than plain should have never been exposed in this
way, as they allow link tables, etc. to be bypassed and serve no useful
purpose. This removes them, and also removes title, as it serves no
purpose without them.

Bug: 60758
Change-Id: I96284ffbe986a9cd92d2bde1ffdb746029bad989
2014-02-03 15:29:39 -05:00
Marius Hoch aa943d5795 Validate the type of the tagName given to mw.html.create
Change-Id: I35f70701ba6156325dfd268903452e6279aca528
2014-01-31 02:08:11 +01:00
Brad Jorsch 5cf28c5a15 Use a child frame in frame:expandTemplate
If we don't do this, then the section edit links point to the wrong page
if we expand a template that contains section headings.

Bug: 55525
Change-Id: I00bda935be3e8b9c0f86fd0f131814207fbb34a7
2014-01-30 22:47:31 +00:00
Marius Hoch 2bcd815d62 Allow changing the title used by Scribunto_LuaEngineTestBase
Change-Id: I8916fc855b96a566d775c947b117a31ad4a49dc8
2014-01-16 17:00:30 +01:00
jenkins-bot b1d7bac7b0 Merge "Add protectionLevels variable to mw.title output" 2014-01-15 20:21:48 +00:00
Jackmcbarn 9902ec1284 Add protectionLevels variable to mw.title output
Include a protectionLevels variable in the output of the mw.title.*
functions, containing the contents of the title's mRestrictions
array (i.e., its protection levels)

Change-Id: I79c9fed64bacfc90aee1d411a3e1b47e44c99755
2014-01-15 14:53:18 -05:00
Brad Jorsch f52136eada Handle invalid keys in Lua-to-PHP calls for LuaStandalone
PHP can't handle having arrays/objects or functions as keys in its
arrays, so make sure we don't try to pass them from Lua. Booleans aren't
really well-handled either, so let's disallow them too.

Also, add tests for proper stringification of floats and infinities when
those are used as keys.

Note this behavior change is needed to match the change in LuaSandbox
for fixing bug 54527, but isn't itself a security issue.

Change-Id: I1e2951bbe8cb78358650ad377bf7119fcac4485d
2014-01-14 10:03:42 -05:00
Marius Hoch ac62e34952 Add mw.html to Scribunto
A module for building complex HTML from Lua using a
fluent interface. The module is originally from enwiki,
but the authors allowed us to reuse it under GPLv2+
(as stated in the file).
The module will be loaded per default and comes with
unit tests.

As discussed on wikitech-l:
http://lists.wikimedia.org/pipermail/wikitech-l/2013-December/073320.html

Change-Id: I7c8d4378091c13d5ace0dd1fcbb4e27163e8c896
2014-01-03 00:04:34 +01:00
Brad Jorsch 0763e22292 Expose PPFrame::getTitle to Lua
This field already exists in PHP with exactly the content requested in
bug 47089, so we may as well expose it on the frame object.

Bug: 47089
Change-Id: I672820589f6ebc7c4daad29b5eb156733a5bc5cc
2013-12-10 04:26:03 +00:00
jenkins-bot a18a762fd7 Merge "Add mw.isSubsting()" 2013-11-06 18:26:56 +00:00
Brad Jorsch 8872dd3eda Update phpunit @group annotations
Some tests weren't being run as they should have been when phpunit's
--group option was used.

Change-Id: I29ff9a04322b91cc085247e5663dfc7bc67d3439
2013-11-01 11:56:24 -04:00
Brad Jorsch d4331f0c96 Add mw.isSubsting()
It's already possible to detect whether the current template is being
substituted via ParserFunctions (see [[en:Template:Ifsubst]]), and a
similar trick works with frame:preprocess. So we may as well provide the
flag directly.

Bug: 47828
Change-Id: Id06d27c6283ee589a8830b78c04e56978e0ac6da
2013-10-31 22:05:00 -04:00
jenkins-bot 47fc1462c4 Merge "Properly handle spaces/underscores in mw.site.namespaces" 2013-10-31 19:36:03 +00:00
Brad Jorsch ec2f67c7ea Regularize whitespace handling in mw.title
Specifically:
* String conversion in non-URL contexts (e.g. .prefixedText) uses spaces
  instead of underscores.
* Setting .fragment now applies the same transformations that are done
  (in PHP) by mw.title.new.

Bug: 56217
Change-Id: I12e354636bcde3327864088175fb4de61aecc81a
2013-10-31 11:54:26 -04:00
Brad Jorsch f31aa37adc Properly handle spaces/underscores in mw.site.namespaces
The PHP call that makes mw.site.namespaces work case-insensitively
doesn't handle non-standard spaces/underscores. So standardize them
before the call.

Bug: 56216
Change-Id: I4758478b126858fb581614f64eb15472f42fef51
2013-10-31 11:09:16 -04:00
jenkins-bot 99b96d8b14 Merge "Add frontier pattern (%f[set]) to ustring" 2013-08-30 06:22:40 +00:00
Brad Jorsch d8314539da Fix mw.ustring edge cases
A few edge cases were being incorrectly handled:
* mw.ustring.sub( 'abc', 1, 0 ) returned 'a', not ''.
* mw.ustring.codepoint( 'abc', 1, 0 ) returned 97, not no results.
* mw.ustring.codepoint( 'abc', 4, 4 ) returned 99, not no results.
* mw.ustring.gcodepoint had the same issues as mw.ustring.codepoint.

Change-Id: Ib8c0ef5a8073106eb7d90d0aa0513be4525dca08
2013-07-03 11:49:52 -04:00
Brad Jorsch ad3e5a7c48 Fix LuaSandbox mw.ustring.byteoffset with negative 'i'
Negative values for 'i' in mw.ustring.byteoffset are supposed to count
from the end of the string. But in LuaSandbox, it was actually counting
from two bytes before the end of the string due to a typo.

Fix that, and add some tests for it.

Bug: 50176
Change-Id: Iceee1022a55abd7a08df1ea7843e1277eb02798b
2013-06-25 10:06:01 -04:00
Brad Jorsch 82820aafc8 Add frontier pattern (%f[set]) to ustring
The "%f[set]" frontier pattern has been in Lua 5.1 since the beginning,
but was undocumented until Lua 5.2. And the code is even unchanged from
5.1.0 to 5.2.1. So there's no reason not to implement it in ustring too.

Note the changes to UstringLibrary.php are somewhat large, because it
splits the "convert a Lua bracketed charset to PCRE" code into a
separate function and it changes the handling of mw.ustring.find's and
mw.ustring.match's 'init' parameter from "substring, match from 0, then
add back on $init" to "use preg_match's $offset and use \G instead of ^
where this matters". Both of these are necessary to properly support
%f.

This also fixes a bug in the pure-Lua code (not used in Scribunto)
exposed by the unit tests for %f where %z was matching '\1' rather than
'\0' and %Z everything except '\1' instead of everything except '\0'.

Bug: 48331
Change-Id: Ie0b95ef5b734db53d6adc9de5dae4874f8944c08
2013-05-12 10:27:36 -04:00
Brad Jorsch d6f3633428 (bug 47365) Fix edge cases in mw.ustring.find, mw.ustring.match
The following errors are fixed:
* PHP warning and wrong return value with empty pattern and plain
* Incorrect offsets returned when init is larger than the string length
* Incorrect captured offsets returned when init is excessively negative

Bug: 47365
Change-Id: I9741418287dc727747326d6a19678370ce155a2b
2013-05-10 06:00:02 +00:00
Brad Jorsch adb7b5f939 Make pairs() work with various library objects
Users are reporting disappointment that using pairs on title objects
doesn't let them see the fields available. It's easy enough to add a
__pairs metamethod to allow pairs to work on title objects, so let's do
that.

The same can be done for mw.uri objects.

For mw.message objects, we can easily enough change the implementation
to be like mw.language objects, which doesn't have this problem.

For mw.site.stats, we may as well just remove the load-on-demand feature
since it will be demanded as soon as the environment is cloned for the
first module.

Change-Id: Ie5a3b5684b2bb6c090f9994baa03977687688929
2013-04-02 17:23:42 +00:00
jenkins-bot a85275592c Merge "Add text module" 2013-03-27 16:55:07 +00:00
Brad Jorsch 3d4a81653c (bug 46405) Fix errors in mw.title.new( pageid )
mw.title.new( pageid ) should not throw an error for an nonexisting
pageid, just return nil. Similarly, it should always return nil for 0,
rather than returning the last non-existent title created.

Change-Id: I3cdbb24fc785aef0f8e75fba1feccd26ac5b7370
2013-03-21 09:39:25 -04:00
Brad Jorsch 0db3d7c6d2 Add text module
This exists for some common text-processing functions that aren't
included in string (and therefore also aren't in mw.ustring), as well as
a logical place for the "unstrip" function requested in bug 45085.

Bug: 45085
Change-Id: I47356215fcc8ddeed5f901cd933a30021394bd78
2013-03-20 10:10:15 -04:00
Brad Jorsch 541e61a1c4 (bug 41769) Add frame:callParserFunction() and frame::extensionTag()
Requires change I339b882010dedd714e7965e25ad650ed8b8cd48f to
mediawiki/core.

Bug: 41769
Change-Id: I0138836654b0e34c5c23daaedcdf5d4f9d1c7ab2
2013-03-18 13:46:26 -04:00
Brad Jorsch b6ea64f132 Make mw.getCurrentFrame() work in console, add frame:newChild()
It would be helpful for debugging if a frame object could be gotten in
the console. To that end, add an empty frame when running in the console
and allow it to be returned by mw.getCurrentFrame().

It would also be helpful to be able to create frames with arbitrary
arguments, again for testing. Fortunately support for creating child
frames with arbitrary arguments already exists in core, so we can just
use it.

And for good measure, be sure to restore the $engine->currentFrames
array even if the Lua code throws an exception.

Change-Id: I1dc8602d63af75424f267c42a3743fabbc1827f7
2013-03-12 14:42:36 +11:00
Brad Jorsch 5faa00150c (bug 45684) Don't count argument parsing time towards Lua limit
Currently, the time taken to parse the arguments passed to a Lua
function from #invoke will be counted against Lua's 10-second limit.
This is counterintuitive, and can remove incentive for users to convert
templates to Lua since they may have to convert a whole stack at once.

Note this requires change I11881232 to mediawiki/php/luasandbox to
actually have any effect.

Bug: 45684
Change-Id: I773950e4c399b8a1cfa6d1cde781a069d286b3bd
2013-03-12 03:18:05 +00:00
jenkins-bot ec86b161cd Merge "Fix unit test engine↔parser assignment" 2013-03-07 19:44:54 +00:00
Brad Jorsch b926846a7a mw.title:getContent() should record in templatelinks
Since the content of a page is being loaded, this should be recorded in
templatelinks so things can be updated properly when that page is
edited.

Change-Id: I3e720fee2705f6c08ac0456c3cab0ed4ede84536
2013-03-07 13:02:41 -05:00
Brad Jorsch 981ea4c39f Fix unit test engine↔parser assignment
In the unit tests, the engine is given a parser, but the parser is not
being associated back with the engine. So in some cases, that could lead
to *another*, default engine being created for certain operations.

Fix that.

Change-Id: I79995c2635d9e470931b84dc1854dae26772bbe3
2013-03-07 01:42:19 +00:00
Brad Jorsch 201bf237b3 Fix Title library breaking other unit tests
Apparently assigning to $wgHooks at the top of the file breaks every
other hook, somehow. So we need to set it in setUp(), but prevent the
interwiki prefix from being queried beforehand when the dataProvider is
run. Fun.

Change-Id: I8535942e3a0e47d9a5b2ea11c1b0b63c5acf2cc5
2013-03-06 20:39:08 -05:00
Brad Jorsch e004b415d6 (bug 45512) Add mw.title:getContent()
Bug: 45512
Change-Id: I630033d367a47f7f80809a0918d3e6feede3fe41
2013-03-05 19:05:00 +00:00
jenkins-bot fc766f205a Merge "Fix mw.uri error with empty fragment or query string" 2013-03-05 18:49:06 +00:00
Brad Jorsch fcef54e9d9 Fix ustring errors
* mw.ustring.sub( '', 1 ) errors in LuaStandalone
* Default value for ustring.maxStringLength and ustring.maxPatternLength
  should be infinity, not nil
* mw.ustring.find() returns one value instead of two in "plain" mode.

Change-Id: I5e65c4ec3a05f0e6930ce7ab7fd4ac72bea95e7f
2013-03-05 12:22:15 -05:00
Brad Jorsch 580cda00d0 Fix mw.uri error with empty fragment or query string
Off-by-one error.

Bug: 45655
Change-Id: Ic6e0dfe8dbf392b1fe9e225f7906b4dd84f4febf
2013-03-05 01:44:10 +00:00
Brad Jorsch 307a5b1be8 Clean up lualibs
Clean up the modules in engines/LuaCommon/lualib:
* Fix luabit/bit.lua to return its table instead of trying to set the
  global directly.
* Fix luabit/hex.lua to return its table instead of trying to set the
  global directly.
* luabit/noki.lua is useless for our purposes
* luabit/utf8.lua is redundant to mw.ustring
* stringtools uses coroutines, which we don't support

Also fix a parser test that has apparently been broken for a long time.

Change-Id: I1284cddb6e9b94327964cb1077d8dbdf7def6d06
2013-02-28 18:39:16 -08:00
Brad Jorsch 04a0a580e3 Add mw.title library
Note that fetching any title besides the one for the current page is
considered "expensive". It also records the title fetched in the
ParserOutput so it will be listed in pagelinks, just like #ifexists.

This also moves the ToString test formatter into TestFramework.lua, so
TitleLibraryTests.lua can use it too.

Change-Id: I799f3289a37fe1349b6bca5758829acf82cb718f
2013-02-28 11:57:11 -08:00
Brad Jorsch ba09ba3fde Add mw.message library
Change-Id: I12ca84f848c34f1227ee8acdc8bc04bdfd0b2d97
2013-02-26 22:20:04 -08:00
Brad Jorsch 2b08f5a5c0 Fix NaN unit test in phpunit 3.7.14
In phpunit 3.6.10, assertEquals considered two NaNs to be equivalent.
Somewhere between that and phpunit 3.7.14, this behavior changed so NaNs
are no longer considered equivalent.

Change-Id: I2c664498eb34cf5119a2eaaa96a6be57b821ab94
2013-02-25 16:56:15 -08:00
Brad Jorsch 00d4b711ed Add mw.loadData, for static data loading
People sometimes want to load large tables of constant data from a
module. Using require(), this has to be reparsed every time, which can
be slow.

mw.loadData() will load the just data once, and return a table with a
metatable cleverly designed to give read-only access to the loaded data.

Change-Id: Icec192bdbe6cfca7902fd5cb5d0e217bd8399637
2013-02-21 23:03:53 +00:00
jenkins-bot 2980cf4c11 Merge "Fix mw.ustring.gmatch and patterns with '^'" 2013-02-15 08:22:11 +00:00
Brad Jorsch 1efe182e40 Hide mw.makeProtectedEnvFuncs from modules
Allowing a module to call mw.makeProtectedEnvFuncs() lets it bypass the
allowEnvFuncs setting. It can also be used to manipulate the global
tables that other modules' sandboxes will be copied from.

And for paranoia's sake, let's tighten up what setfenv is allowed to
set. This requires changing a unit test, because it is no longer
sane to do something like

 env.setfenv, env.getfenv = mw.makeProtectedEnvFuncs( { [env] = true }, {} )

Nothing real does this, it was only in the unit test.

Change-Id: I8e0d83bb0980ee869af3ac4413afd211717ca92f
2013-02-14 16:49:15 -05:00
Brad Jorsch 4dcac2fcd9 Fix mw.ustring.gmatch and patterns with '^'
The Lua manual says this:

 For this function, a '^' at the start of a pattern does not work as an
 anchor, as this would prevent the iteration.

I had interpreted that to mean that a pattern starting with '^' would
never match in gmatch. But further testing reveals that the '^' is just
treated as a literal character: string.gmatch( "foo ^bar baz", "^%a+" )
will match "^bar".

Change-Id: Id91d6ee2db753ce1d6a4f6ae27764691d9e9fdc4
2013-02-14 14:25:55 -05:00
Brad Jorsch e66ccb89b4 Test for leaked global variables
It's easy to forget a 'local' somewhere and accidentally leak a global
variable. Add a unit test to catch that.

Change-Id: I3a8dda22f108d88039f9562a1da7a739850bb14b
2013-02-14 04:02:48 +00:00
Tim Starling 2d9e3c74ba Merge "Lua ustring implementation" 2013-02-13 03:32:43 +00:00
Brad Jorsch 0a8757baba Lua ustring implementation
This is a reimplementation of Lua's string library with support for
UTF-8.

The entire ustring library is implemented in pure Lua. PHP callbacks are
also available for overrides: in LuaSandbox these are used for almost
all functions, while in LuaStandalone they are used only for the pattern
matching. Also, ustring.upper and ustring.lower are overridden using
mw.language's .uc and .lc if available.

It also includes a bunch of unit tests.

Note that if you download the normalization tests, they may fail under
LuaSandbox if you have PHP's intl extension installed and libicu on your
system is too old.

Change-Id: Ie76fdf8d3a85d0a3d2a41b0d3b7afe433f247af0
2013-02-12 14:26:29 -05:00
Brad Jorsch d6116fa6ba Make pairs and ipairs work with frame.args
Using the 5.2 compatability added in I37efc59a, we can now make
pairs( frame.args ) work.

Change-Id: Iefdca8805b08ea222251f28514b2c92182a7feb3
2013-02-12 06:41:33 +00:00
Brad Jorsch ce46700ec9 Fix unit tests after I62259b76
The unit tests never set the title on the engine.

Change-Id: Iefa11cacd4f10a1daa185b2e2a0d9620bbbfa6c6
2013-02-07 15:59:38 -05:00
Brad Jorsch db9fd2b39b Add mw.uri library
Change-Id: I1d94a8c288537ada038f24f2ec26922d95f14785
2013-02-07 13:31:24 -05:00
Brad Jorsch bd03237246 Add mw.site library
Contains various constants and functions to access site info.

Change-Id: I944938f9af0203c16d1a3fb2046f332045dec4d9
2013-02-06 09:20:59 -05:00
Brad Jorsch 337abb295f Make Lua engine tests modular
Rework the LuaEngine tests to be entirely modular, so that every library
need not add itself to one monolithic file. This also allows other
extensions that add Lua modules to make unit tests without having to
somehow inject them into a test class owned by Scribunto.

The approach taken is similar to that used for Selenium for running
tests against multiple browsers.

Change-Id: I294b2a8195759c0e4fa211f879305a8eb66d9c9a
2013-02-06 09:10:57 -05:00
Ori Livneh b2d340bf71 (Bug 37957) php_sapi_name() => PHP_SAPI
Change-Id: I8ccb645fa11f001fc1ca1c95f4e9476f84870714
2013-02-04 20:27:01 -08:00
Brad Jorsch 8b27dc742d Add incrementExpensiveFunctionCount() methods
To allow Lua libraries to mark functions as expensive, add an
incrementExpensiveFunctionCount() method to Scribunto_LuaEngine that
will call the corresponding Parser method and throw an error if the
limit is exceeded.

Also allow libraries to do the same thing from Lua by calling
mw.incrementExpensiveFunctionCount().

Change-Id: I56fded32b1077eff3980371e9abc9b3b7581f7b5
2013-02-01 15:55:16 -05:00
Brad Jorsch 0a8a07de2f Fix package.loaded and other cleanups
The listing of the standard modules in package.loaded seems to have been
removed to avoid leaking information to loaded modules. However, since
the *entire* environment is cloned, *including* package.loaded itself,
this does not seem to actually be a problem. But for good measure, also
add a unit test to verify that the version of the standard module tables
referenced from package.loaded is the same as that in _G.

This change also cleans up some unused local variables and an unused
local function from the package module.

Change-Id: I7ec8227b3273059e8f65ad735c215bfd0c623e64
2013-01-30 17:24:09 +00:00
Brad Jorsch 6b4cfd5b94 Fix LuaStandalone nil handling
In Lua, a table entry with a nil value is the same as a table entry that
doesn't exist. So when serializing for transfer to PHP, these keys will
be skipped. For a table as an associative array this isn't much of a
problem, but for a table as a list it means we have missing indexes.
Some of Lua's functions for handling "lists" (i.e. tables with numeric
keys) also have a problem when the list contains nils.

To work around these issues when passing argument lists and return value
lists, pass the number of elements along with the sparse list. On the
PHP end we can use this to fill in the missing nulls, and on the Lua
end we can pass this count to unpack() to avoid the problems on the Lua
side.

Change-Id: I858e3905a06e377693301da2b8bc534808f00e3e
2013-01-30 10:12:23 -05:00
Brad Jorsch 9f17590f47 Fix unit test parameter order
Trivial fix, the parameter order to assertEquals() was backwards so if
the test failed it would indicate the "Actual" results as "Expected" and
vice versa.

Change-Id: Ibfe12591a58b10e0321aafea576c36cfa674f51d
2013-01-28 19:05:41 -05:00
Demon c9781606fa Merge "Fix reference handling in LuaEnvironmentComparisonTest" 2013-01-28 20:33:24 +00:00
Brad Jorsch dd5241273b Support __pairs and __ipairs
Lua 5.2 introduces a nice feature where a metatable can override the
standard behavior of the pairs() and ipairs() functions. That would be
very useful in allowing a more standard syntax for our frame.args, and
it's very easy to do both in C and in Lua.

Change-Id: I37efc59a0c8876ee16184807e15fafbc07e2d288
2013-01-24 15:46:33 -05:00
Brad Jorsch 10aa7dce46 Fix reference handling in LuaEnvironmentComparisonTest
The Lua engines cannot handle circular data structures. Avoid returning
one for this test.

Change-Id: I7483c29fa440110bc4567bdd4807b035d7475ffc
2013-01-24 15:21:53 -05:00
Brad Jorsch bdacaabf21 Fix string extending
Built-in modules (e.g. the upcoming ustring) should be able to extend
the string metatable, but user modules (and the debug console) should
not be able to.

But currently built-in modules cannot extend the string metatable in
LuaStandalone, while in LuaSandbox the string metatable can be extended
in the console. Fix this and add unit tests.

Change-Id: I15f5598fed318f2fe26b08ec47e16053dddc13c4
2013-01-09 15:54:44 +00:00
Brad Jorsch c84d699e9b Refactor unit tests
The existing unit tests work, but the setup is really not amenable to
the addition of additional tests in a modular fashion. This splits
things out so there is a framework for tests in Lua, and all a module
has to do on the Lua side is supply a list of functions to call and
results to expect. And then on the php side, only one array entry and
two short functions need to be added to LuaSandboxEngineTest to run the
tests.

Change-Id: Ib241b246aa0c7223c33887b38a5858582d7d31b0
2013-01-09 15:54:25 +00:00
Brad Jorsch 374972c924 Add Scribunto_LuaInterpreter::wrapPhpFunction
Currently, the only way for PHP to pass a function to Lua is to pass
back a function that it received from Lua. This means, for example, that
PHP cannot implement a Lua iterator function except by registering a
library holding the function or by using loadString.

This changeset adds Scribunto_LuaInterpreter::wrapPhpFunction (and
implements it for both LuaSandbox and LuaStandalone), which takes a PHP
callable and returns a Lua function wrapping it.

Note that fallback code is included so this does not depend on
I2e552799.

Change-Id: Ic0a98eec7cc17ef4b1acee032c0f42d617b998d2
2013-01-09 05:54:36 +00:00
Chad Horohoe 590d40e145 Make sure to call parent::setUp()
Change-Id: I9da9bbf803ee37d48b954e2e82407fae4dfe464c
2012-12-20 11:43:03 -05:00
Brad Jorsch 27f9cb5d91 Unit test: verify LuaSandbox and LuaStandalone are equivalent
We should include a unit test to try to ensure that we don't add
features to one environment and forget about the other.

Change-Id: I72b1acf8eea4a05e05fed1efeb0663a3eff9278a
2012-12-13 13:53:29 -05:00
Brad Jorsch eabb081c4f Handle non-finite numbers
There's no reason LuaStandalone cannot handle NaN and Inf.

Also, add some unit tests to check this.

Change-Id: I8570242a792e212489ad24dfc04fb1f2940190a5
2012-12-13 13:40:18 -05:00
Tim Starling d5bd1102d0 Fixed transmission of \0 from Lua standalone
* string.format() truncates the string at a null character, causing a
  deadlock when Lua attempts to send null characters to PHP. Use
  concatenation instead.
* Added test.
* Fixed an error reporting issue in the console, which I happened to
  notice at the same time as the above bug.

Change-Id: I2e6061a04512557492bffbd04bc09ca3bc1d80d6
2012-12-03 17:46:11 +11:00
Antoine Musso a179bd6182 tests: skip CPU limit test on Darwin
The Darwin does not support POSIX timer which is being used to limit CPU
usage. Running the tests on Mac OS X would never end so we are just
skipping it.

Change-Id: I56c3e8cd1cba15d33256192aa7e721e6448c7a2e
2012-09-12 17:46:47 -07:00
Tim Starling 89cf9f187d Fixed parser tests broken by I97f7732010ebfebcf494ac9e308275b34dbb9609
Change-Id: Ifd46b5dd25edd912d3fc2ebb92326da11c9df67b
2012-08-28 16:23:52 +10:00
Tim Starling 441943bd9b Do not allow access to setfenv() and getfenv() by default
Optionally remove setfenv and getfenv from the global environment in
which user code runs. This will improve the forwards-compatibility of
user code with Lua 5.2.

Porting to Lua 5.2 would still be a daunting project, of questionable
value, but at least only the internal code would need updating, and not
thousands of on-wiki modules. Compared to the environment changes, the
rest of the Lua 5.2 changes are relatively easy to simulate for
backwards compatibility.

Removed module() from the package module, since it depends on setfenv().
The native version of it is deprecated in Lua 5.2 for that reason.

Change-Id: I978903ca98943ac941833da13fe5027949f6b429
2012-05-31 15:02:04 +02:00
Tim Starling 6bc11ff615 New parser interface
* Implemented the new parser interface based on a frame object, as
  described in the design document and wikitech-l.
* Added parser tests for the new interface.
* Removed {{script:}} parser function
* Allow named parameters to {{#invoke:}}
* Don't trim the return value
* If a function invoked by #invoke returns multiple values, concatenate
  them into a single string.
* If there is an error during parse, show the error message as an HTML
  comment as well as via JavaScript. This makes parser test construction
  easier, and probably makes debugging easier also.
* Rename mw_internal to mw_php to clarify its role. It is now strictly a
  private Lua -> PHP interface function table.
* Protect mw.setup() against multiple invocation.
* Fixed a bug in Scribunto_LuaStandaloneInterpreter::receiveMessage():
  large packets caused fread() to return with less than the requested
  amount of data, which previously caused an exception. It's necessary
  to check for EOF and to repeat the read to get all data. The receive
  function on the Lua side does not suffer from this problem.
* In the standalone engine, fixed a bug in the interpretation of null
  return values from PHP callbacks. This should return no values to Lua.
* Updated the Lua unit tests to account for the fact that functions are
  now forced to return strings.
* Updated the getfenv and setfenv tests to account for the extra stack
  level introduced by mw.executeFunction().

Change-Id: If8fdecdfc91ebe7bd4b1dae8489ccbdeb6bbf5ce
2012-05-22 14:18:49 +10:00
Tim Starling c9c0aa6fce Peak memory usage, debug.traceback
* Use LuaSandbox::getPeakMemoryUsage() from r115086
* Fixed the debug.traceback function from da06273e, was nil

Change-Id: Iae4b195ffe25a522d4c37f9c8341e1d6ea3ae106
2012-04-30 16:37:41 +10:00
Tim Starling 41b93dd7e1 Fixed setfenv() across a tail call
Fixed the issue noticed during testing of da06273e, and which resulted
in satest.setfenv1() being disabled. It's not possible to protect
environments by iterating through every stack level, calling getfenv()
at each one, because if any of the stack levels is a tail call, an error
is raised.

Such a tail call was introduced in da06273e, which is why the test broke.

Instead, just protect the actual specified environments, not their
callers. The callers will have to protect themselves.

Change-Id: If39104010ff2663c1bae5105cc8d37e276532100
2012-04-24 12:33:06 +10:00
Tim Starling da06273ede Nicer errors with backtraces etc.
* Added error backtrace collection to MWServer:handleCall()
* When there is an error on parse, show a short and simple inline error
  message to the user, which when clicked, expands to a full error with
  HTML-formatted backtrace.
* When an error is encountered during module validation, have the code
  editor jump directly to the line. Requires r115011.
* Expose the code location of most errors to Scribunto, by parsing the
  standard error message format.
* During module validation, abbreviate the error location if the error
  is in the same module.
* Do not execute the module during validation, just parse it. Execution
  does not really work without an active parse operation in progress.
  It already caused a fatal error if you called require() from the main
  chunk, and problems would have become more visible as more
  parser-related APIs were added.
* LuaSandbox does not yet provide backtraces, but this is planned.

Change-Id: Id9f6564a41b310792b3fe3ebb527cbf8f8771bd1
2012-04-23 21:58:30 +10:00
tstarling b68cae904a More tests and some related bug fixes
* Added tests for the engine classes.
* Added some tests that run under Lua.
* In the chunk names, fixed truncation of module names at 60 bytes
  by using an "=" prefix instead of @.
* Fixed a bug in mw.clone() which was causing the metatable to be set on
  the source table instead of the destination.
* Put restricted setfenv/getfenv in the cloned environment rather than
  the base environment, they work better that way.
* In setfenv(), check for getfenv() == nil, since that's what our own
  restricted getfenv returns.
* Fixed getfenv() handling of numeric arguments: add one where
  appropriate.

Change-Id: I2b356fd65a3fcb348c4e99a3a4267408fb995739
2012-04-19 17:48:20 +10:00
tstarling cebe775ee8 Added more Lua environment features
Package library:

* Added a simulation of the Lua 5.1 package library.
* Removed mw.import(), replaced it with a package loader. Packages can be
  retrieved from the wiki, using require('Module:Foo'), or from files
  distributed with Scribunto, using require('foo'). The "Module:" prefix allows
  for source compatibility with existing Lua code.
* Added a couple of libraries from LuaForge: luabit and stringtools.
* Made fetchModuleFromParser() return null on error instead of throwing an
  exception, to more easily support the desired behaviour of the package loader,
  which needs to return null on error.
* Renamed mw.setupEnvironment() to mw.setup() since it is setting up things
  other than the environment now.
* In MWServer:handleRegisterLibrary(), remove the feature which interprets dots
  in library names, since LuaSandbox doesn't support this.

Improved module isolation and related refactoring:

* Expose restricted versions of getfenv() and setfenv() to user Lua code.
  Requires luasandbox r114952.
* Don't cache the export list returned by module execution for later function
  calls. This breaks isolation of #invoke calls, since the local variables are
  persistent.
* Removed ScribuntoFunctionBase and its children, since it doesn't really have
  a purpose if it can't cache anything. Instead, invoke functions using a module
  method called invoke().
* Removed Module::initialize(), replaced it with a validate() function. This is
  a more elegant interface and works better with the new module caching scheme.
* Use a Status object for the return value of Engine::validate() instead of an
  array. Use the formatting facilities of the Status class.

Other:

* Removed "too many returns" error, doesn't fit in with Lua conventions.
* Use the standalone engine by default, so that the extension will work without
  configuration for more people.
* Added an accessor for $engine->interpreter
* Fix mw.clone() to correctly clone metatables
* If the standalone interpreter exits due to an error, there are some contexts
  where the initial error will be caught and ignored, and the user will see the
  error from checkValid() instead. In this case, rethrow the original error for
  a more informative message.
* Load mw.lua into the initial standalone environment, to reduce code
  duplication between mw.lua and MWServer.lua.
* Fixed a bug in Scribunto_LuaStandaloneInterpreter::handleCall() for functions
  that return no results.
* Fixed a bug in encodeLuaVar() for strings with "\r". Added test case.
* In MWServer.lua, don't call error() for internal errors, instead just print
  the error and exit. This avoids a protocol violation when an error is
  encountered from within handleCall().
* Added lots of documentation. Lua doc comments are in LuaDoc format.

Change-Id: Ie2fd572c362bedf02f45d3fa5352a5280e034740
2012-04-18 13:46:18 +10:00
tstarling 889f4e42a5 Allow PHP functions to return multiple values to Lua
Corresponding luasandbox commit is r114920.

Change-Id: I10fe726795623e2976c9ddadaa4d9b6d44b30bed
2012-04-16 15:43:34 +10:00
tstarling b0f00103e2 Added tests and fixed bugs
* Added unit tests for the two Lua interpreter classes
* Fixed a bug in checkType()
* Have Scribunto_LuaSandboxInterpreter throw an exception on construct
  when the extension doesn't exist, to match the standalone behaviour.
* In Scribunto_LuaSandboxInterpreter, removed debugging statements
  accidentally left in.
* Convert LuaSandboxTimeoutError to the appropriate common error
  message.
* Moved the option munging from the sandbox engine to the interpreter,
  so that the interpreter can be unit tested separately.
* Use /bin/sh instead of bash for lua_ulimit.sh, since dash is smaller
  and still supports ulimit.
* Use exec to run the lua binary, so that the vsize of the shell doesn't
  add to the memory limit.
* Added a quit function to the standalone interpreter. Unused at present.
* Don't add a comma after the last element of a table in a Lua
  expression.
* Make the SIGXCPU detection work: proc_open() runs the command via a
  shell, which reports signals in the child via the exit status, so
  proc_get_status() will never return a valid termsig element.
* In MWServer:call(), fixed a bug causing the return values to be
  wrapped in an array.
* Fixed a misunderstanding of what select() does.
* In MWServer:getStatus(), fixed indexes so that vsize will be correct.
  Removed RSS, since it wasn't used anyway and turns out to be measured
  in multiples of the page size, and I couldn't be bothered trying to
  fetch that from getconf. Return the PID and vsize as numbers rather
  than strings.
* Added a simple table dump feature to MWServer:debug().
* Fixed brackets in MWServer:tostring().
* Added missing Linux 32-bit binary.

Change-Id: Ibf5f4656b1c0a9f81287d363184c3fe9d2abdafd
2012-04-16 14:41:08 +10:00