To run the Selenium tests you will have to install Ruby (for the exact version see Gemfile), the latest versions of RubyGems and Firefox. The easiest way to install Ruby on *nix is RVM (https://rvm.io/) and on Windows RubyInstaller (http://rubyinstaller.org/). Open terminal in tests/browser. Update RubyGems and install required gems with: gem update --system gem install bundler # on Mac OS X Yosemite append ` --user-install -n~/bin`, # where ~/bin is the install folder bundle install Environment variables MEDIAWIKI_USER and MEDIAWIKI_PASSWORD are required for tests that require a logged in user. For local testing, create a test user on your local wiki and export the user and password as the values for those variables. For example: export MEDIAWIKI_USER= # Linux/Unix/Mac set MEDIAWIKI_USER= # Windows export MEDIAWIKI_PASSWORD= # Linux/Unix/Mac set MEDIAWIKI_PASSWORD= # Windows In addition to this create another user which will be reserved for new uploads "Selenium_newuser". The password for this user should be the same as MEDIAWIKI_PASSWORD Tests that use the "Given I create a random page using the API" step need to set the MEDIAWIKI_API_URL environment variable, e.g. export MEDIAWIKI_API_URL=http://en.wikipedia.beta.wmflabs.org/w/api.php Run the tests from the Popups directory with: make cucumber If you want to run a single set of tests, go to the tests/browser directory and call 'bundle exec cucumber' with the path to the test file. For example, to run only the watchlist tests: bundle exec cucumber features/watchstar.feature XML report (for Jenkins) is created at tests/browser/reports/junit. Jenkins is hosted at https://wmf.ci.cloudbees.com/ and it drives browsers at http://saucelabs.com/ For more information about running Selenium tests please see https://github.com/wikimedia/mediawiki-selenium