This is the team responsible for Release Engineering/Management and QA at the Wikimedia Foundation. We predominately use the QA mailing list along with conversations on the appropriate IRC channels (especially #wikimedia-tech ^connect, #wikimedia-operations ^connect, #wikimedia-qa ^connect, and #wikimedia-dev ^connect).

User:Greg (WMF)/Example/Wikimedia project status line{{{Add new Wikimedia engineering project status update}}}

• February 2014
• November 2013
• August 2013

We maintain a list of projects that would be great to have done, but are not on our roadmap in in the near term. See the wishlist here.

See /Meetings for notes from the Showcase meetings of the Release and QA Team.

See the WMF Engineering 13-14 goals page for the yearly view.

• Process through all (useful) pain points from the Dev/Deploy review session (Greg)
• Scap incremental improvements
  • 1) Refactor existing scap scripts to enhance maintainability and reveal hidden complexity of current solution (Bryan)
  • 2) create matrix of tool requirements per software stack (MW, Parsoid, ElasticSearch) (Greg)
  • 3) Use above matrix to add/fix functionality in scap (or related) tooling for ONE software stack, prioritized by cross stack use (Bryan)
• Use the API to create test data for given tests at run time. (Jeff, Chris, Željko)
• Create the ability to test headless (Željko, Jeff, Chris)
• Run versions of tests compatible with target test environments (Chris, all)
• Make database(s) in Beta Cluster emulate production (set up db slaves) (Antoine)

• Status:     in-progress - Process through pain points
  • Status:     Done - security patches
  • Status:     Done - LD SWAT team
  • next...

• LD SWAT team
  • Status:     Done - Kick off
  • Status:     Done - PushBot? - NO (without more dev than Greg can provide right now)

• Status:     Done - create blockdiag version of flow chart
  • commons:File:Wikimedia_development_and_deployment_flowchart.png
  • https://gerrit.wikimedia.org/r/#/c/116750/

• Status:     in-progress - Refine deployment system requirements with Bryan
  • Deployment_tooling/Notes/Deployment_system_requirements

• Status:     Done - quarterly post-mortem review
  • See the review
  • Status:     Not done - set up morgue?

• Status:     In progress Fix up VisualEditor browser tests and make it voting in Gerrit
• Status:     In progress Makes CirrusSearch browser tests voting
•  Done Publish sphinx documentation (use jenkins jobs). Examples:
  • https://doc.wikimedia.org/mw-tools-releng/html/devdeployflow/
  • https://doc.wikimedia.org/mw-tools-scap/docs/_build/html/
• Status:     In progress Migrate beta cluster from pmtpa to eqiad!
  •  Done Varnishes instances creation + puppet passing
  •  Done Application servers created + puppet passing
  •  Done Apache config in git (operations/apache-config.git branch: betacluster)
  •  Done Bunch of files / git repos copied from pmtpa to eqiad
  • Status:     In progress Sean Pringle to create MariaDB
  •  Done Ariel Glenn to set up the swift emulator (copy pasted instance)
  • Status:     In progress last minute sync of files
  • Status:     In progress add Jenkins slaves to sync Parsoid and MW code
  • Status:     In progress write jobs to update Parsoid and MW code on eqiad

• Status:     In progress Scap refactor/python port
  •  Done Extract common logic for making a command line interface into a class
  •  Done Fixed sync-wikiversions to use common dsh arguments
  •  Done Converted sync-wikiversions to python
  •  Done Invented scap-purge-l10n-cache script to cleanup l10n cache on unused branches
  •  Done Documented process for retiring a branch from tin
  • Doing... Setup test environment in beta (carried forward from Feb; harder than I'd hoped)
    •  Done Salt master in eqiad beta project
    •  Done Puppet master in eqiad beta project
    •  Done logstash host in eqiad beta project
    • Doing... scap host in eqiad beta project
  • Continue to enhance and simplify scripts
• Status:     In progress Next generation deployment tooling
  • Doing... Organize requirements into wiki page (carried forward from Feb; pairing with Greg)
• Status:     In progress Support train deploys
  •  Done Deploy 1.23wmf16 to group1
  •  Done Deploy 1.23wmf17 to group0
  •  Done Deploy 1.23wmf17 to group1
  •  Done Cleanup old l10n cache files on cluster
  •  Done Deploy 1.23wmf18 to group0
  •  Done Deploy 1.23wmf18 to group1

Bug management/Task list

Continuing from Feb:

• Test and announce general availability of feature to check for ResourceLoader upon page load: ResourceLoader error checking more globally. Status:     Done

• Test and announce general availability of feature to use the API to create wiki pages and users: create-page (and create-user) API. Status:     In progress

• Investigate how release branches are described in make-deploy-notes Status:     In progress
  • Tracking this at https://bugzilla.wikimedia.org/show_bug.cgi?id=62509 . I have an idea about how to make this work. Status:     In progress

Also:

• Update documentation on mw.o to reflect the refactoring and consolidation of recent times. Much of what exists now is long out of date. Status:     In progress

User:Zeljko.filipin(WMF)

• Process through pain points Status:     in-progress
  • security patches Status:     Done
  • Kick off LD SWAT team Status:     Not done
  • next...
• create blockdiag version of flow chart Status:     Done
  • commons:File:Wikimedia_development_and_deployment_flowchart.png
  • https://gerrit.wikimedia.org/r/#/c/116750/
• Refine deployment system requirements with Bryan Status:     in-progress
  • Deployment_tooling/Notes/Deployment_system_requirements
• quarterly post-mortem kickoff Status:     Not done
  • set up morgue?

• Status:     In progress Complete integration of browsertests for VisualEditor
  • One build worked and triggered two successful scenario!!
  • Parsoid is stopped by the job now, was not previously :/
•  Done Integration of browsertests for CirrusSearch. Cause me to slightly rethink the browsertests infra to closely match production.
•  Done Train Zeljkof on Jenkins Job Builder script
  • Zeljkof started on it. Will pair with him to finish up.
  • Zeljkof found his way through JJB arcanes \O/
•  Done Help migrating Cloudbees Jenkins template to JJB YAML templates
  • Zeljkof started on it. Will pair with him to finish up.
  • Zeljkof found his way through JJB arcanes \O/

• Status:     In progress Scap refactor/python port
  •  Done Local test environment in a MW-Vagrant instance
  •  Done scap converted to python
  •  Done scap-1 converted to python
  •  Done add detailed duration timing for scap & scap-1
  •  Done remove external script dependencies from scap-1
  •  Done add progress bar for dsh commands
  •  Done Scap logs in json format for easy parsing
  •  Done Scap logs sent to florine via udp2log
  •  Done Scap logs sent to logstash via udp2log
  •  Done Converted mwversionsinuse to python
  •  Done Deleted obsolete scripts: scap-1, scap-2, find-nearest-rsync, scap-old
  • Doing... Setup test environment in beta (will pair with Antione)
  • Continue to enhance and simplify scripts
• Status:     In progress Next generation deployment tooling
  •  Done gather requirements via etherpad and Ops-l mailing list
  • Doing... Organize requirements into wiki page (will pair with Greg)
• Status:     In progress Support train deploys
  •  Done Deploy 1.23wmf16 to group0

Bug management/Task list

• Investigate how release branches are described in make-deploy-notes Status:     In progress

• Investigate using ResourceLoader error checking more globally. Status:     Done

• Refactor tests to use the create-page (and create-user) API. Status:     Done

User:Zeljko.filipin(WMF)

These are here just for historical reasons, we no longer use this format.

• /Checkin-20131202
• /Checkin-20131119
• /Checkin-20131105
• /Checkin-20131008
• /Checkin-20130924
• /Checkin-20130917
• /Checkin-20131022
• /Checkin-20130910
• /Checkin-20130903