• Summary
• Background
  • Infrastructure required for open source collaboration
  • Criteria for selecting project infrastructure
  • Criteria for deploying project infrastructure
• Current NumFOCUS infrastructure
• Proposed new NumFOCUS infrastructure
  • Overview
  • Capabilities of current vs. proposed resources
  • Recommended implementation
  • Overview of cost, self-hosted vs managed hosting
  • Deployment methodology
  • Timing and blockers
  • A note about scaling
  • Risks
  • Costs of not deploying this infrastructure

31 May 2017

Summary

Because the advantages conferred by working in the “open source way” apply as significantly to projects that are not primarily about code, I plan to run the Sustainability Program in the same way that I would run an open source software project.

Open source projects, particularly because they are conducted by geographically and often institutionally distributed teams and individuals, require a minimum set of collaboration tools (infrastructure) in order to do their work. The Sustainability Program requires this as well.

NumFOCUS already has access to and utilizes some of these tools, but not all. And in some cases NumFOCUS is utilizing options for a given collaboration tool that are not optimal for open team work.

I propose allocating approximately $1,500 annually and 4-8 hours of staff/volunteer time per month to deploy and maintain the following services to close this gap: Zulip for synchronous text chat, Discourse for asynchronous discussion, Etherpad Lite for real-time note taking, and Mediawiki for meta documentation and recording institutional memory.

The benefits to NumFOCUS and the Sustainability Program for this expenditure include:

• Participation in the sustainability program is maximized and and we benefit from valuable contributions and perspectives.

• Outreach about the sustainability program is maximized and we leverage valuable opportunities to raise our visibility in the community and with sponsors.

• NumFOCUS further demonstrates our commitment to transparency and alignment with the scientific open source communities we serve.

Background

Infrastructure required for open source collaboration

Open source projects require a minimum amount of infrastructure in order for viable and engaged collaboration to occur and work to be produced. These include:

1. A way to host, share, and collaborate on the work product. For most projects, this takes place on a hosted (D)VCS such as GitHub, GitLab, Bitbucket, Subversion etc.

2. A way to collect, triage, and prioritize bug reports and feature requests. For most projects this takes place on an issue tracker such as GitHub issues, Jira, Trac, or Bugzilla. Some projects utilize additional tools intended to provide a friendlier experience to for end-users such as Uservoice.

3. A way to coordinate and prioritize product development. Some projects use their issue tracker for this, others use different tools like Trello, Asana, Kanbanery, etc.

4. Ways to communicate (synchronously and/or asynchronously) with other project contributors, potential contributors, and users about the project. IRC used to be the de facto standard for real-time text chat among open source projects. Now Slack and Gitter are common tools for this. Mailing lists (using mailman or Google groups) have long been the most common asynchronous platform. Discourse is emerging as a popular, more accessible and user-friendly alternative.

5. Ways to collaboratively conduct synchronous meetings. Open source projects run meetings in myriad ways. Some projects never meet in real-time and conduct all their work over mailing lists or issue trackers. Others conduct meetings exclusively via text-chat with IRC, Slack, or Gitter. Others take advantage of voice and/or video teleconference solutions such as Google Hangouts, Skype, Vidyo, GoToMeeting, Appear.in, Talky.io, etc. Many open source projects use Etherpad (Lite) for collaboratively drafting agendas and taking meeting notes. Some use Google docs for this.

6. A way to authoritatively communicate what the project is, including its mission, who is involved and how to get involved. Most projects have some kind of homepage for their project. For projects that do not have a stand-alone website, it is common to use the Readme.md file in the root of their GitHub repository, GitHub pages, the landing page of their documentation on Read the Docs, or something similar.

7. A way to host contributor-focused documentation. Most projects include this directly in their source code and use GitHub markdown, GitHub pages, or ReadTheDocs to render and display. Some larger projects have separate repositories and stand-alone websites dedicated to documentation and contributor on-boarding.

8. A way to collaboratively record and share the institutional memory of the project (e.g. best practices and other collective knowledge, meeting notes, process log, decision log, history, etc.) of the project. Many projects use a wiki for this. The most common wiki software among open source projects are Mediawiki and Confluence, followed by GitHub wiki pages. Other projects use their contributor documentation for this.

As a project grows, and depending on its particular needs, additional tooling might be required such as continuous integration, build and test servers, incident response documentation, monitoring, backups, dev/staging/production servers, authentication, etc.

Criteria for selecting project infrastructure

These are among the criteria I use when evaluating which open source project infrastructure tools to adopt:

• Is the project free or open source software? FOSS isn’t a hard requirement, but it is a preference.

  • If yes:

    • Does the community behind the project have a good reputation?

    • Is the project mature? (I don’t exclude newer projects out of hand, but prefer to reduce risk with well-established projects.)

  • If no:

    • Can open source clients be used with the product?

    • Are necessary clients freely available?

• Are desktop clients available for Linux, Windows, and macOS? Are mobile clients available for both iOS and Android?

• Is the product primarily designed to be used with open teams or closed teams? Does the product support “open by default”? Can the tool’s content be made publicly available without requiring permission or approval?

• Does the product have a permissions model that supports open source project collaboration?

• Are there self-hosted/self-managed options as well hosted/managed ones? Are both options affordable? What SLAs and support options are available for hosted/managed options? What hardware and devops resources are required for self-hosted/self-managed options?

• Is the tool mobile friendly? This is a requirement.

• Is the product respectful of users’ security and privacy?

  • SSL is requirement.

  • Free products are to be avoided because they generally rely on selling some aspect of user data.

• Does the product support multiple authentication methods, or must you have an account on X network (Google, Facebook, GitHub, etc.) in order to use it?

• Is the product accessible and user-friendly?

• Does the product support both GUI/web interfaces as well as commandline ones?

• Does the product support multiple languages? (Nice to have, not a requirement for NumFOCUS at this time.)

• Are open source contributors generally familiar with the tool, or is it something very new to the ecosystem?

• Does the product utilize open standards? Is it interoperable with other platforms and services? Is it easy to import and export data from the product? Does the product have an open API?

Criteria for deploying project infrastructure

Whenever practical, self-hosted project infrastructure should be deployed and maintained following good devops / sysadmin practices. This includes using some kind of configuration management with reproducible scripts stored in version control along with good documentation. More than one staff person or volunteer should understand how to access servers, and how to deploy and maintain them.

Current NumFOCUS infrastructure

At present, NumFOCUS infrastructure includes the following:

• GitHub organization with ability for private repositories (I am unsure if paid tier or not). This includes issue tracking, boards, GitHub pages, and wiki pages.

• Main website using Wordpress served by WP Engine managed hosting.

• Slack account (either free or with NPO discount, I need to confirm which).

• Rackspace cloud account with limited amount of free credit for limited amount of time.

• G Suite for numfocus.org (limited to employees and board members), providing mail, calendar, address book, hangouts, and docs.

• PyData hosting and deployment infrastructure, the details about which I am unfamiliar.

• Namecheap for domain registration and some DNS.

I believe we also have existing accounts with CloudFlare and Amazon Web Services, but don’t know the specifics.

Proposed new NumFOCUS infrastructure

Overview

Although the work of the Sustainability Program is not entirely about code, it will produce some. And it will produce documentation. And we need a way to coordinate, conduct, and share our work. All of the collaboration infrastructure needs mentioned above apply to the Sustainability Program.

Existing infrastructure within NumFOCUS can meet some of these needs, but there are significant gaps that need to be addressed before I can make an effective call for participation.

Current infrastructure that is a good fit for present needs include: GitHub for DVCS, issue tracking, product development, and contributor-focused documentation; Google Hangouts and GoToMeeting for voice and video conferencing; and Wordpress / WPEngine for website hosting.

New infrastructure I propose we allocate budget and staff resources for include: Zulip for synchronous text chat, Discourse for asynchronous discussion, Etherpad Lite for real-time note taking, and Mediawiki for meta documentation and recording institutional memory.

Capabilities of current vs. proposed resources

Below is a table detailing current NumFOCUS resources and new resources I propose we invest in spinning up.

Recommended implementation

I estimate that self-hosting the above infrastructure to be the most cost-effective option, requiring an annual budget of about $1,500 and requiring 4-8 hours of technical staff or volunteer time per month to maintain.

I propose we utilize Digital Ocean virtual machines for hosting all services except for email transactions, for which I recommend we use Amazon SES (Simple Email Service).

I am recommending Digital Ocean over Rackspace or Amazon Web Services for virtual machine deployment because:

• My most recent experience is with Digital Ocean. While I have experience deploying to both Rackspace and AWS, I currently host all my personal websites with Digital Ocean. This means I am familiar with their offerings and already have deployment tools I can repurpose for use with NumFOCUS. I do not have this advantage with AWS or Rackspace and so it would take more of my time to deploy there.

• Digital Ocean is less expensive than Rackspace. While it’s true we currently have an allotment of credits with Rackspace, I think everyone is concerned about the stability of Rackspace and its desire to continue providing free services to open source projects. If we deploy now to Rackspace to save a buck, that decision may end up costing us more if/when free credits go away forcing us to pay a premium to stay with Rackspace or orchestrate a migration to a less expensive provider.

I recommend Amazon SES for email transactions because I have used it before and it offers competitive pricing. Plus I believe we already have an account with AWS. However, I am open to suggestions for other providers. (We absolutely do need a service to transact emails. All of the services I’ve proposed need a way to send and in some cases receive email. I don’t want to setup and administer a mail server and running independent mail servers and effectively dealing with spam issues is becoming harder and harder.)

Overview of cost, self-hosted vs managed hosting

Deployment methodology

Deploying and maintaining the above services will utilize devops best practices whenever feasible. I plan to use configuration management platform Chef and thoroughly document the deployment structure and process. The goal is to deploy in such a way that I am not a bottleneck and others can contribute to maintaining the infrastructure.

Timing and blockers

I can start on this right away. We would need to create a NumFOCUS account with Digital Ocean and attach a payment method to it.

Amazon SES also needs to be in place prior to deploying any of the services. If we already have an AWS account, I need to be granted access to it with appropriate permissions and then I can setup and configure the SES component. Verification will require access to our DNS records, which I believe I already have for numfocus.org through Namecheap.

Once email is configured, I can start deploying the other services. I estimate it will take about one (1) day each to deploy the services and another 1-2 days to complete tooling and documentation.

I would like to have Discourse and Zulip in place prior to a public announcement and call for participation for the Sustainability Index.

A note about scaling

The deployment solutions I’ve proposed above will accommodate the load of NumFOCUS specific work and then some. That is, if it turns out our member projects would like to use any of these tools, they will be able to.

To get an idea of how costs might scale: The next bigger size of VM from the one I have proposed above is 4GB is runs ~$57/month ($40 base + $8 weekly backups + $9 daily snaps). If disk space becomes the limitation, we can explore if block storage is a more economical option.

Risks

I’ve tried to be thorough about thinking through the risks of deploying these services and how we can manage and/or plan for them. Here’s what I have so far:

Christie can’t figure out how to deploy a service or it takes much longer to do so than she has anticipated. A reasonable concern. While not a full-time sysadmin or devops person, I do have significant experience over nearly 20 years of administering linux-based machines. I’m pretty good at it considering it’s not my day job. And I have a substantial network I can tap if I get stuck or otherwise in trouble. If I run over my time estimate, it will likely be by a day or two, not weeks.

I leave NumFOCUS and there is no one available to maintain the services. If I do my job right (e.g. in the ways I’ve described above) it will be straightforward to on-board new administrators and I won’t stay the only person who can help maintain the servers for very long.

Deploying and maintaining these services takes too much of Christie’s time and distracts from the Sustainability work. I’ve spent some time noodling on this one to ensure I’m not falling into the trap of doing fun work at the expense of needed work. What I’ve concluded is that having good collaboration tools is integral to our sustainability work and therefore worth the wise use of my time and energy.

Even though Christie really likes these services, they aren’t widely adopted and we end up paying for something very few people are using. If after some amount of reasonable adoption time (6 months?), very few people are using any of these services, we can decommission it. There is no financial penalty for doing so since we aren’t signing any contracts agreeing to a duration of service. Even keeping something running for a year means we’re out at most a few hundred dollars.

Some of all of these services become very popular and we need to upgrade (spend more money on hosting) to meet demand. This is a good problem to have! It means we are providing a needed service to our contributors. I don’t anticipate that any of these services will suddenly require an exponential cost outlay in order to meet demand. It will be gradual and something we can plan for.

Costs of not deploying this infrastructure

While it’s true to a certain extent that I could organize the work of the Sustainability Program with existing tools, I believe there is a significant downside to not improving our infrastructure at this time. These include:

• Christie is frustrated and not able to do her job as effectively or efficiently.

• Participation in the sustainability program is hampered and we lose out on valuable contributions and perspectives.

• Outreach about the sustainability program is hampered and we lose out on valuable opportunities to raise our visibility in the community and with sponsors.

• NumFOCUS misses out on an opportunity to demonstrate our commitment to and alignment with the open source community.