Tuesday, July 21, 2020

The consequences of archiving open source repos

For this year's vacation, my plans were to add more integration testing on the FieldDB prototype and start converting it to work offline again due to the Chrome store discontinuation of support for apps

Instead, my plans became to update the services to use Node 12. We had switched to using es6 in the web site in 2016, and most by now build tools are intentionally (or unintentionally) dropping support for pre-es6 versions of Node. 

The path to the Node 12 update was relatively short, though bumpy. 

The starting point for this work was that we were using a yarn.lock on the webserver app, but the EC2 instance didn't have yarn installed, and so when installing using npm we were getting install failures due to node-sass. 

While we are at it, let's update everything.

NRC deleted their Github repo

At some point between the last successful build October 29 2017 and July 15 2020, the NRC had deleted their repo https://github.com/nrc-cnrc/InuktitutToolkit. The result was test failures when running the uqailaut.jar (because the jar was not downloaded in the postinstall).

7) /v1
556 GET inuktitut
557 farley's Uqailaut
558 should analyze aaqkiksimalaunngilaq:
559 Uncaught Error: Command failed: ./lib/uqailaut.sh aaqkiksimalaunngilaq
561 at ChildProcess.exithandler (child_process.js:303:12)
562 at maybeClose (internal/child_process.js:1021:16)
563 at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)





Once we got the jar from another source, the build was back up and we were able to verify and merge the Node 12 upgrade. 


ECO is archived

The last service to be updated was Dative, an application written in Coffeescript. We found we were unable to compile the .eco templates.

Eco lets you embed CoffeeScript logic in your markup. It's like EJS and ERB, but with CoffeeScript inside

Using Node 12 (Node 10 and below still works) resulted in what looked to be a parse error. 


Running "eco:files" (eco) task
245File .tmp/scripts/templates/active-server.js created.
246File .tmp/scripts/templates/add-user.js created.
247File .tmp/scripts/templates/alert-dialog.js created.
248File .tmp/scripts/templates/app.js created.
249File .tmp/scripts/templates/application-settings-edit.js created.
250File .tmp/scripts/templates/application-settings-header.js created.
251File .tmp/scripts/templates/application-settings-view.js created.
252>> Error in app/scripts/templates/application-settings.eco:
253>> [stdin]:95:24: error: unexpected end of input
254>> _print _safe '\n' +




We found there was a release candidate 1.1.0-rc-3 we could try, but it contained a breaking change where the compile returned a function rather than the output. 

Running "eco:filesTest" (eco) task
226>> Error in tests/fixtures/advanced-example.eco:
227>> TypeError: eco.compile(...).replace is not a function
228Warning: Eco failed to compile tests/fixtures/advanced-example.eco. Use --force to continue.
230Aborted due to warnings.
231The command "grunt" exited with 6.
cache.




As a first step, we added travis to eco using a fork, since https://github.com/sstephenson/eco was archived. 



Node < 6 errored due to the EJS issue. Node 6-12 passed but passing and unable to reproduce the parsing error using the sample templates in the test suite.
JobNode.jsOSState
 12.10.8Linuxerrored
 12.20.10Linuxerrored
 12.30.12Linuxerrored
 12.46Linuxpassed
 12.58Linuxpassed
 12.612Linuxpassed
We were back to looking for root cause and options. After some "instrumentation" in the node_modules/eco/lib directory, the root cause appeared to be the output of the Coffeeescript circa Node 0.10 was likely not having the same closure as Node >= 12.  One option we explored was finding if anyone else had tried making a new release for eco using a recent version of Coffeescript. 


Looking for others who tried to update eco

Using the network graph we could see quite a few forks had added new commits, some of which were using the existing pull requests and/or building on them. 



Using npm link with one of the forks resulted in a successful local build of the templates. Now the ultimate question, do we try to publish a new release? This would also require updating grunt-eco. 


We also looked around for alternatives to eco, and found http://ectjs.com/ and https://github.com/netzpirat/haml-coffee neither of which would be a quick conversion.

EJS introduced es6 in a patch version

We found we were no longer able to install nodeunit in eco with older node versions, because EJS had accidentally introduced a let on the 2.x release before bumping to 3.x. I made an issue to help others find the root cause and a demo pr that could fix it if they want to make a 2.x release branch. 

Nodeunit recommends switching to a modern test framework

Nodeunit is pretty old, and not recommended or used for new projects but it was quite popular in the early days of node before version 0.12. Nodeunit itself is very stable and hasn't needed any code changes in years. However since February, it's no longer installable on older node versions due to the EJS breaking patch. The result is that older projects will loose their testability if it is not able to accept patches in the future. 



Why Archive Github repos?

After finding two projects archived, and one deleted in the space of one week I started to wonder, why are repo owners deleting and archiving their projects, since this didn't seem to match my expectations of what are best practices for open source projects.

To answer my question I went to the source for node module best practices: NPM. 

NPM recommends transferring a project to them if you don't want to maintain it. That matches my expectation that a package should be transferred to someone else who can help maintain it, if current maintainers not available: https://docs.npmjs.com/deprecating-and-undeprecating-packages-or-package-versions#transferring-a-deprecated-package-to-npm

If you are no longer maintaining a package, but other users depend on it, and you’d like to remove it from your user profile, you can transfer it to the @npm user account, which is owned by npm, Inc.


Github now lets you archive projects, but that is because they want to help prevent deletion. This matches my expectation that codebase which are consumed by codebases should stay available (of course code deletion can be necessary if there are important reasons like a security leak or other sensitive information which needs to be removed from the repo history). 


Just because a repository isn’t actively developed anymore and you don’t want to accept additional contributions doesn’t mean you want to delete it. 

This means that the desire to archive projects isn't coming out of a recommendation or best practice, but rather simply because maintainers want to prevent distraction. 

https://tommcfarlin.com/why-im-archiving-my-github-repositories/ 

I think my time can be better spent on other work. And I think I want to spend my time on other things.

Maintaining old open source projects, particularly those which are still consumed, can take a lot of extra work when the PRs are often proposed by newbies, and/or there is no/limited automated testing. 

But what about projects which have Travis set up for CI, or projects where there are hundreds of forks which might include someone who can help make new releases?

Ask for maintainers rather than archiving

The first step when a project is creating distraction, is not to archive it or delete it, but rather to ask around if anyone who is relying on the project is able to help, or put a notice on the project saying that you are not available to maintain it and the community will likely self organize to offer maintenance help in the original repo rather than having to use forks.


If you need to step away from your project, either on hiatus or permanently, there’s no shame in asking someone else to take over for you.


Do your best to find support for your users and community while you’re away from a project. If you can’t find the support you need, take a break anyway. 

In my case, finding viable forks of eco which had an update that resolved our parse issue took me three days and a lot of npm installs. If eco wasn't archived, this process of communication would have been easier for those who had also been looking for new releases: one of the three who had made a new releases would have been able to help maintain the original repo, rather than each doing it through several forks. 

For consumers of a project, archiving a Github projects comes down to cutting the centralized collaborative aspect of the codebase; arguably one of the most significant aspects for which Github became the defacto strategy for maintaining new open source projects nearly 10 years ago.