Tuesday, July 21, 2020

The consequences of archiving open source repos

For this year's vacation, my plans were to add more integration testing on the FieldDB prototype and start converting it to work offline again due to the Chrome store discontinuation of support for apps

Instead, my plans became to update the services to use Node 12. We had switched to using es6 in the web site in 2016, and most by now build tools are intentionally (or unintentionally) dropping support for pre-es6 versions of Node. 

The path to the Node 12 update was relatively short, though bumpy. 

The starting point for this work was that we were using a yarn.lock on the webserver app, but the EC2 instance didn't have yarn installed, and so when installing using npm we were getting install failures due to node-sass. 

While we are at it, let's update everything.

NRC deleted their Github repo

At some point between the last successful build October 29 2017 and July 15 2020, the NRC had deleted their repo https://github.com/nrc-cnrc/InuktitutToolkit. The result was test failures when running the uqailaut.jar (because the jar was not downloaded in the postinstall).

7) /v1
556 GET inuktitut
557 farley's Uqailaut
558 should analyze aaqkiksimalaunngilaq:
559 Uncaught Error: Command failed: ./lib/uqailaut.sh aaqkiksimalaunngilaq
561 at ChildProcess.exithandler (child_process.js:303:12)
562 at maybeClose (internal/child_process.js:1021:16)
563 at Process.ChildProcess._handle.onexit (internal/child_process.js:286:5)





Once we got the jar from another source, the build was back up and we were able to verify and merge the Node 12 upgrade. 


ECO is archived

The last service to be updated was Dative, an application written in Coffeescript. We found we were unable to compile the .eco templates.

Eco lets you embed CoffeeScript logic in your markup. It's like EJS and ERB, but with CoffeeScript inside

Using Node 12 (Node 10 and below still works) resulted in what looked to be a parse error. 


Running "eco:files" (eco) task
245File .tmp/scripts/templates/active-server.js created.
246File .tmp/scripts/templates/add-user.js created.
247File .tmp/scripts/templates/alert-dialog.js created.
248File .tmp/scripts/templates/app.js created.
249File .tmp/scripts/templates/application-settings-edit.js created.
250File .tmp/scripts/templates/application-settings-header.js created.
251File .tmp/scripts/templates/application-settings-view.js created.
252>> Error in app/scripts/templates/application-settings.eco:
253>> [stdin]:95:24: error: unexpected end of input
254>> _print _safe '\n' +




We found there was a release candidate 1.1.0-rc-3 we could try, but it contained a breaking change where the compile returned a function rather than the output. 

Running "eco:filesTest" (eco) task
226>> Error in tests/fixtures/advanced-example.eco:
227>> TypeError: eco.compile(...).replace is not a function
228Warning: Eco failed to compile tests/fixtures/advanced-example.eco. Use --force to continue.
230Aborted due to warnings.
231The command "grunt" exited with 6.
cache.




As a first step, we added travis to eco using a fork, since https://github.com/sstephenson/eco was archived. 



Node < 6 errored due to the EJS issue. Node 6-12 passed but passing and unable to reproduce the parsing error using the sample templates in the test suite.
JobNode.jsOSState
 12.10.8Linuxerrored
 12.20.10Linuxerrored
 12.30.12Linuxerrored
 12.46Linuxpassed
 12.58Linuxpassed
 12.612Linuxpassed
We were back to looking for root cause and options. After some "instrumentation" in the node_modules/eco/lib directory, the root cause appeared to be the output of the Coffeeescript circa Node 0.10 was likely not having the same closure as Node >= 12.  One option we explored was finding if anyone else had tried making a new release for eco using a recent version of Coffeescript. 


Looking for others who tried to update eco

Using the network graph we could see quite a few forks had added new commits, some of which were using the existing pull requests and/or building on them. 



Using npm link with one of the forks resulted in a successful local build of the templates. Now the ultimate question, do we try to publish a new release? This would also require updating grunt-eco. 


We also looked around for alternatives to eco, and found http://ectjs.com/ and https://github.com/netzpirat/haml-coffee neither of which would be a quick conversion.

EJS introduced es6 in a patch version

We found we were no longer able to install nodeunit in eco with older node versions, because EJS had accidentally introduced a let on the 2.x release before bumping to 3.x. I made an issue to help others find the root cause and a demo pr that could fix it if they want to make a 2.x release branch. 

Nodeunit recommends switching to a modern test framework

Nodeunit is pretty old, and not recommended or used for new projects but it was quite popular in the early days of node before version 0.12. Nodeunit itself is very stable and hasn't needed any code changes in years. However since February, it's no longer installable on older node versions due to the EJS breaking patch. The result is that older projects will loose their testability if it is not able to accept patches in the future. 



Why Archive Github repos?

After finding two projects archived, and one deleted in the space of one week I started to wonder, why are repo owners deleting and archiving their projects, since this didn't seem to match my expectations of what are best practices for open source projects.

To answer my question I went to the source for node module best practices: NPM. 

NPM recommends transferring a project to them if you don't want to maintain it. That matches my expectation that a package should be transferred to someone else who can help maintain it, if current maintainers not available: https://docs.npmjs.com/deprecating-and-undeprecating-packages-or-package-versions#transferring-a-deprecated-package-to-npm

If you are no longer maintaining a package, but other users depend on it, and you’d like to remove it from your user profile, you can transfer it to the @npm user account, which is owned by npm, Inc.


Github now lets you archive projects, but that is because they want to help prevent deletion. This matches my expectation that codebase which are consumed by codebases should stay available (of course code deletion can be necessary if there are important reasons like a security leak or other sensitive information which needs to be removed from the repo history). 


Just because a repository isn’t actively developed anymore and you don’t want to accept additional contributions doesn’t mean you want to delete it. 

This means that the desire to archive projects isn't coming out of a recommendation or best practice, but rather simply because maintainers want to prevent distraction. 

https://tommcfarlin.com/why-im-archiving-my-github-repositories/ 

I think my time can be better spent on other work. And I think I want to spend my time on other things.

Maintaining old open source projects, particularly those which are still consumed, can take a lot of extra work when the PRs are often proposed by newbies, and/or there is no/limited automated testing. 

But what about projects which have Travis set up for CI, or projects where there are hundreds of forks which might include someone who can help make new releases?

Ask for maintainers rather than archiving

The first step when a project is creating distraction, is not to archive it or delete it, but rather to ask around if anyone who is relying on the project is able to help, or put a notice on the project saying that you are not available to maintain it and the community will likely self organize to offer maintenance help in the original repo rather than having to use forks.


If you need to step away from your project, either on hiatus or permanently, there’s no shame in asking someone else to take over for you.


Do your best to find support for your users and community while you’re away from a project. If you can’t find the support you need, take a break anyway. 

In my case, finding viable forks of eco which had an update that resolved our parse issue took me three days and a lot of npm installs. If eco wasn't archived, this process of communication would have been easier for those who had also been looking for new releases: one of the three who had made a new releases would have been able to help maintain the original repo, rather than each doing it through several forks. 

For consumers of a project, archiving a Github projects comes down to cutting the centralized collaborative aspect of the codebase; arguably one of the most significant aspects for which Github became the defacto strategy for maintaining new open source projects nearly 10 years ago. 

Thursday, June 28, 2018

Alpen Tage 6 Obergurgl - Rotmoosjoch

Dotted lines: #notatrail

We went through Rotmoosferner  (Redmoss Glacier) in late June 2018 and found that climbing from the glacier to the ridge required snowshoes (slope > 70 degrees) due to deep unpacked snow. Visiting the Rotmoosferner trail is recommended to see the amazing rock debris left by the glacier melt. 

However, continuing the trail to cross over the mountain ridge to Rotmoosjoch (Redmoss Notch) should only be done with proper equipment and in July or August. The snow and rock slides are unstable and the ridge does not appear to have existing rock climbing attachments to clip into, unlike other ridge crossing trails in the Alps. We do not recommend attempting this crossing in June, or without proper equipment. 

Here is our GPS trace for this hike:

GPS trace recorded by MyTracks.

Here is a video of the hike which allows you to see the angle of assent/decent:




We followed a dotted trail which is shown on some maps to cross the ridge to get to the Zwickauer Hütte. Note: dotted trails are seasonal unmarked mountaineering trails which require rock climbing equipment.


The trail we followed was a dotted line trail, we found out later that means an unmarked seasonal mountaineering trail which requires rock climbing equipment.


Locals said it is recommended to try to cross there only in July or August, and only with a Bergfahrer (mountaineer) who can secure the attachments. 

It is an unmarked footpath (missing red/white markers on the ground). After the ground turns to snow and shifting rock, the path can be followed using GPS from the Ratmoosferner to the Ratmoosjoch. Once you get up to the ridge you will find the ridge does not appear to have existing rock climbing attachments to clip into at the GPS location of the joch (notch).

We climbed up until we could see the sun behind ridge through the fog. We could also see from Google Maps that we had reached the ridge. One of us went up to the rock face to look for attachments (blue dot in the image below). After what felt like 30 minutes of exploring the rock face, realized we would be unable to make the crossing, so we slid back down. 


At one point, we were able to see the sun was unblocked by further mountains.



GPS trace from Google Maps, show section from the Ratmoosferner (Redmoss Glacier) trail to the Ratmoosjoch (Redmoss Notch).


After we had slid down the glacier and we were in a stable location, I took a screenshot of where the GPS showed us. I forgot to snapshot it while we were on the incline, which showed we had made it across the Italian border, possibly due to GPS drift.

We spent a while at the ridge, the GPS trace shows lots of slipping (speeds of 5km/hour without changes in altitude.)



On a sunny day we would have been able to see the notch.



Wednesday, June 27, 2018

Alpen Tage 5 Huben - Langenfeld - Huben

This day we hiked to the next town and back as a day of rest. 

A day of rest, 9km walk from one village to another.

Tuesday, June 26, 2018

Saturday, June 23, 2018

Friday, June 22, 2018

Alpen Tage 2 Kemptner Hütte - Memminger Hütte

We hiked 29km from the Kemptner Hütte to the Memminger Hütte.


Overview of the hike:

GPS trace 

Video overview of the hike: 



Sleeping with ear plugs was magical. We had bread an muesli mit milch und milch cafe for breakfast. We got our things and went outside to fill the water bag and prep the bags for the day.

Well organized to maximize sleeping capacity.


We climbed out of the crater to a amazing view with shrubs. To the first crossing to another country into Austria by walking. We walked down, about halfway down passed a cafe which wasn't open yet (it was still only 8 am).
Morning view of hike out of the crater.


 
We saw the sign taking about the taxi from the town to Madau base of the mountain, we decided to take the walk to at least the Bach town, and re-evaluate there since it would be our longest day and didn't appear to be normal to walk it.

The hills and fields were beautiful, we got to town, changed from our boots to our shoes and called for info about the taxi. The dispatch said they didn't know when the taxi would be back. We went to the market to re-supply a bit, got meat and cheese and nectarines. We read the sign for the taxi and it said 60 euros for up to 4 people, we decided to keep going (we knew we could do the distance, and we are that cheap).

View of the village in the valley.



Leaving Bach towards Madau and found that the train was a pilgrimage trail with icons and benches along the way. We saw signs saying the village or the road would be closed that weekend. We were running really low on water for what we had left to do, with only 700ml of water left. Our trail led us on the west of the river, until we got to the turn in the road and saw that the GPS trace we were following wasn't a path, this was the first mishap of the Komoot app, we turned around and went back to where the village was supposed to be, and saw 1-2 signs for Gasthof, but no restaurants or markets on the road, and still no water.

We made some hiking staffs and started the gradual incline towards the mountains. We ate some  wurst and talked to a nice older couple who were also on their way to the Memmingen hütte. We found a horse/hiker watering trough with running water, as we had seen elsewhere in villages and filled our water pack. We got to the end of the road and saw the cable car for taking packs and supplies up to the house. We read the signs 16€ per pack, that you can call with the telephone up to the hütte, but we decided we had enough energy to just keep the bags since they weren't very heavy.

We are going up there!


We started the climb at around  3:30, expecting to be at the top way before 9pm (sunset) since we had only 2km left to do, but 1km vertical also. We met a group coming down who warned us that it would take a trip least 2 hours, we smiled. Probably they thought we were going for a day hike as well. Yes we know...

Every 100 meters in altitude on our GPS we paused and had cheese, fig, tomatoes or wurst in alternation. This system, and the cold water from the mountains spicket gave us energy and we made it to the top by 7pm!

We crossed a few waterfalls and ice flows. It was windy, we even put on our tuque and gloves. We got to see mountain goats in person and even saw them battle (or play) for king of the boulder.

Mountain goats


We finally arrived at 7pm just before the kitchen closing of 7h30, it took us way longer than expected to do that last 2km.

Hutte in the distance.


While waiting for dinner I started getting the chills and, and I did put on m my winter clothes, but still kept shivering. We sat with a mixed group of folks, one of them (I think the guide of the group) said Mathieu didn't speak German. The rest of  the table gave him an eye and said be careful, I had been speaking with them in German for over an hour before he arrived. He recovered by criticizing me of speaking hoch Deutsch. I shrugged. We ate most of our food to get energy and went out for a walk to enjoy the sunset and view the crystal mountain lake.

The view was breathtaking, and amazing!

A wheat beer to enjoy the view.

Breathtaking view of the glassy lake at the top.


Again to bed early, tomorrow would be a short day only 14km, almost all downhill so we put in our earplugs and planned to wake up at 7:30... but of course at sunrise at 5.30am UP we were!

Thursday, June 21, 2018

Alpen Tage 1 Obersdorf - Kemptner Hütte

We hiked from Obsersdorf to the Kemptner Hütte.

GPS trace 

Video overview of the hike: 




We woke up early to get the 8am train. We walked by the trout to say goodbye, got coffee and our tickets.

The train platform listed two trains stopping at 8am, on the same platform. While we waited in the sun the train arrived. After waiting a few minutes we asked the conductor who was getting off his shift if that train went to Oberstdorf, he said yes, almost missing the train that was in our faces, in that zugteil, fortunately right on the platform where we had been waiting.


Inside the train car, the destination confirmed that it was going to Oberstdorf. Basically we found out it wasn't two trains that stopped, but instead one train with two engines, which would split mid way, the front engine continuing to au.. while the back engine to Oberstdorf!

We walked through the ski village of Oberstdorf, bought a pretzel bun tomato and cheese sandwich and then found the foot path that led up into the mountains.


We heard our first cacophony of cow bells in the hills.

Cow with bells that could be heard from several hundred meters.


After a little while, we stopped at a Gasthof for a beer and tea.

A beer in the valley.


We began to climb, saw our first rock slides, hid from sprinkles under a rain-forest style tree and had lunch. We saw a group retirees who passed us with their day-packs without realizing that we were there eating on a rock under a tree 1 meter away from them.

Climbing was slow going, getting used to the altitude, saw our first melting ice flows.

Finally around 7 pm we made it to the top of the mountains and we could see the Kemptner hut above us.

Kemptner Hütte above us.

We sat with two guys who told us that they were taking a taxi the next day, instead of the 29km we had planned. Dinner included svíčková and haluski, my favorite.

Walk after dinner.


We also got Glühwein and of course Witbier. We had schlafplatz near the door, with ear plugs slept the night through under heavy rain until it was time to wake up at sunrise 5:30.

Sunday, June 17, 2018

Hiking the E5

Next week we are going for a hike, a hike across the alps.


Sunday, November 26, 2017

Geogian diaspora

This month a friend and I looked into the results of Recognizing Speech on Android.

In 2014 we created open source libraries and tools to facilitate usage of the Georgian language by Georgian speakers. One of these tools was Gismet, an Android application which can be used by Georgian speakers to train their Android smartphones to recognize their speech using PocketSphinx. The software was made freely available to the public and also open source on GitHub.


Since 2014 over 1,000 users have used the application to train the default language model to their voices. The location of the recordings is determined via GPS technology, and informed consent for the anonymous analysis of their voices is required as part of the software installation process.



Georgian diaspora
  • Red: Android location with < 100 accuracy range
  • Blue: IP geolocation



georgia
Georgian Diaspora, shown by GPS location of anonymous recordings.


The resulting datasets of anonymous elicited training recordings is similar to datasets obtained in an experimental setting. Below are a couple of spectrograms from the anonymous training recordings.


georgia

Specrogram of "რა ტემპერატურაა დღეს?" with careful yet natural prosody.

georgia
Specrogram of "რა ტემპერატურაა დღეს?" with audience-less prosody.

Tuesday, July 19, 2016

Hiking in Chiang Mai

It turns out that there are plenty of trails around Chiang Mai which you can take to get to the summit of any mountain. The only challenge is finding them. We used GPS traces of other foreigners which we found after a fair bit of Googling.

One of our our recorded GPS trace

Elevation/Speed plot recoded by My Tracks
Trail info along the river.
The trails we found were mostly used by locals for pilgrimage to shrines, most mountains in that region have a shrine on top, so there is almost always a trail.

One of the trails was mostly a dried riverbed.
Hiking in March this year turned out to be a very hot endeavour. March is the dry season, and most of the mountains are control burned so that the land is black and the leaves shrivel and fall off. This makes very little cover from the sun.

The burning also makes the air have a high pollution index, and you can't see very far in the distance.

Saturday, July 9, 2016

Authenticated Systems

This week I will be giving a talk at NY JavaScript on how to build authenticated systems which have multiple services and multiple clients.

In this talk we will look at a sample auth service which lets users login via numerous OAuth2 providers (Google, Facebook, Twitter etc) or using username and password.



The auth service uses a bower component to present a static HTML 5 UI, instead of using  server side templating.

The auth service also shows how to use JWT instead of sessions to identify and enforce the user's permissions for the requested resource.
The auth service shows how you can persist users using sequelize (which lets you learn with sqlite and use postgres later).
The auth ui uses a simple form submit to hit the auth service (it also shows an example of an Ajax request using fetch).

There is a sample client which is an HTML5 bower component in vanilla JavaScript. The client uses JWT to detect if you are logged in and have permissions to view resources.

There is a library that contains the the system's public key and JWT options which can be used both client and server side to verify tokens.

Github codebase:


Tuesday, March 31, 2015

Running Mac as a production server

For one of my open source projects we were given a server, a mini mac to be precise. Sure, we could have put linux on it, but because all the stack we are using runs on Mac so we decided we could probably use this server, why not. We sent it to a mac mini server farm to host it.

Image result for mac mini server 2013

There were a few wrinkles.

Sleep.
We delegated the server setup to an intern so they could learn how to set up the machine, they forgot to turn off sleep so the server was oddly unresponsive for the first week or so of testing.

Updates.
Macs like updating, and they seem to use RAM to do so. If you don't update the server you can have a significant chunk of your RAM hanging out waiting for you to do the update. We haven't figured out how to accept XCode downloads from the apple store, so this means we had to learn how to turn on VNC.

Turning on VNC
sudo  /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -activate -configure -access -on -clientopts -setvnclegacy -vnclegacy yes -clientopts -setvncpw -vncpw mypasswd -restart -agent -privs -all

Turning off VNC

sudo /System/Library/CoreServices/RemoteManagement/ARDAgent.app/Contents/Resources/kickstart -deactivate -configure -access -off


Using pm2 vs forever
At the same time we did the upgrade, we decided wouldn't it be nice to have a new node process manager? So we switched our webservices to use pm2. After a month of sleep like behavior, after we had solved the sleep like behavior, we decided maybe switching process managers at the same time as switching OS was a bad idea. We switched back to forever and now the server is almost normal.

These were a few reasons why we regret using the Mac server instead of formatting it with Linux, although now that its sort of stable with ~97% uptime we aren't planning formatting it just yet.

Wednesday, August 20, 2014

Recognizing Speech on Android

Tonight I'll be giving a talk at Android Montreal. In this talk I’ll show you how to use speech recognition in your own Android apps.

The talk will have something for both beginner and advanced Android devs, namely I will show two ways to do speech recognition: the easy way (using the built-in RecognizerIntent for the user’s language) and the hard way (building a recognizer which wraps existing open source libraries if the built-in RecognizerIntent can’t handle the user’s language).


While I was in Batumi my friends and I built an app so that Kartuli users (code) (slides) (installer) could train their Androids to recognize SMS messages and web searches.



Recognizing Kartuli is one of the cases where you can’t use the built-in recognizer. Kartuli spoken by only 4 million people in the country of Georgia: that is roughly the population of Montreal and surrounding areas.

CaucasusLayout_rev.png

The talk will start with a demo of our Kartuli trainer app to set the context for the talk, and then dig into the code and Android concepts under the demo.
  • * How to use the default system recognizer’s results in your own Android projects,
  • * How to use the NDK in your projects,
  • * How to use PocketSphinx (a lightweight recognizer library written in C) on Android

Live broadcast on YouTube:

http://youtu.be/XNIS0hTzGIA

Friday, March 28, 2014

SMILE is patent pending

This morning Google now let me know that something new had been posted on the Sphinx site.

Google Now has started letting me know if websites I visit often have posted new content,
pretty awesome...


It was an Android app that lets users embed voice data in pictures (along with sharing pictures and doing speech recognition). I went to the play store, the app looked medium unprofessional so I visited their website. They were "seasoned" machine learning experts, but their mobile site had no indication of who they were.

"Seasoned entrepreneurs ... with deep experience in AI and machine learning"

So I said, I like Sphinx, let's see how they used it. I installed their app and giggled out loud at the "patent pending" splash screen. What are they patenting, certainly not Sphinx, certainly not speech recognition, certainly not the ability to take a picture on a smartphone and share it. It must be the embedded voice metadata, which I assume is why they aren't using the built in speech recognition which is far more accurate than Sphinx.
"Patent Pending" technology by seasoned technologists...
It even comes with a splash screen :)
I sent them a few bug reports, it looks as though their app was built maybe a year ago, or at least by someone with limited Android experience and wasn't tested using the camera on Android 4.3...

"Unfortunately Smile has stopped..."