Erik L. Arneson — Writer and Software Developer

Process Automation for an Oregon Charitable Foundation

2026-05-22T00:00:00+00:00

The Oregon Scottish Rite Education Foundation awards scholarships annually to students across Oregon who are entering into college or trade schools. Over the past few years, their foundation has grown, requiring them to distribute more scholarships than ever. However, since the foundation board is small and composed entirely of volunteers, they were looking for some way to automate their application process to reduce the number of manual steps required.

Initial Approach

Members of the board approached me with a wish list of features and requirements. They wanted a system to integrate seamlessly with their website, while incoming applications should be automatically checked, sorted, and categorized by county. They already had a spreadsheet system for tracking applicants as well as a cloud storage system for storing supporting documents; they wanted the automation system to integrate with these existing components.

Since the board was already utilizing Google Business products for cloud documents and storage, I decided that the best approach would be to lean into Google Cloud services. Following their list of requirements, I implemented a Google Cloud Run service in Go that processed incoming applications from their website, then updated the spreadsheet while preparing their cloud storage system for each application’s supporting documents. The spreadsheet now contained all the application data, including a link to an automatically generated PDF version of the application and a link to the cloud storage location.

This initial approach cut out an enormous amount of manual work. Previously, applications were submitted via the website and emailed to a board member, who then manually sorted them by county and generated PDFs for storage and later distribution.

Implementation Challenges

The application submission window for the Oregon Scottish Rite Education Foundation runs from December through March of each year, which gave us a limited amount of time to observe incoming live data and how the automation was performing.

My initial approach worked fairly well, and handled 94% of incoming applications properly. However, the last 6% ran into various problems that prevented correct processing. One error arose from a misspelling in a county name, which was caught early and corrected. Another series of errors arose from processing errors with the Google Sheets API, which has both rate limiting and more mysterious, inexplicable errors. I added additional error correction and error checking, but there was still some additional manual work to enter data from a handful of applications.

By the end of the application period, we had managed to reduce the amount of manual data entry and processing by a large percentage, maybe as much as 60%. After discussing the current process automation with the board, we began to see opportunities for further automation and improvement.

Additional Development

After the first year’s application submission window, I began working with one of the board members to plan out how the process could be improved even more. My list of improvements were primarily on the backend, and involved robuster error detection and reporting, as well as a backup data store that would allow the automation to recover from the Google Sheets API errors more elegantly.

The board proposed an additional feature set that would further reduce manual data processing, by allowing applicants to upload supporting documents such as letters of recommendation and high school transcripts. These uploaded documents would then be processed and stored in the proper cloud storage location, thus reducing the amount of manual file management required by volunteers.

Implementing these two new features went very smoothly. It required the development of a new frontend, using Typescript and Svelte. This frontend allowed applicants to upload their supporting documents via a unique link given to each of them by automated email. It used both Google Cloud Storage and the Google Drive API to store these files.

The improvements on the backend used Google Cloud Datastore and a more robust error recovery system. It now backed up all incoming scholarship applications until they could reliably and more slowly be added to the spreadsheet with the Google Sheets API, thus nearly eliminating all rate limiting errors. In addition, progress reports and error output could now be viewed using a new frontend administrative interface.

Results

This round of improvements produced significant results. Over the entire span of the second year of application submissions, error rates dropped from 6% to just one unrecoverable error. While I had been hoping for improvement, even I was surprised by how much of an improvement we achieved!

The board reported that the amount of manual processing was reduced by another 80-90%, with only a few applicants resorting to email for sending in their supporting documents. While we have discussed further improvements to the process automation for the next application submission window, the board is satisfied with the improvements provided by the level of automation implemented so far.

Lessons

This project benefited a lot by reviewing requirements and expectations frequently with the client. They were great at providing me with very clear goals from the beginning, and as we progressed and it became clear where the most volunteer time was being spent, we were able to identify more tasks for automation.

On the development side of things, three things were clear. First, cloud services really make the development and deployment of process automation tools like this a breeze. Costs can be kept low while still providing robust infrastructure for hosting.

Second, I learned once more the importance of unit tests. I wrote unit tests for most elements of the automation software, which allowed me to catch breaking changes and test features very quickly. Test-driven development can be a great way to run a project, and though I did not follow all of its parameters strictly, that particular philosophy did guide my decision.

Third, once more I was reminded of how the best-planned software runs into difficulties when it collides with the real world. I failed to anticipate the Google Sheets API issues and did not implement enough comprehensive error correction early on. That is not an error I shall repeat again!

Finally, I learned that even with a project that seems as straightforward as this one, it is important to control expectations early. This turned into a multiyear project because we saw room for improvement. I think it is important for a business or organization looking for process automation to understand that it can be an ongoing series of improvements; not everything will be accomplished or perfected after one pass.

Overall, I found this to be a rewarding project. I enjoyed working on software that was aiding a good cause: helping high school graduates fund their education. It was also very nice to work for a charitable foundation instead of a for-profit business; it made all of my programming feel a lot better. I hope to work for more charitable foundations in the future.

Interviewed on "Prot Speaks"

2025-08-18T00:00:00+00:00

Well-known Emacs package creator Protesilaos Stavrou interviewed me for his video podcast series, “Prot Asks.” We talked about a wide variety of topics, from Emacs to podcasting to Portland to public transit. I thought it was a great time, and perhaps you will like it, too!

You can read and view the video on Prot’s website and blog, or you can go straight to YouTube, or you can watch it embedded here below.

Note that Prot is looking for people to talk to on this series! You can join in by signing up on his website.

P.S. In the video, I promised that I would do another Emacs Carnival blog post, and I will, I swear! It is in my org-mode to-do file.

maybe: A command-line tool that succeeds sometimes

2025-02-13T00:00:00+00:00

I just released version 1.0.0 of maybe, a command-line utility that succeeds some of the time. It is written in Go and is not very many lines of code, but I do hope that it can be useful for people.

Some History

I recently got a Korg Volca Drums drum synthesizer. It has a sequencer that allows you to program the probability of a particular instrument playing on a particular step, and I thought that was a really cool feature. This inspired me to think about cron jobs that happen only some of the time. Maybe the cron job would run, maybe it wouldn’t.

This inspired me to write maybe, a command-line utility that returns either success or failure, based on flags you pass it. It defaults to succeeding 50 percent of the time, but you can change that. This means you could make a cron job like this, which would run do-something.sh 40 percent of the time.

# At midnight every Sunday, do something sometimes.
0 0 * * 0     maybe -chance 40 && do-something.sh

The Future of maybe

I don’t think I will stop at version 1.0.0. I have already considered what may happen to maybe in the future. Here are some things that I would like to do:

Add cryptographically secure randomness. There is no reason to use Go’s default random library. Let us try to be as random as possible.
More granularity to the randomness. What if you want to run something 40.5 percent of the time?
Experiment with GitHub actions, such as building binaries for every operating system that maybe should support.

As you can see, the opportunities for improvement are many. And maybe (ha ha!) you will see improvements soon!

Perhaps this is a good time to visit maybe on GitHub and star the repo so others can find it?

Configuring Emacs 29.1 for Go Development

2023-08-01T00:00:00+00:00

Now that I have installed Emacs 29.1, I needed to get it set up for Go development for a project. I was interested in taking advantage of both the new Tree-Sitter integration, and the new Eglot language server client. However, they were mildly tricky to set up! Here is what I did.

Configuring Tree-Sitter for Go

If you follow the excellent How to Get Started with Tree-Sitter instructions from Mickey Peterson, you will have a great head-start on getting Tree-Sitter working for most of your favorite languages (and probably Java, too). However, those instructions didn’t cover everything I needed for Go. When I tried running M-x go-ts-mode, Emacs complained about a missing gomod module. Baffling!

I couldn’t find any information in the Emacs documentation about where to find this missing module. I looked around on the net and found Camden Cheek’s tree-sitter-go-mod, and added that to my list of recipes. My treesit-language-source-alist then looked like this:

(setq treesit-language-source-alist
 '((bash "https://github.com/tree-sitter/tree-sitter-bash")
   (cmake "https://github.com/uyha/tree-sitter-cmake")
   (css "https://github.com/tree-sitter/tree-sitter-css")
   (elisp "https://github.com/Wilfred/tree-sitter-elisp")
   (go "https://github.com/tree-sitter/tree-sitter-go")
   (gomod "https://github.com/camdencheek/tree-sitter-go-mod")
   (dockerfile "https://github.com/camdencheek/tree-sitter-dockerfile")
   (html "https://github.com/tree-sitter/tree-sitter-html")
   (javascript "https://github.com/tree-sitter/tree-sitter-javascript" "master" "src")
   (json "https://github.com/tree-sitter/tree-sitter-json")
   (make "https://github.com/alemuller/tree-sitter-make")
   (markdown "https://github.com/ikatyang/tree-sitter-markdown")
   (python "https://github.com/tree-sitter/tree-sitter-python")
   (toml "https://github.com/tree-sitter/tree-sitter-toml")
   (tsx "https://github.com/tree-sitter/tree-sitter-typescript" "master" "tsx/src")
   (typescript "https://github.com/tree-sitter/tree-sitter-typescript" "master" "typescript/src")
   (yaml "https://github.com/ikatyang/tree-sitter-yaml")))

Note that the package is named go-mod but go-ts-mode expects it to be named gomod. I wish this were documented somewhere! In any case, I was then able to use M-x treesit-install-language-grammar for both go and gomod. Finally, M-x go-ts-mode worked!

After going through this process, I found Robert Enzmann’s post about automatically using Tree-Sitter. He has created the treesit-auto package, now available on MELPA, that does most of this work for you. It is a much faster way of solving the gomod mystery, so give it a shot!

Configuring Eglot for Go

I’d been using lsp-mode for ages, but with Emacs 29.1 including Eglot, I decided to make the switch.

In my Go project, I ran M-x eglot and was immediately met with an error:

[eglot] Server reports (type=1): Error loading workspace folders (expected 1, got 0)
failed to load view for file:///path/to/my/project: err: go command required, not found: exec: "go": executable file not found in $PATH: stderr: 

I’ve got Go installed in /usr/local/go, and /usr/local/go/bin is definitely in my exec-path variable in Emacs. It looked like Eglot wasn’t propagating exec-path down to its subprocesses. How annoying! I did a quick search through the list of Eglot-related variables and the Eglot documentation and no solution seemed immediately forthcoming.

So I took the cheap way out and made a symlink. In my shell, I ran:

sudo ln -sf /usr/local/go/bin/go /usr/local/bin/go

It is a dumb trick, and I am sure there is a better way to solve it. Do you know of one? Please comment and let me know!

What Else?

My exploration has revealed that there’s a lot of work left to do in the Emacs Tree-Sitter world. There are plenty of languages major modes that don’t yet have a ts-mode equivalent, and plenty of others that still need a lot of work.

It’s too soon for me to say if this setup is preferrable to my previous configuration. But I am really looking forward to playing around with Eglot’s features and exploring the capabilities of Tree-Sitter.

Link Rodeo: Go Package Management and Boring Technology

2015-04-06T15:47:00+00:00

Here are a number of interesting topics for you to think about this week.

I’ve been learning the Go programming language recently, and in the process I’ve been having conversations about it with friends and colleagues. Go has a unique package management system that has already caused me a number of headaches. The recommended method for taking care of package dependencies is lacking, at best. Over at Nerdbucket, my buddy Nerdmaster has written a thoughtful piece about it.

While I’m talking about new technology, I’d also like to contradict myself by agreeing with Dan McKinley’s great piece, “Choose Boring Technology.” He argues that a project should be careful about adopting lots of new tools and technologies. It reminds me of a time recently when I was looking for a Node.js programmer, and one of the replies I got back was, “For us, Node.js is glue. Its ecosystem is still too young to support anything long-term. Libraries and packages move too fast to build a product that will need actual maintenance.” I’ve had the same feeling about many technologies I’ve wanted to try, such as Ocsigen for OCaml, which has a build system and API that is always several steps ahead of its documentation.

This post’s featured photo is courtesy of Flickr user MunicipioPinas.

Create a Document From Twitter with TweetBook

2015-04-01T17:48:56+00:00

I just released TweetBook, a web application to transform a Twitter stream into a simple document that can easily be turned into a photo album.

The Problem

A client came to me complaining that there was no available solution to export a Twitter feed as a document filled with images. There are various other Twitter-to-document converters out there, but they all focus on the text. The client asked me to build a tool to fill this void.

The Solution

Since I’ve been working with Google App Engine a lot lately, I used Go to write a tool that retrieves Tweets from a search or timeline. It provides a number of formatting options, and then creates a simple document suitable for exporting into an HTML or PDF file.

Try It Out!

As of 2022, this application is seven years old and probably doesn’t work anymore.

The client told me to do whatever I wanted with the code, so I’m making it available for use! Click here to try TweetBook! If you find it useful, please consider funding further development by contributing a small amount with this button:

Do You Have a Suggestion? A Bug Report?

If you have additional ideas for TweetBook, please visit my Contact page. I’m eager to hear your feedback and get suggestions and bug reports!

Prime Number Service on Google App Engine

2015-02-20T22:37:41+00:00

As I mentioned earlier this week, I’ve decided to learn the Go programming language. I’ve also been very interested in Google App Engine, which lets you deploy applications to the cloud from a development sandbox. It’s like magic for web and mobile applications!

Since I’m so fond of prime numbers, I thought I’d build a web service for calculating them. It does some other fun stuff, too, like figuring out if a number is happy. It also caches primes in the Google Cloud Datastore with some minimal statistics. If you’d like to see the source code for the prime number service, it’s available on GitHub.

The next thing I did was write the simplest of all WordPress plugins to query the service and return the latest prime number in the sequence. This all happens in a post using the [[nextprime]] shortcode. ((Including this information has the side-effect of revealing how much traffic this page gets.))

[nextprime]

You can also visit the app itself and read about querying the service to get prime numbers for your own web page!

Learning Go and writing for the Google App Engine has been really fun. I’m ready for a new challenge! If you have a web application in mind, contact me and let’s figure out how to build it!

Random Link Rodeo

2015-02-17T16:43:32+00:00

I’ve been working on a few projects, but nothing I can share publicly. However, I’ve also been out in the world of the Portland tech community, which has been excellent. If you haven’t heard of Calagator yet, go check it out.

Last week, I went to a meet-and-greet for developer evangelists from Google Cloud Platform. After talking to one of the Google devs, I was convinced to give Go a try. He told me that Go has type inference, and while I was dismayed to learn that it only has type inference on variable declarations, it still looks like a good language. I’m playing around with sawsij and Google App Engine and should have something to share in a few weeks.

Last night, I went to a PDX WordPress Meetup, where I saw Kronda Adair of Karvel Digital give a presentation called “5 Common Website Owner Mistakes and How to Fix Them.” After the talk, there was some interesting talk about website optimization and the “noisy neighbor” problem in cloud computing.

Next week, I’m looking forward to the PdxDevOps meeting, mostly because I’ve realized that this is what I do most of the time. When I build websites, I tend to handle everything from hosting and system administration through software development.

I’m really impressed by how full and rich the Portland tech community is. So far I’ve met a lot of great people in it, and I look forward to meeting more.