Erik L. Arneson — Writer and Software Developer

Process Automation for an Oregon Charitable Foundation

2026-05-22T00:00:00+00:00

The Oregon Scottish Rite Education Foundation awards scholarships annually to students across Oregon who are entering into college or trade schools. Over the past few years, their foundation has grown, requiring them to distribute more scholarships than ever. However, since the foundation board is small and composed entirely of volunteers, they were looking for some way to automate their application process to reduce the number of manual steps required.

Initial Approach

Members of the board approached me with a wish list of features and requirements. They wanted a system to integrate seamlessly with their website, while incoming applications should be automatically checked, sorted, and categorized by county. They already had a spreadsheet system for tracking applicants as well as a cloud storage system for storing supporting documents; they wanted the automation system to integrate with these existing components.

Since the board was already utilizing Google Business products for cloud documents and storage, I decided that the best approach would be to lean into Google Cloud services. Following their list of requirements, I implemented a Google Cloud Run service in Go that processed incoming applications from their website, then updated the spreadsheet while preparing their cloud storage system for each application’s supporting documents. The spreadsheet now contained all the application data, including a link to an automatically generated PDF version of the application and a link to the cloud storage location.

This initial approach cut out an enormous amount of manual work. Previously, applications were submitted via the website and emailed to a board member, who then manually sorted them by county and generated PDFs for storage and later distribution.

Implementation Challenges

The application submission window for the Oregon Scottish Rite Education Foundation runs from December through March of each year, which gave us a limited amount of time to observe incoming live data and how the automation was performing.

My initial approach worked fairly well, and handled 94% of incoming applications properly. However, the last 6% ran into various problems that prevented correct processing. One error arose from a misspelling in a county name, which was caught early and corrected. Another series of errors arose from processing errors with the Google Sheets API, which has both rate limiting and more mysterious, inexplicable errors. I added additional error correction and error checking, but there was still some additional manual work to enter data from a handful of applications.

By the end of the application period, we had managed to reduce the amount of manual data entry and processing by a large percentage, maybe as much as 60%. After discussing the current process automation with the board, we began to see opportunities for further automation and improvement.

Additional Development

After the first year’s application submission window, I began working with one of the board members to plan out how the process could be improved even more. My list of improvements were primarily on the backend, and involved robuster error detection and reporting, as well as a backup data store that would allow the automation to recover from the Google Sheets API errors more elegantly.

The board proposed an additional feature set that would further reduce manual data processing, by allowing applicants to upload supporting documents such as letters of recommendation and high school transcripts. These uploaded documents would then be processed and stored in the proper cloud storage location, thus reducing the amount of manual file management required by volunteers.

Implementing these two new features went very smoothly. It required the development of a new frontend, using Typescript and Svelte. This frontend allowed applicants to upload their supporting documents via a unique link given to each of them by automated email. It used both Google Cloud Storage and the Google Drive API to store these files.

The improvements on the backend used Google Cloud Datastore and a more robust error recovery system. It now backed up all incoming scholarship applications until they could reliably and more slowly be added to the spreadsheet with the Google Sheets API, thus nearly eliminating all rate limiting errors. In addition, progress reports and error output could now be viewed using a new frontend administrative interface.

Results

This round of improvements produced significant results. Over the entire span of the second year of application submissions, error rates dropped from 6% to just one unrecoverable error. While I had been hoping for improvement, even I was surprised by how much of an improvement we achieved!

The board reported that the amount of manual processing was reduced by another 80-90%, with only a few applicants resorting to email for sending in their supporting documents. While we have discussed further improvements to the process automation for the next application submission window, the board is satisfied with the improvements provided by the level of automation implemented so far.

Lessons

This project benefited a lot by reviewing requirements and expectations frequently with the client. They were great at providing me with very clear goals from the beginning, and as we progressed and it became clear where the most volunteer time was being spent, we were able to identify more tasks for automation.

On the development side of things, three things were clear. First, cloud services really make the development and deployment of process automation tools like this a breeze. Costs can be kept low while still providing robust infrastructure for hosting.

Second, I learned once more the importance of unit tests. I wrote unit tests for most elements of the automation software, which allowed me to catch breaking changes and test features very quickly. Test-driven development can be a great way to run a project, and though I did not follow all of its parameters strictly, that particular philosophy did guide my decision.

Third, once more I was reminded of how the best-planned software runs into difficulties when it collides with the real world. I failed to anticipate the Google Sheets API issues and did not implement enough comprehensive error correction early on. That is not an error I shall repeat again!

Finally, I learned that even with a project that seems as straightforward as this one, it is important to control expectations early. This turned into a multiyear project because we saw room for improvement. I think it is important for a business or organization looking for process automation to understand that it can be an ongoing series of improvements; not everything will be accomplished or perfected after one pass.

Overall, I found this to be a rewarding project. I enjoyed working on software that was aiding a good cause: helping high school graduates fund their education. It was also very nice to work for a charitable foundation instead of a for-profit business; it made all of my programming feel a lot better. I hope to work for more charitable foundations in the future.

Some Great Fish Shell Plugins

2022-09-23T17:44:00+00:00

As a long-time Linux user, I am pretty comfortable with command line interfaces. However, as I started learning more about automation and how important it is to get your computer to do more work for you, I leaned toward wanting my command line shell to do more. Eventually I switched to fish shell, a very user-friendly shell with excellent scripting capabilities that is far more readable and less obscure than bash.

Plugins for fish

There have been a few plugin managers over the years, but the one that seems to be the best maintained and most usable is fisher, created by Jorge Bucaran. It allows you to easily install plugins straight from GitHub repositories, while updates can be executed with a simple fisher update on the command line.

z

One of the greatest command-line time savers ever is the z command. Originally a bash script, it has been ported to fish and, frankly, I love it.

What does it do? z provides shorthand for visiting directories based on a combination of frequency and recency—or "frecency".

If you frequently visit the /opt/calibre directory, a z calib command will allow you to jump straight there.

If you already have fisher installed, just use fisher install jethrokuan/z and start playing around with z.

You can read more about z here.

pisces

Your IDE has been handling your parenthesis for ages. Now your shell can do the same! The pisces plugin has automatic matching symbol management for parenthesis, braces, quotes, and others.

Install it with fisher using the command fisher install laughedelic/pisces and be matching parens in seconds!

plugin-emacs

Automation can help you interact with your editor from your shell. The Emacs plugin for fish was originally written for an unmaintained plugin manager called Oh My Fish. However, I wanted the plugin to work with fisher, so I made my own fork.

How does it work? This plugin adds easy commands like ef and ed that allow you to open files and directories in Emacs quickly.

Install it quickly using the command fisher install pymander/plugin-emacs and read more about it here.

vfish

Emacs users have probably heard of vterm, which is the Emacs package that interacts with libvterm. It is a powerful terminal emulator that runs in an Emacs buffer. I wrote a fish plugin called vfish that simplifies using fish in vterm.

How does it work? Inside a vterm session, this plugin adds fish commands like vf and vd to open files and directories quickly. It mirrors the Emacs plugin.

Install it using the command fisher install pymander/vfish and read more about it here. Note that it will require some code to be added to your Emacs startup file.

apt

For Debian and Ubuntu users, the Oh My Fish apt plugin works seamlessly with fisher.

What does it do? The apt plugin adds a wrapper around the apt and apt-get commands that simplify package management from the command line.

Install this plugin quickly with fisher install oh-my-fish/plugin-apt and then read more about it.

What's your favorite fisher plugin?

I love finding plugins that can help me automate more of my everyday tasks and workflows. What's your favorite fisher plugin?

Leave a comment below and let me know! Be sure to include a link.