Process Automation for an Oregon Charitable Foundation

2026-05-22T00:00:00+00:00

The Oregon Scottish Rite Education Foundation awards scholarships annually to students across Oregon who are entering into college or trade schools. Over the past few years, their foundation has grown, requiring them to distribute more scholarships than ever. However, since the foundation board is small and composed entirely of volunteers, they were looking for some way to automate their application process to reduce the number of manual steps required.

Initial Approach

Members of the board approached me with a wish list of features and requirements. They wanted a system to integrate seamlessly with their website, while incoming applications should be automatically checked, sorted, and categorized by county. They already had a spreadsheet system for tracking applicants as well as a cloud storage system for storing supporting documents; they wanted the automation system to integrate with these existing components.

Since the board was already utilizing Google Business products for cloud documents and storage, I decided that the best approach would be to lean into Google Cloud services. Following their list of requirements, I implemented a Google Cloud Run service in Go that processed incoming applications from their website, then updated the spreadsheet while preparing their cloud storage system for each application’s supporting documents. The spreadsheet now contained all the application data, including a link to an automatically generated PDF version of the application and a link to the cloud storage location.

This initial approach cut out an enormous amount of manual work. Previously, applications were submitted via the website and emailed to a board member, who then manually sorted them by county and generated PDFs for storage and later distribution.

Implementation Challenges

The application submission window for the Oregon Scottish Rite Education Foundation runs from December through March of each year, which gave us a limited amount of time to observe incoming live data and how the automation was performing.

My initial approach worked fairly well, and handled 94% of incoming applications properly. However, the last 6% ran into various problems that prevented correct processing. One error arose from a misspelling in a county name, which was caught early and corrected. Another series of errors arose from processing errors with the Google Sheets API, which has both rate limiting and more mysterious, inexplicable errors. I added additional error correction and error checking, but there was still some additional manual work to enter data from a handful of applications.

By the end of the application period, we had managed to reduce the amount of manual data entry and processing by a large percentage, maybe as much as 60%. After discussing the current process automation with the board, we began to see opportunities for further automation and improvement.

Additional Development

After the first year’s application submission window, I began working with one of the board members to plan out how the process could be improved even more. My list of improvements were primarily on the backend, and involved robuster error detection and reporting, as well as a backup data store that would allow the automation to recover from the Google Sheets API errors more elegantly.

The board proposed an additional feature set that would further reduce manual data processing, by allowing applicants to upload supporting documents such as letters of recommendation and high school transcripts. These uploaded documents would then be processed and stored in the proper cloud storage location, thus reducing the amount of manual file management required by volunteers.

Implementing these two new features went very smoothly. It required the development of a new frontend, using Typescript and Svelte. This frontend allowed applicants to upload their supporting documents via a unique link given to each of them by automated email. It used both Google Cloud Storage and the Google Drive API to store these files.

The improvements on the backend used Google Cloud Datastore and a more robust error recovery system. It now backed up all incoming scholarship applications until they could reliably and more slowly be added to the spreadsheet with the Google Sheets API, thus nearly eliminating all rate limiting errors. In addition, progress reports and error output could now be viewed using a new frontend administrative interface.

Results

This round of improvements produced significant results. Over the entire span of the second year of application submissions, error rates dropped from 6% to just one unrecoverable error. While I had been hoping for improvement, even I was surprised by how much of an improvement we achieved!

The board reported that the amount of manual processing was reduced by another 80-90%, with only a few applicants resorting to email for sending in their supporting documents. While we have discussed further improvements to the process automation for the next application submission window, the board is satisfied with the improvements provided by the level of automation implemented so far.

Lessons

This project benefited a lot by reviewing requirements and expectations frequently with the client. They were great at providing me with very clear goals from the beginning, and as we progressed and it became clear where the most volunteer time was being spent, we were able to identify more tasks for automation.

On the development side of things, three things were clear. First, cloud services really make the development and deployment of process automation tools like this a breeze. Costs can be kept low while still providing robust infrastructure for hosting.

Second, I learned once more the importance of unit tests. I wrote unit tests for most elements of the automation software, which allowed me to catch breaking changes and test features very quickly. Test-driven development can be a great way to run a project, and though I did not follow all of its parameters strictly, that particular philosophy did guide my decision.

Third, once more I was reminded of how the best-planned software runs into difficulties when it collides with the real world. I failed to anticipate the Google Sheets API issues and did not implement enough comprehensive error correction early on. That is not an error I shall repeat again!

Finally, I learned that even with a project that seems as straightforward as this one, it is important to control expectations early. This turned into a multiyear project because we saw room for improvement. I think it is important for a business or organization looking for process automation to understand that it can be an ongoing series of improvements; not everything will be accomplished or perfected after one pass.

Overall, I found this to be a rewarding project. I enjoyed working on software that was aiding a good cause: helping high school graduates fund their education. It was also very nice to work for a charitable foundation instead of a for-profit business; it made all of my programming feel a lot better. I hope to work for more charitable foundations in the future.

Create a Document From Twitter with TweetBook

2015-04-01T17:48:56+00:00

I just released TweetBook, a web application to transform a Twitter stream into a simple document that can easily be turned into a photo album.

The Problem

A client came to me complaining that there was no available solution to export a Twitter feed as a document filled with images. There are various other Twitter-to-document converters out there, but they all focus on the text. The client asked me to build a tool to fill this void.

The Solution

Since I’ve been working with Google App Engine a lot lately, I used Go to write a tool that retrieves Tweets from a search or timeline. It provides a number of formatting options, and then creates a simple document suitable for exporting into an HTML or PDF file.

Try It Out!

As of 2022, this application is seven years old and probably doesn’t work anymore.

The client told me to do whatever I wanted with the code, so I’m making it available for use! Click here to try TweetBook! If you find it useful, please consider funding further development by contributing a small amount with this button:

Do You Have a Suggestion? A Bug Report?

If you have additional ideas for TweetBook, please visit my Contact page. I’m eager to hear your feedback and get suggestions and bug reports!

Prime Number Service on Google App Engine

2015-02-20T22:37:41+00:00

As I mentioned earlier this week, I’ve decided to learn the Go programming language. I’ve also been very interested in Google App Engine, which lets you deploy applications to the cloud from a development sandbox. It’s like magic for web and mobile applications!

Since I’m so fond of prime numbers, I thought I’d build a web service for calculating them. It does some other fun stuff, too, like figuring out if a number is happy. It also caches primes in the Google Cloud Datastore with some minimal statistics. If you’d like to see the source code for the prime number service, it’s available on GitHub.

The next thing I did was write the simplest of all WordPress plugins to query the service and return the latest prime number in the sequence. This all happens in a post using the [[nextprime]] shortcode. ((Including this information has the side-effect of revealing how much traffic this page gets.))

[nextprime]

You can also visit the app itself and read about querying the service to get prime numbers for your own web page!

Learning Go and writing for the Google App Engine has been really fun. I’m ready for a new challenge! If you have a web application in mind, contact me and let’s figure out how to build it!

Erik L. Arneson — Writer and Software Developer