Malleable Computing, Emacs, and You

It all started with a routine task. I use GitHub issues for all of my public-facing projects but my preference is keeping track of things in Org Agenda. To reconcile the two, I would manually copy over the GitHub issue to an Org file, typically the title and description, like an animal. Despite this separation, the duplicated issue allowed me to treat it as a scratchpad for anything I could express in Org. In this way, I used the duplicated issue in Org as both a dedicated area to take notes and a “staging” area to compose follow-up comments that I’d want to share in the public facing issue. I manually copied for far longer than I care to admit. Repeat a task enough times in Emacs and the inevitable thought arises: “I should automate this.” This post recounts how I automated this task and in doing so, highlight the malleable computing capabilities of Emacs. It should also be considered a follow-up to my earlier post “In Emacs, Everything Looks Like a Service”. Requirements An essential question to ask in any automation exercise is “What do I want done?” I wanted to be able to: Easily copy a GitHub issue (title, description, some metadata) as an Org task that can be tracked in an Agenda view. Work primarily from Emacs to minimize the context switching between it and a web browser. Express my thoughts in Org syntax. Create a new GitHub issue. Avoid dealing with GitHub authentication. From Emacs, open a GitHub issue in a web browser. Another essential question to ask is “What do I not want to do?” Install or write a full-featured GitHub client. Worry too much about synchronization logic between local (Emacs) and server (GitHub) state. Spend a lot of time on this (ideally have something working in a day, no more than a week). Specification With the above requirements in place, the next question is “how do I build this?” For this particular exercise, I decided to leverage my existing install of the GitHub command line utility gh. The benefits for this are: GitHub authentication is delegated to gh; no need to mess with it directly from Emacs. Emacs can treat gh as a REST service to GitHub as shown in the diagram below To flesh out the rest of our tool, we can leverage using different Elisp packages and programs: For the user interface, the Transient and Variable Pitch Table (vtable) packages are used for menus and display respectively. For Org to Markdown translation, ox-gfm will be used. For Markdown to Org translation, Pandoc will be used. Elisp native JSON support will be used for deserializing the JSON responses from gh. Implementation The implementation of the above is published as the package fj with its source available for examination in the file fj.el. Of note is the function fj-request-issues which does the work of retrieving GitHub issues via gh as shown below. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16(defun fj-request-issues (repo) "Request issues for REPO." (let* ((fields fj-browser-fields) (cmd-list (list "gh" "--repo" (format "'%s'" repo) "issue" "list" "--limit" (number-to-string fj-request-issue-count) "--json" (string-join fields ",")))) (json-parse-string (shell-command-to-string (string-join cmd-list " ")):null-object nil))) Consider the high degree of abstraction provided by fj-request-issues: The list cmd-list forms the request (in this case, the arguments to invoke gh with). shell-command-to-string dispatches the request to gh. The returned JSON response is handled by json-parse-string to deserialize the JSON into an Elisp hash table. All of the above is accomplished in less than 20 lines of code. The returned hash table result is subsequently processed to populate a vtable as shown below. From the vtable, the user can navigate the list of issues with an ancillary window updated to show the details of a selected issue. Multiple commands and functions working with said hash-table were made to satisfy the requirements above. They are accessible via the Transient menu shown below: Malleable Computing Observations As Elisp is a dynamic programming language, the above function (or some variant of it) can be coded and evaluated within a running Emacs session. In Emacs it is routine practice to prototype code behavior without the need to restart it. Contrast this with tools that are built with a static language and have no extensibility, where the edit-compile-debug development cycle must be applied to exercise behavior, provided the source code is available. Emacs provides numerous ways to edit and evaluate Elisp code, among them: A scratch buffer An Elisp file An Org source block IELM REPL Eshell eval-expression (M-:) Because there is no isolation between loaded Elisp code, all of it can be orchestrated together in an improvised fashion. Any program accessible to Emacs via shell further adds to this mix. Provided high abstractions, the amount of code needed to be written for desired behavior can be small. At the time of this writing, fj.el has approximately 400 lines of code has been measured by cloc: 1 2 3 4 5 6github.com/AlDanial/cloc v 2.08 T=0.01 s (146.4 files/s, 76550.1 lines/s) ------------------------------------------------------------------------------- Language files blank comment code ------------------------------------------------------------------------------- Lisp 1 93 38 392 ------------------------------------------------------------------------------- It took me about 2½ hours to build the basic behavior I wanted (request issues from GitHub, then display) and the rest of the day to cover all the original requirements. Everything after that was just refactoring. Motivated readers are invited to examine fj.el to understand its details. At this point though, I’d like to take the opportunity to muse on software engineering and malleable computing. Software Scope - Some Percentage Anecdotes The 90/90 rule posits that “The first 90 percent of the code accounts for the first 90 percent of the development time. The remaining 10 percent of the code accounts for the other 90 percent of the development time”. Closely related is the Pareto principle (aka the 80/20 rule) as applied (or more mis-applied) to software development: only 20% of your features will be actually used by 80% of your users. From control theory there is the concept of bounded input, bounded output (BIBO) stability. If the system is BIBO stable, then whatever input that is bounded will have an output that is also bounded. Coupling these ideas together, if the functionality you want (BIBO stability) is within that 20% deliverable then you are positioned to get a desired result faster. Unfortunately, this observation is not available for many producers shipping a tool that must satisfy a large audience. In Sinofsky’s post “What is Software Bloat, Really?”, he describes the product definition issues with shipping Microsoft Office, particularly with feature set. From their user research, they were led to this finding: “… the data was entirely conclusive: Most of Office was used. But no one person used the entire product.” When discussing product definition, it is helpful to distinguish two informal dynamics: supply and demand. With software that is only supply-provided (such as the case with MS Office), the weight of product definition is shouldered by the producer. If the producer wants to serve a vast audience, it is likely their feature set and corresponding development scope will be vast as well. On the demand-side, consumers can make requests for new features, but their prioritization is controlled by the producer. In this situation, the roles of producer and consumer are strongly delineated. Malleable software, where users have agency to adapt and reshape their digital tools, offers a different possibility: producers provide the building blocks and let consumers make their own tools. In this, both existing code and programs are recombined to make new behavior. Stated another way, malleable software takes advantage of combinatorial explosion in that a desired subset of behavior can likely be found within the state space of combining existing libraries (in this case Elisp) with different programs. With malleable software, the roles between producer and consumer are less delineated. A consumer building a new tool out of malleable building blocks now must shoulder the weight of product definition. Build for 1 or N The scope to build for 1 vs N can be an order of magnitude with the kicker that N need not be large. Choosing to write code for another person (N=2) escalates concerns that could otherwise be brushed aside: Error handling Code maintainability Modularity/Reuse Documentation Unit and Integration Testing Packaging Distribution Both a blessing and curse of malleable techniques is that they allow for “just good enough” capability. Malleable techniques encourage building for 1 as reaching a state of “it works for me” is often sufficient to declare victory. For most software that is only supply-provided, building for N is a requirement. For most software built with malleable techniques, building for N is a choice. Malleable Software and User Agency Returning back to fj, the benefits of building for 1 are evident: From within Emacs I can now easily: Peruse GitHub issues for a specified repository. Copy a GitHub issue into a Org file. Create a new GitHub issue using Org syntax. Open a GitHub issue in a web browser. Implementing fj in Emacs was straightforward as I could leverage Elisp packages (both built-in and 3rd party) and external applications (gh, pandoc) to build it. Within a day I had a tool that did what I wanted. I did not have to ask for permission nor required privileged materials (source code) to build fj. With Emacs, I could just do it. Such individual empowerment advanced by malleable techniques is liberating in contrast to working with siloed applications with little to no integration between them. Closing Thoughts This post muses about the malleable computing capabilities provided by Emacs, showing by concrete example (fj.el) how new behavior can be created leveraging both code and program reuse in an improvised fashion. With measured expectations (requirements, feature set, audience), malleable techniques allow for building tools that would otherwise be infeasible without it. Links https://github.com/kickingvegas/fj “What Is Software Bloat, Really?”, Steven Sinofsky. emacs org mode