The Slow March of Progress in Programming Language Tooling

We’re Earthly. We help you build software faster and simpler using containerization. While talking about tooling, Earthly is a great complement that can optimize your dev experience. Check it out.

The 2022 Stack Overflow developer survey is out!

And what’s fascinating to me is which popular programming languages are either loved or dreaded.¹

Last time the developer survey came out, I wrote a post about Green VS Brown languages. In it, I said that when people like a new programming language, they judge it from a biased perspective because new things get used for new development, whereas the older things get used in older projects, which are less exciting. But that misses something big: Language tooling is getting better².

Raising the Stakes - Go vs Rust

The internet is full of fights about Go vs. Rust. But I think the fascinating thing is how similar their developer tooling experience is. They both have a modern, batteries-included take on development that is very different from many of the languages in the DREAD list. You don’t need to wonder what the best tool to use for testing, fuzzing, packaging, or linting is. All of these things are standardized and included. And I think this is one reason they are both at the top of the Loved list.

That is, a big difference between working in Go or Rust vs. some of the languages on the dreaded list has nothing to do with the specifics of the language syntax. Instead, the difference is the tooling and supporting ecosystem.

Tooling Is the New Syntax

My thesis is that the tooling and developer experience for programming languages is improving over time, but mainly in new languages. It goes like this: Tooling innovation happens, new languages adopt and standardize on it, and end up incrementally better than existing languages. If you add up enough of these increments, the older languages, which may have pioneered some of these innovations, seem painful and antiquated.

It will make more sense once I give some examples. So here is a partial list of programming language innovations that aren’t the language’s syntax or semantics.

Batteries-Included Standard Library

I’m not sure whether it’s a blessing or a curse, but expansive standard libraries [are a big improvement]. It’s possible to get a lot done with PHP, Python, and Go without needing to install any third-party library. They come with json, http client and server, and even database access for the most part.

Amir Saeid

Standard Library History

A standard library is a library of common things that comes with the language. C has libc, and C++ has libcpp, but both feel very minimal compared to the common conception of a batteries-included standard library.

The history is a little unclear to me, but it seems like Python (1991) was the first language to take the stance of really having an extensive standard library. Java 1.0 (1996) also came with an extensive standard library (The Java Class Library), and many other languages would follow suit.

It’s hard to overstate the benefits of getting simple things done without implementing foundational stuff yourself or having to reach for third-party dependencies.

Standard Library State of the Art: GoLang

Most modern languages (that aren’t JavaScript) now ship with extensive standard libraries. In particular, though, Go puts special emphasis on its standard library. It promises backward compatibility, performance, and well-thought-out implementations. Because of this, Go developers lean on the standard library more than many other communities and generally hold it in high regard.

Third-Party Package Repositories

Third Party Packages: History

Around the time more extensive standard libraries became a thing, the world-wide-web was also taking off, and the internet proved to be pretty good at fostering collaboration.

Eventually, you outstrip even the most inclusive of standard libraries and have to build something yourself. Or do you? Perl popularized the idea of a global collection of packages with CPAN; since then, nothing has been the same. I think it’s fair to say that people who used and contributed to CPAN knew it was a game-changer.

CPAN was launched in 1995 (based on CTAN), and by its height in 2003, it had established a new way for people to get things done with software. That is gluing together third-party components. So much of modern development now follows this model.

It’s hard to find a commonly used programming language created after 2003 that didn’t come with a third-party package repository of some sort. CPAN moved the line so that from then forward, a “real” programming language needed to have a strategy for third-party package management.

So after CPAN, a great standard library that does most of the things you need to do, augmented with a third-party package repository that is simple to use and straight forward to contribute to became the table stakes for a language.

Documentation Support

Documentation: History

Once you have third-party packages, you need an easy way to document them. Javadoc and it’s generated documentation was the first version of this I encountered. It made it much easier to find what I was looking for in the Java Class Library because I could just click around in the Javadocs on the web. Then you can combine Javadoc with IDE integration and it becomes simple to work with code you’ve never seen before. Exploratory coding becomes possible.

Documentation: State of the Art

Java with Javadocs is no longer the state of the art. Go has godoc, Julia has Documenter.jl and even hackage has pretty good package docs. But the state of the art seems to be Rust with docs.rs.

Write Once, Run Most Places ™

[One improvement I’ve seen has been] J2EE and the standardization of web servers have unlocked the computing foundation on which our civilization depends today. Java and the JVM pioneered that but I don’t think they are being adequately credited for it. Once Java became a thing, the platform you develop on stopped being relevant with regards to the platform you deploy on. It’s obvious knowledge today, but 20 years ago, this was revolutionary.

Cédric Beust

Run Anywhere: History

Java and the JVM really pushed the line for cross platform development. No longer did your developer environment need to match your production environment. With the JVM, you could compile things down to JARs and run it anywhere that had a Java Virtual Machine installed.

Virtualization and then containerization have made this far more common, but the Java was the first major programming language to enable this type of run anywhere workflow.

Run Anywhere: State of the Art

There are of course downsides to the Java approach. One of which is the slow start-up time of JIT code and another is the limitation of not being able to easily call something that isn’t written in Java. GraalVM claims these problems have been overcome but the current trend seems to be ahead-of-time cross-compiling. Both Rust and Go make this fairly simple, assuming you don’t have any c or libc dependencies.

But the state of the art appears to be Zig, which can easily cross compile not just Zig programs but also anything that would normally be build with Clang or GCC.

Package Managers

Package Manager: History

Languages have compilers, and they have lots of flags that you call them with, but this quickly becomes a pain. So things like Make and Autotools came to exist. Now introduce a third-party package ecosystem, and things get even more complex. So you get stuff like Maven and pip. Then you have issues like multiple versions of the compiler or runtime and different programs require different versions of packages and in Python you end up with things like pipenv and virtualenv and something called conda that I don’t even understand.

All this complexity makes it hard for new users to get up to speed. So new languages try to simplify things by bringing all this management under one roof.

I would say that package management and LSP have been the two biggest game changers in my personal experience of programming languages.

Ganesh Sittampalam

Much like how the batteries-included standard library expanded the definition of a language, modern package managers have raised expectations substantially. The upside of this expansion is the ease of onboarding and improved developer experience. The downside is that packaging, vendoring, and build software is not free. The language authors must pour many engineering hours into these tools to solve these problems.

Package Manager: State of the Art

I think one main advantage that cargo has is that it came with the language. Retro active build tools often lack integration into the platform as a whole

Robert Masen

This package manager area is fast-moving. If you invest a lot of engineering hours, you can improve your language onboarding and day to day usage experience substantially. And so the stakes here keep growing. The cargo and rustup documentation for Rust are almost as extensive as the rust book, and that doesn’t include all the cargo plugins.

Can you easily switch compiler versions in your language? Can you quickly run tests? Can you easily do code coverage, and performance testing? Vendor code? Generate documentation? Lint code? Fix the code lints? All of these used to be stand-alone tools in various language ecosystems and now they come with Rust out of the box. And it’s a similar story for Go. I assume other newer languages will try to match or exceed this level of tooling.

Code Formatters

Case Study: gofmt

Code formatters existed before gofmt, just as third-party software packages existed before CPAN, but making something a standard for a community profoundly changes things. For example, no language that was created before Go is likely to achieve the near 100% style conformance that Go has because the existing languages have to deal with existing code, whereas gofmt enforces a single style and has no knobs to tweak. Languages that follow Go can learn this lesson. And so Rust (rustfmt) and Zig (zig fmt) have embraced a strong default code style and an accompanying code formatter, and have gained an edge in developer experience because of this.

So Much More

There are so many other things that I could list in this essay. Some things, like runtime improvements, seem close to direct language improvements. Others like IDEs, LSPs, fuzzing support, and refactoring tools seem closer to developer tools and could have gone on this list. But you have to stop somewhere.

(Tell me what I missed and/or got wrong, and I’ll do an update.)

Conclusion

The tooling that enables developers to get work done is a massive part of what makes a language useable. Tooling is also a changing landscape where standards keep rising.

It goes like this:

When a new developer tooling innovation is discovered, newer programming languages get a chance to bake that innovation into their language tooling. Doing so gives them an incremental advantage, and these increments add up over time to a better developer experience.

So newer languages have one clear, well thought out way to do something, and older languages will have either many contradictory ways, or no ways at all, to do the same thing. And this makes older languages feel old.