[FoRK] The Coming Future of System Administration

Eugen Leitl eugen at leitl.org
Wed Mar 7 02:56:59 PST 2012

(probably a no-brainer for most people here, just a nice writeup)


The Coming Future of System Administration

Date March 6, 2012

Four years ago, I wrote a blog entry titled The Future of System
Administration. It started out with “Here’s a hint…it’s not sitting at a
shell prompt typing out commands.”

In it, I discussed how system administration would become development, much
more like coding than like swapping disks…

    The future is already here — it’s just not very evenly distributed.

    –William Gibson 

Basically, yes, it has come to pass. No, not everyone is using that
particular strategy, but the number of people who aren’t will continue to get
fewer and fewer, especially as cloud-based offerings get more and more
prevalent, accepted, and expected. But all of this is just tangential to what
I really want to talk about.

System Administration has to divide in order to continue to grow. Tom
Limoncelli has been saying it for a few years, but I did’t really get it
until relatively recently. I don’t know if we agree about the underlying
reasons, but I suspect we’re on the same page. Let me lay the ground work,
and then I think you’ll also agree that it’s pretty much inevitable.

System Administration is a relatively recent field compared to pretty much
every other professional endeavor. Medicine has been around in an organized
form for over 100 years. Accounting for 400-500 years, and
other..um..professions…for considerably longer. (Note: this is not to say
that the art and science of these are not much, much older, but I am speaking
here of organized professions).

The profession of System Administration is still coalescing, but that’s only
because we’re still very much in the formative years of the art and science.
It’s exciting, because we get to experience the transition from humors and
leeches into germ theory and beyond.

As we’ve progressed, something has become clear; a single person is unable to
handle all aspects of the job. We’ve all seen this, of course. There are
administrators specific to networks, storage, virtualization, applications,
and more. This division has been caused by the domain-specific knowledge
required to adequately perform a certain aspect of running IT. Not everyone
can know everything…

It is this last point that is going to cause the division that I speak of.
Currently, the field is very roughly divided into junior and senior
sysadmins. The Job Descriptions for System Administrators short booklet by
USENIX is a good effort to add some qualification to the distinctions between
the various job levels, but it still supposes a continual advancement from
the very entry level administrator up through senior, level IV System
Administrator. I don’t think this can continue for much longer, particularly
in an infrastructure of any complexity.

As time progresses, here’s what I think will need to happen (and in some
cases, it’s already happening, but it’s going to become more widespread).

We’re going to start seeing two career paths for people who work with
servers. At the top of the professional career paths, there will be the
people who are in charge of physical infrastructures, and there will be
people who are in charge of the operational infrastructures. The inevitable
move to cloud-like infrastructures is going to mean that, to the operational
architect, the physical hardware is almost completely irrelevant. At the same
time, the physical administrator is going to be concerned with those aspects
of running the infrastructure, such as physical Moves, Adds, and Changes
(MACs) as hardware becomes more numerous and anonymous.

Almost no one is the size of Google, so when machines die, they will need to
be replaced. They will need to be monitored, and their performance,
independent of the software installed on them, will need to be monitored for
efficiency, with underperforming equipment being fixed. The cloud layer of
abstraction will allow machines to be pulled in and out of service to be
repaired and replaced, but certainly a degree of expertise will required to
engineer the best, most efficient datacenter solutions. This is what the top
tier physical infrastructure will concern themselves with.

The other side of the coin is the operational administrator, who concerns
themselves primarily with ensuring the business operations and continuity. If
the physical infrastructure admin’s concerns are hardware, the operational
administrator’s concerns are data and software. The operational admin won’t
ever have to consider what disk array to purchase, or how large the storage
needs to be, but will instead concern themselves with things like the
performance growth rate and the number of CPU units required for a particular
task. Infrastructures will be developed like software is now, using the same
methodologies, sharing some of the tools, if not the languages. Someone in a
senior operational role will need business intelligence, as well, because
they will be responsible for translating business directives into

These specialized roles will have junior members, of course. An entry level
physical infrastructure admin will probably enter the field in the same way
that many of us did, as an apprentice, then moving to a journeyman type of
level while learning more. I imagine that more education will be necessary to
fill the truly senior physical roles, but it’s not inconceivable to learn
everything on the job.

In the operational role, it’s harder to imagine many people making the same
transition from non-admin to admin. My own path, from Tech Support to
SysAdmin, won’t be nearly as common. Many current admins were able to make
the transition between non-admin and admin because the operating environments
are similar on their laptops and their production infrastructure. Once the
operating system ceases to be important, this easy path will disappear, and
the only people who make that transition are those who actively work at it.

Because the operational aspects have such an emphasis on programming, it’s
easy to see that more administrators will become sysadmins directly out of CS
programs at universities. Hopefully the people who dictate the curriculum
will take notice of this and start adding operational electives.

I don’t have a time frame for this, but I suspect that in another four years,
we’ll see more and more companies coming online with this model. Over the
past couple of years, I’ve seen a lot of blog entries and online articles
talking about the disappearance of System Administrators. These articles are
mostly hyperbole. The sysadmin of today is not going to disappear, but they
are going to be fewer in number.

Companies who, right now, have their own physical infrastructure are not
going to go through a lot of expense to get rid of it, but companies who come
online in the future are going to see more and more reason to use a public
cloud service, which means that they’ll only be employing the operational
administrators. Sure, there will still be desktop support in most cases, but
in terms of sysadmins, for those companies, it’s mostly going to be
architects. Once a company grows to the point where it’s not economical to
host at Amazon, they may build their own cloud infrastructure, in which case
they’ll employ the physical infrastructure admins, too.

Just so you know, this isn’t a dystopian future. Specialization is a good
thing. It’s going to require us all to work together, but the end result will
be that we can accomplish much more than we could before. The knowledge level
required to do our jobs is going to go up, but we should see this as a
challenge to learn more and to be better. 

More information about the FoRK mailing list