worried by wordexp(3) [c, posix]

The function wordexp(3) is a POSIX C standard library function which performs "word expansion like a POSIX shell". wordexp(3) combines the safety of elaborate string parsing in C with the efficiency and robustness of invoking the shell on arbitrary user input. Why does it even exist? And why shouldn't you use it?

1. usage

Probably the most legit use is by init-style programs executing a command line (e.g. finit); though, since many wordexp implementations invoke the shell anyway, these might as well exec sh -c 'exec ...' instead.

Applications typically use wordexp to expand tildes and globs (~/*.txt), and are oblivious to its excessive powers. Mostly, these uses are in places like configuration files the user directly controls, so any disasters as a consequence of wordexp can be considered the user's fault.

More severe are the cases where wordexp's input comes from an untrusted source1 or the program is in question is setuid2. Sometimes people use it when parsing files, even (e.g. this tinygltf issue ended up affecting blender).3

I continue to find code copied from Stack Overflow where it remains recommended, despite it being unsafe, and probably slow, too.

All this would just make wordexp seem like just another call like popen or system where it's obvious that you're opening Pandora's box, except wordexp has a flag, WRDE_NOCMD, intended to prevent the worst abuses of it. The existence of this flag is a mistake, because almost no libc actually tries to make it consistently safe. This flag may imply to people that wordexp is ever safe to use on untrusted input. However WRDE_NOCMD is effectively broken depending on the combination of shell and libc in use.

2. a central problem

Command substitution has two forms in shell: backtick-delimited (`command`) and dollar-parenthesized ($(command)). The former presents more problems for the user, but fewer for the author of the parser: simply scan ahead for a matching backtick, obeying other escaping and quoting.

The latter form is often where first-time shell writers fall into despair. It may be nested, and contain content-dependent unbalanced parentheses 4. The lexer must mutually-recursively invoke the parser just to find the end of the token.

The POSIX standard is ambiguous about a concerning detail of shell syntax: the difference between command and arithmetic substitution. Don't expend too much effort trying to understand this:

If the current character is an unquoted '$' or '`', the shell shall identify the start of any candidates for parameter expansion, command substitution, or arithmetic expansion from their introductory unquoted character sequences: '$' or "${", "$(" or '`', and "$((", respectively. The shell shall read sufficient input to determine the end of the unit to be expanded (as explained in the cited sections). While processing the characters, if instances of expansions or quoting are found nested within the substitution, the shell shall recursively process them in the manner specified for the construct that is found. The characters found from the beginning of the substitution to its end, allowing for any recursion necessary to recognize embedded constructs, shall be included unmodified in the result token, including any embedded or enclosing substitution operators or quotes. The token shall not be delimited by the end of the substitution.

POSIX Shell Command Language, 2.3 Token Recognition

Each shell and wordexp implementation have their own take on how to deal with ambiguous expressions like these:

$((echo a);(echo b))
$((echo "(");(echo ")"))
$((case a in *) echo b;; esac))

An actual shell needs to recursively parse these expressions, whereas most wordexp implementations try to simply match parentheses. The last example in particular offers an opportunity to introduce arbitrary parentheses.

The POSIX rationale does say:

Arithmetic expansions have precedence over command substitutions. That is, if the shell can parse an expansion beginning with "$((" as an arithmetic expansion then it will do so. It will only parse the expansion as a command substitution (that starts with a subshell) if it determines that it cannot parse the expansion as an arithmetic expansion. If the syntax is valid for neither type of expansion, then it is unspecified what kind of syntax error the shell reports.

POSIX Rationale for Shell and Utilities, 2.6

But who's reading that? Clearly not the authors of most popular shells:

shell $((echo a);(echo b)) $((echo "(");(echo ")")) $((case a in *) echo b;; esac))
dash expects ) parse error parse error
bash a b ( ) syntax error
zsh a b bad math expression b
mksh a b ( ) b
hush expects ) syntax error syntax error

And while perhaps zsh is unlikely to ever be /bin/sh, all of the others are reasonable candidates for it that I've seen on other systems.

3. implementations

Now that we know shells handle these expressions inconsistently, how do different libcs implement wordexp?

3.1. OpenBSD

OpenBSD has the best possible implementation of wordexp(3): none. The demerits of the function are discussed in this thread from 2010.

3.2. leveraging the shell

There are implementations which try to keep wordexp simple by shelling out, which was probably the intended behavior when the function was first created. Unfortunately, this means WRDE_NOCMD can't be trusted in these libcs, without the direct assistance of the shell.

3.2.1. musl

musl's implementation is nice and simple, because it shells out. Unfortunately, in this simplicity, there's no way to really enforce WRDE_NOCMD. musl tries, by matching parentheses, but $((echo a);(echo b)) or similar will get around it, as long as /bin/sh supports such contortions (this confuses dash, but bash happily runs these commands).

3.2.2. Apple / FreeBSD

FreeBSD added wordexp in 2002, using the problematic shell-invoking approach. To FreeBSD's credit, they fixed the major issues with this approach circa 2015 by specializing the shell, indeed noting:

Shell syntax is too complicated to detect command substitution and unquoted operators reliably without implementing much of sh's parser. Therefore, have sh do this detection.

macOS has inherited versions of this implementation, with some modifications (1044.1.2, 1534.81.1). An important practical difference, though, is that on FreeBSD, /bin/sh is always their ash, which at the least doesn't suffer from the aforementioned parsing problem, while on macOS /bin/sh has been bash.5

One interesting twist of Apple's implementation is that in the past, they shelled out to perl, via popen. Last time I checked, they use a helper called /usr/lib/system/wordexp, but this did nothing to prevent command substitution – Apple's libc suffered the same problem as musl. (The shell situation on macOS is always evolving in interesting ways so who knows what the state is now.)

3.2.3. Solaris / Illumos

The Solaris implementation (originally from MKS with a copyright of 1985!) is notable for implementing WRDE_NOCMD by leveraging ksh's restricted mode.

Not many people may still be using this implementation, but this is pretty clever and I guess demonstrates that the commercial Unix implementations may not have been bad.

3.3. parsing shell syntax

So, instead of being simple, we can try the herculean task of implementing most of a shell in libc instead. The only libc I know of that does this is glibc, though (often old, broken) copies of its implementation are found widely, both in programs trying to get around systems like OpenBSD as well as other libcs. For example, uclibc has an old version of glibc's wordexp which has the fatal flaw that backtick can still be parsed from arithmetic expressions, so e.g. $[`touch foo`] will execute a command.

glibc avoids calling out to the shell except when it must, for command substitution. This results in an elaborate, 2500-line reimplementation of some of a shell parser, including a single 800-line function to parse parameter expansions.

It omits many details; for example, in arithmetic expansion. Many valid (and useful) expressions like $((1<<16)) will not be recognized.

However it has one great merit. In 2014, Carlos O'Donnel fixed an important vulnerability. Previously, the code tried to enforce WRDE_NOCMD only when it recognized command substitution's two forms, like many of the implementations which lean on the shell for everything. Though it seems obvious in retrospect, glibc's current implementation guards the actual execution of commands with WRDE_NOCMD tests, instead of trying to do this during parsing.

Despite its complexity, it does seem like the only implementation that is safe to use, though it seems a better policy is to declare a ban on wordexp.

4. conclusion

I had never heard of wordexp(3) until I saw it mentioned in the POSIX standard, while I was implementing a shell.

Many programs that just use wordexp("~/foo") could be replaced with glob(..., GLOB_TILDE). (Though keep in mind, anyone can crash your program with a bad glob.)

I'm not sure where it was first implemented; scanning the Unix history repo, it doesn't really appear until the 90s but there's a reference in a manpage from FreeBSD 1.0 (which doesn't implement it). Let's hope it fades into history similarly.

(Post-publishing addendum: a friend alerted me to the POSIX rationale for wordexp (which I should have read to begin with). It seems the problem was that people kept demanding more and more features for glob, so it seems like the committee threw up their hands and added the far-too-broad wordexp in attempt to cover all possible bases.)



jailkit uses wordexp in its restricted shell; to be fair, this is only enabled if you use the allow_word_expansion option which is disabled by default, and there's always the chance it will link with a safer wordexp like in modern glibc. However, it ships with a vulnerable wordexp bundled, for systems without it, and for example jk_lsh -c '$[`touch foo`]' will do bad things in such a case.


sudo has some clever logic to stuff the WRDE_NOCMD flag into calls to wordexp in its noexec mode; unfortunately, as demonstrated in this post, that's not sufficient to prevent execution from happening in all cases.


Debian's excellent codesearch service provides a quick way to find calls: https://codesearch.debian.net/search?q=%5Cbwordexp%5C%28%5Cb&literal=0&page=1


Consider this example:

$(case $(case $x in *) (echo $x);; esac) in (x) $(echo :);; *) $(echo :);; esac)

bash has a (disabled by default at compile time) --wordexp option, which is an attempt to provide this kind of functionality more safely. It tries to disable command substitution everywhere and only invokes the parser and expander immediately. Last time I checked, macOS didn't use this.