Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Type Punning Functions in C (evanmiller.org)
164 points by heliostatic on Aug 10, 2016 | hide | past | favorite | 94 comments


Sir, while the gentlemen and ladies of this fine organization are infinitely pleased with your research into the black magic of the nether regions and the treatise you have presented from aforesaid research, we would ask you please be even more strident in your warnings about practicing this subject. It seems a number of neophytes have disappeared under odd circumstances in the weeks since you presented your work, and the interruption to the experiments they were assisting with has become quite burdensome. Additionally, we expect training and recruitment costs to be quite a bit higher than normal next year, and as you well know our coffers are not unlimited.


...I once wrote a program to summon demons. It didn't work, mind. But I was really proud of my doubly-linked-list pentacle, each node of which contained a pointer to the summoning point in the centre. I thought that was really elegant.

Incidentally, if anyone is thinking of trying this for yourself: always remember to mlock() the memory containing your summoning. The last thing you want is some demon of the pit getting swapped out to disk. Not only is swap notoriously insecure, but sulphur buildup on the disk platter will cause a disk crash. And for heaven's sake, make sure your pentacle is properly aligned. If it's split across two pages, they may end up not being next to each other in physical ram, and then it won't work!

Ideally, use an embedded device where you know what every cycle does. Don't be put off by the low power. You'd be surprised how much evil you can do with an Arduino.


I've found that my demon summoning programs work much better if I sacrifice the ram beforehand.


Nicely done. The first time I came across the information of punning vs code was during the SPARC ABI standards work where the test code was working but the test case was supposed to fail. It happened to be a vendor compiler that optimized certain four parameter function calls in a similar way. I was rather perplexed how a compiler could "work" and "not work" at the same time.


> black magic of the nether regions

Or plain abstraction violation. “Black magic” gives it an undeserved good reputation: “I must aspire to be clever enough to understand and actively make use of this.”


> “Black magic” gives it an undeserved good reputation

From http://catb.org/jargon/html/B/black-magic.html

> black magic: n.

> [common] A technique that works, though nobody really understands why. More obscure than voodoo programming, which may be done by cookbook. Compare also black art, deep magic, and magic number (sense 2).


Maybe. Given it relies on the calling convention of the OS you're on, the CPU architecture you're on, and knowledge of assembly, it could very well be both.


It also relies on implementation-specific behavior in the compiler, since the latter is not actually required to follow established the convention for argument passing, so long as it can prove that the only place the function is called from is the same translation unit.

In fact, so far as I can tell by looking at the first example, the compiler can circumvent this whole thing by noting that:

1. DoubleToTheInt is only ever called in the same unit, so it can be inlined - no need to generate the function.

2. Pointer to DoubleToTheInt is created, but is then immediately cast to an incompatible function type, which is undefined behavior - and so it can be substituted with any random value, say, whatever happens to be in EAX at that moment (any U.B. is substitutable for any other U.B.!), and the compiler can still avoid generating the actual function.


Here's an example I'm rather proud of. It involves setting getpid(2) as a signal handler(2) via JNA.

https://github.com/facebook/buck/commit/48f1e201a966b9e2adab...


Why is setting the signal handler for SIGHUP to getpid preferable to masking the signal?


> But when we exec a child, the kernel resets any signal not set to `SIG_IGN` to `SIG_DFL`, so all our children, atomically, get their `SIGCHLD` handlers set to `SIG_DFL` at the moment of creation. Consequently, when either our process or the kernel sends `SIGHUP` to our process group, all our children die.


SIG_IGN is not the same as the signal mask.

The result is the same though; I forgot that signal mask is not reset on exec. Damn POSIX signals!


If your C compiler doesn't complain bitterly to you about an incompatible pointer type on that assignment, get a better C compiler. If your C compiler does complain bitterly about it, heed those warnings or expect no sympathy.

If you write this in production code, and don't get sent home that day in tears, your company's code review process has failed.


> If your C compiler doesn't complain bitterly to you about an incompatible pointer type on that assignment, get a better C compiler.

The example code uses C style casts, which is C-speak for "fuck off compiler I know what I'm doing." I'd like to see what warnings your compiler produces. VS2015 /W4 /ANALYZE produces nothing, and neither do clang nor GCC from the looks of it.

That people C cast without knowing what they're doing is rather unfortunate tragedy. That this is a common enough problem, that the Win32 API is littered with backwards compatability hacks where someone C-casted away ABI differences turns it into a statistic.

> If you write this in production code, and don't get sent home that day in tears, your company's code review process has failed.

Assuming we're talking "well meaning newbie" instead of "sanity hating self-flagellating freak and/or undefined behavior worshiping cultist", I prefer to carefully explain that they've invoked undefined behavior, that undefined behavior leads to crunching to find optimizer induced heisenbugs in the weeks before we ship, and to impress upon them that this is why we never cast away even "trivial" and "minor" function pointer differences. And then, when they ask what the correct thing to do is, point out the API docs and header both have this little "WINAPI" annotation they can use. Hey! No more casts! Easier than the casts too. Win/win.

However, if we are talking about the "sanity hating self-flagellating freak and/or undefined behavior worshiping cultist", I can only recommend nuking the site from orbit. It's the only way to be sure. If their home isn't part of the radioactive crater, you didn't use large enough nukes, and your company's code review process has failed.


> your company's code review process has failed.

I often remind my fellow engineers to keep their code muggle-friendly. And, mind you, our code base is mostly Python, a language that actively avoids black magic by design.


> keep their code muggle-friendly

What a great phrase, I'm going to borrow that.


I'd hate to work at a company with a blanket prohibition on dirty tricks like this. On rare occasions, there are good reasons to use hacks, and I'd hope my coworkers would at least be open to an attempt at justification. Sometimes silly newbie mistakes actually end up being the right thing to do in rare, specialized circumstances and in experienced hands.


I can't ever imagine a scenario where a hack _like_ _that_ is acceptable.

If you need something like that, just drop down to assembly. It's more portable (in the sense that you're coding to an interface), infinitely more clear about what you're doing and and how you're doing it, much less likely to break, and when it does break is likely to break loudly.

Even if you've never done a lick of assembly programming (I haven't done much) it'd still be infinitely preferable, even if you had to support a dozen architectures with two dozen different function proxies.

"Hack" is a very broad term. By some definitions a significant chunk of extant code qualifies as a hack. But whatever the definition, a hack still implies some quantum of legitimacy. There's nothing legitimate in using function type punning as a way to reorder or control argument values. Not unless this was 1969, where there was precisely one C compiler, used by precisely one guy, and where such code was expected to be thrown out in short order, never to infect code that executed in other than identical circumstances.


The reordering thing is bunk anyway, because if you just define the reordering function:

  double IntPowerOfDouble(int power, double base)
  {
      return DoubleToTheInt(base, power);
  }
...then gcc with -O1 will optimise IntPowerOfDouble to a plain "call DoubleToTheInt", and with -O2 will completely inline DoubleToTheInt into it.


Dirty tricks are almost never worth the long term technical debt they represent when either (1) weird bugs start showing up or (2) the original author leaves the company.

I used to believe that there were times when dirty tricks were the right call in certain cases. The problem was that the long term view always proved me wrong: a more verbose but commonly understood approach is much more maintainable than a concise dirty trick, even with ample commenting and documentation.


> On rare occasions, there are good reasons to use hacks

Yes.

However.

If you wish to convince me that any of the type punning examples from the article, as given, are useful in production code, I will allow you that opportunity. But only after I've had the chance to make myself a bowl of popcorn large enough to share with whoever else might be entranced by the ensuing trainwreck.


And there are different version of 'dirty trick' perception too. Sometime what people perceive as 'dirty trick' is just normal use of the language while the muggles want to talk baby talk all the time.

For example, having to explain the functional use of 'x = !!variable' to a C programmer. There are many like that.

Sometime you just have to decide to raise the bar a little...


I used !! for decades, but since C'99, I've recommended "bool x = variable" or "(bool) variable".


The reason I would prefer !! (or != 0) is that there are far too many err, non-bool bools. Even if you know for sure it's the stdbool.h bool, the next reader might not be aware -- presuming that when you do this you care about the distinction (otherwise I agree that the cast is fine).


It's not practical to make this a warning because of, e.g., dlsym (which is actually worse because void * and void (*)() aren't necessarily compatible types).

Another thing which makes this sort of thing difficult to warn in is the brokenness of autoconf. Autoconf checks that your system has memcpy (or anything that you check via AC_TRY_LINK) by compiling this program:

    void memcpy();
    int main() {
      (void)memcpy();
    }


Wait, undefined parameters are guaranteed to be zero in C? That's something I wouldn't have expected. Certainly nothing I would rely on in my own code.


No, not at all. It doesn't run the program, the program isn't correct, it just compiles and links. Symbols don't know their types (... with some caveats about name mangling in some languages).


In C, a function with no parameters is variadic. You need to add "void" to preclude parameters. This is one of the few breaking changes in C++ where no parameters means no parameters.


True, but I'm not seeing the relevance. Just an interesting aside?


It's legal to declare an external function symbol without specifying the parameters, so long as you actually match the types of real parameters (after all the standard conversions for functions with such prototypes are taken into account, like char -> int; and excluding some corner cases, like varargs) when you call it. If you don't actually call it, it's not U.B.

However, the snippet above is still broken in that respect, because it doesn't match the return type of memcpy - and that is U.B.


> It's legal to declare an external function symbol without specifying the parameters, so long as you actually match the types of real parameters

Yes. And the call here does not match the types of the real parameters, even before we consider the return type.


Right, but (again, ignoring the return type) the call doesn't actually get executed, and U.B. only happens if it does. Though I guess they could have wrapped it in if(0){...} to make this completely unambiguous.


Is your claim "there is no undefined behavior because the program is not run"? While that's true (there's no behavior, much less undefined behavior), in that case I don't see why the return type deserves calling out special. This program exhibits undefined behavior iff executed, and that would be true even if the return type were void * .

> I guess they could have wrapped it in if(0){...} to make this completely unambiguous.

Depending on optimizations, that may mean the executable does not depend on the symbol we're trying to test the presence of.

Indeed, on my local gcc that's the case, even with -O0.

    $ cat test.c
    void *memcpy();
    
    int main() {
    	if(TEST) {
    		(void)memcpy();
    	}
    }
    $ gcc -DTEST=0 test.c
    $ objdump -T a.out
    
    a.out:     file format elf64-x86-64
    
    DYNAMIC SYMBOL TABLE:
    0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 __libc_start_main
    0000000000000000  w   D  *UND*	0000000000000000              __gmon_start__
    
    
    $ gcc -DTEST=1 test.c
    $ objdump -T a.out
    
    a.out:     file format elf64-x86-64
    
    DYNAMIC SYMBOL TABLE:
    0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 __libc_start_main
    0000000000000000  w   D  *UND*	0000000000000000              __gmon_start__
    0000000000000000      DF *UND*	0000000000000000  GLIBC_2.14  memcpy
    
    
    $ gcc -O0 -DTEST=0 test.c
    $ objdump -T a.out
    
    a.out:     file format elf64-x86-64
    
    DYNAMIC SYMBOL TABLE:
    0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 __libc_start_main
    0000000000000000  w   D  *UND*	0000000000000000              __gmon_start__
    
c.f.:

    $ cat test.c
    void *blah();
    
    int main() {
    	if(TEST) {
    		(void)blah();
    	}
    }
    $ gcc -DTEST=1 test.c
    /tmp/cc2Vk5mE.o: In function `main':
    test.c:(.text+0xa): undefined reference to `blah'
    collect2: error: ld returned 1 exit status
    $ gcc -DTEST=0 test.c
    $ objdump -T a.out
    
    a.out:     file format elf64-x86-64
    
    DYNAMIC SYMBOL TABLE:
    0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 __libc_start_main
    0000000000000000  w   D  *UND*	0000000000000000              __gmon_start__
    
    
    $ gcc -O0 -DTEST=0 test.c
    $ objdump -T a.out
    
    a.out:     file format elf64-x86-64
    
    DYNAMIC SYMBOL TABLE:
    0000000000000000      DF *UND*	0000000000000000  GLIBC_2.2.5 __libc_start_main
    0000000000000000  w   D  *UND*	0000000000000000              __gmon_start__


I'm just trying to draw the line between U.B. that is guaranteed to happen at runtime (at which point the compiler is, technically speaking, not even obligated to provide you with a binary to run), and U.B. that will only happen if that particular code is executed, but doesn't trigger if that branch is omitted. It's probably overly pedantic, but given what modern optimizing compilers do, I'm in the "better safe than sorry" camp.

You definitely have a point regarding compiler just ripping that code out. I guess the proper approach would be to do something like if(argv[0][0]), or test a volatile variable.


But in this case, the U.B. is guaranteed to happen at runtime, even with the proper return type. There are no branches involved here...


Yes, you're right. I was thinking aloud about how it could be made conforming, in principle.


Gotcha.


No. Undefined parameters are not guaranteed to be zero in C. Static variables not explicitly set are guaranteed to be initialized to zero in C.


>If your C compiler doesn't complain bitterly to you about an incompatible pointer type on that assignment, get a better C compiler.

Yeah? So which compiler do you have in mind exactly we should use?


No warnings (-W -Wall) with gcc 4.2.1 on OSX :/

gcc 4.8.4 on Ubuntu 14.04 Trusty still gives no warnings, but gives a different result:

  $ gcc -W -Wall x.c -lm
  $ a.out
  (0.99)^100: 0.366032
  (0.99)^100: inf
clang 3.4 on Ubuntu 14.04:

  $ clang -W -Wall -fsanitize=undefined x.c -lm
  $ a.out
  (0.99)^100: 0.366032
  (0.99)^100: inf
:(


    ; gcc -v |[2] grep version
    gcc version 6.1.1 20160802 (GCC) 
    ; gcc -Wall -Werror ./something.c  -lm
    ; ./a.out 
    (0.99)^100: 0.366032 
    (0.99)^100: 0.000000 
    ;


I get 0.366032 on gcc-5.4, gcc-6.1, and clang-3.8 on Linux (Archlinux)


Sometimes you just need to cast and assume that you've been given the right thing. Such as when going anywhere near a Windows message loop.

And sometimes you need to do terrifying, nonportable, ugly hacks in order to get something specific to work. My own contribution to this field is an ARM stack unwinder that requires only one __asm__ inline in order to get the link register and does everything else by interpreting instructions.


The latest GCC and CLANG compilers do not seem to complain with this code. What might be a better compiler?


They are not many C compilers that are widely used in the wild. So, if I had to guess the average C programmer is using either GCC, Microsoft's C Compiler, or Clang.


This is undefined behavior, so you're claim that this is 'in C' is blatantly false. In general, you cannot assume the underlying architecture will pass its arguments in any particular location.


It's specifically presented as not part of C.

I cheated a little bit above — I assumed you're running code on a 64-bit x86 PC. If you're on another architecture, the trick above might not work. In spite of C's reputation for having an infinite number of dark corners, the int-double-argument-order behavior is certainly not a part of the C standard. It's a result of how functions are called on today's x86 machines, and can be used for some neat programming tricks.


But it is part of the C standard: It's explicitly undefined behavior.

There are varieties of type punning that are legal C, so I agree it's confusing to present this UB implementation artifact as "Type Punning ... in C".

Still a neat magic trick, and an interesting explanation of how it works.


In that respect, every possible situation that could happen is part of the standard, as it separates everything into defined and undefined. Cosmic rays twiddling bits in memory is part of the standard by UB under that view. While technically correct, it's useless in practice when taken to this extreme.


twoodfin is correct in saying that it is explicitly undefined behavior: "If a converted pointer is used to call a function whose type is not compatible with the pointed-to type, the behavior is undefined." See http://stackoverflow.com/questions/188839/function-pointer-c...

Your response would make sense if twoodfin claimed it was implicitly undefined behavior. But that's not the case. The standard clearly addresses this case.


Except that the C standard explicitly makes no distinction between explicit and implicit undefined behavior.

    If a "shall" or "shall not" requirement that appears outside
    of a constraint or runtime constraint is violated, the behavior
    is undefined. Undefined behavior is otherwise indicated in this
    International Standard by the words "undefined behavior" or by
    the omission of any explicit definition of behavior. There is
    no difference in emphasis among these three; they all describe
    "behavior that is undefined".
N1570 section 4 paragraph 2.


Well, if it relied on just that then that would be accurate. It also relies on the specific architecture and registers available and the calling conventions of the target format (which use different numbers of registers for function arguments).


are you sure?

i think in your cosmic ray example, you get well defined behaviour producing unexpected results - its a subtle distinction, but that's part of why UB is horrible, its results can be even less reasonable than those of random bit twiddling.

e.g. your standard conforming addition function will still always perform addition - now if by definition that function merely looked like an addition but was actually describing UB, the result could be anything at all, including multiplication (unlikely, but possible) and doing nothing at all (very likely).


>This is undefined behavior, so you're claim that this is 'in C' is blatantly false.

Undefined behavior is undefined behavior FOR C. Undefined behavior makes no sense unless referring to a specific language.

Besides, people have been taking advantage of certain undefined behaviors in actual C programs since the dawn of time -- deliberately I mean.


It's important to note that standards other than C can place constraints on C implementations and make things that are "undefined" according to ANSI C perfectly valid in specific environments.

For example, POSIX now mandates that the NULL pointer has all zero bits: http://austingroupbugs.net/view.php?id=940


The representation of types is either unspecified or implementation-defined in C, depending on the context. See C11 6.2.6; also see Annex J. Relying on particular representations does not make your program per se invalid C. Reliance on undefined behavior does make it invalid.

Representations are readily observable because the standard permits all objects to be cast to char type. Therefore the standard basically says that you can read and even modify objects through a char array as long as you obey house rules.

So POSIX isn't changing what undefined means by mandating that CHAR_BIT is 8, or that null pointers can be represented as all-zero bits.

By contrast, I think the definition of dlsym() does depend on undefined behavior because C does not permit casting function pointers to void pointers or vice-versa. For various reasons that's a significant deficiency. Keeping that behavior undefined is intended to, among other things, preserve the ability to implement tagged pointers in a standards compliant manner, or to even create implementations for Harvard architecture machines. Also, I think the distinction between undefined behavior and unspecified or implementation-defined behavior preserves the ability to implement practical, quality static analyzers that can analyze all C code, whether or not dependent on certain implementation-specific behaviors. Some amount of license is a practical necessity; too much makes the standard useless.

C permits assignment of pointers to functions of incompatible types as long as the functions aren't invoked. But mixing pointers to functions and pointers to objects is a definite no-no. So POSIX will probably add a sister to dlsym that returns a generic function pointer type like (void (*)(void)).


> So POSIX will probably add a sister to dlsym that returns a generic function pointer type like (void (*)(void)).

I think that change would be a mistake: it makes no practical difference on the machines for which POSIX is intended, so this sister function would bloat the API only to satisfy pedantic concerns that don't really matter. Besides, POSIX systems have to keep the traditional dlsym around for compatibility, which further erodes the advantage of your proposed function.

Right now, POSIX mandates by implication that interconversation of function and data pointers works, and I think that's a perfectly reasonable state of affairs. There is nothing wrong with POSIX constraining the C standard.


A year ago I'd probably agree with you. But since then I've researched various avenues of software hardening and I think the future of C programming will begin to increasingly leverage the restriction.

For example, C++ class vtables are a prime target for heap overflow attacks because they store function pointers on the heap. And with the anti-ROP measures coming around the corner it's a target that will be increasingly used to create and execute a ROP chain.

So, for example, I can easily imagine new compiler modes and coding styles that outright ban casting function pointers to object pointers, and ban casting pointers to function pointers to other object pointers. PaX and W^X don't help you here as the issue isn't writing over code, but writing over pointers to code.

Banning those conversions is neither sufficient nor necessary, but nonetheless a very plausible and practical counter-measure. In that context dlsym sticks out like a sore thumb. One of the easiest solutions is to just standardize a routine for returning a function pointer. That's arguably better than forcing compilers to understand the semantics of dlsym.

Note that implementations can still support mixing pointer types, and POSIX could even still require it so existing software can continue using dlsym the same way. But it would simplify things for the tooling and runtimes that enforce the distinction. Given that it makes POSIX more consistent with the C standard, it's a win-win.


> C++ class vtables are a prime target for heap overflow attacks because they store function pointers on the heap

Is that so? The actual pointers should be in .data. The pointer to the vtable is on the heap (or on the stack, or wherever your object lives). In any case, you should be able to use something like [1] to place constraints on indirect calls, right? There are lots of approaches to double checking function calls that don't involve increasing function pointer size.

Besides, there are lots of bits of code that depend on function- and data-pointer interconvertibility --- people can stuff function pointers in raw void* cookies to callbacks.

Rather than make the effort to make existing code work in this (in practice, new) subset of customary (as opposed to standard) C, wouldn't it be better to just write components in a language that's memory-safe to start with?

[1] https://gcc.gnu.org/wiki/cauldron2012?action=AttachFile&do=g...


I didn't mean to imply that you needed to increase function pointer size or otherwise change pointer representations to add such counter-measures. I mentioned tagged pointers in the ancestor post as a classic justification for restricting conversions between object and function pointers.

Restricting void conversions for function pointers is just a pragmatic example where part of the burden of implementing and enforcing new language features can be paid by reference to the C standard and the license it affords. It's loosely related to tagged pointers in that both depend on software not abusing the type system too thoroughly. And as problematic as C's type system is, it's not without rhyme or reason.

We don't _need_ to rely on the the C standard in any way, shape, or form to implement counter measures. But the function of a language standard is to be a locus for these things, and to help ease the burden of coordinating and implementing changes, and pushing adoption among a wide and diverse community.

As for simply just switching to some other language, I don't think that's even remotely realistic. C isn't going away anytime soon, and neither are people going to stop writing new C code anytime soon.

People rave about a language like Rust, but there is no language called "Rust". It's constantly being updated, claimed guarantees regarding compatibility notwithstanding. You simply cannot rely on Rust to create a large and complex ecosystem of software yet, and probably won't for awhile. Conversely, when authors of a language like Go adamantly refuse to evolve the language out of legitimate concern for compatibility, people flip out. Language maintainers are damned if they do, damned if they don't. There's no simple answer or approach.

C has survived for as long as it has partly _because_ undefined and unspecified behavior provide space for it to slowly evolve through controlled experimentation. And it will continue doing so (and should do so) on its descent just as it did on its ascent.


> It's constantly being updated, claimed guarantees regarding compatibility notwithstanding.

Updated does not mean incompatible. And the community bears this out, in the survey, most people had things Just Work, and for those that had issues, they said the upgrade was incredibly minor.

If there's something different than this, I'd love to hear about it: we care deeply about stability.


Regarding the paper you cited, that doesn't really address the core issue and sort of misses the point.

1) The issue isn't just modifying vtables; again, that's just a classic example intended to highlight the issue. Both C and C++ code can and often does juggle pointers to regular functions.

2) The referenced mechanism to protect vtables and calls through vtables _rely_ on the much stricter language semantics regarding method invocations. My entire point was that by enforcing some of the restrictions C already guarantees regarding function pointers, we can more easily implement similar approaches for C and C++ code directly manipulating function pointers.

Your example about void pointer cookies for callbacks is perfect. Restricting conversions between function and object pointers makes it easier to add instrumentation to verify integrity. I'm sure it's possible to do that without such limitations, especially with your particular example, but it adds complexity and removes opportunities for optimizing away the cost. Undefined behavior helps compilers improve performance because it simplifies the model compilers require for proving the correctness of generated code. That same dynamic also makes it easier to implement and enforce security constraints. That simpler model can allow the compiler to assume a pointer is valid, but it also permits it to more easily detect and abort the program when it isn't without violating the contract with the programmer.

Contrast that with a language that permits you to dynamically update code sections by poking and jumping to memory anywhere. That permits all kinds of fun hacks, but it means the compiler can't enforce any meaningful constraints. There's a huge middle ground between that and a language like Rust or Haskell.

I find it odd that people will in the same breath claim that C is a security nightmare and should be abandoned, yet argue to preserve insecure semantics that could be easily removed and in many cases are already disallowed by the standard. That's a harmful cognitive dissonance. To be clear, I'm not accusing you of that; but I find your arguments to be uncomfortably close to that misguided conception of C and the C ecosystem.


Strictly speaking, POSIX mandates that a pointer with all zero bits is a null pointer, not necessarily the other way around.


C guarantees some properties.

The platform ABI is guaranteed to work in a certain way.

In this specific case, the compiler is not allowed to make any assumptions about the legality, but has to follow the ABI.

There's no UB involved (not every platform specific issue may be labeled UB; think accessing bare metal memory on one platform where a specific address is IO mapped and on another it is regular SRAM; it's perfectly valid C, and has perfectly known semantics on both platforms, but will work on one platform and not work on another platform; this is also not UB).


Nope, it's UB[0]. While not everything that doesn't work on every platform is UB, plenty of things are. Consider signed integer overflow, which is a common source of UB.

To be clear, UB means the compiler is allowed to e.g. totally remove the call (or really anything else it wants to do at all) if it can be positive that this is what's happening -- not that it will behave as could be expected on the platform, the term for that would be implementation defined behavior.

[0]: http://c0x.coding-guidelines.com/6.3.2.3.html number 766-768


You're right.


The problem is that determining whether an ostensibly C (syntactically valid and typable) program is actually written “in C” is undecidable.


Almost any interesting property about programs is undecidable. Yet somehow programmers still write nontrivial programs.

Undefined behavior in C sucks, but being its presence or absence being undecidable is not the main reason.


Careful about what you're quantifying over.

Any interesting property is undecidable in the sense that we will be unable to produce an answer for some programs. Nothing says that there can't exist plenty of nontrivial programs for which we can show the property holds.


Excellent point. This misconception is also the root of silly statements like "human intelligence is uncomputable because we can do things like tell whether a program will halt." No, we can solve specific cases of the halting problem, we can't solve the general case any more than a computer can.


> we can't solve the general case any more than a computer can

While almost certainly the case, I'm not sure how we can demonstrate that this is true in principle without begging the question, if we're asking whether human intelligence is computable.

I agree that all we have evidence of is that we can solve specific cases, and that this completely undermines the argument you're critiquing.

This error also occurs more subtly. I had someone tell me that static type checking was basically useless because it couldn't even do something as simple as saying whether a program will halt. Nevermind, if we're demanding this of all programs, that neither can any other method of reasoning we're applying. (And also nevermind, if we're demanding this of an interesting subset of programs, that there are type systems that will tell us exactly that...).


Just like there are plenty of nontrivial C programs where we can be sure that the behavior is defined. That's exactly my point.


> Almost any interesting property about programs is undecidable.

It's only undecidable if the language makes it undecidable. For example, subject reduction is decidable in a type-safe language (EDIT: with decidable type checking, oops!), because the very definition of type safety means that typability implies subject reduction.

Of course, not everything of interest can be made decidable in a general-purpose language, but the integrity of a language's basic abstractions should never be in question.


The formalization of this idea is Rice's Theorem: https://en.wikipedia.org/wiki/Rice%27s_theorem


Rice's theorem is a statement about partial functions, not programs. There exist decidable properties of programs: Is it of size less than N (for any suitable measure)? Does it finish in less than N steps (for a given fixed input)? Two syntactically distinct programs may compute the same partial function.


I wonder whether it'd be feasible to have the compiler issue warnings for explicitly undefined behavior. While this is not something you'd self-inflict by accident, other more subtle traps could be flagged.

I have known of organizations that are happy only when code compiles without warnings.


Code with UB is valid C. Nasal Demons and all. Otherwise, it would be illegal to do this:

  int i = 16 >> 32;


Incidentally, the codebase I'm looking at right now contains gems like this:

    /*VARARGS2*/
    int margin_printf (fp, a, b, c, d, e, f, g)
    FILE *fp;
    char *a;
    long b, c, d, e, f, g;
    {
        ind_printf (0, fp, a, b, c, d, e, f, g);
    } /* margin_printf */
Called (from a different source file) like this:

    margin_printf (outfile, length ? "/* %s */\n" : "\n", storage);
Okay, so that's K&R C, and it's not actually compiled any more (because I've been slowly taking this stuff out and replacing it with things that actually work), but still --- the horror, the horror...


That kind of hackery was necessary because K&R C didn't give you any sanctioned way to write your own printf-like functions. ANSI C introduced <stdarg.h>.


Sure. Also, K&R C tended to target platforms with really simple ABIs, where parameters were always passed on the stack and everything was a machine word; so you could get away with this kind of thing.

K&R C actually has a minimalist elegance to it that later versions of C lost. Not that I'm claiming that it's better, of course. But K&R C had a distinct philosophy to it that was definitely its own.


... we saw how register allocation and calling conventions — supposedly the exclusive concern of assembly-spinning compiler writers — occasionally pop their heads up in C ...

I don't know about "occasionally". At least on Windows, calling conventions used to be a constant headache. The C language default was cdecl, but the Win32 API default (most of the time) was stdcall. Then there was a variant (fastcall?) that tried to make use of registers.

Maybe it's all fixed now... But this being Win32, I rather expect they've managed to accumulate a few more "interesting" gotchas and edge cases as the platform expanded with 64-bit, WinRT, Universal Windows Platform and whatever.


Fortunately, in the amd64 Windows world, there's only one calling convention. (Edit: as cremno points out, there's now a new vectorcall convention. Dammit.) The 64-bit Windows ABI is a big improvement over the old 32-bit one. Other improvements are use of unwind tables for SEH and the requirement that all code be unwindable.


>Fortunately, in the amd64 Windows world, there's only one calling convention.

Not anymore: https://blogs.msdn.microsoft.com/vcblog/2013/07/11/introduci...


> the requirement that all code be unwindable.

i would not call that an improvement in all cases... the extra code produced by enabling exceptions, even for the latest compilers, produces enough bloat and slow down for basically everyone seriously concerned with real-time performance to turn them off.

small simple functions are forced to have prologue/epilogue, use a frame pointer and to preserve the return address on the stack. doesn't sound like much, but if your function does anything simple (float f() { return 1; } bool g(float a, float b) { return a < b; } etc...) you spend all the time paying for exceptions instead of the intended functionality.


You're confusing unwinding mechanisms. The whole point of using unwind tables is to avoid the need to preserve frame pointers. There zero cost in pure C code and minimal cost in C++ code just for preserving the ability to unwind. You absolutely do not need to have extra prologue and epilogue code or to burn a register for a frame pointer.


from what i understood it only gets rid of those requirements in some cases....

there is a table i remember in some doc about this, the only reason i listed from memory was the lack of being able to find it quickly on google

.. its a shame i can't find it. perhaps my memory is imply faulty.


I'm happy to hear that! With a fixed ABI, C++ 1x language features and UWP, it sounds like doing C/C++ on Windows could actually be fun these days.


don't get ahead of yourself... that ABI isn't there yet. :P


its quite intentional that on x64 the default convention for the compiler and the default convention for the platform is identical.

the "constant headache" you allude to played no small part in that decision i would say...


its a shame this only glosses over the subject, and starts with a highly misleading example and then continues with plenty more misleading information.

unix/windows are not interchangeable with compilers targetting those platforms. the article doesn't make the distinction or highlight why it matters - the OS has nothing to do with calling conventions in this context, they are just very conveniently the same in this case.

this example doesn't work with vs2010 out of the box because it targets win32 by default... for example. its the compilers, and only the compilers, which are deciding this. the standard calling convention for the platform is incidental, and it being identical to the default for the compiler is a wise design choice for compilers targetting x64, rather than a necessity - something which is much clearer in the old world of windows and x86 (32-bit) where cdecl was the default out of the box, and stdcall was what windows api calls expected.


Doesn't the compiler at least need to use the platform-standard argument convention when calling OS functions?


Yes. Every Windows API is prefixed with WINAPI, which is a macro for the calling convention.


The macro expands to nothing on Win64, though, because it has one standard calling convention.

(At the ABI boundary, anyway. There's also __vectorcall now. And, of course, the compiler can use whatever it wants for functions that are not exposed at the boundary - even arbitrary register assignments based on where the function is actually called, to minimize the amount of work needed to save registers etc.)


I freaking hate you, OP. I really freaking hate you right now. This is an abomination in the sight of God.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: