docs/README-dynapi.md
changeset 10486 5bf595c48fd4
parent 9756 2069629bd155
child 11720 9cbb45a5874f
     1.1 --- a/docs/README-dynapi.md	Fri Oct 07 17:30:21 2016 -0700
     1.2 +++ b/docs/README-dynapi.md	Fri Oct 07 17:46:58 2016 -0700
     1.3 @@ -1,130 +1,130 @@
     1.4 -Dynamic API
     1.5 -================================================================================
     1.6 -Originally posted by Ryan at:
     1.7 -  https://plus.google.com/103391075724026391227/posts/TB8UfnDYu4U
     1.8 -
     1.9 -Background:
    1.10 -
    1.11 -- The Steam Runtime has (at least in theory) a really kick-ass build of SDL2, 
    1.12 -  but developers are shipping their own SDL2 with individual Steam games. 
    1.13 -  These games might stop getting updates, but a newer SDL2 might be needed later. 
    1.14 -  Certainly we'll always be fixing bugs in SDL, even if a new video target isn't 
    1.15 -  ever needed, and these fixes won't make it to a game shipping its own SDL.
    1.16 -- Even if we replace the SDL2 in those games with a compatible one, that is to 
    1.17 -  say, edit a developer's Steam depot (yuck!), there are developers that are 
    1.18 -  statically linking SDL2 that we can't do this for. We can't even force the 
    1.19 -  dynamic loader to ignore their SDL2 in this case, of course.
    1.20 -- If you don't ship an SDL2 with the game in some form, people that disabled the
    1.21 -  Steam Runtime, or just tried to run the game from the command line instead of 
    1.22 -  Steam might find themselves unable to run the game, due to a missing dependency.
    1.23 -- If you want to ship on non-Steam platforms like GOG or Humble Bundle, or target
    1.24 -  generic Linux boxes that may or may not have SDL2 installed, you have to ship 
    1.25 -  the library or risk a total failure to launch. So now, you might have to have 
    1.26 -  a non-Steam build plus a Steam build (that is, one with and one without SDL2 
    1.27 -  included), which is inconvenient if you could have had one universal build 
    1.28 -  that works everywhere.
    1.29 -- We like the zlib license, but the biggest complaint from the open source 
    1.30 -  community about the license change is the static linking. The LGPL forced this 
    1.31 -  as a legal, not technical issue, but zlib doesn't care. Even those that aren't
    1.32 -  concerned about the GNU freedoms found themselves solving the same problems: 
    1.33 -  swapping in a newer SDL to an older game often times can save the day. 
    1.34 -  Static linking stops this dead.
    1.35 -
    1.36 -So here's what we did:
    1.37 -
    1.38 -SDL now has, internally, a table of function pointers. So, this is what SDL_Init
    1.39 -now looks like:
    1.40 -
    1.41 -    UInt32 SDL_Init(Uint32 flags)
    1.42 -    {
    1.43 -        return jump_table.SDL_Init(flags);
    1.44 -    }
    1.45 -
    1.46 -Except that is all done with a bunch of macro magic so we don't have to maintain
    1.47 -every one of these.
    1.48 -
    1.49 -What is jump_table.SDL_init()? Eventually, that's a function pointer of the real
    1.50 -SDL_Init() that you've been calling all this time. But at startup, it looks more 
    1.51 -like this:
    1.52 -
    1.53 -    Uint32 SDL_Init_DEFAULT(Uint32 flags)
    1.54 -    {
    1.55 -        SDL_InitDynamicAPI();
    1.56 -        return jump_table.SDL_Init(flags);
    1.57 -    }
    1.58 -
    1.59 -SDL_InitDynamicAPI() fills in jump_table with all the actual SDL function 
    1.60 -pointers, which means that this _DEFAULT function never gets called again. 
    1.61 -First call to any SDL function sets the whole thing up.
    1.62 -
    1.63 -So you might be asking, what was the value in that? Isn't this what the operating
    1.64 -system's dynamic loader was supposed to do for us? Yes, but now we've got this 
    1.65 -level of indirection, we can do things like this:
    1.66 -
    1.67 -    export SDL_DYNAMIC_API=/my/actual/libSDL-2.0.so.0
    1.68 -    ./MyGameThatIsStaticallyLinkedToSDL2
    1.69 -
    1.70 -And now, this game that is staticallly linked to SDL, can still be overridden 
    1.71 -with a newer, or better, SDL. The statically linked one will only be used as 
    1.72 -far as calling into the jump table in this case. But in cases where no override
    1.73 -is desired, the statically linked version will provide its own jump table, 
    1.74 -and everyone is happy.
    1.75 -
    1.76 -So now:
    1.77 -- Developers can statically link SDL, and users can still replace it. 
    1.78 -  (We'd still rather you ship a shared library, though!)
    1.79 -- Developers can ship an SDL with their game, Valve can override it for, say, 
    1.80 -  new features on SteamOS, or distros can override it for their own needs, 
    1.81 -  but it'll also just work in the default case.
    1.82 -- Developers can ship the same package to everyone (Humble Bundle, GOG, etc), 
    1.83 -  and it'll do the right thing.
    1.84 -- End users (and Valve) can update a game's SDL in almost any case, 
    1.85 -  to keep abandoned games running on newer platforms.
    1.86 -- Everyone develops with SDL exactly as they have been doing all along. 
    1.87 -  Same headers, same ABI. Just get the latest version to enable this magic.
    1.88 -
    1.89 -
    1.90 -A little more about SDL_InitDynamicAPI():
    1.91 -
    1.92 -Internally, InitAPI does some locking to make sure everything waits until a 
    1.93 -single thread initializes everything (although even SDL_CreateThread() goes 
    1.94 -through here before spinning a thread, too), and then decides if it should use
    1.95 -an external SDL library. If not, it sets up the jump table using the current 
    1.96 -SDL's function pointers (which might be statically linked into a program, or in
    1.97 -a shared library of its own). If so, it loads that library and looks for and 
    1.98 -calls a single function:
    1.99 -
   1.100 -    SInt32 SDL_DYNAPI_entry(Uint32 version, void *table, Uint32 tablesize);
   1.101 -
   1.102 -That function takes a version number (more on that in a moment), the address of
   1.103 -the jump table, and the size, in bytes, of the table. 
   1.104 -Now, we've got policy here: this table's layout never changes; new stuff gets 
   1.105 -added to the end. Therefore SDL_DYNAPI_entry() knows that it can provide all 
   1.106 -the needed functions if tablesize <= sizeof its own jump table. If tablesize is
   1.107 -bigger (say, SDL 2.0.4 is trying to load SDL 2.0.3), then we know to abort, but
   1.108 -if it's smaller, we know we can provide the entire API that the caller needs.
   1.109 -
   1.110 -The version variable is a failsafe switch. 
   1.111 -Right now it's always 1. This number changes when there are major API changes 
   1.112 -(so we know if the tablesize might be smaller, or entries in it have changed). 
   1.113 -Right now SDL_DYNAPI_entry gives up if the version doesn't match, but it's not 
   1.114 -inconceivable to have a small dispatch library that only supplies this one 
   1.115 -function and loads different, otherwise-incompatible SDL libraries and has the
   1.116 -right one initialize the jump table based on the version. For something that 
   1.117 -must generically catch lots of different versions of SDL over time, like the 
   1.118 -Steam Client, this isn't a bad option.
   1.119 -
   1.120 -Finally, I'm sure some people are reading this and thinking,
   1.121 -"I don't want that overhead in my project!"  
   1.122 -To which I would point out that the extra function call through the jump table 
   1.123 -probably wouldn't even show up in a profile, but lucky you: this can all be 
   1.124 -disabled. You can build SDL without this if you absolutely must, but we would 
   1.125 -encourage you not to do that. However, on heavily locked down platforms like 
   1.126 -iOS, or maybe when debugging, it makes sense to disable it. The way this is
   1.127 -designed in SDL, you just have to change one #define, and the entire system 
   1.128 -vaporizes out, and SDL functions exactly like it always did. Most of it is 
   1.129 -macro magic, so the system is contained to one C file and a few headers. 
   1.130 -However, this is on by default and you have to edit a header file to turn it 
   1.131 -off. Our hopes is that if we make it easy to disable, but not too easy, 
   1.132 -everyone will ultimately be able to get what they want, but we've gently 
   1.133 -nudged everyone towards what we think is the best solution.
   1.134 +Dynamic API
   1.135 +================================================================================
   1.136 +Originally posted by Ryan at:
   1.137 +  https://plus.google.com/103391075724026391227/posts/TB8UfnDYu4U
   1.138 +
   1.139 +Background:
   1.140 +
   1.141 +- The Steam Runtime has (at least in theory) a really kick-ass build of SDL2, 
   1.142 +  but developers are shipping their own SDL2 with individual Steam games. 
   1.143 +  These games might stop getting updates, but a newer SDL2 might be needed later. 
   1.144 +  Certainly we'll always be fixing bugs in SDL, even if a new video target isn't 
   1.145 +  ever needed, and these fixes won't make it to a game shipping its own SDL.
   1.146 +- Even if we replace the SDL2 in those games with a compatible one, that is to 
   1.147 +  say, edit a developer's Steam depot (yuck!), there are developers that are 
   1.148 +  statically linking SDL2 that we can't do this for. We can't even force the 
   1.149 +  dynamic loader to ignore their SDL2 in this case, of course.
   1.150 +- If you don't ship an SDL2 with the game in some form, people that disabled the
   1.151 +  Steam Runtime, or just tried to run the game from the command line instead of 
   1.152 +  Steam might find themselves unable to run the game, due to a missing dependency.
   1.153 +- If you want to ship on non-Steam platforms like GOG or Humble Bundle, or target
   1.154 +  generic Linux boxes that may or may not have SDL2 installed, you have to ship 
   1.155 +  the library or risk a total failure to launch. So now, you might have to have 
   1.156 +  a non-Steam build plus a Steam build (that is, one with and one without SDL2 
   1.157 +  included), which is inconvenient if you could have had one universal build 
   1.158 +  that works everywhere.
   1.159 +- We like the zlib license, but the biggest complaint from the open source 
   1.160 +  community about the license change is the static linking. The LGPL forced this 
   1.161 +  as a legal, not technical issue, but zlib doesn't care. Even those that aren't
   1.162 +  concerned about the GNU freedoms found themselves solving the same problems: 
   1.163 +  swapping in a newer SDL to an older game often times can save the day. 
   1.164 +  Static linking stops this dead.
   1.165 +
   1.166 +So here's what we did:
   1.167 +
   1.168 +SDL now has, internally, a table of function pointers. So, this is what SDL_Init
   1.169 +now looks like:
   1.170 +
   1.171 +    UInt32 SDL_Init(Uint32 flags)
   1.172 +    {
   1.173 +        return jump_table.SDL_Init(flags);
   1.174 +    }
   1.175 +
   1.176 +Except that is all done with a bunch of macro magic so we don't have to maintain
   1.177 +every one of these.
   1.178 +
   1.179 +What is jump_table.SDL_init()? Eventually, that's a function pointer of the real
   1.180 +SDL_Init() that you've been calling all this time. But at startup, it looks more 
   1.181 +like this:
   1.182 +
   1.183 +    Uint32 SDL_Init_DEFAULT(Uint32 flags)
   1.184 +    {
   1.185 +        SDL_InitDynamicAPI();
   1.186 +        return jump_table.SDL_Init(flags);
   1.187 +    }
   1.188 +
   1.189 +SDL_InitDynamicAPI() fills in jump_table with all the actual SDL function 
   1.190 +pointers, which means that this _DEFAULT function never gets called again. 
   1.191 +First call to any SDL function sets the whole thing up.
   1.192 +
   1.193 +So you might be asking, what was the value in that? Isn't this what the operating
   1.194 +system's dynamic loader was supposed to do for us? Yes, but now we've got this 
   1.195 +level of indirection, we can do things like this:
   1.196 +
   1.197 +    export SDL_DYNAMIC_API=/my/actual/libSDL-2.0.so.0
   1.198 +    ./MyGameThatIsStaticallyLinkedToSDL2
   1.199 +
   1.200 +And now, this game that is staticallly linked to SDL, can still be overridden 
   1.201 +with a newer, or better, SDL. The statically linked one will only be used as 
   1.202 +far as calling into the jump table in this case. But in cases where no override
   1.203 +is desired, the statically linked version will provide its own jump table, 
   1.204 +and everyone is happy.
   1.205 +
   1.206 +So now:
   1.207 +- Developers can statically link SDL, and users can still replace it. 
   1.208 +  (We'd still rather you ship a shared library, though!)
   1.209 +- Developers can ship an SDL with their game, Valve can override it for, say, 
   1.210 +  new features on SteamOS, or distros can override it for their own needs, 
   1.211 +  but it'll also just work in the default case.
   1.212 +- Developers can ship the same package to everyone (Humble Bundle, GOG, etc), 
   1.213 +  and it'll do the right thing.
   1.214 +- End users (and Valve) can update a game's SDL in almost any case, 
   1.215 +  to keep abandoned games running on newer platforms.
   1.216 +- Everyone develops with SDL exactly as they have been doing all along. 
   1.217 +  Same headers, same ABI. Just get the latest version to enable this magic.
   1.218 +
   1.219 +
   1.220 +A little more about SDL_InitDynamicAPI():
   1.221 +
   1.222 +Internally, InitAPI does some locking to make sure everything waits until a 
   1.223 +single thread initializes everything (although even SDL_CreateThread() goes 
   1.224 +through here before spinning a thread, too), and then decides if it should use
   1.225 +an external SDL library. If not, it sets up the jump table using the current 
   1.226 +SDL's function pointers (which might be statically linked into a program, or in
   1.227 +a shared library of its own). If so, it loads that library and looks for and 
   1.228 +calls a single function:
   1.229 +
   1.230 +    SInt32 SDL_DYNAPI_entry(Uint32 version, void *table, Uint32 tablesize);
   1.231 +
   1.232 +That function takes a version number (more on that in a moment), the address of
   1.233 +the jump table, and the size, in bytes, of the table. 
   1.234 +Now, we've got policy here: this table's layout never changes; new stuff gets 
   1.235 +added to the end. Therefore SDL_DYNAPI_entry() knows that it can provide all 
   1.236 +the needed functions if tablesize <= sizeof its own jump table. If tablesize is
   1.237 +bigger (say, SDL 2.0.4 is trying to load SDL 2.0.3), then we know to abort, but
   1.238 +if it's smaller, we know we can provide the entire API that the caller needs.
   1.239 +
   1.240 +The version variable is a failsafe switch. 
   1.241 +Right now it's always 1. This number changes when there are major API changes 
   1.242 +(so we know if the tablesize might be smaller, or entries in it have changed). 
   1.243 +Right now SDL_DYNAPI_entry gives up if the version doesn't match, but it's not 
   1.244 +inconceivable to have a small dispatch library that only supplies this one 
   1.245 +function and loads different, otherwise-incompatible SDL libraries and has the
   1.246 +right one initialize the jump table based on the version. For something that 
   1.247 +must generically catch lots of different versions of SDL over time, like the 
   1.248 +Steam Client, this isn't a bad option.
   1.249 +
   1.250 +Finally, I'm sure some people are reading this and thinking,
   1.251 +"I don't want that overhead in my project!"  
   1.252 +To which I would point out that the extra function call through the jump table 
   1.253 +probably wouldn't even show up in a profile, but lucky you: this can all be 
   1.254 +disabled. You can build SDL without this if you absolutely must, but we would 
   1.255 +encourage you not to do that. However, on heavily locked down platforms like 
   1.256 +iOS, or maybe when debugging, it makes sense to disable it. The way this is
   1.257 +designed in SDL, you just have to change one #define, and the entire system 
   1.258 +vaporizes out, and SDL functions exactly like it always did. Most of it is 
   1.259 +macro magic, so the system is contained to one C file and a few headers. 
   1.260 +However, this is on by default and you have to edit a header file to turn it 
   1.261 +off. Our hopes is that if we make it easy to disable, but not too easy, 
   1.262 +everyone will ultimately be able to get what they want, but we've gently 
   1.263 +nudged everyone towards what we think is the best solution.