Wednesday, April 5, 2017

Sleeps Don't Scale

We've all done it - something races something else, we don't care if we lose the race, so we just throw a sleep in there and call it done for the day.

Unfortunately, as a system grows, all those buried sleeps start adding up against you. A large complex system simply cannot sustain sleeps, and there are three main reasons why not.

First: a sleep usually is a workaround for a race condition. As your system grows more complicated, the timing of that race condition will change - usually the window grows larger. However, in most cases, the duration of the sleep itself remains the same. A 10ms sleep that worked fine when you only had 5 threads is suddenly right on the edge when you have 20. It may break every time when you have a hundred (eek! But you know it happens.) Good luck finding the sleep that is suddenly causing your issues, you forgot about it months ago, cause everything was working fine.

Secondly, and this is strongly related to the first point: the sleeps will destabilize your code base. Little races that you didn't even know about will come up, because some other function that you had forgotten about and assumed was quick suddenly sits around for a while, giving your new code a chance to run and then suddenly pre-empting the middle of it. You know, when you sort of assumed your main loop was idle.

Finally, they kill performance. As your system scales up, all those sleeps start to add up. Take the example of a simple web service, for instance. It receives a message on a socket, passes it to a waiting worker, and gets the response back. Now imagine you had some dumb little race condition and threw in a 10ms sleep. When you're receiving 1 response a second, who cares about a 10ms sleep? Nobody! What happens when your service gets popular and needs to process 1000 packets a second? It's impossible - you need at least 10 seconds of sleep time to process that 1 seconds worth of traffic.

So what do you do? First off, you don't let them in there in the first place.

Oh sure, there are cases where you can let it slide. A standalone task running on its own thread with no interdependencies with the rest of your mainline code, sure, let it sleep.

Sleep might also be helpful for releasing your timeslice when you're done working, but a timer or signal would be better, since you will get more predictable wakeup times. For instance, if you want to process once every 10 milliseconds, you could do your task, then sleep for 10ms, then wake again. That will work. But if your work takes 1ms, then your cycle time that loop actually becomes 11ms. If you use a timer, you could include that work time in your 10ms cycle with minimal extra effort.

But in your mainline code? Anything that is processing on behalf of another system? Really bad idea. If you are sleeping just because you need to wait for something else to be done, you are far better off finding out what that other thing is and synchronizing with it than blindly sleeping. In long sequential tasks, a state machine approach might be better. Each cycle you can check if its time to work - if yes, do the work. If no, don't. This allows you to interleave the operation of multiple clients through that state machine, rather than blocking on the completion of each individual task.

And I'm bored. No punchline today. ;)

Monday, February 13, 2017

Object Oriented Programming - How We Got Here

While reading "The Mythical Man Month" (which I didn't know was about programming), I was struck by the number of valid points that this book, written to help manage software products in 1975 -- which are still ignored or blatently flaunted today.
I have a bigger project where I hope to distill the most important points I see from that. Later.
But the book I picked up is a later edition with a 1986 addendum titled "No Silver Bullet". And in this article is a section entitled "Object-Oriented Programming - Will a Brass Bullet Do?" And in that section, is a single paragraph that enlightened me entirely as to how we got where we are with Object Oriented Programming.
I'd like to comment on that paragraph.
Now this was written in the infancy of the very concept of Object Oriented Programming, and it's musing about why the concept had not yet caught on as well as the author thought it should. And so he attempts to describe some of the goals of Object Oriented Programming. And it's this one paragraph summarizing "one view" that I can see the germs of thought that have today mutated into rampaging plagues across most of software development. Seriously, I used to wonder.
One view of object-oriented programming is that it is a discipline that enforces modularity and clean interfaces. A second view emphasizes encapsulation, the fact that one cannot see, much less design, the inner structure of the pieces. Another view emphasizes inheritance, with its concomitant hierarchical structure of classes, with virtual functions. Yet another view emphasizes strong abstract data-typing, with its assurance that a particular data-type will be manipulated only by operations proper to it.
Every one of those features is actually pretty good in the original intent - that is - that it is used where it makes sense. The problem is that these guidelines have been mutated in many programs into absolute laws. You absolutely may not access data inside another class. You don't need to see how a class was written, let alone have the right to modify it. Everything is inherited from something else - whether it makes any sense or not (the number of times I've had a basic data type with an inheritance chain six or more classes deep is no longer amusing to me, but rather depressing). And I've literally worked on a project where I was not allowed to store public data in a central database because the database, which existed in the software already, didn't support strong data typing. That was the reason.
The point of Object Oriented Programming was to make it faster and easier to develop pieces of software and bring those pieces together.
Modularity exists so that a component can be developed and tested in isolation. It makes no sense whatever to make a class modular if you still need other classes to make it work. That's not modular anymore, and you probably should consider whether those should be merged into one object, rather than an incestuous mess. And for what it's worth, bool is not a modular class. Don't wrap bool.
Encapsulation is a tricky one to grasp. It's stated so plainly - one cannot see the inner structure of the pieces. But good encapsulation requires two things: a good design and enough runtime to prove that the design actually is good. If you enforce encapsulation to the point of "nobody looks at the code and therefore nobody can change the code" from day one, all that will happen is you will end up with workarounds for missing, obtuse, or broken functionality. Worse, you'll probably try to code for every conceivable case, most of which aren't what people actually want to use, in hopes no changes will be needed. The project will be more complicated and less stable. I've seen people enforce this rule to the point where they are doing this with their own objects. Encapsulation is for stable code, not development code. And you don't need to encapsulate bool. Don't wrap bool.
Inheritance is one of the most powerful features of Object-Oriented Programming and frankly, one of the few features I actually really like. But you inherit where it makes sense. In most cases your inheritance chain should not be any more complicated than the example in most text books -- that being a base class extended to one level. In rare cases you may need two levels for certain objects (but certainly not all of them) and in equally rare cases it may make sense to have multiple inheritance (but certainly not all of them). Good planning goes a long way here. Going nuts with inheritance leads to complicated, incestuous code that is difficult to debug, difficult to modify (without breaking something else), difficult to implement and difficult to document. It's also poorly performing in many cases and in cases where it's not, harder to predict what the code will do. You don't need to start with basic classes like a wrapper around bool and inherit from there. Don't wrap bool.
Strong Abstract Data-Typing was meant to get away from the admittedly sloppy practice of casting objects in C and hoping you got it right. This feature alone is a good reason to port C code to C++, even if nothing else changes (you'll be surprised where you screwed up but it worked anyway ;) ). But it doesn't mean you need to wrap every type of data you want to use in a custom object just so the data-typing will protect your function calls. (In fact, in many cases passing different types of data around is a better job for classes with a common base class and utilizing inheritance...). But simply put, if you have several true or false items, you don't need to wrap bool in different classes to make sure you pass the right kind of bool to the right function. Bool is a bool. Don't wrap bool.
That's all I really wanted to say. I learned a bit about when modern programming missed the left turn in Albuquerque. It was roughly thirty-one years ago. We have GPS now, let's figure out where North is and start getting back on track.

Monday, December 19, 2016

C++ "MetaProgramming" and Why C++ Should Die

I saw a pretty awesome accomplishment on Twitter today - this fellow write a little raytracer that does all its calculations at compile time.
Pretty impressive, even if it takes a while. It's all done with templates and "metaprogramming", which is a fancy term used to excuse the complexity of programming both the computer AND the compiler.
No, I don't like it. And it's a great example of why C++ is done and needs to die.
I've loved C++ for a long time. I've encouraged friends learning in University without realizing the true pain they were experiencing. You see, I've been blissfully ignorant of how far things had gone for a long time. For the most part I ignored C++11 and C++14, until a recent project forced me into the deep end, and I got the O'Reilly book out and started reading.
I was pretty horrified, in general, but we'll focus on this particular aspect.
So the article above is about this fellow learning these new concepts with an ambitious and fairly impressive task, inspired more or less by this example:
Take this example:
template<int base,int power>
struct Pow
const static int result = Pow<base,power-1>::result*base;
template<int base>
struct Pow<base,0>
const static int result = 1;
int main(int argc,char *argv[])
return Pow<5,2>::result;
If we look into the assembly file produced we just see the constant 25 being written to a register.
mov eax, 25
Your formatting is crap, Blogger.
Anyway, so what happens there is that main() invokes a template (Pow<5,2>), which generates a recursive chain of structures, each containing the next power, until the power is zero and the final template is invoked. The compiler runs through all this, and the final result is that single assembly instruction that generates a single const value "25" (5^2 = 25).
Fans of this style of programming point at the amazing efficiency of this resulting code as a major win. It's so much faster, they will tell you, than running the code the old fashioned way. But I call bullshit. Because in "the old way" we wouldn't have done that anyway. Not if performance mattered. Do you know what'd we do? Hell, this goes all the way back to "C", not even one plus!
const static int result = 25;
"Oh! Oh! Oh!" cries the peanut gallery. "But the computer didn't calculate that for you!!"
Of course it did. We did it offline. Or we did it in a separate program. Or we used a calculator. Or we did it at startup and cached the value. Or in the worst case, maybe we used a code generator. (Deliberately ignoring the fact that this example is very simple and didn't need code at all).
But! Isn't using a code generator for the most complex cases exactly what we did here? It's just built into the language now, isn't it?
Well, yes and no.
Yes, you essentially used a code generator to calculate the problem and reduce the code to the important single constant. But this is about the most complicated, difficult to debug way of doing it that I could have imagined.
First of all, you just littered your namespace with three different Pow structures. You didn't need the other two, but the compiler did, and they exist. It was a lot more expensive for the compiler to calculate all other structures and then decide what was really needed than just about any other technique would have been, which means your compile time is increased (substantially, in fact, depending on how many Pow's you need and how deep they have to go!) And suppose you typoed in the base,0 template? Well, then your error code is going to have to reflect the entire chain. In this case, it's a short chain of just three entities, and the error is a single line per entity, since it's very simple.
$ gcc test.cpp -otest
test.cpp:10: error: `into' does not name a type
test.cpp: In instantiation of `Pow<5, 1>':
test.cpp:4:   instantiated from `Pow<5, 2>'
test.cpp:15:   instantiated from here
test.cpp:4: error: `result' is not a member of `Pow<5, 0>'
But real life templates tend not to be so simple. And because of the nature of the templates, you can trigger errors simply by specifying the wrong type of argument to a parameter (for instance, forgetting to std::move can break some templates). The result can be pages of template chain errors, making troubleshooting difficult. And indeed, that is what our experimenter found:
This was the first time I had tinkered with metaprogramming so it was pretty hard at first, basically once one template evaluation fails you get a huge chain of thousands of failures.
The entire direction of the language's development seems to have shifted towards programming the compiler to generate constants for you at compile time. This is a good thing and we've often done it in the past, but it's not ALWAYS the right answer, and today it's being taken to ridiculous extremes. Some of the things I've seen mean that the compile-time ray tracer isn't even that outrageous to me, I've seen things attempted that feel on the same level of complexity. And I don't believe that we should be doing that.
Why not?
Well, this impacts you in several ways:
-compile time is longer. How many of those complicated template chains result in the single constant that the example above shows? You've built the code hundreds of times and never changed that value, have you? Make it a damned constant and save the time.
-typos in the code are MUCH harder to understand. If you've done std or boost template programming, you already know what I mean. If you haven't yet, you will. If you're a god who never makes mistakes, go back to cartoon land. This costs time - a simple typo that may be as simple as a missing modifier goes from a 10 second change to a minute or more, just to determine what line the error actually occurred on. I know people who switch to a different, non-production compiler for testing, just because the error messages are less verbose (meaning an entire compile phase is wasted). This time adds up substantially.
-learning time is longer. If you're using your own complex template chains (in addition to std, boost, or other common ones), then you have a larger and deeper codebase for a new developer to have to come to terms with -- and the two issues above are not going to help with that. Since most developers on most projects are thrown in with little more than an incomplete wiki and a promise to get around to guidance, you'd think that simple, easy to follow code would have some value.
I'm reminded of an old quote by DadHacker (
The future of computing is its own past, mashed-up and remixed by young'uns who have yet to fear the dark corners, the places where us old farts went in with similar bushy-tailed attitudes and came out with ashen-faced, eyes barn-door wide and with fifty new words for "pucker." Heed us. The stove is hot if you touch it. The stove is not only hot, it will incinerate your soul. At some point you will want to make pancakes or wash dishes for a living rather than run another build or merge another check-in or fix another bug...

Wednesday, November 23, 2016

Rock Band Guitar Overdrive Update

I'd been having some trouble with Overdrive on my very abused Rock Band guitars -- to the point where one of them (an original RB1 unit) all but quit working altogether.
I took it apart to see what could be done, and was surprised to see that the tilt sensor actually used little metal balls in a can (this is why the guitars rattle when you shake them). It used two of the sensors wired in series, probably to better filter false positives caused by vibration.
Sensors used were similar to this:
Some testing suggested that one of the sensors was barely responding at all anymore, so to get it going, I shorted one of the sensors out, so that only the other one was needed to trigger. This only sort of worked and got us through the evening.
I ordered a set of mercury tilt switches to replace them. I got a set of 10 little ones from Amazon for $6, so I could install two in each of my guitars.
Since this is the internet, obligatory warning. Mercury is a toxic metal that can be absorbed. If you break one of these, it will be hard to safely clean up. Don't bother if you don't know why that matters.
I had two guitars I needed to update - a Rock Band 1 and a Rock Band 2 -- there are obvious external differences, and a number of internal ones. I'm only interested in the tilt switches here. In this case, Rock Band 2 upgraded the tilt sensor. It still uses the ball bearing type of sensor, but it used two larger sensors (heavier balls), and it wired them in parallel instead of series (so that EITHER switch could trigger it). It then went one further than that and added a port to the side of the guitar so you could plug in an external foot switch for overdrive - this is also wired in parallel.
Testing showed that the ball-based tilt sensors worked, but the connection was iffy. It was bouncy and imperfect. The mercury tilt switch, by comparison, works by immersing two contacts in conductive liquid metal. The connection, compared to the ball switches, was pretty much perfect and very low resistance with no bounce.
So for the Rock Band 1 guitar, it was a straight remove-and-replace. With the switches being so much better I wanted to keep the series connection so that just shaking or bouncing the guitar was less likely to generate an accidental overdrive. I then bent the leads to get the approximate angle I wanted the switches to trip at.
For Rock Band 2's guitar, the switches were wired in parallel. Again, Because I liked the idea of the series connection providing resistance against false triggers, I tied two of the leads together and wired that up, after insulating the PCB to prevent shorts. This gave me a series connection on that guitar as well.

When I went to install them into the guitar, I hit another small snag... the normal orientation of the guitar meant that the switches lay flat instead of tilting, which made them trigger too easily.
Fortunately, the boards mount by means of a slot and are held in with screws with a very wide head. Friction meant I could just lay the board flat on the mount and screw it into place like that -- this worked fine.
And there we go! Hooked it all up and it seems to be working just fine! I probably should have replaced the reed switches while I was in there, but we'll do that next time. ;)

Sunday, November 6, 2016

The Programmer is not an end user

As I continue to modernize my skillset into C++11 and C++14, as well as pick up side toys like Unity, I'm more and more noticing a really disturbing trend. It took me a while to figure out what it was that was bugging me so much, and tonight I finally realized - the programmer is being treated like an end-user.

Phrases like "you don't need to know about that" and "don't worry about that" are backed up with massively complex templates that hide the actual behavior of an object. I've been going through the O-Reilly book on C++11/14, and the number of times it warns that the same line of code can have drastically different effects due to context (because of the complex templates and code features backing them) has exceeded my capacity to remember. Even the simple code editor is getting into the game with concepts like code folding, whose sole job it is to hide code not currently being worked on from the programmer doing the work.

Why? Is it really that distracting? (Actually the conspiracy theorist in me suspects it was first done so that really long files could be managed in really poorly written editors... but that's a different rant!)

The programmer is the first and often the ONLY person who can view the code and tell you what it's going to do. He or she is your first AND last line of defense. Why in the name of all things Boolean would you hide details from that person and leave them in the state of "I don't know"?

It was code collapsing that triggered me to write tonight. I've known about it forever, I even know people who use it. I chose not to, because I prefer to understand what I'm working on. But only tonight did I realize it was probably the simplest and most insidious of this deliberate attempt to "dumb down" the act of writing code, and why it was bad.

So why is that? Even if you are not working on a piece of code, the act of reading through it as you skim past can catch bugs. This has happened to me many times - especially in group projects. You see it, it catches your eye, you go "oh my god!" and you fix it, BEFORE the customer finds it.

Or, you collapse all the code except the one little function you've chosen to put your blinders onto, and the bug goes unnoticed until it takes out the internet because nobody ever bothered to look at it, despite hundreds of eyes passing over that function. Yeah, tell me that's never happened. ;)

People: LOOK AT THE CODE. That's your JOB as a programmer, for pete's sake. Keep things simple enough to actually understand, and TEST that code. Don't say "oh, that's Test Group's problem". No, Test Group's job is black box testing - to make sure the ultimate product as a whole passes a reasonable set of tests and does what is REQUIRED. YOUR job, as programmer, is to test every code path you write to ensure it does what you INTENDED. The two are not always the same, that's why you HAVE a different group.

Rant rant rant...