Wednesday, 24 August 2016

Story Time : Fantastic Rack Mount Mistakes #1

This is the first in a little series, which I'm afraid will have no nice pictures, but trust me, these are all true and if not first hand, at least second hand accounts from trusted sources...

You may or may not have seen a server room in your time, if not then they're basically rooms, generally locked, with at least air conditioning, generally a fire extinguisher nearby and all your company server assets inside.

When I started out in business the server room was a large cupboard with an air conditioner and the 386 based NCR mini-computer inside.  It also had set of aluminium shelves, but it had no rack mounting, everything stood on the floor.

This was true for me, for a few years, the IBM AS400's we moved onto were huge machines on the floor, aside from the colour you felt like you were walking into the set of "Flight of the Navigator"


However, I soon moved over to Compaq Proliant machines and these were rack mounted... I walked into the server room, and took a look at the rack, only to be greeted by the UPS battery pack being at the top of the rack... Yes, the batteries, full of acid, were at the top...

I came out and went to speak to the nice chap in charge, pointing out, as gently as I could, that the batteries may leak, or if there was a fire could be ruptured, and it was not unknown for batteries to get hot swell & burst...

He stared at me, blinking, "What's wrong with that, they're in a case".

And he was damn right, they were... a Plastic case, this particular battery pack had no metal sheeting, no casing other made of metal, just metal corner brackets, it was essentially plastic...And full of acid.

In story number 2, you will hear about the server room fire I'd dealt with the previous year.  But this chap didn't seem to register a problem with having the battery pack at the top of the rack.

I did however lodge my complaint in the service book, and went about my work.  Months passed, and I didn't often go to this site, we moved all the other company assets to rack mounted ethernet switches, a few smaller rack mounted Pentium II based machines and all new desktops.

When one day, a user came to my office door and said they could not get onto the central server, checking my desktop serial terminal sure enough nothing.  Checking a physical VT100 nothing... Oh no.

I call the other site, no answer...

I get a secretary to start calling them every five minutes and I start to check back through our logs, it seems from our end the link was active at 6:03am about two hours before business would ramp up during the day, and the last items were a set of data from the design department at a Derby based location.

So I call Derby, asking "Did your upload complete at 6am?" a little bit of silence and I finally get the right chap on the line, he'd come in very early to get a jump on the day and to the best of my recall he said "No, it suddenly stopped and I could not get back on".

I cut him off as the secretary suddenly pages me (yes, rocking the pager) and I pick up my internal phone to hear I've got the guy from the server site on the line, he's coughing... "What happened?" is my instant question.

"Halon was active when I arrived"...

This is pretty bad, the server room was protected by an automatic halon fire extinguisher, if it detects smoke or heat there is a 10 second alert, the door closes and halon is released... 

Remember the lab scene in Terminator 2?...


A lot like that!... Staff in the room, or entering, need to use a breathing apparatus, which we'd all been trained on.

This guy was short of breath because he'd been in the halon room.

He says that when he arrived (at 7am) the room was in close down, the building fire alarm was going and a fire engine was already present... The security staff had let them in and they'd used breathing kit to enter the server room, to find....  And this was the best he could describe:

"A smouldering brown mess of plastic encasing the top of the Proliant server, stuck in all the front vents and solid into the quick release drive catches, all the network wires above are only copper and the battery pack is black"

I dive in a company car and drive up there, I arrive as the fire brigade are happy it's electrically isolated, that the halon is clear, we've got a guy from the fire suppression system coming to strut his stuff, but in the mean time the company is offline.

So, I set up a 10Mbit ethernet cable from the main input multiplexor, set up a backup system on an AST dual core machine (literally, it was a PC with two 486 CPU sockets) and 16mb of ram... power!

This was to be the tide over machine, we cycled the nightly backup onto that and at around noon, people were working, if extremely slowly.  That machine was our stop gap for the next 48 hours.

So, now to the server room, initially the most obvious problem (because the rack had a glass door closed) was the ceiling and floor... The ceiling tiles were seemingly not flame proof... DOH.... They'd melted at least and there were droplets of melted whatever they were all over the place, blown around by the AC.

The AC itself had valiantly kept fanning the flames, so it had black soot marks neatly blown all down the wall.

And the floor directly around the cabinet base was charred, they were fibre carpet tiles on concrete, but had smouldered...

Our fire plan had fired, and the halon tanks were empty, both had fired, despite perhaps this only needing one... The vent out had worked and we'd used up one of our internal breathing kits, so two spares, one inside one outside.

The room is also electrically dead, nothing is on inside.  We had redundant feeds in from sites, one inside the server room and the one outside down the corridor which the AST machine was now connected to.

We're using torches, so I opened the front door of the server rack, and it's just black, the glass is coated in black, the once cream coloured Compaqs are black, the silver rack arms are black, going up the patch panel above is black but you can see the wires look a lot thinner, reaching up they're charred, the plastic has sloughed off of them and later when I have a step ladder I see it's mostly pooled onto the top of the first computer, but it's also seeped into and set within the vents on top and spilled over into the front of the machine.

Above this is the crispy remains of the UPS, there is a hole about 2 by 3 inches around 8 inches from the left of the case bottom, and I can see into the UPS... It's a charred vision of hell.

We take this UPS out of the rack, as it is the ONLY thing in there that had been put in by the local chap, the ONLY things not on the service list, the ONLY thing I had had an issue with.... And we pop the lid, one battery had gone, catastrophically, it had melted acid out, the batteries lay on their side and so the corrosive acid had eaten the rubber off the outside of the wires inside, shorted and an electrical fire had ensued.

The battery acid had then, xenomorph style, eaten out of the bottom of the case and dripped onto the ethernet patch cables, they'd shorted, got hot, dripped more, the acid had also cut through the plastic on the power cables, more electrical shorts... And the gates of hell had opened swallowing the nice working machine.


The clean up was epic, we lifted the servers out to the cleaner area of the IT room, we cleaned the outside and had a pair of Compaq chaps come and clean them... I think they were called Mark and Jeff, and they did an excellent job.

I ordered new cable and thousands of RJ45 connectors and a crimper, setting about rewiring the room, in a better manner.

And I set the guy in charge of the site, Mr "It'll do" to taking out all the racking, jet washing it, putting it back in, redoing the powerlines with a guy from the engineering department and I also had him remove all the carpet and the ceiling tiles, and find a supplier for a proper fireproof ceiling.

I also checked with the fire extinguisher guy, and he reckons that we were lucky the room was an hour outside of working hours, because the roof was just ceiling tiles and shout never had been fitted with this strange fire system... Despite his being from the same company!... Basically most all the halon had exited via the ceiling tiles!

Two twenty hour days after all this I had one clean Compaq server, one clean rack and enough ethernet connections to put one patch panel and the server back online, this let our users stop moaning about how slow things were.

The other Compaq, the one closest the mess, was going off to be re-cased, the damage was so bad, but the unit itself worked.

We also now had a new fire safe, as the original had proven to not be smoke proof!

And I had a new certification on the walls being fire proof.

It took three months to recover from this, my end report to the owners, was that it'd cost about £8,500.  Not a bad deal considering the whole rebuild of the whole room would have been £40K+

I'd instigated better carpet, better ceiling, better procedure, and I had the brand new branded, supported, service contracted UPS in the bottom of the rack.

The final wash down, was the UPS which had failed, was known to be bad, three weeks before it'd gone up, the site manager had had it out and replaced two of the four battery cells within with a pair he'd bought cheap, he'd done this himself, I was not even aware, and he'd had the bottle to charge the company twice for the batteries, pocketing the difference.

Embezzlement, bad attitude, bad procedures and simply his considering me below him, resulted in his dismissal soon after.

He was the first person I enjoyed firing, he was the first person I'd actually had to actively fire from a role for incompetence, and he was the last person I let dictate to me where kit in a rack sat.

All for the placement of a UPS in a rack...



(Paul, Andy, Max if you recognise this story, you know you were a big part of that clear up, but you were in London, Leicester and Morocco... I was up to my arm pits in that mess!)

Monday, 22 August 2016

Story Time : The £220 Commute

Today; at least the today at the time of writing this, not when I publish it; has been a troublesome day.  I got up very early, the plan being to get petrol and then go to the pony...


(Ours are Gerty in the black center, and Magic on the right)

Anyway, this plan was interrupted after about 50 feet.. I reversed off the drive, pulled forward and opened the windows to air the car out... And I heard an ominous flapping....

Pulling over, immediately, I found that I had a nail in the rear passenger tyre...


Yup, that's the sucker...

And this tyre was flat... I have a spare!... but... but... Looking at the rear drivers side tyre... it's flat too.. I don't have a picture of this, as I didn't want to stand in the road to take it, I'm in the UK, remember, so you can figure out that the drivers side would have been in the road.

So here's another wider shot of my flat passenger side...


It's only 6:45am, plenty of time to get this fixed before I'm late to work...

So, I give up even looking to change the tyre, there's some air in them, get to the tyre place!

Shit, it doesn't open until 8:30am, so I had the highlight of my day, I left the car on their forecourt and went for breakfast.  A long walk, collecting no Pok√©mon I might add, later and I had black coffee and a croissant.. Oh la la.

Walking back however and they were opening up, and they happily helped to get my car on the ramps before they officially opened.

Two punctures?... Nope... Unrepairable, so two new tyres, SHIT!  I'm skint, proper skinty poos... Credit Card to the rescue.

But then... "There's another puncture here mate, looks like a different nail type"... And it was a very different nail type... It must looked something like this before it went in.

So a large outer disc, with a hollow shaft and a much smaller inner disc, which had been deformed into a sort of blob when it'd gone through the rubber wall of my tyre, what had then happened was the tip pierced the first disc took a 3mm diameter of rubber then the bigger head had formed a half decent seal on the outside, so I only had a very slow puncture.

Touching this outer disc with a screwdriver the chap changing the tyre got it to fly across the room impressively pinging off the concrete floor; we'll say nothing about the health and safety worries this might cause some of you reading at home.

Anyway, the result was a third new tyre... So what the hell, in for a penny, in for Two hundred Pounds, do all four!

So, four mid-range tyres and an alignment, I'm two hundred and twenty pounds down but by golly I'm back on the commute, and seeking my desk, not least because I need another coffee and a sit down.


Saturday, 20 August 2016

Coming up later in 2016

To the many many people who stop by everyday, welcome and thank you!

To those of you who might hang around and revisit these pages I'd like to just go over a few items I'm looking at performing over the next few weeks.

Firstly, I have a big technical post about setting up Trac with Subversion, including securing it, hosting it on a virtual server and really getting you up and running with project tracking and management with Trac, from total scratch.

Secondly, I have a big of Dungeons and Dragons stuff to cover, both a bit about the official rules and the in-house rules we play with, and there will hopefully be a live stream with my niece whom is D&D crazy, and even made me a D&D based birthday card, which we will play through together.

Third, electronics, I'm probably not going to get to do the PS/2 connector project previously listed, but as the autumn winds in and and the winter comes across, I hope to get back to the DCC layout stuff, not least because I have a tonne of stuff laying around for it and two part soldered projects ready to go.

Fourth, a bit of computer history, I have a manual (just a manual) for a VAX VMS, I'd like to do some coverage of that for you all to geek out over.

Fifth and perhaps finally, gaming, I'm going to be looking back at Elite Dangerous I think, with the release and massive price tag on "No Mans Sky", I'm going to go back and see if Elite has done anything more interesting.  I've also been looking at buying Squad, the successor to "project reality" and going into that.  Our good friend Chaplain has invited us to play some Arma 3 time with him, and I'm playing World of Warships still.  So we'll go through all of that.

Sixth, so yes, fifth was not final, programming.  I'm looking at a project internal to myself, something I want to actually create, manage & sell.  So we'll see how that goes, but understandably there's not a lot I can share.  You will also find me touting myself for business at www.freelancer.co.uk, and as such I hope to make up some of the wads of cash we've been spending since the wife's surgery.

I'll speak to you all soon, I may start to make a mid-month post like this for you all to catch up on, but we'll see how things pan out.

In the mean time, the pony has a new home, and she has a new friend meet our Gerty and Magic once again:


Tuesday, 16 August 2016

Programming : SDL_Window* into std::shared_ptr

I've had a little bit of a brain fart this morning, so to save you all the trouble, I'll go through it... First of all, I am working on an isolated system, therefore I had no internet, no clues, and importantly no books... This is a drag, because mainly I'll check things out look things up and just make sure I get things right... When things go wrong.

Unfortunately, things went wrong, and it was very early (for me) and I was unable to see the problem.

extern "C"
{
    #include <SDL.h>
}
SDL_Window* theWindow = SDL_CreateWindow (
    "The Window Title,
    SDL_WINDOWPOS_UNDEFINED,
    SDL_WINDOWPOS_UNDEFINED,
    800,
    600,
    SDL_WINDOW_SHOWN);

This code will create an SDL window at a unspecified location, with a size of 800x600 and the imaginative title "The Window Title".

This is fine, however, I hate seeing raw pointers, that "theWindow" variable irks me, it's just wrong to see that in modern C++ code; even if I am using a C based library.

So, I wanted to set about wrapping that into a smart pointer:

#include <memory>
namespace Xelous
{
    using SDLWindowPtr = std::shared_ptr<SDL_Window>;
}

This is my wanted type, "SDLWindowPtr", I know what this means to me, and I wanted to have a creating function wrap this for me:


The trouble?... This is considered an incomplete type, when we pass "SDL_CreateWindow" to the smart pointer ctor, it reports "incomplete type".


You can see the full error above, click to zoom in on the image.

So, what have I done wrong?... Well, the brain fart is to forget that "SDL_Window*" does not contain a standard destructor, it actually needs to be passed to SDL_DestroyWindow to close it down & release all resources.

I had totally forgotten this detail, I was too focussing too much upon the creation step and not thinking about the window objects life cycle.  The result was about 20 minutes wasted time banding my head wondering why this was happening.

Two cups of coffee and a pint of water, plus a wander down the corridor later, and I had that moment of realisation about this.


And so my code evolved, giving the solution to create a nullptr smart pointer result, then reset it to use the newly created window and instruct the smart pointer to use the SDL_DestroyWindow call for the clean up.

The complete code for a user might be more like this:

#include <string>
#include <memory>
extern "C"
{
    #include <SDL.h>
}

using SDLWindowPtr = std::shared_ptr<SDL_Window>;

SDLWindowPtr CreateWindowPtr (
    const std::string& p_Title,
    const unsigned int& p_X,
    const unsigned int& p_Y,
    const unsigned int& p_Width,
    const unsigned int& p_Height)
{
    auto l_result = SDLWindowPtr(nullptr);
    l_result.reset (
        SDL_CreateWindow(
            p_Title.c_str(),
            p_X,
            p_Y,
            p_Width,
            p_Height,
            SDL_WINDOW_SHOWN),
        SDL_DestroyWindow);
    return l_result;
}

int main ()
{
    SDLWindowPtr theWindow = CreateWindowPtr(
        "Hello World",
        10,
        10,
        800,
        600);

    SDL_Delay(5000);
}

We don't need to remember to SDL_DestroyWindow, we don't need to worry about forgetting to clean up, the smart pointer will do it for us when the reference count drops to zero.


Monday, 15 August 2016

Programming : My Thoughts on Kivy

About a year ago, I had to work on Python, suddenly and without warning.  And though I had tinkered with python I neither listed it in my arsenal of programming languages nor thought myself conversant enough in it to deliver a product.  But the powers that be said "he's the linux guy, this is python, we don't want it on Windows, give it him".

So there I was, Python on the Linux server and Python on the Android phone... Two parts of the same project, and they needed a nice UI.

Now I don't know Qt at all, I've used it, never been terribly impressed (though that opinion is for another post and is highly subject to change) with the look of it.  So I set about looking for alternatives.

The alternative I found, which worked best for me, was Kivy.

I found the syntax a little awkward, and the actual creation of the recipe for the WI was very awkward at times, finding what inherited where being the main problem, it was a little.  Well it was a little counter intuitive to say the least, but it worked.

And the product went out there looking black and sleek and cool, and it was on everyone's Samsung smart phone at the launch meeting, and I was quite proud I'd managed it, and Kivy was the key in that delivery.

It has been a year since I created that application, a year of waiting for any feedback, it was a demanding time to get stuff floated and out to users.  I hope they liked it, I certainly did.

Android - Python - Kivy... Pretty much my go-to platform for Android development.

Office Life : Air Con

Someone remind me why this company dropped £35,000 on new air-conditioning earlier this year?... When on one of the few days of the year we need the damn thing, it's hotter in here than it was outside yesterday!


Yes, that's 25 Celsius in here... 77 Fahrenheit for those of you still in the 1970's.

This is warmer than Tahiti today, the same as in Cannes and hotter than it is outside (24 by the weather, 22 by my thermometer)...

It utterly sucks in here right now...


Sunday, 14 August 2016

What to do....

So, what do you do when your wife has had surgery for breast cancer, your mother-in-law has had a heart attack, the game you like to play is full of douche bags and you've got burned to a crisp by the sun?

Well, you play snake of course...


Nokia have a lot to answer for.