Wednesday, 22 February 2017

Development : Scrum & Leveraging Virtualisation (My Hobby Team)

Recently in one of the Scrum teams I oversee as their Scrum Master, I saw a common problem, one or two members of the team were being blocked by mundane issues once or twice a week.  For one member of the team this was a physical PC fault, for the other it was disk-space.

They are running their own equipment, outside the remit of my authority or the company control, therefore I had to clear the decks in some manner, the solution?

Well, the development was all Linux, I created a standard Ubuntu image and issued it to all concerned and asked that these be the ONLY virtual machines run to complete work.

The first week of this resulted in a slow time, I found people were not using; as per the original specification; virtual machines at all, I found they had a mish-mash of development materials, and importantly nearly every person asked me to allow them rights to install software.

I didn't allow this, instead I used the Scrum management software I've put together and collected all their ideas about editors, tools and software to add to the image, and we had an additional 10 minute open discussion going around the group to assess the merits of each piece of software, what it would be used for and how it might leverage either an easier time to task completion route or how it might introduce potential benefits in other ways.

I started with the proposer of the software, then alternated between those in the group whom had indicated they liked the software option and those that had not, in cased where no-one objected to a piece of software, I simply took the concensus benefits and added them as points below each item listed.

In the end we had reduced the pile of suggestions to three, the first a lightweight text editor, many were split between "just using nano, pico or vi" and those "desparate to use sublime".  This came down to a simple choice, one was free, one was not, we have a sub-zero budget since having to source those drive caddys last month... Everyone was going to use my preference "nano".

The second was a performance measuring tool option, to monitor the system, I opted to leave this decision to the testers in the group they have yet to advocate a choice, but I decided to take the option out of the scrum teams scope of interest, their code has to be fast and that's the end of that discussion.

The third was an art package, there were many options, one person wanted to do all their art in Paint.NET on a windows machine then move it into the main project, another wanted to create assets with GIMP and another wanted to use Photoshop. on his Mac, an easy decision GIMP.  It's native to Linux and free.

The standards were set, we would use tar and gzip on the image I had issued, Nano, GIMP, Code::Blocks and the compiler was the GCC for the group.

For diskspace, problems, every individual member was issued a brand new 1TB drive, the Ubuntu Image and they had to host this virtual machine however they wanted, it needed a single CPU core and 1GB of RAM minimum.  Many of the developers were able to tweak and give the machine 4 cores and 4gb of RAM.

Since then the team has run a lot more smoothly, issues with your software, pull the VM image down again... Issues with the source tree?  Delete your local working copy and resync with the repo server!  Need a place holder image?  Fire up GIMP.

This was the first time, since the New Years, that I had reviewed the team in any depth, as a Scrum master in this situation I don't have a lot of blocking or shielding duty, individuals work in their own time and in their own homes, so they have to deliver or fall short.  It is a hobby project after all.

However, our first Sprint feedback is in from yesterday evening, the overwhelming feel is that the team is now working much more quickly, they are verging on out stripping all previous schedule expectations (which were secret from them) and yet they all feel this is easy gliding development time, moving from product point to product point without the hassle of their machine or not being sure how to run an item, or not having the same set up.

My next task?  To do the same with the tester group... What a bag of cats they might be....

Saturday, 18 February 2017

There's a Rat in me Shed, What am I gunna do?

We've had a rat come visit our shed...

Well, when I say paid a visit, I mean he has dug a complex trench system for the war which I am now perusing against him...


This was beneath a patio, and somewhat is a surprise... The first signs of a visitor were these droppings in the shed where the guinea pigs live...


I spotted these and immediately knew they were rat... The curled one on the right is a classic shape for rat droppings...

Anyway, I then decided to take a look around, and I found a black smudge and some trailed mud, in the back of the shed where no-one could go, so it was obvious there was a visitor, I moved everything out of the shed and found a neat hole cut into the back wall... Bastards!

I baited this and nailed it up, then placed more bait inside, after I had jeyes fluid cleaned the whole area and swept up.  As you can see in the second picture there was a lot of dropped matter, bedding and bits from the guinea pigs, but there was also hay from the stables in there and horse feed... Bad place to store them in the bag, rather than dumped into the hard plastic bins we have.

Anyway, inside secure, I set about looking outside, and saw what I thought was a run behind the shed in line with the hole, so I baited that and blocked the main exit into the garden with a tall slab.

I also saw this darker area of soil... Now we have a rabbit who lives free in the garden most of the day, and I figured she had just been moving earth around, as rabbits tend to do... Maybe for a fortnight I've seen this earth moved each evening, and assumed it was the rabbit.

But, it wasn't, seems the clever ratties decided it was a bore to go all the way around the shed, and they cut the above tunnels beneath a storage box... I lifted this and found the tunnels....

I baited the one nearest the shed, blocked another with yet another slab on end, filled in the tunnels then laid new slabs on them and placed the storage box back on this new footing.

The detritus in the middle of that run is all stuff from the shed, mainly guinea pig feed... So we've clearly been spilling it... In fact I know I have, the perfect invite for these little bastards.

Anyway, poison down, earth moved, slabs laid and gnawed shed patched up... The guinea pigs however are spending the night stood under three layers of blankets in the garden as I put so much cleaning fluid down the shed hasn't dried yet.


Thursday, 16 February 2017

People : Co-Workers We Remember

Today everyone has some form of social media, at least in this business, LinkedIn, Facebook, twitter or this blog!  They're all examples of being able to keep in contact with people out there, either bi-directionally in the case of LinkedIn & Facebook, or uni-directional in the case of Twitter and this blog.

I like the latter, I like people to be able to find me; if they know me, or just discover me.  I recently had a message from a chap involved in my first Rack Mount Mistakes posts.  His name is Stuart, and he was a very nice chap, he helped with the clean up and was my second when we had to confront and appoint the blame for the debacle on the chap responsible.

He contacted me to say "hey, I remember this"... Which is exactly what this blog is about, but I had to tell him I didn't clearly remember him...

After a little discussion we came to a conclusion, he found me to be a pain in the bum, fun and good at my job, but I was young, much younger than him, but in charge; which he didn't like.  I was early twenties, he was early thirties.  Now he's entered his fifties he looks back on those days with fondness and even said "I thought back then I had the nous to do your job, that you were pushy, but I know now I could never had been like that with people, never told them what to do and get those jobs done when I was that age".

Two things struck me about this, firstly that he was complimentary, but also I realised why I didn't remember him clearly.  He was one of those rank and file kind of guys he did his job, he fitted in, and he got the task done.

Because he never rose above the surface level he was never a nail head needing me to drop on him like a hammer...

To my fault this made me place him in my "of least concern" perimeter, so as time has gone on I've forgotten that he was diligent, forgotten that he was a good developer, forgotten even his surname.  This is good and bad.

However, taking this tail of thinking further, I realised I remembered lots of the bad ones... The bad egg, the nail which poked up and needed hammering into place, the ones who didn't get on with their jobs, the ones who disappeared from their desks for hours on end, or who missed deadlines, even some whom even today I chase around bad code they've left in their wake.

I don't contact either group of ex-co-workers, very few of them can directly contact me, perhaps this is good for the bad ones, but it's certainly not ideal for all those good ones I've met and worked with over the years....

Andy S
Max B
Paul D
Leon B
Dave P
Ravi S
Steve W
Tony
Andy K
Richard (Dad!)

And the valiant few I still work with and like to work with where I sit today, I salute you all.... Anyone not on the list... I either forgot about you or I don't like you; you decide which!


Tuesday, 14 February 2017

Programming : Python MySQL Connector Debug

Today, I was asked to look at a server for a friend, their problem... "It just stops working after a few days"... A few days turned into "between three and five".  Doing some mathematics I found they had between 125 and 350 unique visits to the server, each unique visit represents one customer or one remote unit of their fleet.

They relay their data from these to individual database instances on one MySQL Server, so there is about 30 customers each with many unique databases.

The problem?... Well, I find this very distressing, as they open one connection for each arriving remote client, use it and then they closed it... Right... RIGHT?!??!!

import mysql.connector

l_total = 0
while (True):
    # Count
    l_total += 1
    l_res = l_total % 100
    if l_res == 0:
        print (l_total)

    # Open a connection
    con = mysql.connector.connect(user='root', password='***', host='localhost', database='Pickles')
    cursor = con.cursor()

    # Query
    query = ("SELECT * FROM VeggiePatch")
    cursor.execute(query)

    # Retrieve the data
    cursor.fetchall()

    # Close the query cursor
    cursor.close()

    # Close the Connection
    con.close()

This is my test code based on the way their production code works, as having read the error log I see the problem is in the connector constructor and delves down into the networking code.

This of course crashes after around 33,000 cycles.

They're not willing to change their script "willy-nilly", I in fact think they're petrified I've found this problem.  Googling around I don't find any official explanation of this error, only anecdotal forum posts about the MySQL Connector not cleaning up after itself and so reusing the sockets fails over time.



The better solution is to garbage collect the connection each cycle...

import mysql.connector
import gc

l_total = 0
while (True):
    # Count
    l_total += 1
    l_res = l_total % 100
    if l_res == 0:
        print (l_total)

    # Open a connection
    con = mysql.connector.connect(user='root', password='***', host='localhost', database='Pickles')
    cursor = con.cursor()

    # Query
    query = ("SELECT * FROM Tickets")
    cursor.execute(query)

    # Retrieve the data
    cursor.fetchall()

    # Close the query cursor
    cursor.close()

    # Close the Connection
    con.close()
    con = None

    gc.collect()



I also tried to garbage collect each time I printed the the "total", each 100 passes, but this still crashed, the fixed loop here has so far done just under half a million cycles without issue....


Sunday, 12 February 2017

People : Do you Talk the Talk?

Having recently been speaking to some technical and none-technical people, both relation to staff I need, and indeed myself looking outwardly, I have come to the conclusion most people interviewing, in the technology sphere, today sadly fall into two general categories...

1. Those who talk the talk 
2. Those who don't...

What is the talk?  Well, as a general technologist I don't talk in specifics, doing so I've always found overwhelms the listening.  For example, "Network Connection"... This is perfectly sufficient to communicate to 99.9% of listeners what I mean, I can draw it on a piece of paper, make it an arrow or communicate it's meaning very simply.

However, I was asked (last week) "what so you mean by connection"... The Socket... The Client-Server connection... The handle to the I/O buffer within my application to the TCP/IP Stack on the machine?!?!?

None of my replies seemed to mollify them.  Knowing this was clearly my not using the correct form of incantation from the necronomicon of buzzwords, I asked them what they thought I meant... Their reply stunned me, a little...

"I mean the post stream packet set from after the first packet, containing the Syn flag, and the following series of packets over the TCP Pipe".

I think I actually laughed, asking them, if that's what they really meant to ask as they were testing me, or that's how they seriously expected me to express myself... They didn't have an answer to this.

"Network connection" was perfectly adequate to express the object of our mutual attention, but clearly this person had no wiggle room, no faith perhaps that they were not talking to a moron; despite their apparently having read my humble blog pages here.

So, where did this confusion come from?  I think, putting it simply, I don't want to compartmentalise technology, I don't want to say "this is the interface", "this is the control", "this is the communication" and only one person or one team be involved into that one area.  For a development team, or a single project, this can lead to almost an incestuous proclivity to sharing information with others.

Rather, I see technology as a huge ecology within which we all live and work, move a team member to other libraries, or other parts of a project now and then, if they're not confident in the graphics slowly make them a junior member of graphics, if they are a strong outward thinking communications body, let them lead communications but have them chaperone those less confident with it through the same code.  Share tools, and time and attention on one another.

This means perhaps means I don't talk "the" talk, I never jump to nuts and bolts, it's not how I want to communicate technologically, and was never how one communicated when teaching Karate, you had to build one another up to the same level of understanding, both at a personal and a working level.

So, I will never talk about the raw metal, or the raw software libraries, instead I'll talk about "building the Exchange layer", I won't talk about "the CCTalk byte code interface" I'll talk about the hardware abstraction to simplify all these calls, and give it a name, such as the namespace within the code, so those driven and interested can go take a look at it.

If you take a look around this blog you will see the hundred of posts, they do explain an awful lot of information, sometimes they contain my opinion; I think today they're more professional looking, and perhaps are for a niche audience, but they will contain a lot of technical information... They don't contain "the talk"... But over 300 people read these pages every day, and most of the feedback I receive is about how clear the tutorials are, or how clearly I've expressed something about development, which was previously hidden in layers of this "talk".

If you speak to me about technology, I will want know what you are talking about, rather than buzz words, acronyms, or specifics.  I don't know what you are working on, it could be PDF file generation from Geographic data, it could be time and attendance clocking systems, it could be automotive displays.  The unifying thing isn't the minutia, it's being able to be generic.

Is this my problem?

Well, no, this is my blog and my little corner of the internet so I can pretty much say exactly what I like... I make you welcome to talk to me in the comments below, and I'm pretty sure with the hundreds of comments and hundreds of thousands of visitors we've each understood each other.

My question then is, why do some IT related folks, when they invite you into their little corner of the planet, why do they not understand me or even one another?

They seem to jump immediately to a check list of words they desperately need, to hear, they don't listen, they don't engage, they (I'm pretty damn sure) just want to follow their script rote to complete their task.  Where as I want to speak to people who are as enabled and interested in technology as I am, without having to experience ten years working in a room with them before we synergy.

Likewise, I'd like to think presented with all these pages, knowing they were going to speak to me, that they would do me the courtesy of visiting this; my humble corner of the inter-webs; before asking me to engage in their merry dance around the knives of indifference.

Friday, 10 February 2017

Programming : Obscure Languages I Learned

This year marks the twenty fifth anniversary of my learning a proper programming language (that is a language which was not BBC or Commodore BASIC, but required a compiler or interpreter and a code file to run).

This is longer than two or my team have been alive... Which blows them away, but which I take as quite worrying.  I worry as I wonder if my skills are out of date, despite my regular passed through modern papers & books on new technica... This is related to my "Talk the Talk" post scheduled for Sunday, so come back to that.... However, back to the here and now...

During my time as a programmer I have learned, mostly for academic or edge case scenarios, some strange languages, here are my top three strange, annoying or otherwise pointless languages...


3.  Haskell
Controversial to come, but I hate Haskell, it was to be learned as part of a  course module on "Functional Programming" during my degree; the difficulty being I was very much in a different linear mind-set for my then main language (Pascal) and learning to apply the Object Orientated Paradigm with C++ and Java.

I didn't like the language, found the interpreter to be very slow, and there seemed to be no clear connection between the book we had been set (which I still have somewhere) and the version of the language the course defined interpreter required us to use.

The task assigned was to quad-tree compress an image, I don't remember finishing, but have created image processing tools since with C++ and the CImg library.


2. Occam
Set as part of a course on parallel processing during my degree Occam was meant to teach us how to strip a process down to it's bare minimum and then run this in parallel with another process.  The problem?  The machines we had at hand to perform the course were old 386 PC's, whilst the Occam compiler was for the Motorola 68000, the solution... Cross compile and upload the Occam into a 68K daughter-board which was perched on top of the PC case tethered from flying off by a flaky ribbon cable.

The up shot was more time spent in the sweltering hot room, fanning these over heated, junks clunky boards than actually cutting code or learning anything... Perhaps I should revisit Occam?

If only the lecturer had let us use the just released Linux OS and pthreads!?!?!


1. Z
No, not Z++, Z... A formal description language!?!? I hear you cry, but no I have a book titled "The Z Programming Language" and we had the honour of trying to decipher this system description language into a series of function testing calls.  Not a true programming language per say, but still a horrible, horrible thing to behold.

I just have to mention "Z" to any of my peers from that same Uni class and they shudder, not least as we learned the following year would not study "Z" at all... Again, I still have the book, I may take a re-read, but for now Z languishes as the most unloved language I ever had to touch; really, I much rather had used FoxPro and SSADM, to that's how much I hated Z.



Coming up, my exploration of using the "R" language for data processing, in regards to which I have just had a massive book!  Yay.

Wednesday, 8 February 2017

Development : The Art of Fungibility

Fungibility is about being able to substitute one thing for another seamlessly, let us say you have a paint sprayer which contains red paint today, but tomorrow you want to paint the wall blue, you simply swap the paint... The unit remains unchanged... The paint is fungiable.

When it comes to software, especially in the past, it was of paramount importance to write your code to conform to the system, you had to write code which was compiled for one processor only, you had to conform to the calling conventions of the underlying architecture and most importantly the OS... Software was generally not very portable.

High level languages, like C, were created to get over this problem and indeed the Unix system as created by Dennis Richie could be recompiled from source on various platforms, however, things were sill very specific and the "portability" was the case of recompiling and overcoming the localisation quirks of each underlying platform.

However, with the advent of portable interpreted languages like Java or Python, and manageable tool-chain assistants like Docker and Kivy you can write your code once and have it move between systems without effort.  The first of these which I encountered really was Java, I had this run on both a Sun Sparc station and a PC, and when I wrote my degree dissertation (Parallel Computing in an Open Environment) in order to open up that platform the interop module was all Java, running on a Windows PC or any other Java 1.1.4 (AWP) functional system.

Little did I understand, at undergraduate level, how important a trend this cross platform context for an application would become.  Smart Phones, Tablets, Consoles, various PC Hardware, Linux, Windows... At one point I was even writing work in Personal Pascal on the Atari ST, then porting it to Turbo Pascal on the PC for profit, when I didn't own a PC myself, so one had to be careful and thoughtful.  There were not so many avenues for a system to truly be the same on one platform or the next.

So, as I learned to program you had to sit down with a pencil & paper, with a process flow chart stencil and work out how you wanted to lay the system in order.  To order what was being written when.

This lead to bring frugal with your time, and ultimately lead me to see fungiability as a key component of my way of working.  Because it was such hard work to cover all the bases and to make your code truly portable across machines.

As great as Java was, and python is now, or technology like Docker & Kivy are though, I believe they are pushing fungiability into decline, sometimes with it not even being considered, take my post of a few weeks prior, replacing an older system with a new, I simply looked at the API being exposed, boiled it down to a few function calls then re-implemented them and the system came back up.  Replacing the code inside the functions was hard work, but the API swapped over for me.

I've since learned that other, younger, developers would simply have re-written all the code, they'd not have cut out what wasn't needed; because they have the tools, they have the machine power (ram and processor) to support continuing with the added load, they don't pair things down, they don't binary chop what's not required, they only create and ever expanding gaggle of code (If you are young and don't do this, please don't think I'm painting all with the same brush!)

So as you start to consider scale-ability, on demand computing and cloud computing, keep fungibility in mind, think and design like for like replacement of systems, can you swap the service back end from an instance of Ubuntu on AWS to running on your local XEN host?  Can your developers in Canada send their data to dedicated local machine today, but deploy to a virtual instance hosted by yourself in the UK tomorrow?

Consider your file system advanced RAID arrays offer a form of Fungibility, that you can swap bad disks out and replace them with new, ZFS does this in software in the same manner, so long as you include your disks into the pool by disk-id.

Fungibility, in code, in provision, in hardware... It's an art which is sadly not often mentioned, indeed it's not even in my spellchecker's dictionary.