Monday, 26 January 2015

Elite Dangerous - Trading Tools - OCR - Progress Pt7

Help Sponsor me and this project now! Click here for details:

Over the weekend I've been working on the collection of data, so I've been to a few more systems and stations, I've been checking the algorithm for finding the best route, and I've been giving myself some feedback...

The first two things I immediately wanted was the program to remember the last folder used, and also as I captured data, to be able to refresh the loaded information without having to close & open the tool again.  Basic stuff.

So, loading the data looks like this now:

Lets say we've got courier missions from LHS 1914 to FK5 2550, we select the start system, so LHS 1914:

This instantly populates the Stations and total list of commodities in columns 2 and 3 there.  We can select the specific station we're docked in as well:

Reducing the number of commodities, now we could just select Fk5 2550 from the destination system list, just to see what would bring us the most profit in running into that system:

The answer here is then given in the top right box, that result will stay there now until we change system or destination system, we can get the same search just without the station select from the second column also, but then the search would start from all stations in the source system on the left.

This search alone wasn't enough however, because I needed to be able to quickly find a supply of certain commodities, hence the new controls in the right bottom.

Lets say we want to find Beryllium for a specific mission goal, we can select that commodity by name from the drop down:

The results are the source of Beryllium from the database in best price (for purchase) order.  Obviously, if you've selected a commodity you've never seen before then you get no reply, unfortunately in the current context there is no distance, or connectivity in the database, so we don't know if these items are close or not, that is still a player decision*.

The other feature added there is the "Demand For" search, this simply lists those commodities in the database which are wanted somewhere, the results are given in best price (for sale) order, this allows you to quickly marry up any profit (if any) there maybe to be had, I find it most useful if the upper algorithm driven box comes up with an illegal selection.

* However, I do envisage changing the data collection so as data is captured from system to system we have a basic web of where it connected to where by input, this basic webbing could then be used to add "how many jumps" a commodity is from the selected source or destination systems.  However, to add all this functionality I need your help!

Story Time - The A3000 Break-in (My Story of Computing at pre-GCSE Level)

Yes, I'm going to tell a story again, this one pertains to my early teens, when at School I really discovered coding computers.  I went to a state Comprehensive, not a bad one, but not a fabulous one, named for the council estate I grew up on it was nothing special.  But in 1990 it was one of the first schools to start shedding it's older BBC Micro's for Acorn Archimedes Computers.

These caught my attention as just a couple of years earlier I had my Atari ST, my brother and I picked the ST because our cousin had one, and he got all the pirated games available, so we had that and loved it, but until after college I never really programmed it, the Archimedes at school however, they had a programming language built in... they had BBC Basic, just as the Micro's before had.

I also knew they were this "RISC" thing, which was big news in the 90's it was going to revolutionise computing, by making things easier and simpler to code.  "Reduced Instruction Set" after all, sounded simpler!

However, when it came time to pick my GCSE options I took one look at the IT (now called Information Communication Technology - ICT) syllabus and thought it a load of crap.  Who wanted to learn about Ceefax, or weather, or news broadcasts, I wanted to sit at the computer and pick through it's electrons.

So, I avoided ICT and IT, and went onto other things, as the years went on however, they moved two of the science rooms from old labs on the third floor down into another block, and in the vacant rooms they had nice power strung around the walls and they installed a suite of new Archimedes 3000 computers, with 2MB of RAM!...

I remember being entranced by these machines, they had the same form factor as my ST, but I only had 512K of RAM, with 2MB of RAM and a built in OS which was much more full featured than GEM was, I got to grips with these things quickly.  Not that I ever ever had a simple lesson with them... Oh no, because, I used to sneak in there.

I would actively bunk off of Physics and go into this often empty computer suite, I soon found built into the machine a RAM disk, so I could put 512K of ram aside as a disk!... And the speed of that was phenomenal!  Ensuring I had a script to write my files back to my disk, and then another script on another disk to load the RAM disk and copy the files in, I had a basic "boot & sync" mechanism such as many USB sticks come with today, but this was in 1991, on 2MB of RAM using 3.5" 720K DS DD Floppy disks!

Pretty soon I had a whole troop of school friends who would bunk off to the computer lab with me, Daniel, Nicola, Tracey, even the most buttoned down hard working lad in our shared physics class Gurpal bunked off to the Computer Lab with us.

I also used my ST at home for reports, especially doing Integrated Humanities, so I would be given permission from that teacher, to go write up my report on the Computers, it was only a text file so it worked a treat copying the data on and off the disk, even if it was formatted for my ST, the ST, IBM PC and Acorn machines could all read one another's disks... Unlike those poor Amiga Schmucks who had to jump through hoops to get their machines to change the disk idents and headers (yes, yes this was a dig - Atari Forever, Amiga Never - hehe).

Anyhew, leading this rag tag bunch, and exploring the inner workings of these Acorn A3000's one day I was found, in the supposedly locked room, by a teacher, I think his name was Atherton, he was a strange one, but what strikes me now is the manner of his reaction, he asked what I was doing, he saw the pages of code, he saw my sync program as I desperately tried to move the RAM disk back into the safe storage of the floppy, because he had his finger on the master trip switch - the teachers, if they caught you playing a game, or messing around had a tactic; and on machines without hard-drives this was perfectly sound; of hitting the master trip to isolate the power to the bench ring supplying the computers, everyones machine went dead and they had you bang to rights with a game disk in the machine.

This teacher however had the nous to see this wasn't a game, but he was only ICT syllabus trained, he had no idea what I was doing, or the power of the machines on these desks, he thought of them as a teaching aid, not a gateway to the future I now have as a programmer.

I explained I was coding, and I just said "I have a machine at home", and this is where the strangeness began, he assumed I meant an A3000 or similar, which were expensive back then, my Atari was not cheap, but it was already nearing five years old, the A3000 was brand new on the market, and cost nearly £800 each.  He'd not found me playing, it looked like work, he didn't know what code looked like.  Quite a damning statement for a school with him as the lead of IT subjects, with this fancy new computer suite, in the dawning of the 90's where the digital age and computers were fast on the ascendancy.

But he really had no clue, his punishment... yes, there I said it, rather than ask, or encourage me, or see what I was up to and realise I knew more about the computers than he did, he decided not to encourage, or help - which tells you a lot about that teacher - he decided I'd done wrong, being in the computer lab without permission.  So his punishment was... Well it was weird.

I don't think he believed I had one of those machines at home, so to challenge me, he locked the lab - stopping me getting into them - and he took me to his office, where he handed me a copy of Lemmings, yes the compute game... And a pack of 10 blank disks... He made me sign for the disks... And he said "Copy that disk onto all those disks"...

Yes, copyright theft via pirating was to be my punishment... I didn't know what he wanted the copies for, there was no internet, so no e-bay, so I guess he couldn't sell the copies... Maybe he had a class to show a game to, ironic considering the stance the school took for having games on machines etc.

So, off I went, I got home and spent a good hour copying these disks, it took ages on a 512K ST, you had to read the first half of the disk into RAM, then write it out, then read the second half in, and write it out, it was one of those tedious tasks, and I'm pretty sure even today my muscle memory is enough to let me swap disks onto the side slot mounted drive of the ST or Acorn machines at speed.

Anyway, with five disks done, we had to go out training that evening, so in the morning I had only half the task done and this bizarre punishment still at hand... My solution was easy, there were two ways into the computer lab at school, the first was through the door, the second was via an English classroom, into the store cupboard at the back, and then on my hands and knees through the floor level vent, this led into the store room in the computer lab, if it's door was unlocked I could get back in...

It would be risky, but I entered the English classroom before the first bell, and moved a pile of books out from under a table to expose the vent, unscrewed it and moved it aside, then found two T ring binder clips and hooked them through the vent, climbed through and pulled the vent back into place from the other side.

Dusting myself down it was pitch black, so I moved slowly with my feet trying to find the door and not hit anything, finding the handle I turned and it opened, thank god!  Because to go back out now would look very dodgy, as the English classroom had resumed.

I quickly fired on the power and brought the nearest five machines up, and started then making copies of the copies I already had.  It was an agonizing two or more minutes to copy the files into RAM, but the machines had RAM enough to hold the whole disk, so I just had to do one mass swap and press Space to continue.

Writing the files back out was even slower, and made more noise, suddenly I heard a key rattle in the door, someone was coming in... I ducked back into the store cupboard, as a teacher came in, a women, she spread out a group of first years to each machine, except the five which were on, thank goodness...

I decided to play it casual, I rattled a few things and emerged from the cupboard, "Morning Miss"... And I just got on with my machines, she didn't even ask why I was there, as I collected up my five disks, and she never queried why I was in the room with a locked door.  I don't think she had the nous.

Anyway, I had my booty, the ten copies, plus the original, and I went off to hand them into the bloke before break - which was his deadline - I knocked on his door and handed them over, he had a class so could not really talk to me, but he did ask - "Did that take long?", "yes Sir", "Let that be a lesson to you"...

You know, to this day, I'm not sure what lesson he was trying to teach, I look back on using 3.5" floppies with a nostalgia, and I look back on that day the sheer balls I had to get away with using the lab, now locked, and their attitude, it stank... They should have encouraged me, I'd have been somewhere other than I was, my skills and fluency with a computer were delays reaching my education by at least three years, I didn't learn anything more than BASIC and the prompting shell script languages until I learned my first 3rd Generation High Level Language (PASCAL) at college.

By then the internet revolution was burning it's way through the stock markets, but I always think, I could have been writing tools I use and wrote five years later that little bit earlier, I could have had an easier time at college and university by earning cash from the tools I had, or even giving them away to make a name or addition for my CV.

Friday, 23 January 2015

The World Reacts to the Mighty Jingles - #5 Space The Final Frontier

We've all wondered, haven't we, what happens when Jingles releases a new Video?... In this new series I aim to help solve that mystery.... This week, the Final Frontier

Check back each Friday at 18:30 UK time for the next thrilling instalment, where might we find Jingles being Enjoyed?

Elite Dangerous - Market & Trading Tools


Today I have taken time to put together a Patreon page, this is intended to help establish my project with some real backing, the pledges are Monthly for just $1 a month you can help speed up the development of this suite of tools:

You can opt to support me with higher pledges per month as well, but all the time you can choose your level of support for the project.

If you can't afford to help, then please just show the love in the comments below.

Thursday, 22 January 2015

General Update -Elite Dangerous - Houses - Money

So it's been a long couple of months, our Christmas was severely cut back, and we've been waiting for the house sale to go through... It should complete Friday... So today I'm busy taking my excess hours off of work and going down to give the place a once over and then to the estate agents to hand in the keys ready for hand-over.

It has been a rather stressful time, not least with the ex-tenants we had being chased from pillar to post for debts they've racked up.. The best one being the illegal sub-tenant being prosecuted by Derbyshire City Council over a parking ticket... She appeared in court - how they summonsed her I don't know... But she still put our property address down as her own, in court, in early January 2015, despite not living in our property since October 2014...

I'd call that perjury, but I'd also call it incompetence by the court because the wife and I are on the council tax & electoral register, it'd have taken minutes to discover this and shows they didn't confirm the guilty party's address!

Needless to say, when the notice of impending removal of goods arrived, the wife ripped it open and nearly fainted, I've sent them a letter now, but we're still expecting burly blokes with bad attitudes to arrive, so the wife has her ID out and ready to show them through the window.

With all this going on Money has been an issue, hopefully tomorrow it gets a lot freer as the house funds get cleared to us, my plan is to wire cat6 shielded network wire into the upstairs and remove buy new springs for my X45 control stick.  All to aid my enjoyment of Elite, as well as WarThunder and some other games.

I've not been able to play any simulator battles in WarThunder for a while, because the spring on my stick is now as slack as a wizard sleeve, so straight and level flight is almost impossible, accurate gunnery is impossible.

I'm looking at ordering a set of springs from a place in Redditch and have a friend; a Redditch native; bring them up.

Coding wise, as you can see I've got the OCR tools working, last night was the first time I didn't do any collecting of data, I sat down with the 5 systems I had done, and I just used the tool (as shown in yesterdays post - pt6)... It was amazingly good, I tweaked one trade I had been taking Gold about for $45,000 profit (in my cobra with 36 cargo), but found I had a palladium source and consumer upping that first leg to $49,500.

The return leg I had been carrying Liquor for around £11,000 profit, I upped that to $18,500 by swapping to Power Generators.

Rinse and repeat, the viewer tool as you see in the previous screen shot worked great, a lot better than the first revision I'd put together.

Running the game in a window (though full screen border less) also gave better interactive results, I could have the collector running, and then slide the mouse out the client to use the data.  I need to add a refresh - or even an auto-refresh which detects file changes as data is collected - but for now this suits my purposes.  And it achieved results I earned a little over $800,000 credits trading in the short two hours I had to play last night... Taking me from Peddler to Dealer ranking to boot.

I am looking for testers, or backers, if you want to help, or offer advice, or offer support (in the form of servers for data collection or just money to pay for the hosting I think we'll need, just drop me a line in the comments below).

Wednesday, 21 January 2015

Elite Dangerous - Trading Tools - OCR - Progress Pt6

Update: You can now support this project at Patreon!

Armed with some of my data I've had two stabs at making use of the data, and so here's the first peek at the tool I've thrown together to let me make use of the information, one simply selects the start system and then the destination system and the best commodity (for profit) is calculated.

You can also refine the start point by station or specifically a commodity, before you select the destination to give a specific start point and be given the best profit item.

The next feature I'm going to add is a box to enter the name of a commodity and find where it's sold as a list of system & station names, just as a helper for what's already in the game's galaxy map.

Sunday, 18 January 2015

Elite Dangerous - Trading Tools - OCR - Progress Pt5

Update: You can now support this project at Patreon!

So, last night I had my first long run and tuning up session, capturing system name, station name and commodities as I played the game... As you can see, I have quite a few captured commodities now:

I'm now going to throw together a separate reader, to load the data set and let me do comparisons or round-trip plotting of best prices.

But, I thought i'd explain how this application works, as I've posted it into the streams of a few Twitch chaps playing Elite, and everyone seems to have their own preconceptions....

1) This is a stand alone application, it runs in windows completely separately to Elite.

2) It captures the screen images, it DOES NOT open the processes memory.

3) The areas of the screens captured are filtered, and then passed to an optical character recognition (OCR) system.

4) The OCR system I'm using is Tesseract

5) The image processing is triggered by keys:  In game you just play, and then Press F1 whilst in the navigation menu, and the program will take a shot and try to work out the system.

6) Press F2 whilst in the commodities screen, and the station name is captured.

7) Press F3 and this toggles on the capturing of the area the commodities are listed in, it takes pictures until you press F3 again, so you slowly scroll down the list, then press F3 and the program passes the images taken for processing.

8) The data is written out as XML and contains the System/Station name, then the Commodity Name, the buy and sell prices, and then the supply & demand levels.

I have had to do some manual fixing of things, such as correcting for the OCR getting the strange minerals spelt wrong.

My plan once I have the data coming in, is to have it upload to a server, which I can then view from anywhere and use to make searches of the commodities.

I would however like some help completing the application, specifically I require testers, if you are interested, please contact me in the comments below.

Finally, lots of people ask "Why", why make this tool, why not use one of the others out there... Well, I have nothing against the others, I looked at a couple, including EliteOCR and they showed that Tesseract could get the job done, but none of those tools were mine, I've played and used a lot of tools for game, I'm a professional programmer, so I figured I should make a tool which is exactly what I want.

Will I never make this publically available?... Perhaps, but I've had little to no interest thus far.