Pondering P6

In the list of points in my previous post was point P6

p6. Current models start with random sets of connections to neurons. This seems odd. Wouldn't a more efficient starting point be to add neurons with connections matching patterns as they occur during network development.

I've played around with a couple of small models that attempt to place nodes (neuron models) to recognise patterns that occur at an occurrence rate above that expected in a random input signal. It has showed some promise and I shall probably take it further given time.

The implementation so far uses an array of nodes that keep track of occurrence rates of sets of inputs, but these are not weighted connection nodes like the normal neuron modelled nodes in a neural net, they are generated by occurring pattern within a domain of inputs and then produce an output when that pattern occurs a set level above chance. The output from these nodes can then be used by other nodes to find combinations of patterns that occur above chance and to place neuron nodes and connections in places where pattern recognition should occur.

Where as nodes in a neural net have some basis in neuroscience, I could not see a basis for this new type of node until I did some further reading. I was taught that neurons where the data processors in the brain, held in place by glial cells that form the scaffolding for the cortex. However these cells also play a role in directing the placement and connection of new neurons and connections. Maybe these cells could be performing the process that my pattern tracking nodes are doing.

Of particular interest are Astrocytes and Radial glia.

Please accept my confession. Neural Nets and me.

Please accept my confession and absolve me of my sins, it has been four years since my last post.

My current area of interest is Neural Nets and I've been playing...

My first adventures in AI started with my first computer, the ZX Spectrum. I developed simple programs a bit like Amazon's Alexa that were simple question and answer applications with words extracted from sentences and substituted into set replies. I dreamt of a day when I could hold a conversation with my computer in natural language. It appears it is quite a hard nut to crack :-)

I've been interested and played with Neural Nets since I was 17, which is quite a long time ago. My first attempt was building an electronic analogue circuit as my A level electronics project. It was too ambitious for both my skills and my budget, but it did look cool and complicated. The examining board were not so easily impressed. I should have stuck to a motor controller.

I then went to University to study Physics and Psychology but soon realised that Physics was not for me and that Psychology might be the place to find some answers to the problem of how to build good AI. It was, but I was also expected to learn about a lot of things that I had no interest in so it did not go as well as it could have. My final thesis was a poor attempt at showing how genetic algorithms could be used to evolve AI systems (I did learn from this that University cleaners will unplug your computer if the socket is needed for the vacuum cleaner even in a computing lab, and that a Z88 computer does not have the power to run a decent genetic algorithm in useful amount of time).But enough of blaming my tools, I hope that now my skills and knowledge have grown, and I can show that GAs can be used to create useful Neural Nets.

After a few different fill in jobs I ended up working as a software developer and that where I have stayed for 25 years. I have had plenty of time to hone my skills and increase my understanding of how the brain might work. I've followed developments that have changed the playing field considerably, from MRI scanners that remove the reliance on head injuries to investigate brain processing, to the cloud computing revolution making super computing power within the grasp of the amateur.

Neural Networks are fashionable and the increased speed and size of computers have made large networks feasible even for tinkerers like myself, so I have decided to return to the area that first interested me all those years ago.

I've always been most interested in AGI (Artificial General Intelligence) and particularly learning without an external teacher. The current large scale networks relate little to what we know about how the brain learns. They require thousands of items of training data tagged with the information within them that we wish the network to recognise.

Below are the current areas I am working on broken into three sections. The main problems that I feel need to be overcome, the elements of human cognition that have not yet been addressed and the experiments I am running to fix the problems and address the issues.

Problems that need to be overcome

p1. We, as humans, do not need thousands of instances of a patterns to group them. Often just a couple are enough. Evidence:

p2. We do not need to be told that patterns belong to a particular group to place them in groups. Evidence:

p3. We can recognise visual patterns as being the same no matter where in the visual field the appear or which orientation they are presented in. Evidence:

p4. There is not overseer that can monitor, place and alter neurons individually. Neurons and their connections are the only micro control mechanism. Chemical transmitters can send macro signals to and from other areas of the body (including brain). Neurons and those they are connected to are on their own as far as micro decisions go.

p5. Back propagation, used to train networks by altering connection strengths, require a teaching signal to provide a score based on neuron responses. It is similar to behaviourism in psychology. It only takes you so far, at some point you have to supply a trainer system.

p6. Current models start with random sets of connections to neurons. This seems odd. Wouldn't a more efficient starting point be to add neurons with connections matching patterns as they occur during network development.

Elements that have not been addressed

e1. We, and therefore our brains, have evolved...

e2. Humans go through stages of learning that have evolved. Evidence:

• Visual system learns to recognise edges after birth and before a set age.
• The kindergarten effect

e3. We apply our current internal model onto input from the external real world. Evidence:

• The Swiss Cheese illusion
• The Spotty Dog illusion
• The speech without consonants illusion

e4. We have layers of neurons in the cortex. More than one and less than those in a deep neural net.

e5. Areas of the brain seem to have specialised functions. Evidence:

• Face recognition

e6. Some elements of behaviour have evolved, while others are learnt, yet others we have probably evolved to learn. How do these two elements interact to create a network.

 My Current Experiments

Ex1. Learning pattern recognition without access to the training labels.

See p1,2,3,5,6

In this experiment, I am building a NN system that learns to differentiate handwritten digits from 0 to 9 from a data set called MNist but without the use of the labels provided.

The network builds itself to differentiate the different elements of the patterns in the set.

My theory is that given the correct set of simple rules, a network can be built that will have distinct nodes (neurons) that fire when, and only when, a specific digit is displayed but can generalise to all examples of that digit.

Whereas current networks are given the training labels (eg. given an image of the digit 8 and told it is a digit 8), this network will only be given the digit 8 and not given the label. Once a level of training (or in this case experiencing) has elapsed, the nodes are searched for ones that only fire for a specific digit and connected to output nodes. This differs from current networks in that we do not pre-define where the recogniser for a specific pattern (digit) should occur, but let it develop and then at a later time associate the recogniser with an output using a function called MagiMax. Another difference is that we do not start with an initial layer of nodes with large numbers of connections with random weights, but rather build the nodes and layers based on patterns submitted to the network.

The network built is of multiple layers from an input layer (layer 0 that is pre built to match the 28x28 matrix used in the MDist data set) to an output layer (5). Nodes in a layer only connect to a limited domain of nodes in the layer above. The training data set is applied once as each layer is built, so layer 1 is built based on a full training set, then layer 2 is built by processing a full data set etc.

Initial simple applications of this process gave a number of good recognisers for some digits, but others failed to appear. The networks ran out of space for new nodes.

Ex2. Evolving solutions to problems using neural net populations

See e1,6

This a repeat of previous experiments to evolve a network of nodes, connections and weights to recognise MNist data set digits. This is really just to check that the implementation of the GA and NN work.

Evolution stalled on an above chance but poor recognition score.

Ex3. Evolving builders of neural nets to solve problems

See e1,6

Currently the work on this problem is concentrated on building a set of elements that can be randomly arranged to build networks that can then be scored on their ability to build networks.

The elements need to be resilient to crossover and mutation, so that breeding new genomes gives a viable network builder in most instances.

Ex4. Building matrix transformations based on experience to solve spacial, 2d rotation and 3d rotation of patterns for recognition.

See p3

Work on this is still in the planning phase. The idea is that by presenting moving patterns to the input of a visual recognition network, the network can arrange itself into a configuration where new patterns can be recognised independently of where they appear in the field of vision and in any orientation.

It is possible that the result of Ex3 may be used to try and evolve a solution to this problem.

Ex5. Path finder nodes

An investigation into whether a network of nodes can be used to keep track of patterns that have happened frequently with a view to then placing nodes and connections where patterns occur significantly often.

Crazy Idea of the day

Having been reading about Google's Omega project on wired, I was thinking about how I could build a rival super computer using my meager resources. How could I get my hands on thousands of processors to perform large parallel computations.
How about using browsers as nodes on a web wide computer.

1. A visitor to my server opens a page and confirms that they are will to take part in the process.
2. The web page polls the server for tasks.
3. When a set of tasks are available, the tasks are shared by the server among polling browsers.
4. The tasks are sent as small JavaScript programs to run.
5. The results are sent back to the server for collation.

The web page would display details of the tasks being run.

The service could be offered as a peer to peer service, with a web interface offering the ability to upload jobs to be run.

Simple Magnetic Imaging

This is an idea I have been playing with. Passing a low energy magnetic pulse through an object such as a part of your body and then using a sensor array to create an image of its internal structure. Like X-Ray but using low frequency radiation. The problem is that the low wave length restricts the resolution of any image created. I have an idea that using a genetic algorithm against data from a known object , such as a saline filled bag, and a large set of readings, a system can be trained to create high resolution images from a relatively small set of sensors.
I've started experimenting and will post back any findings.

Prediction 2

In my new (self defined :-) ) role as a futurologist, I thought I should add my next prediction / suggestion of a new technology.

With the growth in renewable, sustainable electricity generation, particularly wind and solar, comes the problem of matching generation to demand.

Storage is the answer. Not small scale storage like a battery bank, but large scale storage. This is already achieved to an extent by such projects as Dinorwig Power Station, where off peak electricity is stored as water pumped to a high reservoir, then released through turbines during peak demand periods.
There are only so many sites where this kind of geography exists, what is needed is a large scale storage solution that can be placed at any location, preferably near to the place where the electricity will be consumed to keep transmission losses low.
One solution that is being considered, is storage of energy as compressed (liquid) air which is then used to power a gas turbine as the air is allowed to expand during periods of demand. This does seem like a good solution, but has inherent dangers due to the storage of compressed gasses, and technical difficulties due to the generation of ice during decompression.

Another solution that has been investigated (in New Zealand) is the large scale Redox battery. This works in a similar way to traditional battery charging, where energy is stored chemically in an electrolyte. In the Redox battery, electrolyte is 'charged' and then stored in a tank. Only a small unit is needed for the relatively expensive battery electrodes, but a large amount of electrolyte can be stored in relatively cheap tanks. The New Zealand system uses Vanadium.

This is the kind of technology I am interested in.  In preference to vanadium, I am investigating iron  as my storage chemical using activated carbon electrodes. The benefits of this medium, is its low cost, relative abundance and low toxicity. During my experiments it has also come to my attention that the charged iron slurry that is produced during charging of the battery can be attracted by a magnet, and pumped around using a pump constructed like a linear motor. The basic chemistry is that iron oxide is converted to iron in an activated carbon slurry using electrolysis. This can be pumped and stored in tanks. When electricity is required, the slurry is oxygenated to oxidise back into iron oxide. This acts as an iron air battery, creating usable electricity.

I'll write more on this when I have progressed further. Family calls :-)

I have a lot of pretty crazy ideas, some of which turn out to be less crazy in hindsight. Often I try to follow them up by building prototypes and reading up on the subject to see if there is an invention or business opportunity waiting for me there. On a number of occasions my idea has proved to be following the Zeitgeist and others prove the idea has merit before I do, and often this is due to the level of funding and resources available to them. This is not my whining about how unlucky I have been, but I intend to start writing about some of these ideas here in the future. That way when I moan about this or that being 'my idea' :-) I can point back to a blog here to prove it.

So my areas of interest at the moment are:
3D printing. I know it is being touted as the next big thing for the consumer market, but my interest take me in a different direction.

A 3D printer has four essential elements, a machine that can move a print head in three planes (x,y, and z), a print head that deposits a material at any point, some software that takes a model and converts it into instructions for the printer and some software that allows a human to define that model.

The two target areas where innovation will take place are:

The print head.

Currently 3D print heads print in a single medium such as a plastic, starch based material or even metal. I believe a future development will be to increase the number of media that can be used by the printer. This could be achieved by having multiple, automatically replaceable print heads to embed different materials and elements into the model. This may be as simple as harder materials for an outer shell or generic electronic components that can be made to simulate different devices such as a FGPA type chip, and a print head to print the required circuit board in a material such as graphite infused plastic.
Another development may be to add post processing heads to the printer. Presently the printer makes a single x,y scan for each layer of a model being printed before progressing to the next layer in the z axis. The print head could quite easily be adapted to allow a post printing scan of the object to spray on colours or an external coating, to cut away areas where this could not be achieved during printing or drilling an adding fixings.
Multiple print heads could also be used to print elements of buildings. I have seen this is already being investigated, I feel that a loom based arrangement suspended above a printed building structure could also incorporate z oriented fibres into the materials as they are printed. Improvements in the print heads for this application would use dry concrete as the print medium, mixed with water at the tip of the print head rather than the current use of shotcrete.

The 3D modeling software

The current breed of 3D modelling software takes a lot of practice and training to use to make even the simplest models, I envisage a piece of software that allows you to create items from a pattern book of simple parts. Anyone can make a model out of lego. 3D modeling should be that easy. A standard set of virtual parts that can be snapped together in the software, but as this is a virtual model and not lego, the standard parts can then be modified, skinned and altered to make the desired model.

As a simpler way of printing consumer items, I envisage that models will be automatically translated into a set of parts like a kit that can be stuck together by the user once printed much like a plastic model kit. This would reduce the complexity of printing and reduce the print area required to make larger items.

I'll stop there for the moment and come back with my other ideas soon.

RenSMART has been running as a business for a while now and is now in profit !
I've decided that I shall no-longer be posting to this blog about RenSMART and will just use this as a place for random musings.

RenSMART's new blog can be found here: http://rensmart.blogspot.co.uk/