The next leap forward: Parallel Processing

One of the nice things about having a blog is that I can always point to a previous post and say, "I predicted this!".  Unfortunately, predicting parallelism as the next great step for computational processing is no great feat.  The serial nature of our current CPU's is inhibiting.  The processors are fast, but they are primarily single threaded, meaning they can only process one instruction at a time.  The human brain by contrast is slow in comparison to today's CPU's, but the human brain is multi-threaded and can handle multiple requests at once, making the brain more efficient at processing information per cycle.

Last week at the OOPSLA (Object-Oriented Programming, Systems and Languages) conference in Montreal, the programming demi-gods of our age got together to celebrate the 40th anniversary of Simula 67, the first object oriented programming language. 

Programming Superstars Eye Parallelism

 

Included in the panel were: Steven Fraser, director of engineering at Cisco Research, who chaired the group; James Gosling, a Sun Microsystems vice president, Fellow and creator of Java; Anders Hejlsberg, a Microsoft distinguished engineer and creator of C#; and Bertrand Meyer, founder of Eiffel Software and creator of the Eiffel programming language.
 

 

"Going forward, the big challenge we have today is we have to really think deeply about programming for concurrency," Hejlsberg said. "The models we have today for concurrency don't work."
 

 

Hejlsberg said developers need to move an abstraction level up. He said he has been speculating on the area of querying with his LINQ (Language Integrated Query) project.  

 

"The LINQ work we've done is especially useful in parallel computing," he said. "We have an implementation called PLINQ," or Parallel LINQ, which automatically optimizes and parallelizes query operations based on dynamic runtime information.
 

First off, let me say that I think its great that we have arrived to this point at all, and I credit the moves of the Java revolution with getting us here.  Java removed the platform, the network, and in many ways the user interface, as obstacles to programming language ubiquity.  But its not long after diving into the Java threading api's, and dealing with blocking and synchronization, that you begin to realize the shortcomings of our current CPU/Programming architecture.

To move parallel processing forward several technologies will need to become more closely coordinated, namely our programming languages and our current processor architecture.  I've spoken about the parallel processor issue in the past, so I'll focus on the programming languages today.  Parallel processing offers unique challenges to the way we store and access mutable data, and maintain state.  The demi-gods offers us Functional Programming in languages such as LISP, Erlang, and Haskell.

Functional languages differ from structured languages in that they emphasis the application of functions instead of emphasis on changes in state.  In functional programming, everything is a function and each function exists in an ordered class hierarchy.  Typically, a main function is defined as a series of sub-functions, which then contain their own sub-functions until at the very bottom functions are defined as language primitives.

Functional programs contain no assignment statements so once variables are given a value, they never change (immutable).  Another aspect is the benefit of containment across functions.  A function call can have no other effect other than to return its results.  This eliminates a major source of bugs, and also makes the order of execution irrelevant - since no side-effect can change the value of an expression, it can be evaluated at any time.

Functional programming has been around since the dawn of the computer age, but has been relegated to academia and research since then.  I believe this probably has more to do with the productivity of writing programs in a teaming environment than the languages ability to perform.  Look for new ground to be broken in the functional programming area here in the next few years.  If we're ever going to successful in simulating the parallelism of the human brain, we'll need to look towards functional languages, especially for neural net development.  Functional Java anyone?