nanogui: Thread: Thread-safety, wrapping globals in structs, RTEMS

Some time ago there was mention of the toils required to wrap all of our
globals into structs so as to give us thread-safety in our nano-X clients.

I'm using nano-X and RTEMS, and for some time I've been shutting-off
preemption before every graphics call in order to avoid clobbering nano-X
globals.  RTEMS has provisions for giving each rtems_task a private copy
of global variables, and I haven't yet played with this and so I haven't
yet identified the gotchas I'd encounter by trying to privatize nano-X
globals.

Much more elegant would be the wrapping of globals into structs.  Has
anyone tried tackling this, or given thought to doing so?  Or how to do so
without destroying everything?  and if not, is anyone interested in
giving it a shot with me?  I'd start in about two weeks.

-- Kaben Nanlohy

: RTEMS has provisions for giving each rtems_task a private copy
: of global variables, and I haven't yet played with this and so I haven't
: yet identified the gotchas I'd encounter by trying to privatize nano-X
: globals.

I'd like to get your list of globals used.  There are two basic
types, static's per file and shared globals.  I've had some
thought on moving some of the g_ globals in engine/devopen.c into
the PSD struct, for various reasons.  Others, including all the
statics, are really global and shouldn't be per-task nor should
they affect the operation of Microwindows in a multi-threaded
environment, unless you're wanting to be able to draw simultaneously
from multiple tasks.

:
: Much more elegant would be the wrapping of globals into structs.  Has
: anyone tried tackling this, or given thought to doing so?  Or how to do so
: without destroying everything?  and if not, is anyone interested in
: giving it a shot with me?  I'd start in about two weeks.

I like the idea, but what really does wrapping into a struct buy
us from a multi-tasking perspective?

Regards,

Greg

On Mon, 18 Dec 2000, Greg Haerr wrote:
> Trying to think deeply about this, I came up with the following.  Making
> Nano-X or Microwindows work fully reentrant (which is essentially
> what you're talking about with state-variables in a per-task structure)
> isn't really the best solution, unless you really are running multiple
> CPUs.  That is, with a single CPU, all of this modification ultimately
> just becomes a "programming methodology" rather than an actual
> speed enhancement, since there's only one CPU anyway.
> 
> So, I think a better idea isn't to make Microwindows reentrant,
> but, rather, thread-safe.  Here, I am assuming that you're not using
> the client/server Nano-X style, since it already works by serializing
> requests read by the socket in the server. (And RTEMS doesn't have
> processes, only tasks).  So, in the linked-together version, all we
> really need is a couple of global semaphores and then wrapping
> them around most graphics calls in order to protect the server's
> state.  These could be built as macros and left out of a non-threaded
> version.  What do you think of doing it this way?
> 

I think that this would fix things for me.  Thanks.

I'll share when it works.

-- Kaben.

I've been working on porting Microwindows (both Mwin32 and
NanoX) to a VxWorks platform.  VxWorks is a small, thread-based,
shared-namespace kernel, and it sounds like we share some of the
same issues as RTEMS.

"Greg Haerr" ####@####.#### wrote: 
> 

> : RTEMS has provisions for giving each rtems_task a private copy
> : of global variables, and I haven't yet played with this and so I haven't
> : yet identified the gotchas I'd encounter by trying to privatize nano-X
> : globals.
> 
> I'd like to get your list of globals used.  There are two basic
> types, static's per file and shared globals.  I've had some
> thought on moving some of the g_ globals in engine/devopen.c into
> the PSD struct, for various reasons.  Others, including all the
> statics, are really global and shouldn't be per-task nor should
> they affect the operation of Microwindows in a multi-threaded
> environment, unless you're wanting to be able to draw simultaneously
> from multiple tasks.
 >

> : Much more elegant would be the wrapping of globals into structs.  Has
> : anyone tried tackling this, or given thought to doing so?  Or how to do so
> : without destroying everything?  and if not, is anyone interested in
> : giving it a shot with me?  I'd start in about two weeks.
> 
> I like the idea, but what really does wrapping into a struct buy
> us from a multi-tasking perspective?

I did successfully get the engine code "de-globalized" by
moving all of the globals (and statics) into the PSD.
What this gets you (in my case, anyway) is the chance to
support multiple independent screen devices with just a
little more init code (ie, each screen will have its own
root window, fore/back-ground colors, cursors, clipping regions,
etc.).  I do have this code available, but have not yet
sent in any patches (see next Paragraph for why!).  If you
would like to see this code, just ask and I'll package it
up for you.

But when I moved onto the Mwin32 and NanoX code, there was
a *lot* more work to be done.  Especially problematic are
the Win32 calls which don't pass in any pointers, so you
*have* to use globals to get anywhere.  So I dropped work
on this and am re-evaluating how to support multiple
independent screens.

As for making Microwindows thread-safe (which is what
Kaben was originally asking for -- I think! ;-)  I don't
think you necessarily need to "de-globalize" the code.  As
long as your system can share globals, we just need a
semaphore type of lock/unlock every time we access one of
the globals (or, at a more gross level, every time we enter/leave
a Microwindows function).  I think it would be useful to
add some access control functions (ie, semaphore lock/unlock)
to all of the Microwindows functions.  By default, they
can just be empty functions (and so, with compiler optimization,
would be optimized out and the code should work just as
fast as it does today).  But if an implementor over-rode these
functions with real semaphore lock/unlocks for a particular
OS, then you would have a multi-tasking thread-safe
Microwindows implementation!

I'm not a Linux-guru, but how do Linux handle multiple
Microwindows processes?  I have experience in OS-9 (a real-time
process-oriented OS) and in that environment we would package
up all of the globals/statics to be shared into a "data module"
which all of the apps could then link to and use (with proper
semaphore access control, of course!)  Is this what you
mean, Kaben?  In this case, you would need both the
"de-globalization" and semaphore control added to Microwindows.

Well, that's enough of me yapping for now.  What do you
guys think?

BAT
-- 
Brent A. Thompson, Next Level Communications <www.nlc.com>
1776 22nd Street #100, West Des Moines, IA 50266-1444
EMail: ####@####.#### Phone: (515)991-3853

On Wed, 20 Dec 2000, Thompson, Brent wrote:

> I did successfully get the engine code "de-globalized" by
> moving all of the globals (and statics) into the PSD.
> What this gets you (in my case, anyway) is the chance to
> support multiple independent screen devices with just a
> little more init code (ie, each screen will have its own
> root window, fore/back-ground colors, cursors, clipping regions,
> etc.).  I do have this code available, but have not yet
> sent in any patches (see next Paragraph for why!).  If you
> would like to see this code, just ask and I'll package it
> up for you.

I had separate nano-X clients running in separate threads in an rtems
(linux-posix) application.  This was using the client/server setup; each
task had its own vt52 text-out panel (modified nxterm).  It worked
smoothly as long as the nano-X server ran as a separate linux process.  
But as it was only an experiment, and a series of ugly hacks, I tabled it
some months ago.

It was done by encapsulating client globals and statics into structs
created by GrOpen(), where GrOpen() stored the struct pointer in an
rtems-notepad of the client task.  When I tried to run the server as a
separate thread in the same application, deadlock resulted on sockets (I
think) on the linux side.


> But when I moved onto the Mwin32 and NanoX code, there was
> a *lot* more work to be done.  Especially problematic are
> the Win32 calls which don't pass in any pointers, so you
> *have* to use globals to get anywhere.  So I dropped work
> on this and am re-evaluating how to support multiple
> independent screens.
> 
> As for making Microwindows thread-safe (which is what
> Kaben was originally asking for -- I think! ;-)  I don't
> think you necessarily need to "de-globalize" the code.  As
> long as your system can share globals, we just need a
> semaphore type of lock/unlock every time we access one of
> the globals (or, at a more gross level, every time we enter/leave
> a Microwindows function).  I think it would be useful to
> add some access control functions (ie, semaphore lock/unlock)
> to all of the Microwindows functions.  By default, they
> can just be empty functions (and so, with compiler optimization,
> would be optimized out and the code should work just as
> fast as it does today).  But if an implementor over-rode these
> functions with real semaphore lock/unlocks for a particular
> OS, then you would have a multi-tasking thread-safe
> Microwindows implementation!

I'm running rtems on a single M68332, so all threads can see globals.

Making microwindows thread safe rather than reentrant, using the ideas you
suggest, was suggested by Greg, too.  And I don't need to have one nano-x
client per rtems task, so the solution is a good one for me at the moment.

But I can see, somewhere down the road, the need to display data from from
multiple devices on multiple displays.  One device sending data to
multiple displays on a local network, multiple devices sending data to a
single display.  Etcetera.

That's way down the road if it happens at all.  I perked right up when I
read the discussion of networking microwindows/nano-X.

So while right now I'm looking for thread safety, I was hoping to not
paint anybody into a corner with a thread-safe solution that precludes
reentrancy if it begins to look attractive.


I realize that this is an ambitious discussion, and perhaps not the route
intended by others for nano-X.  But my current rtems/nano-x project makes
for a 290k executable, with a tiny gui and signal processing and stuff,
and I'm very pleased with both rtems and nano-x.  When we see what sort of
networking is given to us by Alex Holden, I expect we'll also see that
multiple clients in multiple threads, connected to nano-x servers on a
network, isn't a far reach.

> I'm not a Linux-guru, but how do Linux handle multiple
> Microwindows processes?  I have experience in OS-9 (a real-time
> process-oriented OS) and in that environment we would package
> up all of the globals/statics to be shared into a "data module"
> which all of the apps could then link to and use (with proper
> semaphore access control, of course!)  Is this what you
> mean, Kaben?  In this case, you would need both the
> "de-globalization" and semaphore control added to Microwindows.
> 

I don't know if this is what I mean.  I think I mean, for now, multiple
threads getting preempted without clobbering globals in the linked-in
server model.  I think, later, distinct clients, potentially one per
thread, each with its own set of client globals.

In linux nano-x, as far as I can tell, each process that's a microwindows
client can make one connection to a server process on the same machine
using a local socket (or shared memory).  The server keeps track of the
clients and the objects they own.  But as far as the application
programmer is concerned, at least at the moment, there's no difference
between the client/server and the linked-in server models.

***


Right now I'm listing and trying to organize the globals and statics used
in a linked-in server application.

Questions for the moment:

Call these hooks "Gr(Un)LockSomethingOrOther()" or
"GrObtainSomeSemaphore()"...  Does anyone want to name these functions
after their favorite pet or anything like that?

Giving them the same kind of abstraction used in the drivers seems
excessive, but it would be pretty.  Using makefile or macro magic or just
leaving hooks for implementors to fill would be clunkier but easier.  Are
there preferences?

-- Kaben.

: Call these hooks "Gr(Un)LockSomethingOrOther()" or
: "GrObtainSomeSemaphore()"...  Does anyone want to name these functions
: after their favorite pet or anything like that?

Use
    LOCK(mutex)
and
    UNLOCK(mutex)

where LOCK and UNLOCK are #define's.  In addition,
the mutex itself can be declared as a MUTEX, allowing
any different means of critical section handling routines
to be used without having to re-hack Nano-X all the time.

Since we're talking about code that's internal to Nano-X,
we won't use the GrXXX functions, since they're not allowed
to be called by users.

At this point, I'd recommend a #include "mutex.h" in serv.h,
with the 

#if RTEMS_MUTEX
#include <pthread.h>
#define MUTEX            pthread_mutex
#define LOCK(mutex)   pthread_lock(&mutex)
#endif
etc 

or whatever in mutex.h.

Then, a single global mutex can be used in each GrXXX routine
to guarantee thread safety.

Regards,

Greg

On Wed, 20 Dec 2000, Greg Haerr wrote:

> Use
>     LOCK(mutex)
> and
>     UNLOCK(mutex)
> 

I don't know anything about the internals of microwindows or nano-X or
anything, so take this thought with a grain of salt.  Wouldn't it be
better to use rwlocks?  They exist in the thread.h library in solaris, but
linux uses pthreads, hence no thread.h.  However read/write locks can be
emulated using multiple semaphores, correct?  Do you think that'd be worth 
the effort, or no?  Do rwlocks exist natively on the other platforms
supported by microwindows & nano-X?  Just a thought.

jason