In order to coordinate development / improvement of serial-pyio, I'm
opening this thread (Otherwise, it could be done in the sourceforge
forum, but that would require another login, etc...)

Use this thread for discussion - if you have bug reports or feature
requests, please post them directly at serial-pyio's project page:
https://sourceforge.net/tracker/?group_id=196146&atid=956289

hi rooker

thanks for your OSS project.
it worked for me perfectly nice, out-of-the-box.

great that i didnt had to mess around and could directly start into PD 
stuff.
thanks a lot.

i just wanted to mention 2 things:

1) i would suggest to package it in a root folder, containing version 
number, enclosing all files.
'cause fo instance in my case i wget your package to my already quite 
stuffed /usr/local/src and it took me a look into your CVS repository to 
find out that there's a startscript and README included.

2) i just added 2 lines to my run.sh to find the ttyUSB? in the 
/var/log/messages
i attached my version, and would check it into CVS if u dont mind

Thanks for the "thanks" :) - you're welcome.
Glad to hear that it worked out-of-the-box. I'm curious: Which distro 
are you using it with?

about your suggestions:
1) You mean that the tar-file should unzip into its own folder like 
"serial-pyio-1.1" instead of directly into the current folder? Good 
idea. I'll do that. I have to pack another release anyway, because as 
I've mentionend in the news on the sourceforge page, I've fixed some 
bugs.

2) Your run.sh will only work if there's just 1 xxh device attached. But 
since this will be the case for 99.9% of all users, I'll put your 
version of run.sh into the next release. Thanks.


You mentioned PD: Last week I made a patch for playing / looping samples 
using my 40h and pd. It's quite a mess, so I haven't published it here, 
yet - but if you're interested, you can have it.

Hey rooker I would be glad to get a version of your patch ;)

Hello all,

I have been working on serial-pyio for the past two weeks and there are 
a couple of new features:

- virtual monome: you can launch a (several) virtual monome that can be 
used together with an actual monome, used alone... A possible cool usage 
would be to display the virtual monome on a separate screen so that your 
audience sees what happens on the monome. Provided this is a touch 
screen then the audience can collaborate with you...

- displaying messages: I added some code to display simple characters on 
the monome. I will add some code to display messages, like on a ticker. 
A possible usage is to display small help messages in complicate apps 
.If you have multiple "pages" or mode in your app you can display mode 
name. Generally, this can be useful to give additional feedback to the 
user. This will need a new OSC message to enable app creator to use this 
feature.

- coding simple applications: it is also possible to code simply, in 
Python, standalone applications, without relying on the OSC layer. This 
is useful to code games for example.

Comments and others ideas are welcome !

Julien

just downloaded the 0.1.1 version to give it a try on winxp. seems like 
the run.bat is broken. when i changed the python line to :

python serialpyio.py COM2 -t

it worked. i dont really know how to use cvs, otherwise i would update 
it.

Just wait a bit. I will commit my changes and we'll make a new release 
with Rooker. Hold on a bit.

Concerning the run.bat script. I cannot really help, I am on Linux. But 
their should be a basic user interface in the next release that will 
make things a bit easier.

Julien

@jah: Yes, you're right. There's a typo in "run.bat" in the current 
release. It calls "serial-pyio", but should call "serialpyio" (no "-"). 
I've fixed this already, but haven't checked it out into the CVS.
(Sorry for the typo, but I had to write this "blind" because I had to 
Windows at hand to verify it)


@Julien: If I can do some testing on the new version over the weekend, 
we might be able to pack a new release at the beginning of next week. 
(Note to myself: I must write a clever script to make the release easier 
and more failsafe)

and "great thanks" btw.

Here's the first screenshot of the new GUI version of serial-pyio, 
running on Ubuntu / Gnome:
http://wiki.monome.org/view/PictureOfSerialPyioProcessingGameOfLife

I'm already in love with the virtual monome, although I must read myself 
a little bit more into its code to figure out how to add keyboard as 
input for at least some of its buttons.

Today, I've fixed the problem with "run.bat", as well as added the 
device-finding code by  3-14159 to "run.sh". These and some other minor 
improvements have been comitted today into the CVS. As soon as I have 
finished testing serial-pyio with my 40h (sorry, but haven't been at 
home lately), I'm planning to pack an official new release.


Until then, I've put "pre-releases" marked as "Beta" here:
http://entenhausen.at/~quack/apps/serial-pyio-0.2.0b.zip
http://entenhausen.at/~quack/apps/serial-pyio-0.2.0b.tar.gz
http://entenhausen.at/~quack/apps/serial-pyio-0.2.0b.tar.bz2


Enjoy!

!!!

very nice looking!  sorry i havent gottent o do some testing on windows 
yet, been very busy lately (im always saying that, but its true).

the plugin engine you guys started is a great idea.

Brilliant stuff, this serialpyio! Trying it out on win xp now, working 
fine. Inspiring, I hope I can get some Pd apps working soon...

I just run python serialpyio.py -d COM2 from command line and it's fine. 
Note, though, that the run.bat is still not executing okay. I think you 
must add the ".py" extension to the where it just says "serialpyio" 
today.

The virtual monome is a really cool idea. But it only sends button press 
messages and not release messages right now, right? That is, when I 
press the upper leftmost button of the virtual monome with my mouse it 
sends "/40h/press 0 0 1", but when I release the mouse it doesn't send 
"/40h/press 0 0 0".

Regarding the keyboard input on the virtual monome - I hacked the 
fake_40h and set up the keyboard to trigger virtual monome buttons, with 
this mapping:
row 1: 12345678
row 2: qwertyui
row 3: asdfghjk
row 4: zxcvbnm,
row 5: 12345678 (with caps lock on)
row 6: qwertyui (with caps lock on)
row 7: asdfghjk (with caps lock on)
row 8: zxcvbnm, (with caps lock on)

I am using a swedish keyboard. I would love to have this same mapping in 
the serialpyio virtual monome!

/anton

edit: the python line in the run.bat file i think should read:
python serialpyio.py -d COM2

Hey Jah,

Thanks for the input and the feedback. I'll try to incorporate as soon 
as possible  your suggestions.

THe keymapping was on the todo list. I'll also add the key release.

Thanks to correct the run.bat, we don't have a windows box to test it, 
it is made blindly.

You can also try the small standalone apps, they are not that useful but 
fun:

- go into serial-pyio dir.
- type into the cmd.com window: set PYTHONPATH=src (I am not sure the 
syntax under cmd.com)
- then : python example/something.py

These apps mostly demonstrate how to make standalone apps with the 
monome in python.

Julien

@Jah: "run.bat": Sorry. my fault. will check again. As already 
mentioned: I did this blindly, due to lack of Windows. :)

no problem, I'm your win xp beta tester :).

serialpyio is great work, it's becoming my main serial app!

a quick feature suggestion: i'd love to be able to set a default 
(initial) prefix with a command line argument.

@Jah: I personally prefer to do this from within the application 
connecting *to* serialpyio, because then I don't have to modify my 
startup script. :)

But I'll see what I can do - shouldn't be too much of a problem to add 
that. Actually, I want to have a config file anyway (take a look at the 
commandline arguments... -F is already there, but currently not 
implemented).

Which patches/apps are you already using serialpyio with - and have you 
tried v0.2.0, yet? I'm asking because I'm wondering how much impact 
Julien's threading code has on improving response time.

rooker:
when i come to think of it this is no problem at all. you can scrap that 
feature request. i'll just bake some /dev/prefix messages into my 
patches.

i noticed that serialpyio and monomeserial work differently in this 
regard:

monomeserial (win xp) only takes messages like these:
"/sys/prefix /midi_slide"

while serialpyio only works with "/" omitted from the prefix, like this:
"/sys/prefix midi_slide"

i am not sure what is the correct way of doing things, but the new 
_40h_midi_slide and _40h_midi_press in the base lib is using 
"/sys/prefix /midi_slide" and "/sys/prefix /midi_press"

i prefer serialpyio to monomeserial right now, monomeserial has been a 
bit buggy and takes longer time to load. i've tried serialpyio with mlr 
and my own ableton live max/msp patches, and with some pd patches i'm 
cooking up - it's working fine, though i've had some trouble when i want 
to exit and press the quit button. i usually just kill the command 
prompt instead. i'm using 0.2.0 beta posted above :)

@Jah:
First: Thanks for doing the Windows-Testing. :)
Second: I'll check the syntax for /sys/prefix. I might have interpreted 
the docs, but certainly monomeserial is the reference-app, so I'll have 
to change it.
Thanks for pointing it out. (I've never seen or used monomeserial, since 
I don't have a Mac, and it wasn't available for windows when I started 
serial-pyio).

hey guys. i'm testing on os x. great great app!

the one thing (which is pretty major) i see as a problem is latency and 
the possibility of dropped serial packets.

running my standard 40h test patches, very familiar button/led response 
tests that are usually quite snappy with monomeserial are a bit laggy 
with serial-pyio.

with sound-related actions very small timing discrepancies are very 
noticeable.

is there any way i can tweak/tune the python setup to give it higher 
priority?

i'm really excited about this, great thanks.

Concerning the latency problem. It is probably possible to improve this. 
Problem is tuning. If you poll to often serial-pyio starts to consume 
too much CPU, if you poll too less latency appears. I will try to tune 
it. But you already try yourself too, it is easy.  in the file 
device_xxh.py change the line time.sleep(0.001) to something like 
time.sleep(0.00001) and see if it improves latency. Normally it should, 
since OSC packets are sent directly from the thread that polls the 
serial port. This avoids adding the latency of thread switching.

@Jah: for the problem of sending "40h" or "/40h" as a prefix. I added 
some code so that you can send both. If the / is missing serialpyio adds 
it itself.

Ok I made a quick test to see how many times per second a polling thread 
can be active, while not hogging all the processor. This is the program, 
I guess non-python programmer can get it:

import threading,time
class Test(threading.Thread):
    def __init__(self,pause):
        super(Test,self).__init__()
        self.pause = pause
    def run(self):
        self.stopThread = False
        self.i = 0
        while(self.stopThread is False):
            self.i += 1
            time.sleep(self.pause)


test = Test(0.001)
test2 = Test(0.01)
test.start()
test2.start()
time.sleep(2)
test.stopThread = True
test2.stopThread = True
print test.i, test2.i

This gives something like 240 activations per seconds while keeping the 
CPU usage of the python interpreter below 1%. This means that the 
latency of serial-pyio cannot go below 4ms.

The result of the test is that it is useless to change the line code, I 
mentioned in my previous message. Going below 0.001 sleep time does not 
change anything, under Linux. Below 0.00001 the thread simply does not 
sleep and the interpreter consumes all the CPU resources.

Anyway, if you have time Tehn you can run the above program and give me 
your results. You try different sleep time to see the max values you can 
get.

julien, thanks for the solid explanations. i'll run these tests when i 
get a chance later today.

any chance you can avoid polling altogether?  does python support 
os-level threading constructs like event-signaling (whih i use in monome 
serial XP) or mutexes (which is used in monome serial osx)?  or, even 
perhaps a simple while loop calling a blocking serialport read, on a 
differant thread?  this requires the serialport object to have an 
overlapped mode, which im sure it does anyways.

Yes I wanted to do something like that. I have look a bit more into the 
serial lib we use.

Hello all,

So I modified serial-pyio to improve latency.

- I removed polling and use blocking reads. Actually reads were already 
blocking so I just removed the sleep. As expected processor usage is 
still very low with this approach. Thanks Kid sputnik for the suggestion 
!
- Coordinates are now transformed using a single matrix multiplication. 
This enables to take into account cable orientation and coordinates 
shift (for multiple units) by just multiplying a 3x3 matrix with a 
vector.

I attach an archive for you to test. If someone could tell me if it is 
snappier than before, I would be glad. I made a small app to test it 
visually and it seems snappy. However, I don't have a working pure data 
installation, so I cannot test with an audio app.

WARNING: this archive is only for testing purpose, it contains new code 
to handle multiple applications, that is not fully tested/implemented. 
However the usual 40h messages work. These modifications will be 
included in a future release of serial-pyio.

Julien

substantial timing improvement!

i suspect the 40h i'm using has some button trigger issues (i 
disassembled it the other day to show one of our manufacturers, and 
probably re-assembled it sloppily.)

i don't imagine there's much of a chance that incoming (button) serial 
packets could get dropped?

No very little chance. The serial lib of python is just a wrapper on top 
of the C api of the OS. So, it is quite fast. Moreover, there are the os 
buffers in between. Loosing a packet would mean that the kernel starts 
dropping packets. I don't know how this can happen, except sending a lot 
of info to the serial port with no app requesting it. Which is not the 
case here. Button presses do not generate that much traffic. Concerning 
info coming from the ADC, we'll have to test this later. I don't know 
how verbose are these outputs.

This button press droppings will need more investigation. If you have 
some time to test with another unit, this can be useful. Did someone 
else noticed button dropped?

At least, I am glad timing improved !

Julien

how are you handling the reads, btw?  are you reading blocks of 2 when a 
packet arrives?  ive found for monome serial, the best/most stable 
method is to literally jjust read 1 byte at a time whenever a new serial 
event is detected, going in a do/while loop, adding each byte to an 
array/vector and sending it out when done.  just curious what method you 
use.

i figured the pythion serial lib wraps the OS serial methods, as you 
said, but one thing to think about is that, on Windows NT-based systems, 
there are a few differant methods of reading serialdata.  they are 
mainly ReadFile() or (what i prefer) the CommEvent API (probably not the 
correct term for it).  both use the OverLapped IO File API.

I haven't seen Julien's new version of the serial code, but my initial 
code queries the serial-API about how many bytes are in the current 
queue of the serial port and only return that a command was received if 
a full message (40h = 2bytes) could be read.

that sounds much cleaner than polling, i think i read that Julian fixed 
this in some other thread though.

what ive found though, is that if i just try reading 2 bytes max i will 
lose packets.  as i said, i usually read 1 byte at a time from the input 
buffer, and load those into a new List<byte>, one byte at a time.  this 
sounds less efficient compared reading 2 bytes at a time, but for my 
code it always seems to work the most effective.  i cant just read 2 
because when i press multiple buttons, there is more than 2 byte 
packets.  i do the read by calling WaitCommEvent(), this uses a single 
wait-object, as well as Overlapped File IO, to ensure that even if the 
call returns false/empty, if there are bytes coming late due to 
threading issues, they will still get checked.  i remember brian said 
something about press data getting lost in his serial-pyio testing, 
although it might have been related to polling, reading 2 bytes only can 
also do it if the input buffer has more than 2 bytes at a time.  this 
got me too in my original serialiodotnet code, what i did when using the 
all-managed .NET serialport class is (note my new unmanaged .NET wrapped 
File/Serial IO Wrapped is similar, but i needed to impliment the 
serialPort.DataReceived event myself).  The following code is in C#, 
which is VERY similar to Java (which is itself based on


// class-scope variables
private byte[] data = new byte[INPUT_BUFFER_LENGTH];
private byte[] packet;
public event EventHandler<PressReceivedEventArgs> PresstReceived;
public event EventHandler<AdcReceivedEventArgs> AdcReceived;
...
// event-handler method
serialPort_DataReceived(object sender, SerialPortDataReceivedEventArgs 
e)
{

    serialPort.Read(data, length, 0); // buffer, length, offset

    // read thru the input buffer, 2 bytes at a time.  Send a new Press, 
Adc or Enc
    // event, depending on the packet value (data[i] >> 4)
    for(int i = 0; i < length; i += 2)
    {
        packet = new byte[3];

        switch (data[0] >> 4)
        {
            case 0:
                packet = new byte[3];
                packet[0] = data[i + 1] >> 4;
                packet[1] = data[i + 1]  & 0x0F;
                packet[2] = data[  i  ] >> 4;

                OnPressReceived(new PressReceivedEventArgs(packet));
                break;
            // same for adc and encoder
        }
    }
}


without the for-loop part, it didnt work, even if i repeatedly tried 
reading from the serialPort with the read method serialPort.Read(buffer, 
2, 0).   also, ive found it necessary to save the serialPort's length in 
an int variable, since each call to read will lower the value, so 
calling it in a forloop header will always create major issues.

the moral here if you are using a Stream-based SerialPort object, do the 
read once on all bytes in the buffer, and handle the bytes afterwards. 
you will never lose packets.  btw, for the INPUT_BUFFER_LENGTH, i used 
136.  why?  64 buttons, plus either 2 encoders or 4 adcs, with 2-byte 
packets, so (64 + 4) * 2 = 136.  of course the odds of hitting all 
buttons and adcs at the same time, and them all hitting the internal 
input stream's buffer at the same time is pretty much null, its still a 
reasonable value to think about

Thanks for your help, kid-sputnik!

I just wanted to add, that I'm waiting for ">= MSG_LENGTH" and then I'm 
taking only MSG_LENGTH bytes out of the buffer, actually assuming that 
the rest would stay in that queue, picking it up on the next iteration.

If that's where bytes get lost, I'd say that the serial-code is behaving 
strangely :). That would make your approach more stable, since I cannot 
assure that the serial-code is behaving the same on all different 
platforms (as mentioned above, it's a wrapper of native serial-API).


I'm currently in the office, but I'll take a closer look at your 
suggestion in the evening.

This is the code to read the serial port I made in the last version:

 while self.stopThread is False:
            cmd = ''

            # reads two bytes from the serial port
            # blocking call
            cmd = self._ser.read(2)

            if len(cmd) == 2 :
            # handle the received command....
            ....

This assumes that the read returns only when it can get 2 bytes from the 
serial port. Reads are blocking. I have to check if the read sometime 
returns more than 2 bytes which could generate packet losses here.

Concerning the C# code:

To me, it seems that this call serialPort.Read(data, length, 0); modify 
the length var to set it up to the number of read bytes. So, if this is 
right,  the call serialPort.Read(data, 2, 0); cannot work because the 
function expect a reference to a variable as second parameter and not an 
actual value. The read method in python is different, the parameter is 
the number of bytes you want to read. In this sense, this simplifies 
things a lot.

@Rooker:

If you make test try to make them on the version I attached couple of 
message ago. It is already an "improved" version.

Julien

@Julien:
I'll use the new version when doing tests.
There's something in your code example up there that I'm not comfortable 
with: You use the number "2" instead of MSG_LENGTH. ts ts ts....
This will fail with devices having more than 8 buttons per row, since 
they will have larger msgs. :(

Yes, I made this quickly to see if it improved latency. It is easy to 
clean this.

Sorry.
It's just that I'm a "beaten child"(*) regarding hardcoding of values so 
I just wanted to clean such things out as early as possible. :)
(I hope it didn't sound offensive or anything)


(*) In german there's a phrase, which literally translated is: "I'm a 
beaten child" - meaning that one has seen/learned things in a "hard and 
uncomfortable" way and thus anxiously shrugs everytime when coming 
across something related. ;)

Yeah no worries !

Btw, I progressed on the proxies. I am in the final phase. I am adding 
the GUI to manage it. I hope to finish this during the week end and then 
the multi-app framework will be there.

julian, osrry if that code was flaky, i typed in 'on the fly' on a 
differant computer than the one my code is on.

i actually dont use the C# serialport class anymore, but just because of 
a stupid bug (if a device gets unplugged, it removes if from the list of 
available comports until the app is restarted, no matter what you do). 
my own wrapper class is much simpler than the .net one, and is geared 
really towards monome only (no functions for modems, and no underlying 
stream object).