Blog my Mind my Map

I made a todo list in the form of a mindmap. There are other informations in there, not just about this website. But I limited the info about my job and personal life, and tried to make a fairly complete list of the ideas I am thinking of implementing in the near future on this website.

Today I mainly did some simple debugging. I introduced a couple of filters to make sure that the files uploaded were .xml and .mm. I  know they are not going to stop the bad guys, but they might stop the mindless ones.  I am starting to understand why there are so many warning signs in this society! I also started to work on map 6. I  tried to make a  map of the first 500 posts: no luck. 400: idem; 300: same same; 200: nop; 100: no again; 50: YES! So the first 50 posts could be mapped. I think there must be a couple of different problems with this map. First a character set problem. Second a size problem. I will first try to work on the character set problem. Then I might tackle the size problem. The problem is that if a map has n tags, I need to make n dictonaries of size n. But I suspect that I might simplify the whole thing taking a n*n matrix. And if not, do we really need to map all the tags? I mean the tags follow a power law, of those n tags probably 50 are relevant, and the rest are just the long long tail. Tags with only one entry. Maybe I could, for example, only consider the tags that have more than one post. That might actually work. And the result might even be nicer.

I finally managed to change the code, so that now every new map that is created is first transposed to MySQL, and no big data is stored in the RAM. This was needed to permit bigger maps to be calculated. Also for the further versions of the mapmaker. Still the program is less then perfect. With all the work I did, it still did not manage to be able to compute data 6. Also all this use of MySQL is actually slowing down the performance for everybody else. Not clever. In the next days I shall work in adding indexes, and other changes that should make the code faster, and MySQL less subject to the weight of the mapmaker. I also need to look very carefully in the whole topic of the various character sets. Quite a lot of times the map is not being generated because the data are in the wrong character set.

I also noticed that it was not possible if you were not an administrator to make a map from some data that have not been uploaded by you. This was against the spirit of the site, so I corrected it, and now everybody can use everybody else data.

I will be out for a few days. In the meantime, I took away a few other bugs (do they actually proliferate?). In particular there was a small nasty that made sure that if you were trying to memorise a map, he would make you memorise the data with the same id, and viceversa. The result of this might have driven a few people away, who knows.

In the meantime we got a new user: user Thomaspoto which  generously inserted his xml data, but then did not went further to actually make the map. Also the previous user followed the same pattern, so now I am starting to be paranoid that it is not clear enough. Maybe I should spell somewhere that after you upload an xml file you still need to click on the [Make Standard Map] button.
In any case I went again in and clicked the button myself. It is also strange that no one seem to makke anybody else mindmap. I would be so curious.

But beside all this I am now working on why I can’t get mindmaps made for systems that are too big. And too big, is really not that big at all. 500 bookmarks or so. After some testing I reached the conclusion that I am not hitting the CPU limit, I am not hitting the length of process limit (I got a process running for 20 seconds before I killed it myself), so I must be hitting the RAM limit. And this is hard, so to walk around it, trying to make the calculation without being to heavy on the RAM right now I am studying how to store a whole xml in mysql. And then interacting very closely with the MySQL. Without ever holding so much data in the RAM all at once. Hopefully, eventually it might work.

In the meantime I am off, treat well the lady (the website).

Pietro

I am going further, and to test some .py code in my machine I had to add mysqldb. Then, opening it, he refused to work unless ‘easy install‘ was installed too. They must have been friend. He tried to get his mate by himself and didn’t suceed. So there I was, searching the net to get the second package. That was a .egg file. Now .egg file are in general archives. Not this. I actually had to call “sh thefileIjustdownloaded”. Something that I happen to discover in some obscure page. Now with the easy install installed, I could go one step further to install mysqldb. The next step was discovering that he was not willing to let me install mysqldb unless I had a version of mysql running in loco.  How patient am I! In any case I wanted to have a local mysql, so there I was looking for the right version. Firsl locate mysql.com, then proceed over the terrorist notice: “unless you are a god in programming buy a local guide”. Now what kind of Mac have I got? I know it is not Intel, so it is PowerPc. 32 or 64? Google doesn’t help. Wikipedia neither, andthe About this Mac is similarly obscure. I just know that my cpu is PowerPC G4 (1.1). No reference to 32-bit or 64-bit. It is probably a question too stupid to ask? Let’s try. I download the 64. Install. Run from the System Preferences. It does not start. But he did not protest when I installed it, it MUST be the right type… right? I actually need to add a few things in the PATH. Look he does not even find the mysql. But here they are. Let’s try to run it, you know just for sport.

>WRONG CPU TYPE.

Hmm, looks like I have a 32 bit. Let’s download the 32 bit.  Read Me: unfortunately installing a new version of mysql does not disinstall the previous… Great! So google: how do I unistall a mysql that was wrongly installed (although-I-can’t-understand why-I had to instal-it-in-the-firstplace-since-I-only-wanted-to-connect-to-the-far-away-server). Answer. Worked fine. Except that in the meantime I had alread installed half of the 32 bit version. Let’s delete all and start again. Ok, installed. Now let’s go back to mysqldb.

Same problem, he cannot find “mysql_config”. I use spotlight, I can’t find it either. I use quicksilver (they promised me that it was the best), … not in this. I use google. Only one person had the same problem (and is mirrored all over the place), but the suggestion don’t seem to apply.

Someone suggests: you need to tell the setup where to find mysql_config. Well, how can I tell him if I don’t find it myself. It continues: normally is in …/mysql/bin/

“normally” works fine for me. So I found it, how do I tell him? The error was raised by setup_posix.py. Here you go:

f = popen(”%s –%s” % (mysql_config.path, what))
and no way is  mysql_config.path defined. I change it into:

f = popen(”%s –%s” % (’/usr/local/mysql/bin/mysql_config’, what))

and it works!

python setup.py install
sudo python setup.py install

A long list of things that happened, and it ends with something similar to:

mysqldb installed with easy install.

“…with easy install”: fuck off!

There must be a better way to pass the sunday. In any case now that mysqldb is installed I just discovered I cannot run the code because he cannot connect to the mysql. It must be because of the university proxy. It doesn’t matter, I shall use test it directly on the net.

After discussing with a friend of mine I was convinced to require everybody who upload any data in MMM to also release it into the Public Domain. This will permit a few things:

a) In any case any data that is uploaded in Mind My Map can be easily downloaded. Also the xml data can be regenerated from the mindmap. So I really don’t want to give to the user an impression of a security that is not there.

b) I want to use the xml data to calculate the distance between mindmaps as I have described in my English blog. It is already one year and a half that I have described those algorithms and no one has stepped forward to use them. So I will.

c) If the data is in the Public Domain, then anybody can use anybody else data. And the sitebecomes a big toy store. Yeah! Anybody can make maps from anybody else xml data. And for now only standard map are available, but soon…

So I had to work to change the data structure. Now each map and data stores the creator. And then you have a list of the data that you have decided to remember. Creating some data automatically makes you remember it. And the user page now only shows the data you have chosen to remember.

This brings up a few problem:

when should the data be deleted, if ever?

should you ‘remember’ any new map that has been generated from data you have created or remember?

Hmm…

Not to make errors, for now no data gets deleted (I took away the buttons). But eventually I will need to reorganise them.

And we also have the first user who uploaded uncorrupded data. Welcome to janzo. Using his data I made a really pretty map.

I made some changes in the code. Now the map maker directly connects to python. And directly stores the data to MySQL. I am not sure if it makes such a huge difference in terms of speed, but it surely it does in the complexity of the code and of the structure. Before I would read the data in php, store it as a file, call the py program, write the result file from python. read the file from php and store it in mysql. A mess. Now it is all done through pyhton. This might also open the door toward the idea of running the program in background.
At the moment the speed of the mysql is the bottleneck of the whole service. Only people with few delicious bookmarks can have their map done. I don’t like this, and I have in mind a few ideas to solve this situation. The first one will just be to try to make the calculation run in background, so the user is not forced to keep a windo open the whole time. They might still be killed by DreamHost, though. So it is only moving the bar, not really solving the situation.

Some of the information uploaded from the user has been corrupted. Mostly I, without realising how small a BLOB is, have saved it, just to realise that only the first 64 kb were saved. Please try to upload it again, and let me know if now it works better.
I also fixed a few bugs and I passed some time in writing a system that directly connects to delicious. Eventually I want it not to connect only to delicious but to every other bookmarking service that offers the whole set of bookmarks in their API, but for now I shall start with delicious.

A couple of users have tried the site, enough to register. Too bad the website was not working at the time.

I discovered a few bugs. Some related with the encodings, some with the mysql. If you tried to use the system before and it did not work please delete the old data and try again. Some bugs I have spotted are unfortunately a bit more complex, and might require a bit of restructuring of the code. In any case for the first time the system generated a readable mindmap. Yes!

The service is still very buggy. It is pre alpha, or similar. For now you can pretty much upload a map, sometimes download it, sometimes visualize it. But not always. You can upload data from delicious, and call the program to make the mind map out of it. But it does not always work, and in general it mostly work when the mind map that wold be generated is quite small. I am doing my best to correct all the above, and make the whole system smooth, but it will take some time. In the meantime, feel free to test it, and give me feedbacks.