Since my last blog, I have been working on adding a module to the Stingray library that would enable us to run blocks of code in parallel, evenly distributed across a machine's cores.
I do believe that it would be a great asset, once it is done right.
Unfortunately, the first challenge is performing multiprocessing in Python. I've always heard about the ugly Python monster named "GIL" (Global Interpreter Lock). I never understood why some people hate it so much. But not anymore, every strategy I came up with to try and work around the GIL, it always comes back and hit me.
We tried to use some external libraries to help us. But the downside of doing so is whenever there is an error, you must dig deep inside the external libraries code and figure out what is causing the problem, not to mention that debugging a multi-threaded program is already a challenge.
However, my mentor suggested using the built in multiprocessing module in Python, which basically works around the GIL by creating multiple Python processes.
It actually achieved good results, and the fact that I can customize it as much as I want without much limitation is great. But of course it has to have a downside, and that would be that absolutely no errors are handled automatically, all kinds of exceptions\errors must be handled explicitly.
My goal is to have this module tested and ready before the final deadline. If I do make it in time I believe it will have a great impact on Stingray, I also plan on continuing to develop it even after GSoC ends.
GSoC is a really great experience, it teaches you a lot. Not just on a technical level, working on a huge sized project with developers from all across the world, it helps your work to be mature enough to meetup with the standards.
I do believe that it would be a great asset, once it is done right.
Unfortunately, the first challenge is performing multiprocessing in Python. I've always heard about the ugly Python monster named "GIL" (Global Interpreter Lock). I never understood why some people hate it so much. But not anymore, every strategy I came up with to try and work around the GIL, it always comes back and hit me.
We tried to use some external libraries to help us. But the downside of doing so is whenever there is an error, you must dig deep inside the external libraries code and figure out what is causing the problem, not to mention that debugging a multi-threaded program is already a challenge.
However, my mentor suggested using the built in multiprocessing module in Python, which basically works around the GIL by creating multiple Python processes.
It actually achieved good results, and the fact that I can customize it as much as I want without much limitation is great. But of course it has to have a downside, and that would be that absolutely no errors are handled automatically, all kinds of exceptions\errors must be handled explicitly.
My goal is to have this module tested and ready before the final deadline. If I do make it in time I believe it will have a great impact on Stingray, I also plan on continuing to develop it even after GSoC ends.
GSoC is a really great experience, it teaches you a lot. Not just on a technical level, working on a huge sized project with developers from all across the world, it helps your work to be mature enough to meetup with the standards.
No comments:
Post a Comment