I've found and fixed a big problem with the http audio streaming at Magnatune. Several people in Europe had complained about being unable to reliably stream our music.
My own tests found transfer rates of 50k/second from he3.magnatune.com but 3mb/second from www.magnatune.com. Since I know Apache (which is on the www server) is slower than mathopd (which is on the he3 server), either there was a software bug in mathopd or a machine/networking problem.
The problem was that the 100mbit Intel Ethernet card was set to autosense its speed, and wasn't doing that correctly, so we were getting 1/60th the speed we were supposed to get. This new problem appeared few months ago when I moved from http://he4 to http://he3 -- he4 didn't have this problem (he3 had more disk space, which we needed). For the Linux geeks out there, the solution was adding "options e1000 Duplex=2,2 Speed=100,100" to the /etc/modprobe.conf file and rebooting. This is spelled out by Intel docs at http://support.intel.com/support/network/sb/CS-009209.htm#speed_duplex
I'm now getting 5.5M/second transfer rate from http://he3, which is the machine that feeds all our audio streams, and this seems like a more sane speed (I believe that's 44mbits, on a 100mbit line).
Magnatune fans: please let me know if you see an improvement in either drop-outs or time-to-start-playing-the-song.
On a separate topic, I thought the problem might be the "mathopd" web server I'm using, and investigated and benchmarked several high performance alternatives. I finished by using thttpd http://www.acme.com/software/thttpd/thttpd.html because the comparison charts http://www.acme.com/software/thttpd/benchmarks.html they created show mathopd a close running up to them (in comparison, Apache is 1/5th as fast with large files) which agrees well with my own tests 2 years ago, where I found mathopd to be the fastest.
My own use of the "ab" benchmark tool finds thttpd to be about 25% faster than mathopd with a moderate load (10 users) and a mp3-sized file (3megs).
To run my benchmark, thttp took 14 seconds, mathopd took 19 seconds, and apache took 78 seconds. This is why a specialized web server is a useful thing when serving lots of multi-megabyte files.
Whoops, turns out thttpd was piggy with its memory and he3 was running slowly this morning, with a Unix load average of 1.5 and mult-second latency on the command line (telnet).
So, now I'm back to mathopd. I found a new web page command in mathopd ("dump") which can show me how many simultaneous listeners we have. The server was set to what I thought was a very generous 1000 listeners, but it turns out we were hitting maximum that most of the day. Now it's set to 5000 max listeners, and I'm storing the # of listeners on a per-minute basis to see if we ever max out.
I also found that I had set the per-connection buffer at 64k, rather the default 12k, and this was causing mathopd to use lots of memory, which caused memory swapping and thus high machine load. I couldn't find any setting like this for thttpd, which is unfortunate because I think the high load on the machine was due to the high memory usage of thttpd.
Mathopd reports right now:
Uptime: 5354 seconds
Active connections: 814 out of 5000
Max simultaneous connections since last dump: 814
Forked child processes: 0
Exited child processes: 0
Requests executed: 73791
Accepted connections: 64473
Pipelined requests: 285
Posted by: John Buckman | February 11, 2006 at 03:13 PM
What is with this lady's voice at the end of every preview song? I know it is a preview song 'cause all the tag fields has the word "preview" in big letters. I also know which artist it is as it has the artist/album in the tags. Then why this announcement? It kinda ruins the experience.
Posted by: Manish | February 13, 2006 at 09:40 PM
John decided to put this anouncement at the end of each track because magnatune songs can also be heard on shoutcast radio stations, and people who listen to magnatune radios can't know which artist is playing when they hear something great (plus for other reasons I don't remember, I think). He explained that in this blog and in the forum.
Posted by: Glukx Ouglouk | February 14, 2006 at 10:28 AM
Wikipedia uses lighttpd for their high-load image-server, and it seem to do the job very well. Funny, that thttpd doesn't compare their browser to lighttpd.
http://www.lighttpd.net/benchmark/
Conclusion:
lighttpd is 4-6 times faster in every setup than apache and outperfoms thttpd for large files.
Might be worth a try.
Posted by: | February 16, 2006 at 04:46 AM