Firefox keeps a list of every file you download. If you haven't cleaned it up, it can become QUITE large and slow down the download startup. Go to your tools menu in firefox and click downloads, then click the "clean up" button. It will not delete the files you have downloaded, just clear the list. That should speed it up a bit.

If that doesn't help you can try defragmenting your hard drive.