A summit implies importance of the attendees.  One does not have a summit of Star Wars fans, for example.  :)  It generally implies a decision or output, as well.

A forum also has a connotation of importance, but not quite the self-agrandizing level that summit implies.  Forum also mixes in a couple of other connotations.  It implies learning, equality of participants, and exchange of ideas.

A summit generally implies participation is by invitation only, whereas a forum usually means many people can at least attend events, if not participate directly in them.