This is a good post for beginners.
To edit the side bar:
Ok. When looking at your website in edit mode you can simply click on edit side bar under the actual side bar! Simple as that.
Another way to edit the side bar is:
More > manage > site layout > you can edit the side bar from there.
Let me know how you get on.