@jeeves... re "I am confused as to where (ie. primary aside) to insert the html as a text widget. And it doesn't really flow well with the page"...
There are various ways to do it. I'm a minimalist and like to keep my blog designs clean and simple so here is a live example you might be able to learn from.
This is a blog project in progress so it's not finished yet but you can see what the Feedburner email subscription looks like in the top right of the sidebar at...
http://zebrowskiconsulting.com/blog/
- in the primary aside area I've added a text widget
- in the title area of the widget I added the text "get updates by RSS or email"
- I placed the email icon in the image folder of my child theme and
- used this code to import the icon and connect it to Feedburner
<img src="http://www.ZebrowskiConsulting.com/zebrowskiconsulting/blog/wp-content/themes/zebrowski_750/images/email.jpg" alt="" style="border:0"/>
If you have questions or need more detailed help, just ask!