Before you can get a website up and running, you need to have a place to put it. Paying for web hosting is, basically, like renting a small amount of space on someone's server and paying what it costs them to send your web pages to your customers. Fortunately for you, though, web hosting has never been cheaper.
Domains and Hosting Together?
Many domain name companies have taken to offering you hosting when you buy your domain from them. This is generally an expensive option, and a bad idea - you'll be getting few features compared to what you're paying. Few people who are serious about web hosting get it from the same place they get their domains.
So Where Should I Start?
Well, that all depends on what your website is going to need. How many visitors do you expect to have? Are you going to have lots of large graphics on the site? Do you have a lot of articles or products that you want to put in a database? Do you want to have an email address at your website (email@example.com)? On and on it goes. Each host you look at will offer you different combinations of features at different price points, and finding the one that's right for you can be quite a task. Here's a technical-to-English guide to what you should be looking for.
MB storage. The more MB of storage you have, the more you can put on your website. For most websites, this number can be really very small without it being much of a concern - the pages would be too big for anyone to download and see before they'd be too big to store. You only really need to worry if you're planning to put something apart from plain pages on your site. If you want to make a gallery for your digital photos or let people download ebooks from you, for example, this number needs to be higher.
GB bandwidth per month. This is a limit on how much data your website can transfer each month. For small websites, you don't need to worry too much, but as you get more visitors the amount you need will increase sharply, especially if each one looks at lots of pages or downloads large files from the site. The amount of bandwidth your site needs is generally considered to be the deciding factor in how 'big' it is, and how much it will cost you.
MySQL databases. The number of databases your website will have to store things in. It will make it much easier for you if you have one. Don't pay more to get extra, though: one database is all you need. It's worth noting that if your host may offer some other kind of SQL instead of MySQL (for example, PostgreSQL). You should usually avoid anything apart from MySQL, unless you know what you're doing.
PHP, Perl, ASP, JSP, ColdFusion, Python, Ruby. These are all scripting languages, used to write your website. You should make sure your host offers the languages that any software you plan to use is written in. If you don't have specific requirements, then you should be fine with just Perl and PHP.
Subdomains. These allow you to split your website into more sections than just 'www' - you might decide, for example, that you would people to be able to go to 'shop.yourdomain.com' and 'news.yourdomain.com' and see pages there. You don't really need these, though, as doing the same thing with subfolders ('www.yourdomain.com/shop') is usually just as effective.
FTP accounts. An FTP (File Transfer Protocol) account is what you'll use to upload your website to your host. You'll always get one of these. The only situation when you'll need more is if you want to let someone alter things on your site without giving them the master password.
POP3 accounts. POP stands for 'Post Office Protocol', which is just fancy-speak for email. The more POP3 accounts you get, the more email addresses you can have: useful if you want to have firstname.lastname@example.org for new customers and email@example.com for existing ones, for example.