As far as I know, you'll be able to put up a certain amount of auctions per a certain amount of time for free. Any auctions you put up beyond that amount within that timeframe will cost you a certain amount of real money, regardless of whether they sell or not. If the item sells, the transaction fee will be higher.
In either case, the auction house system will deduct a nominal fixed transaction fee from the seller, the amount of which is determined by whether or not the item was sold (see below).
How is the transaction fee determined?
A nominal fixed transaction fee will be deducted from the seller for each item listed in the auction house. This fee consists of a fixed charge to list the item, which is assessed whether or not the item is successfully sold, and an additional fixed charge that is assessed only if the item is sold. Because the listing portion of the fee is charged even if the item doesn't sell, it will be in the seller's interest to list items he or she believes other players will be interested in, and to do so at a competitive price. Specific details related to the transaction fee for the currency-based auction house will vary by region and will be announced at a later date.
Please note that we plan to waive the listing portion of the fee for a limited number of transactions per account. In other words, for these transactions, the seller will only pay a transaction fee if the item is successfully sold, and that fee will not include the listing charge. We'll have further details on this as well at a later date.
So, let's say that we get ten auctions per week, which cost us money only if the item sells. Every item we put up for auction beyond those ten within that particular week costs us let's say 50 cents. This means any farmers or bots will only hurt themselves if they flood the RMAH with items, as they will lower the prices and sell absolutely nothing, and they will lose real money each time, which in the long run isn't such a great idea.
Also, add the fact that any item bought from the RMAH will have a cooldown period during which it can't be relisted, and farming/botting for common items in D3 won't be a very lucrative business at all.