Is there any way to have inventory for one product but not others?

No, inventory is site-wide and is either on or off. You could set the inventory for items that you don't really want to track to 9999 and set the INV LIMIT setting to a lower number, like 999 and then those items won't ever show any inventory levels.