The business software maker Novell posted a quarterly loss but its results were better than expected, helped by an increase in sales of Linux software.